(note: this got an update, I'm now using dataclasses with run-time type enforcment. see bottom :3)
So, it's been a year and I'm going to answer my own question. I don't quite like answering it myself, but: this will mark the thread as resolved which in itself might help others.
On the other hand, I want to document and give reason to why I chose my solution over proposed answers. Not, to prove me right, but to highlight the different tradeoffs.
I just realized, that this got quite long, so:
tl;dr
collections.abc contains powerful abstractions and you should use them if you have access to it (cpython >= 3.3).
@property is nice to use, enables to add documentation easily and provides read only access.
Nested classes look weird but replicate the structure of deeply nested JSON just fine.
Proposed solutions
python meta-classes
So first off: I love the concept.
I've considered many applications for where they prove useful, especially when:
- writing a pluggable API where meta-classes enforce correct usage of derived classes and their implementation specifics
- having a fully automated registry of classes that derive a from a meta-class.
On the other hand, python's meta-class logic felt obscure to wrap my head around (took me at least three days to figure it out). While simple in principle, the devil is in the details.
So, I decided against it, simply because I might abandon the project in the not so far future and others should be able to pick up where I left off easily.
namedtuple
collections.namedtuple is very efficient and concise enough to boil my solution down to several lines instead of the current 800+ lines. My IDE will also be able to introspect possible members of the generated class.
Cons: the breverity of namedtuple leaves much less room for the awfully necessary documentation of the APIs returned values. So with less insane APIs you will possibly get away with just that.
It also feels wierd to nest class objects into the namedtuple, but that's just personal preference.
What I went with
So in the end, I chose to stick to my first original solution with a few minor details added, if you find the details interesting, you can look at the source on github.
collections.abc
When I started the project, my python knowledge was next to none, so I went with what I knew about python ("everything is a dict") and wrote code like that. For example: classes that work like a dict, but have a file structure underneath (that was before pathlib).
While looking through python's code I noticed how they implement and enforce container "traits" through abstract base classes which sounds far more complicated than it really is in python.
the very basics
The following is indeed very basic, but we'll build up from there.
from collections import Mapping, Sequence, Sized
class JsonWrapper(Sized):
def __len__(self):
return len(self._data)
def __init__(self, json):
self._data = json
@property
def raw(self): return self._data
The most basic class I could come up with, this will just enable you to call len on the container. You also can get read-only access through raw if you really want to bother with the underlying dictionary.
So why am I inheriting from Sized instead of just starting from scratch and def __len__ just like that?
- not overriding
__len__ will not be accepted by the python interpreter. I forget when exactly, but AFAIR it's when you import the module that contains the class, so you're not getting screwed at runtime.
- While
Sized does not provide any mixin methods, the next two abstractions do provide them. I'll explain there.
With that down, we only got two more basic cases in JSON lists and dicts.
Lists
So, with the API I had to worry about, we we're not always sure what we got; so I wanted a way of checking if I got a list when we initialize the wrapper class, mostly to abort early instead of "object has no member" during more complicated processes.
Deriving from Sequence will enforce overriding __getitem__ and __len__ (which is already implemented in JsonWrapper).
class JsonListWrapper(JsonWrapper, Sequence):
def __init__(self, json_list):
if type(json_list) is not list:
raise TypeError('received type {}, expected list'.format(type(json_list)))
super().__init__(json_list)
def __getitem__(self, index):
return self._data[index]
def __iter__(self):
raise NotImplementedError('__iter__')
def get(self, index):
try:
return self._data[index]
except Exception as e:
print(index)
raise e
So you might have noted, that I chose to not implement __iter__.
I wanted an iterator that yielded typed objects, so my IDE is able to autocomplete. To illustrate:
class CourseListResponse(JsonListWrapper):
def __iter__(self):
for course in self._data:
yield self.Course(course)
class Course(JsonDictWrapper):
pass # for now
Implementing the abstract methods of Sequence, the mixin methods __contains__, __reversed__, index and count are gifted to you, so you don't have to worry about possible side-effects.
Dictionaries
To complete the basic types to wrangle JSON, here's the class derived from Mapping:
class JsonDictWrapper(JsonWrapper, Mapping):
def __init__(self, json_dict):
super().__init__(json_dict)
if type(self._data) is not dict:
raise TypeError('received type {}, expected dict'.format(type(json_dict)))
def __iter__(self):
return iter(self._data)
def __getitem__(self, key):
return self._data[key]
__marker = object()
def get(self, key, default=__marker):
try:
return self._data[key]
except KeyError:
if default is self.__marker:
raise
else:
return default
Mapping only enforces __iter__, __getitem__ and __len__.
To avoid confusion: There is also MutableMapping which will enforce the writing methods. But that's neither needed nor wanted here.
With the abstract methods out of the way, python provides the mixins __contains__, keys, items, values, get, __eq__, and __ne__ based on them.
I'm not sure why I chose to override the get mixin, I might update the post when it get's back to me.
__marker serves as a fallback to detect if the default keyword was not set. If somebody decided to call get(*args, default=None) you won't be able to detect that otherwise.
So to pick up the previous example:
class CourseListResponse(JsonListWrapper):
# [...]
class Course(JsonDictWrapper):
# Jn is just a class that contains the keys for JSON, so I only mistype once.
@property
def id(self): return self[Jn.id]
@property
def short_name(self): return self[Jn.short_name]
@property
def full_name(self): return self[Jn.full_name]
@property
def enrolled_user_count(self): return self[Jn.enrolled_user_count]
# [...] you get the idea
The properties provide read-only access to members and can be documented like a function definition.
Altough verbose, for basic accessors you can easily define a template in your editor, so it's less tedious to write.
Properties also allow to abstract from magic numbers and optional JSON return values, to provide defaults instead guarding for KeyError everywhere:
@property
def isdir(self): return 1 == self[Jn.is_dir]
@property
def time_created(self): return self.get(Jn.time_created, 0)
@property
def file_size(self): return self.get(Jn.file_size, -1)
@property
def author(self): return self.get(Jn.author, "")
@property
def license(self): return self.get(Jn.license, "")
class nesting
It seems a little weird to nest classes in others.
I chose to do that, becaue the API uses the same name for various objects with different attributes, depending on which remote function you called.
Another benefit: new people can easily understand the structure of the returned JSON.
The end of the file contains various aliases to the nested classes for easier access from outside the module.
adding logic
Now that we have encapsulated most of the returned values, I wanted to have more logic associated with the data, to add some convenience.
It also seemed necessary to merge some of the data into a more comprehensive tree that contained all of the data gathered through several API calls:
- get all "assignments". each assignment contains many submissions, so:
- for(assignment in assigmnents) get all "submissions"
- merge submissions into respective assignment.
- now get grades for the submissions, and so on...
I chose to implement them seperately, so I just inherited from the "dumb" accessors (full source):
So in this class
class Assignment(MoodleAssignment):
def __init__(self, data, course=None):
super().__init__(data)
self.course = course
self._submissions = {} # accessed via submission.id
self._grades = {} # are accessed via user_id
these properties do the merging
@property
def submissions(self): return self._submissions
@submissions.setter
def submissions(self, data):
if data is None:
self.submissions = {}
return
for submission in data:
sub = Submission(submission, assignment=self)
if sub.has_content:
self.submissions[sub.id] = sub
@property
def grades(self):
return self._grades
@grades.setter
def grades(self, data):
if data is None:
self.grades = {}
return
grades = [Grade(g) for g in data]
for g in grades:
self.grades[g.user_id] = g
and these implement some logic that can be abstracted from the data.
@property
def is_due(self):
now = datetime.now()
return now > self.due_date
@property
def due_date(self): return datetime.fromtimestamp(super().due_date)
While the setters obscure the wrangling, they are nice to write and use: so it's just a trade-off.
Caveat: The logic implementation is not quite what I want it to be, there's much interdependance where it should not be. It's grown from me not knowing enough of python to get the abstractions right and getting things done, so I can do the actual work with the tedium out of my way.
Now that I know, what could have been done: I look at some of that spaghetti, and well … you know the feeling.
Conclusion
Encapsulating the JSON into classes proved quite useful to me and the project's structue and I'm quite happy with it.
The rest of the project is fine and works, although some parts are just awful :D
Thank you all for the feedback, I'll be around for questions and remarks.
update: 2019-05-02
As @RickTeachey points out in the comments, pythons dataclasses (DCs) can be used here, as well.
And I forgot to put an update here, since I already did that some time ago and extended it with pythons typing functionality :D
Reason for that: I was growing tired to manually check if the documentation of the API I was abstracting from was correct or if I got my implementation wrong.
With dataclasses.fields I'm able to check if the response does conform to my schema; and now I'm able to find changes in the external API much faster, since the assumptions are checked during run-time on instantiation.
DCs provide a __post_init__(self) hook to do some post-processing once the __init__ completed successfully. Pythons' type hints are only in place to provide hints for static checkers, I built a little system that does enforce the types on dataclasses in the post init phase.
Here is the BaseDC, from which all other DCs inherit (abbreviated)
import dataclasses as dc
@dataclass
class BaseDC:
def _typecheck(self):
for field in dc.fields(self):
expected = field.type
f = getattr(self, field.name)
actual = type(f)
if expected is list or expected is dict:
log.warning(f'untyped list or dict in {self.__class__.__qualname__}: {field.name}')
if expected is actual:
continue
if is_generic(expected):
return self._typecheck_generic(expected, actual)
# Subscripted generics cannot be used with class and instance checks
if issubclass(actual, expected):
continue
print(f'mismatch {field.name}: should be: {expected}, but is {actual}')
print(f'offending value: {f}')
def __post_init__(self):
for field in dc.fields(self):
castfunc = field.metadata.get('castfunc', False)
if castfunc:
attr = getattr(self, field.name)
new = castfunc(attr)
setattr(self, field.name, new)
if DEBUG:
self._typecheck()
Fields have an additional attribute that is allowed to store arbitary information, I'm using it to store functions that convert the response value; but more on that later.
A basic response wrapper looks like this:
@dataclass
class DCcore_enrol_get_users_courses(BaseDC):
id: int # id of course
shortname: str # short name of course
fullname: str # long name of course
enrolledusercount: int # Number of enrolled users in this course
idnumber: str # id number of course
visible: int # 1 means visible, 0 means hidden course
summary: Optional[str] = None # summary
summaryformat: Optional[int] = None # summary format (1 = HTML, 0 = MOODLE, 2 = PLAIN or 4 = MARKDOWN)
format: Optional[str] = None # course format: weeks, topics, social, site
showgrades: Optional[int] = None # true if grades are shown, otherwise false
lang: Optional[str] = None # forced course language
enablecompletion: Optional[int] = None # true if completion is enabled, otherwise false
category: Optional[int] = None # course category id
progress: Optional[float] = None # Progress percentage
startdate: Optional[int] = None # Timestamp when the course start
enddate: Optional[int] = None # Timestamp when the course end
def __str__(self): return f'{self.fullname[0:39]:40} id:{self.id:5d} short: {self.shortname}'
core_enrol_get_users_courses = destructuring_list_cast(DCcore_enrol_get_users_courses)
Responses that are just lists were giving me trouble in the beginning, since I could not enforce type checking on them with a plain List[DCcore_enrol_get_users_courses].
This is where the destructuring_list_cast solves that problem for me, which is a little more involved. We're entering higher order function territory:
T = typing.TypeVar('T')
def destructuring_list_cast(cls: typing.Callable[[dict], T]) -> typing.Callable[[list], T]:
def cast(data: list) -> List[T]:
if data is None:
return []
if not isinstance(data, list):
raise SystemExit(f'listcast expects a list, you sent: {type(data)}')
try:
return [cls(**entry) for entry in data]
except TypeError as err:
# here is more code that explains errors
raise SystemExit(f'listcast for class {cls} failed:\n{err}')
return cast
This expects a Callable that accepts a dict and returns a class instance of type T, which is something what you'd expect from a constructor or a factory.
It returns a Callable that will accept a list, here it's cast.
return [cls(**entry) for entry in data] does all the work here, by constructing a list of dataclasses, when you call core_enrol_get_users_courses(response.json()).
(Throwing SystemExit is not nice, but that's handled in the upper layers, so it works for me; I want that to fail hard and fast.)
It's other use case is to define nested fields, then the responses are deeply nested: remember the field.metadata.get('castfunc', False) in the BaseDC? That's where these two shortcuts come in:
# destructured_cast_field
def dcf(cls):
return dc.field(metadata={'castfunc': destructuring_list_cast(cls)})
def optional_dcf(cls):
return dc.field(metadata={'castfunc': destructuring_list_cast(cls)}, default_factory=list)
These are used in nested cases like this (see bottom):
@dataclass
class core_files_get_files(BaseDC):
@dataclass
class parent(BaseDC):
contextid: int
# abbrev ...
@dataclass
class file(BaseDC):
contextid: int
component: str
timecreated: Optional[int] = None # Time created
# abbrev ...
parents: List[parent] = dcf(parent)
files: Optional[List[file]] = optional_dcf(file)
cls.membersto be defined at a class level.selfdoesn't make sense in that context ofproperty– Craytoncomparative-reviewtag. – Gustavo