How to organize Python API module to make it neat?

Question

I'm writing a Python library that represents some web API. Right now, my library directory looks close to this:

• __init__.py
• Account.py
• Order.py
• Category.py
• requests.py

In __init__.py, I have something like this:

from .Account import Account
from .Order import Order
from .Category import Category
from . import requests

This allows to use import cool_site and then cool_site.Account(…) and so on, but it has a following problem: when I play around with my code in IDLE, the object is then called cool_site.Account.Account, which I feel is bad.

1. Is there any way to avoid class name duplication and still have separate file for every class?

The next thing I don't feel great about is my code organization. Right now, my Account class takes credentials on initialization, creates a requests.Session object and then handles all communication with server, i.e. searching for orders and so on. This Account class instance will then pass itself to all other instances, for example to Order - so the order's instance will have .account property holding the Account instance which created it. When another class instance itself has to do something, for example change an order's comment (by calling o.comment = 'new comment', so by @comment.setter decorator in the Order class), it forwards that to an Account object which is passed to it on initialization, and then uses for example self.account.set_order_comment(new_comment). This method will then use all the web requests to achieve that goal.

2. Is it better to hold server communication logic in one class or to spread different aspects of it to classes that are affected by them?

The last thing I'd like to ask about is how and where to keep low-level request templates. Right now I have it in cool_site.requests submodule, and there are different functions for different requests, for example SetOrderComment for the case mentioned above (it's a function, so it should be lowercase, but in this case I think it resembles a class in a way - is that OK?). The Account.set_order_comment will use it like this:

r = cool_site.requests.SetOrderComment(order.id, new_comment)
response = self._session.request(**r)

because this function returns a dict with arguments to Session.request function from requests library. The authentication headers are already set in the _session property of Account class instance. I feel it's a little bit ugly, but I don't have any better idea.

3. How to organize web requests to keep it all clean?

Post scriptum

I'm sorry this question is so long and covers many aspects of API library design, but all the tips will be appreciated. In a way, all of the three questions above could be expressed as "How to do it better and cleaner?" or "How most of the Python developers do it?", or maybe even "What would feel most Pythonic?".

Throw at me any little tips you can think of.

Biweekly

Files names are mostly not capitalized, especially not Python modules. This would mean you have something like cool_site.account.Account. Doesn't really solve your questions, but maybe it helps to know that many core modules have similar naming problems: e.g.: datetime.datetime, copy.copy, ... (note that datetime is a class even if it is not capitalized, similar to int or float) – Glyco

This is the kind of question I so rarely hear around me at work, and its even more rare when coming from interns. Thinking about how it should be used and how it should be done, who's responsible of what. Problem is that it could attract opinion-based answers, whereas everyhing is always about tradeoff. Which raises other questions : what are the tradeoff criterias ? what is the room for maneuver ? – Benilda

@LoneWanderer: well, this library it's actually just for me, but I'd really, really like to do it like it's big, popular and everything — just to learn how it should be done and develop good habits. – Biweekly

I guess it sounds a little silly when it's phrased like this, but really, the main objective here is to learn that good habits even in the internal design, which often may look like not the most important thing — but I think I shouldn't just settle for whatever came first to my mind. – Biweekly

Answer 1

I've been thinking about very similar things lately working on wistiapy. Examples of my current thinking about client code organisation are in there. YMMV.

1. "One class per file" is more of a Java style guideline than Python. Python modules are a legitimate and important level in the code hierarchy, and you shouldn't worry about having more than one function or class in the same module. You could put all the model classes in a .models module, and then from .models import (Account, Order, Category) in __init__.py.

2. More-or-less common practice for client libraries seems to be to have a client module, containing something like a MyServiceClient class. (eg the Segment client). This is where the networking logic goes. If you want to have the public interface be module-level functions, you can do some clever stuff with creating a default client and having the functions call methods on that.

Functions should be snake_case, classes should be PascalCase. Doing anything else tends to cause more confusion than benefit.

It seems like the big question you're dealing with is trying to choose between the "Active Record" pattern (some_account.set_order_comment(comment)), and the "Data Mapper" pattern (set_order_comment(account, comment)). Either will work and they each have their benefits and drawbacks. I've found the data mapper pattern -- using smart functions to manipulate fairly simple data classes -- simpler to begin with.

I find it helpful to design the public interface concurrently with something that uses that interface. In the calling code, you can write what you'd like to have to call, and then implement the client code "outside-in".

Falsehood

Not a "case" expert, but I think this is known as snake_case. – Groundsheet

Thank you for your answer. However, about the "active record" / "data mapper" pattern — well, I do not intend to choose between two of them. I tried to make that clear in my question, but maybe it wasn't after all. The only reason I called account.set_order_comment is because I keep communication logic in Account class. Outside, I'd like to use simpler things, like the @comment.setter decorator. But the bigger question here is where to store communication logic — in Account, which has its own credentials already, in Order, which would pull them from Account, or somewhere else. – Biweekly

Communication logic should all be kept in the same place - within a class called Client or API or something. That class should have methods that you can call from methods of your models like Account and Category, if that's what you want the external interface to look like. – Falsehood

Answer 2

1) no upper case in names of .py file (also try to avoid _) so your files should be

__init__.py
account.py
order.py
category.py
requests.py

2) if you want to use like cool_site.Account you need to add to __init__.py

from .account import Account
from .order import Order
from .category import Category

__all__ = [
    'Account',
    'Order',
    'Category',
]

3) SetOrderComment is bad name, use set_order_comment

4) If you write a python wrapper for communication with API, make method that do authorisation and other other stuff that is same in every API request. This method should take as params part of requests kwargs that are different for different API calls

for example

class API:
    def __init__(self, endpoint:s str, api_key: str):
        self.endpoint = endpoint
        self.api_key = api_key

    def _get_auth_headers(self) -> Dict[str, str]:
        return {
           'Authorization': 'Bearer ' + self.api_key,
        }

    def get(self, path, params)
         resp = requester.get(
            self.endpoint + path,
            headers=self._get_auth_headers(),
            params=params,
            timeout=30,
        )
        self._check_api_response(resp)
        payload = resp.json()
        return payload

5) If you write a python API look at flask and django frameworks and projects with API written on them. You should find some good ides overthere.

Elapse

About 4: that seems like a nice idea, but it still doesn't quite address my question. Which object should call that API in my example with changing order's comment – an account which an order is bound to, or maybe an order itself? Where should requests be kept and in what form? – Biweekly

Answer 3

I can throw in an easy tip: You could use __all__ = ["Account"] in your Account.py module and then use from .Account import * in your __init__.pyfile. I believe that addresses your first question. You can read up on the __all__ magic method here. In short, you can specify what methods etc you would like to import when you use from x import *. You can hide 'private' methods from being imported with _one_leading_underscore.

Saffren

That seems like a nice trick, I'll definitely try that! Thank you. – Biweekly