How to design an application in a modular way?

Question

I am looking for pointers, suggestions, links, warnings, ideas and even anecdotical accounts about "how to design an application in a modular way". I am going to use python for this project, but advice does not need to necessarily refer to this language, although I am only willing to implement a design based on OOP.

Here's some context to understand where I come from and what I am trying to achieve...

---

My project will be a small application that will consume web services and display the results in a variety of ways, including:

• notification popup containing just the result of the call
• tab in the main window of the application with graphics plotted from retrieved raw-data
• buffer of messages (visible on domand) where results from various services will pile up

The application will be released as free (as-in-speech) software, and for this reason I would like to make it really easy for other devs to write plugins/modules that will extend the functionality of the main application without needing to change the core code.

At this point in time, plugins should essentially enable a developer to activate a new webservice, by defining the provider, the data manipulation (if any) and the way the data will be presented to the user.

I have extensive experience in developing with drupal which has a powerful modular approach, but that also follows a non-object-oriented design, so I suspect that for python, drupal design might not be the optimal solution.

If this is of any importance - the core will be natively developed for GNU/Linux.

Thank you in advance for your time!

Schlesien

Answer 1

Try to keep things loosely coupled, and use interfaces liberally to help.

I'd start the design with the Separation of Concerns. The major architectural layers are:

• Problem Domain (aka. Engine, Back-end): the domain classes, which do all the actual work, have domain knowledge implement domain behaviour
• Persistence: storage management for domain classes, database/filesystem layer
• User Interface: the GUI, which talks to the domain classes
• System Interfaces: talking to other systems, eg. networking, web services

The domain classes do the work, but don't know about the UI. The persistence layer knows about the domain classes, enough to save/load as required. The system interface layer abstracts away external systems, which lets you plug a simulator in behind while testing. The UI should ideally use MVC, for maximum flexibility.

Without putting too fine a point on it, one would not ordinarily look to Drupal as an exemplar of good architectural design. It has grown rather organically, and there have been many upheavals of the design, as evidenced by the regular plugin breakage upon system upgrades.

I would also echo what MicSim said, regarding carefully designing the plugin interface and writing multiple different plugins to exercise it. This is the only way to really flesh out the issues of how the app and plugins interact.

Equilibrate

It took 1+ years to decide this was the best answer. But here you go! Marked as "accepted" today! ;) – Schlesien

Thanks - better late than never! – Equilibrate

Answer 2

As you will deliver some basic functionality with your app, make sure that you code the part that should be extendable/replaceable already as a plugin by yourself. Then you'll best get a feeling about how your API should look like.

And to prove that the API is good, you should write a second and third plugin, because then you will discover that you made a lot of assumptions when writing the first one. Normally things clear up a bit after doing this 2nd and 3rd step.

Now, you should write one more plugin, because the last plugins you wrote resemble the first one in type, input data and presentation (maybe yet another weather webservice). Choose something total different, with absolutely different data, and you will see your API being still too tailored. (Else you did a good job!)

Efficient

Thanks for this (+1). Actually your answer resemble closely one of the points given in the google presentation linked by "The MYYN" in his answer! :) – Schlesien

Answer 3

Well, probably the first place to start is to sit down and figure out what the plug-in might need to fulfill its purpose.

You'd want to consider two main aspects in your design.

• How will your framework pass requests / receive responses from the plug-in?
• What helper classes or modules might be good to provide?

And probably also, since this sounds like a learning project.

• What do you want to write yourself, and what are you happy just to pick out of an existing library?

I'd also recommend developing some basic plugins as you design the API. The experience of having to actually use what you design will allow you to see where a given approach might be making things harder than they need to be.

Pentecostal

Thank you for your answer (+1). Some additional points: (1) "How will your framework pass requests / receive responses from the plug-in" is probably the main focus of my question. I am used to a system for which the core "invokes" hooks in all registered modules, but I do this procedurally, and I was interested in finding out how to do it in a OOP. Any hints on that? (2) This is a learning project [meaning: I do it for fun, for my own learning]. The main focus for me is modular design. I am happy to use libraries for those parts like plotting data, notification popups, etc... – Schlesien

(3) My idea is to have the "core" already shipped with some "core modules" providing example services. So I will go with your advice of using the API myself! :) – Schlesien

Re 1,2: One solution would be to create a PlugIn class to be used as a base class for all plugins. It should provide default implementations for all of your 'hooks' (Or throw NotImplemented for mandatory hooks) which can then be overridden by the plugin. – Pentecostal

Answer 4

• design the api for your app, carefully (How To Design A Good API and Why it Matters)
• make everything, which could be used independently a module, then group and build larger parts out of the simple parts (KISS)
• don't repeat yourself (DRY)
• write/publish short documentation frequently, for yourself and others (open source mantra) ...

Monolingual

Thank you for the answer (+1). The guy in the video is a bit verbose [he starts off saying the subject does not land itself to brief presentations... but I feel like disagree on that!], but it gives some concrete advice that I will try to stick to! :) – Schlesien

Answer 5

Look into the listener-subscriber pattern. Sooner or later, your app will be complex enough that you need to implement callbacks. When you hit that limit, use listener-subscriber (there's an implementation in wxPython).

For example, several modules will want to watch for new data from a number of feeds. Modules that link together might want to update themselves, based on new data.

Alialia

Thank you for this (+1). I was looking up on the GoF the Observer pattern (which I notice now is also known as "Publish-Subscribe"). I will research on the listener-subscriber one, then. It looks like a promising lead! – Schlesien