Python on an Real-Time Operation System (RTOS)

Question

I am planning to implement a small-scale data acquisition system on an RTOS platform. (Either on a QNX or an RT-Linux system.)

As far as I know, these jobs are performed using C / C++ to get the most out of the system. However I am curious to know and want to learn some experienced people's opinions before I blindly jump into the coding action whether it would be feasible and wiser to write everything in Python (from low-level instrument interfacing through a shiny graphical user interface). If not, mixing with timing-critical parts of the design with "C", or writing everything in C and not even putting a line of Python code.

Or at least wrapping the C code using Python to provide an easier access to the system.

Which way would you advise me to work on? I would be glad if you point some similar design cases and further readings as well.

Thank you

NOTE1: The reason of emphasizing on QNX is due to we already have a QNX 4.25 based data acquisition system (M300) for our atmospheric measurement experiments. This is a proprietary system and we can't access the internals of it. Looking further on QNX might be advantageous to us since 6.4 has a free academic licensing option, comes with Python 2.5, and a recent GCC version. I have never tested a RT-Linux system, don't know how comparable it to QNX in terms of stability and efficiency, but I know that all the members of Python habitat and non-Python tools (like Google Earth) that the new system could be developed on works most of the time out-of-the-box.

Godliman

can you give a hint on timing requirements? What frequencies/response times do you need? seconds or microseconds? Looking at your RTOS I assume you have either a PC or a powerful embedded platform. Is this right? – Indiscipline

For most of the measurements 1Hz sample rate is satisfactory. However there are instruments which need to be sampled at high rates around 100Hz. Usually super-fast measuring devices (such as a Cloud Particle Imager) comes with their dedicated data system --which these are beyond the scope of my initial intention. And yes the current system runs on a PC for the acquisition tasks where lots of boards on it to interface with various equipments. I think it would be right to call it as an embedded platform rather than just a typical desktop PC. – Ballista

Answer 1

I can't speak for every data acquisition setup out there, but most of them spend most of their "real-time operations" waiting for data to come in -- at least the ones I've worked on.

Then when the data does come in, you need to immediately record the event or respond to it, and then it's back to the waiting game. That's typically the most time-critical part of a data acquisition system. For that reason, I would generally say stick with C for the I/O parts of the data acquisition, but there aren't any particularly compelling reasons not to use Python on the non-time-critical portions.

If you have fairly loose requirements -- only needs millisecond precision, perhaps -- that adds some more weight to doing everything in Python. As far as development time goes, if you're already comfortable with Python, you would probably have a finished product significantly sooner if you were to do everything in Python and refactor only as bottlenecks appear. Doing the bulk of your work in Python will also make it easier to thoroughly test your code, and as a general rule of thumb, there will be fewer lines of code and thus less room for bugs.

If you need to specifically multi-task (not multi-thread), Stackless Python might be beneficial as well. It's like multi-threading, but the threads (or tasklets, in Stackless lingo) are not OS-level threads, but Python/application-level, so the overhead of switching between tasklets is significantly reduced. You can configure Stackless to multitask cooperatively or preemptively. The biggest downside is that blocking IO will generally block your entire set of tasklets. Anyway, considering that QNX is already a real-time system, it's hard to speculate whether Stackless would be worth using.

My vote would be to take the as-much-Python-as-possible route -- I see it as low cost and high benefit. If and when you do need to rewrite in C, you'll already have working code to start from.

Epidemic

I haven't used Stackless before, nor I have any idea about its integration with many core scientific tools. One of the reason that I want to stay with an RT-Linux is that it has all the Python tools and 2D / 3D visualization libraries working very well. However on the QNX side, lots of libraries are missing and I am sure making them work on QNX would take an immense amount of effort. Any comments on this would be greatly appreciated. – Ballista

Stackless only modifies an existing Python installation -- it replaces a couple core files that shouldn't affect any Python libraries. It moves away from using the C stack, but it shouldn't affect even libraries written in C that interface with Python. Play around with Stackless a little bit (really easy to install on Windows), see if it suits your needs. I haven't built Stackless from source, so I can't comment on any foreseen difficulties with QNX. – Epidemic

Answer 2

I've built several all-Python soft real-time (RT) systems, with primary cycle times from 1 ms to 1 second. There are some basic strategies and tactics I've learned along the way:

1. Use threading/multiprocessing only to offload non-RT work from the primary thread, where queues between threads are acceptable and cooperative threading is possible (no preemptive threads!).

2. Avoid the GIL. Which basically means not only avoiding threading, but also avoiding system calls to the greatest extent possible, especially during time-critical operations, unless they are non-blocking.

3. Use C modules when practical. Things (usually) go faster with C! But mainly if you don't have to write your own: Stay in Python unless there really is no alternative. Optimizing C module performance is a PITA, especially when translating across the Python-C interface becomes the most expensive part of the code.

4. Use Python accelerators to speed up your code. My first RT Python project greatly benefited from Psyco (yeah, I've been doing this a while). One reason I'm staying with Python 2.x today is PyPy: Things always go faster with LLVM!

5. Don't be afraid to busy-wait when critical timing is needed. Use time.sleep() to 'sneak up' on the desired time, then busy-wait during the last 1-10 ms. I've been able to get repeatable performance with self-timing on the order of 10 microseconds. Be sure your Python task is run at max OS priority.

6. Numpy ROCKS! If you are doing 'live' analytics or tons of statistics, there is NO way to get more work done faster and with less work (less code, fewer bugs) than by using Numpy. Not in any other language I know of, including C/C++. If the majority of your code consists of Numpy calls, you will be very, very fast. I can't wait for the Numpy port to PyPy to be completed!

7. Be aware of how and when Python does garbage collection. Monitor your memory use, and force GC when needed. Be sure to explicitly disable GC during time-critical operations. All of my RT Python systems run continuously, and Python loves to hog memory. Careful coding can eliminate almost all dynamic memory allocation, in which case you can completely disable GC!

8. Try to perform processing in batches to the greatest extent possible. Instead of processing data at the input rate, try to process data at the output rate, which is often much slower. Processing in batches also makes it more convenient to gather higher-level statistics.

9. Did I mention using PyPy? Well, it's worth mentioning twice.

There are many other benefits to using Python for RT development. For example, even if you are fairly certain your target language can't be Python, it can pay huge benefits to develop and debug a prototype in Python, then use it as both a template and test tool for the final system. I had been using Python to create quick prototypes of the "hard parts" of a system for years, and to create quick'n'dirty test GUIs. That's how my first RT Python system came into existence: The prototype (+Psyco) was fast enough, even with the test GUI running!

-BobC

Edit: Forgot to mention the cross-platform benefits: My code runs pretty much everywhere with a) no recompilation (or compiler dependencies, or need for cross-compilers), and b) almost no platform-dependent code (mainly for misc stuff like file handling and serial I/O). I can develop on Win7-x86 and deploy on Linux-ARM (or any other POSIX OS, all of which have Python these days). PyPy is primarily x86 for now, but the ARM port is evolving at an incredible pace.

Affinal

Late to the question but thanks for this detailed answer. For me this was the most interesting part: Careful coding can eliminate almost all dynamic memory allocation, in which case you can completely disable GC. Can you please give some examples about that careful coding part? – Tigon

Answer 3

Generally the reason advanced against using a high-level language in a real-time context is uncertainty -- when you run a routine one time it might take 100us; the next time you run the same routine it might decide to extend a hash table, calling malloc, then malloc asks the kernel for more memory, which could do anything from returning instantly to returning milliseconds later to returning seconds later to erroring, none of which is immediately apparent (or controllable) from the code. Whereas theoretically if you write in C (or even lower) you can prove that your critical paths will "always" (barring meteor strike) run in X time.

Menashem

I agree with almost all of this, but critical path (and particularly worst case execution time) analysis in any language is a really horrible problem, and depends significantly on both hardware platform (caches, pipeline, branch predictors, page faults) and software (OS services, interrupts, blocking time). If timing is that critical Ada is probably essential – Backsaw

Yeah, take with as many grains of salt as you like. I think maybe my argument should be "if all of this stuff sounds like wacko nonsense to you, then you're not realtime enough to worry about what language you're writing in". – Menashem

At this point no lower than C. and no intention of calling assembly opcodes from Python using a tool like CorePy. Time criticality is important, but not that important to switch me to use Ada. – Ballista

Answer 4

Our team have done some work combining multiple languages on QNX and had quite a lot of success with the approach. Using python can have a big impact on productivity, and tools like SWIG and ctypes make it really easy to optimize code and combine features from the different languages.

However, if you're writing anything time critical, it should almost certainly be written in c. Doing this means you avoid the implicit costs of an interpreted langauge like the GIL (Global Interpreter Lock), and contention on many small memory allocations. Both of these things can have a big impact on how your application performs.

Also python on QNX tends not to be 100% compatible with other distributions (ie/ there are sometimes modules missing).

Backsaw

and python for QNX is hard to find in an up to date version... and not always easy to build for QNX – Freese

@Fuzz, 2.5 is on QNX 6.4.x, and if my memory serves me right PyQt 4 binaries should be somewhere on a third-party repository. I haven't seen a NumPy port as of yet, also haven't tried to build any library myself. I am sure lots of hacking would be needed to make some of them half working. That's why I consider using a RT-Linux solution over a QNX, but I need more input and sources on this. – Ballista

@Andrew, could you please give me some pointers of where to find such projects utilizing Python - C approach on a RTOS system? Apparently, this is somewhat hidden on the web, need to dig deeper or just take to the trial and error approach. – Ballista

The builds for python on QNX are provided and hosted by Crank software cranksoftware.com/downloads. You'll get some good advice on OpenQNX, openqnx.com, and at Foundary27 community.qnx.com/sf/sfmain/do/home As for getting started, it's pretty much dependent on what you're looking todo. We build cross platform, and so our use of QNX specific features is minimal. But if I could give one suggestion, make sure that if you've lots of time critical IO (via COM ports, custom hardware or IPC), make sure you do some prototyping to get your design right. – Backsaw

Answer 5

One important note: Python for QNX is generally available only for x86.

I'm sure you can compile it for ppc and other archs, but that's not going to work out of the box.

But