Machine learning in OCaml or Haskell?

Question

I'm hoping to use either Haskell or OCaml on a new project because R is too slow. I need to be able to use support vectory machines, ideally separating out each execution to run in parallel. I want to use a functional language and I have the feeling that these two are the best so far as performance and elegance are concerned (I like Clojure, but it wasn't as fast in a short test). I am leaning towards OCaml because there appears to be more support for integration with other languages so it could be a better fit in the long run (e.g. OCaml-R).

Does anyone know of a good tutorial for this kind of analysis, or a code example, in either Haskell or OCaml?

Tucci

Just a comment to say you can integrate C program (or even Fortran) into R relatively easily this may be a more sensible approach than forgetting R altogether :) – Sharpsighted

Just for completeness sake, the issue of programming languages for machine learning is the subject of an interesting discussion here. – Phrasal

You should also check out FACTORIE, a Scala machine learning framework. – Revanchism

I recommend you scikit-learn with python. There is no much diference between R and scikit in perfomance issues. – Parenteral

"R is too slow" … people should check out pqR and snow / foreach / doParallel . – Irregular

Answer 1

Hal Daume has written several major machine learning algorithms during his Ph.D. (now he is an assistant professor and rising star in machine learning community)

On his web page, there are a SVM, a simple decision tree and a logistic regression all in OCaml. By reading these code, you can have a feeling how machine learning models are implemented in OCaml.

Another good example of writing basic machine learning models is Owl library for scientific and numeric computations in OCaml.

I'd also like to mention F#, a new .Net language similar to OCaml. Here's a factor graph model written in F# analyzing Chess play data. This research also has a NIPS publication.

While FP is suitable for implementing machine learning and data mining models. But what you can get here most is NOT performance. It is right that FP supports parallel computing better than imperative languages, like C# or Java. But implementing a parallel SVM, or decision tree, has very little relation to do with the language! Parallel is parallel. The numerical optimizations behind machine learning and data mining are usually imperative, writing them pure-functionally is usually hard and less efficient. Making these sophisticated algorithms parallel is very hard task in the algorithm level, not in the language level. If you want to run 100 SVM in parallel, FP helps here. But I don't see the difficulty running 100 libsvm parallel in C++, not to consider that the single thread libsvm is more efficient than a not-well-tested haskell svm package.

Then what do FP languages, like F#, OCaml, Haskell, give?

1. Easy to test your code. FP languages usually have a top-level interpreter, you can test your functions on the fly.

2. Few mutable states. This means that passing the same parameter to a function, this function always gives the same result, thus debugging is easy in FPs.

3. Code is succinct. Type inference, pattern matching, closures, etc. You focus more on the domain logic, and less on the language part. So when you write the code, your mind is mainly thinking about the programming logic itself.

4. Writing code in FPs is fun.

Dior

"FP supports parallel computing better". Only in theory. In practice, functional languages like OCaml and Haskell have some of the worst support for parallel programming in existence. Try to write an efficient generic parallel quicksort in either of those languages, for example. It is incredibly hard (for no good reason) and you cannot attain competitive performance with them. – Catadromous

@JonHarrop --the big strength of quicksort is that it is in-place, which is hard to translate to functional languages. On the other hand the "my first mergesort" in Haskell is parallel with only a single par – Sneer

Looks like all his stuff is in haskell now? – Hermaphroditism

@PhilipJF "the 'my first mergesort' in Haskell is parallel with only a single par". The sole purpose of parallelism is performance. What is the absolute performance of that parallel mergesort in Haskell like on the widest multicore desktop available today? Probably 1,000x slower than Sedgewick's quicksort in C... – Catadromous

@JonHarrop That is possible, but I think this is a bit silly. First of all, "absolute performance" difference is not the right metric--since it depends on both the comparison function and the size of the data to be sorted. Also, the "my first mergesort" works with linked lists, which will add a huge performance cost to the C code also. Something like QSort is fast, but not that as fast as a non generic sort because of all the dynamic calls to the comparison function (std::sort in C++ avoids this problem, as can the Haskell with a specialize pragma). So? Optimized code tends to be faster... – Sneer

@PhilipJF: "which will add a huge performance cost to the C code also". The objective is to speed up the Haskell code, not slow down the C code. – Catadromous

Answer 2

The only problem I can see is that OCaml doesn't really support multicore parallelism, while GHC has excellent support and performance. If you're looking to use multiple threads of execution, on multiple calls, GHC Haskell will be a lot easier.

Secondly, the Haskell FFI is more powerful (that is, it does more with less code) than OCaml's, and more libraries are avaliable (via Hackage: http://hackage.haskell.org ) so I don't think foreign interfaces will be a deciding factor.

Seidule

Wow, the irony here is amazingly thick. I sincerely hope Cuoq and Harrop aren't representative of the OCaml and F# communities. – Strachey

I don't think I can agree with your friend since all languages are written by programmers. :) – Precast

...but not all programmers are bad and languages are often written by good ones – Wordless

for machine learning applications, CPU multicore parallelism is not that important though. For performance gain, algorithms often resort to run massively parallel operations on gpu instead, and a good ML library will allow you to do that – Sexcentenary

Answer 3

As far as multi-language integration goes, combining C and Haskell is remarkably easy, and I say this as someone who is (unlike dons) not really much of an expert on either. Any other language that integrates well with C shouldn't be much trickier; you can always fall back to a thin interface layer in C if nothing else. For better or worse, C is still the lingua franca of programming, so Haskell is more than acceptable for most cases.

...but. You say you're motivated by performance issues, and want to use "a functional language". From this I infer you're not previously familiar with the languages you ask about. Among Haskell's defining features are that it, by default, uses non-strict evaluation and immutable data structures--which are both incredibly useful in many ways, but it also means that optimizing Haskell for performance is often dramatically different from other languages, and well-honed instincts may lead you astray in baffling ways. You may want to browse performance-related topics on the Haskell wiki to get a feel for the issues.

Which isn't to say that you can't do what you want in Haskell--you certainly can. Both laziness and immutability can in fact be exploited for performance benefits (Chris Okasaki's thesis provides some nice examples). But be aware that there'll be a bit of a learning curve when it comes to dealing with performance.

Both Haskell and OCaml provide the lovely benefits of using an ML-family language, but for most programmers, OCaml is likely to offer a gentler learning curve and better immediate results.

Strachey

Answer 4

It's hard to give a definitive answer on this. Haskell has the advantages that Don mentioned along with having a more powerful type system and cleaner syntax. OCaml will be easier to learn if you coming from almost any other language (this is because Haskell is as function as functional languages get), and working with mutable random access structures can be a little clunky in Haskell. You will also likely find the performance characteristics of your OCaml code more intuitive than Haskell because of Haskell's lazy evaluation.

Really, I would recommend you evaluate both if you have the time. Here are some relevant Haskell resources:

• http://hackage.haskell.org/package/hslibsvm
• http://hackage.haskell.org/package/HSvm
• Real World Haskell: this is a great freely available book for Haskell
• Learn You a Haskell: this tutorial is just plain fun to read

Oh, if you look further into Haskell be sure to sign up for the Haskell Beginners and Haskell Cafe lists. The community is friendly and eager to help out newcomers (is my bias showing?).

Accomplished

You might like to mention what these resources are. For example, HSvm is an old Haskell binding to a C++ library that never made it out of alpha release. – Catadromous

BTW, your statement about "powerful" type systems doesn't really make sense. OCaml can infer algebraic data types, has structural types and subtypes and a vastly more powerful higher-order module system. They are just different. – Catadromous

4.0 + 3.0;; Error: This expression has type float but an expression was expected of type int – Accomplished

In Haskell, Foo gives Not in scope, data constructor Foo. Haskell failed to infer the type... – Catadromous

Answer 5

If speed is your prime concern then go for C. Haskell is pretty good performance wise but you are never going to get as fast as C. To my knowledge the only functional language that has bettered C in a benchmark is Stalin Scheme but that is very old and nobody really knows how it works.

I've written genetic programming libraries where performance was key and I wrote it in a functional style in C. The functional style allowed me to easily parallelise it using OMP and it scales linearly upto 8 cores within a single process. You certainly can't do that in OCaml although Haskell is improving all the time with regards to concurrency and parallelism.

The downside of using C was that it took me months to finally find all the bugs and stop the core dumps which was extremely challenging because of the concurrency. Haskell would probably have caught 90% of those bugs on the first compilation.

So speed at any cost ? Looking back I'd wish I'd used Haskell as I could stand it to be 2 - 3 times slower if I'd saved over a month in development time.

Dollarbird

As an update, I did rewrite my library in Haskell and the code was just beautiful in Haskell with the core library going from 1200 lines of C code to just over 100 lines of Haskell. Performance was about 4 times slower than C but I'm now looking at using the GPU accelerated Data.Array library to massively parallelise key parts on many GPU's. I had also looked at doing this in C but it would have meant a huge painful rewrite. – Dollarbird

Answer 6

While dons is correct that multicore parallelism at the thread level is better supported in Haskell, it sounds like you could live with process level parallelism (from your phrase: ideally separating out each execution to run in parallel.) which is supported quite well in OCaml. Keith pointed out that Haskell has a more powerful type system, but it can also be said that OCaml has a more powerful module system than Haskell.

As others have pointed out, OCaml's learning curve will be lower than Haskell's; you'll likely be more productive more quickly in OCaml. That said, learning OCaml is a great stepping-stone towards learning Haskell because many of the underlying concepts are very similar, so you could always migrate to Haskell later and find a lot of things familiar there. And as you pointed out, there is an OCaml-R bridge.

Undercarriage

Answer 7

As an examples of Haskell and Ocaml in machine learning see stuff at Hal Daume and Lloyd Allison homepages. IMO it's is much more straightforward to achieve C++-like performance in Ocaml, than in Haskell. Through, as already said, Haskell has much nicer community (packages, tools and support), syntax&features (i.e. FFI, probability monads via typeclasses) and parallel programming support.

Allethrin

Answer 8

Having revamped OCaml-R, I've got a few comments to make on integrating OCaml and R. It might be worthwile to use OCaml to call R code, it works, but is not yet exactly straightforward. So using it to pilot R is worthwile. Integrating R functionality much more thoroughly is still cumbersome as, for example, much remains to be done to export R's type system and data to OCaml in a seamless way (you will have work to do). Moreover, the interaction of R's GC and OCaml's GC is a delicate point: you free n values in O(n^2) time, which isn't nice (to solve this point, you either need a more flexible R API, as far as I understand it, or to implement a GC in the binding itself as a big R array for proper interaction between GCs).

In a nutshell, I'd go for the "pilot R from OCaml" approach.

Contributions on the GC interaction layer and on mapping R datatypes to OCaml are most welcome.

Cartie

Answer 9

You may want to take a look at this : http://www.haskell.org/pipermail/haskell-cafe/2010-May/077243.html

Shedevil

I'd be interested to know if you get much help. – Catadromous

We're a few people involved, and I've been busy with exams but I'm likely to kick-start some development very soon -- it's hard to get free time with the GSoC on-going. – Shedevil

Answer 10

Late answer but a machine learning library in Haskell is available here : https://github.com/mikeizbicki/HLearn

This library implements various ML algorithms who are designed to have a much faster cross-validation than the usual implementations. It is based on the following paper Algebraic classifiers: a generic approach to fast cross-validation, online training, and parallel training. The authors claims a 400x speed-up compared to the same task in Weka.

Macadam

Answer 11

for haskell, consider checking hasktorch (which I managed to use for my AI thesis). for ocaml there seem to be tensorflow bindings.

Agrestic

Good to hear you managed to use Hasktorch for your thesis! From my limited experience with it, it is however unfortunately not really a good choice, specifically with memory leaks on the GPU. At the same time, it doesn't get Haskell's type system into the game nearly as well as I'd like. So, alas, kind of worst of both worlds between Haskell and Python. (That's not meant to undercut the effort that the Hasktorch developers made – I think it's great they did that, but I also feel I should be honest so people don't expect to much. If your experience differs, I'd like to read more about it.) – Flagella

@Flagella hey, thanks for commenting! I haven't used GPU much yet, tho I admittedly ran into a memory leak issue when I tried implementing weight decay as well. as for types two interfaces are offered so you have some choice there, but I guess for either bugs or types, note it is still a work in progress, and the team there would be glad to receive your feedback. :) – Agrestic