Large scale machine learning - Python or Java? [closed]

Question

I am currently embarking on a project that will involve crawling and processing huge amounts of data (hundreds of gigs), and also mining them for extracting structured data, named entity recognition, deduplication, classification etc.

I'm familiar with ML tools from both Java and the Python world: Lingpipe, Mahout, NLTK, etc. However, when it comes down to picking a platform for such a large scale problem - I lack sufficient experience to decide between Java or Python.

I know this sounds like a vague question, and but I am looking for general advice on picking either Java or Python. The JVM offers better performance(?) over Python, but are libraries like Lingpipe etc. match up with the Python ecosystem? If I went this Python, how easy would it be scaling it and managing it across multiple machines etc.

Which one should I go with and why?

Gwendolin

Jython :-) – Quadrangular

The algorithms used are as important (or more) than language in large scale ML tasks; for example, eigenvector based techniques may be unfeasible when dealing with a lot of data. So while it is worth thinking about each language's available tools for large scale problems, the ML algorithms used may dominate performance. – Ambivert

Your question is very interesting but also very very broad, I don't even know what I could put in an answer. A word of advice would be to use Mahout (project from Hadoop) which directly addresses your concern. Try to make your question a lot more explicit and detailed, or it will probably be closed even if it has potential. – Shanitashank

Answer 1

As Apache is going strong producing excellent stuff like Lucene/Solr/Nutch for Search, Mahout for Big Data Machine Learning, Hadoop for Map Reduce, OpenNLP for NLP, lot of NoSQL stuff. The best part is the big "I" which stands for integration and these products can be integrated with each other well as of course in most situations they (these products) complement each other.

Python is great too however if you consider above from ASF then I will go with Java like Sean Owen. Python will always be available for the above but mostly like Add on's and not the actual stuff. For example you can do Hadoop using Python by using Streaming etc.

I partially switched from C++ to Java in order to utilize some of the very popular Apache products like Lucene, Solr & OpenNLP and also other popular open source NoSQL Java products like Neo4j & OrientDB.

Ardys

Answer 2

I think one big thing Java has going for it is Hadoop. If you really mean large scale, you'll want to be able to use something like that. Generally speaking Java has the performance advantage, and more libraries available. So: Java.

Rambling

Answer 3

If you are looking at NoSQL databases fit for ML task, then Neo4J is one of the more production ready (relatively) and capable of handling BigData, it is native to JAVA but comes along with a beautiful REST API out of the box and hence can be integrated with the platform of your choice. JAVA will give you an performance edge here.

Trimester