Natural Language Processing - Word Alignment

Question

I am looking for word alignment tools and algorithms.
I am dealing with bilingual English - Hindi text, and currently working on

• DTW (Dynamic Time Warping) algorithm
• CLA (Competitive Linking Algorithm)
• NATools
• Giza++

Could you please suggest any other algorithm/tool which is language independent and which could achieve Statistical word alignment for parallel English Hindi Corpora and its evaluation.
Some tools are best for certain languages; could you please tell me how true that is and, if so, could you please provide an example of what would be better suited for Asian languages like Hindi. Counter-examples of what one shouldn't I use for such languages is also welcome.

I have heard a bit about Uplug word aligner... Could someone tell me if this tool is useful for my purpose.

Thank you.. :)

Gressorial

;-) For a linguist, and even if English isn't your first language, you should work on sentence structure... The original question was "one long sentence"... – Agulhas

Point Sir...:) will keep it in mind from next time for sure.... – Gressorial

Answer 1

The Berkeley Aligner is very good. By doing joint training of the IBM word alignment models, it's able to get a much lower alignment error rate (AER) than older packages like GIZA++.

It also supports some more advanced features such as syntactic distortion (i.e., using parse tree information to get better alignments). For this, you'll only need parse trees for one of the language pairs. So, you should be okay doing Hindi<->English, since there are plenty of freely available and good English parsers.

If you decide not to go with the Berkeley Aligner, you should probably just use GIZA++. For years, it has been essentially the standard word aligner in the machine translation community.

Pollute

Thank you for your answer..:) I would like to know if any other tools for word alignment could be used, if you know some other tool please let me know.. – Gressorial

Answer 2

Uplug is a great tool, I have been using it for aligning English<->Macedonian texts. It essentially builds on the Giza++ by adding the so-called clue alignments. It's advanced setting actually combines the the clue alignments and Giza++ and performs 3 such iterations. The more clues (pos-tags, lemmas ... ) you provide better the results will be. But I have to mention that you should not expect to get fundamentally different results then by just using Giza++.

Anyway, if you plan to seriously study the topic of SMT, I suggest that you read the paper (phd thesis) about Uplug, it will be very beneficial for you.

Solifluction

Answer 3

Moses is a statistical machine translation suite you might want to take a look at. Its word alignment component is built on GIZA++ but may be tweaked to work better with certain language pairs than pure GIZA++. Their mailing list and the resources you can find on http://www.statmt.org/ may also be a better place to ask questions on this topic than SO. One thing you didn't say anything about but which I would consider even more problematic is where to get a parallel corpus Hindi <-> English.

Mattingly

I have an existing parallel EMILLE corpus for use Sir.. So that was really not an issue.. Thank you for your answer..:). I shall check out the link you have suggested. – Gressorial

Answer 4

You have a vague and broad question.

Try: http://scholar.google.com/scholar?q=algorithm+language+independent+statistical+word+alignment&hl=en&safe=off&client=firefox-a&hs=hJt&rls=com.ubuntu:en-US:official&um=1&ie=UTF-8&oi=scholart

for a list of papers in this area.

Crabwise

Thank you for your answer, somehow I didn't am not convinced why you found this question vague.. it might be different and distinct... anyways..:) thank you for your answer Charles Sir.... – Gressorial