Assembly language for Reverse Engineering [closed]

Question

What should I choose NASM or MASM for learning assembly. I want to learn assembly, motivation being Reverse Engineering.

So that when I disassemble some executable, I can understand the code by looking at disassembled code.

Update: I think I dint make my self clear.. I understand those are assemblers, but to understand the output of a disassembler I need to know assembly and that's the reason I'm asking where to start(with MASM or NASM)

Dashpot

Pick any and get going. Once you actually learn a bit of assembly, you'll realise that this question isn't very useful. – Unprovided

@KerrekSB- But I have heard that MASM has a lot of higher level constructs.. which I wont see when I disassemble, but its not the case with NASM. What will you say about this? – Dashpot

@questions: Then skip the parts of MASM you are not interested in. That would be most of the pseudo-ops. However, there are some important pseudo ops: those to create various memory segments like code, data, bss, etc. It is important to know about those in all assemblers. – Fifine

@wallyk- Being a beginner.. how do I know if something is MASM specific and skip it? Anyway, so choosing any would be good? – Dashpot

A question with a great potential :) – Fant

@ZoZo123- Thanks.. but am not still getting a precise answer.My previous question was similar, dint get many answers there too.. – Dashpot

@questions: I doubt that you will be able to understand any large project by reading the disassembler... – Canthus

@DavidRodríguez-dribeas- I agree.. it will be hard, but isn't saying that you wont be able to understand discouraging? Anti-virus companies understand complicated viruses(ex. Stuxnet) by looking at its disassembled code. – Hemato

@Dashpot - You are actually asking what writers course is best for learning how to read books. That is not the right question to ask. – Derwon

possible duplicate of NASM is pure assembly, but MASM is high level Assembly? – Derwon

This question would not get closed if it was phrased more carefully. E.g. "how to learn reverse-engineering on Windows" -- and the description would be "I want to be able to reverse-engineer a Windows program -- what are the key things I need to learn and in what order?" Also, should have made it a "community wiki"-type question -- these get less criticism for being too open-ended. If you want to try and ask another question, phrased the way I said, I will repost my answer over there and we'll see what other useful answers we can get. – Ullyot

Answer 1

Assuming you want to learn how to do reverse-engineering on Windows, here's how. The Linux Way is very different (although eventually you get to more or less the same place):

You want to start with something very simple, like 8080 processor. You can find an online emulator here: http://www.tramm.li/i8080 -- this is a legacy OS called "CP/M". Click on "Start EMU8080" link -- you will get a command line of "SID" -- a simple ancient debugger. It only has like 10 commands -- to browse memory at any address, to enter new assembly commands etc. Type HELP to see them all. You can try and google for a SID manual if you want. Then google for 8080 assembly manual -- you will find a bunch of PDFs -- they will all be copies of the same 2 manuals written by Intel in 1970s. Read both -- they will give you a good start into 808x CPU basics. All modern processors used in PCs are derived from 8080. Play with SID and 8080 until you feel like you got the idea.

Next step is 8086. For this you'll need to get a copy of Window XP and run DEBUG program in command line. DEBUG is Microsoft's rip-off of SID. It will be very easy to learn once you know SID. The documentation is here: http://thestarman.pcministry.com/asm/debug/debug.htm -- read as much as you can, also make sure to follow the links to other pages -- the same site contains bunch of stuff on basic 8086 programming. If by that time you will have internalized the 8080 manuals, 8086 will feel super-easy. (If you skip the 8080/SID phase, everything will feel weird and the puzzle will take more time to sort-out, trust me.)

You may also want to play with http://ladsoft.tripod.com/grdb_debugger.html -- an improved clone of DEBUG, easier to work with.

Next step, get a book by Peter Abel called "IBM PC Assembler Language and Programming" -- the older edition, the better. You can find one on Amazon for like 1 buck (I'm not kidding!). Read first 7 chapters. Get yourself assembler called "FASM" (forget MASM and NASM crap). Read FASM documentation. Write a program in 16 bit real mode. You can still run them on Windows XP command line.

Next step, read http://www.drpaulcarter.com/pcasm/ -- this will help you understand protected mode. Then read the actual Intel 386 reference documentation: http://www.logix.cz/michal/doc/i386/

Then buy a book called "Reversing. Secrets of Reverse Engineering". It's pretty lame, but will give you some ideas on what you need to learn next. Like low-level Windows programming (messages, handlers etc.)

Once you got through all this, only then can you download http://www.ollydbg.de/ and try playing with it. Be sure to get version 1.x, not 2.x. Go to Youtube and find some tutorials. You are welcome to skip over to this step, but I can guarantee you will get frustrated and won't understand anything you see on the screen unless you follow through all of the above.

Good luck. If you have any questions you can contact me via my blog (the link is on my profile page).

Ullyot

Wow.. that was in great detail. I'm following that.. Thanks zvolkov :) – Dashpot

Updating a bit the thing, DOSBox is a very good place to run 8086 programs. But they can also run on Windows 10 or 11 or whatever 64-bit --> EMU8086, check that program. Emulates the 8086 on new computers. Amazingly slow to compile bigger programs though, but I just used to as an interactive debugger (to run to test, it would be on DOSBox) while writing the programs with Sublime Text and compiling with MASM (in my case 2 or 3 years ago when I learned Assembly). – Oversew

Answer 2

You're talking about disassembly here. The tools you've both listed are assemblers, which won't help you understand or make sense of existing code. They'll be useful for the re-assembly phase, however!

For disassembly, I highly recommend IDA Pro. It has all of the tools, documentation, plugins, and community help that you seek. As a bonus, the older versions are also free.

Commoner

I agree that they both are assemblers.. but for understanding the output of IDA Pro, I need to know assembly, right? And that's the reason I'm asking for which assembler to choose. – Dashpot

@Dashpot I agree with KerrekSB on this: pick one and go with it. They're both fairly robust. As for starting to learn assembly for the first time (I've been in this position!), I strongly recommend pulling down Intel's Software Developer Manuals to figure out how it all goes together. As a bonus, you can have them ship you hard copies of the complete set, entirely for free. :) – Commoner

But I have heard that MASM has a lot of higher level constructs.. which I wont see when I disassemble, but its not the case with NASM. What do you say about this? – Dashpot

@Dashpot I would simply agree with wallyk. :) – Commoner

So choosing any would be good? – Dashpot

It doesn't matter if you learn MASM, NASM, TASM, FASM, ROASM, CRAPASM. If you are going to reverse LEARN x86 Assembly INTEL or AT&T syntax which ever the debugger you are using uses. Every Assembler now has "High Level" constructs to make life easier, MASM will even allow you to write in opcodes if you wanted to. – Pathic

Answer 3

I think you got it backwards. I learned assembly by debugging the output of a high level language compiler.

Dannydannye