Tools to help reverse engineer binary file formats

Question

What tools are available to aid in decoding unknown binary data formats?

I know Hex Workshop and 010 Editor both support structures. These are okay to a limited extent for a known fixed format but get difficult to use with anything more complicated, especially for unknown formats. I guess I'm looking at a module for a scripting language or a scriptable GUI tool.

For example, I'd like to be able to find a structure within a block of data from limited known information, perhaps a magic number. Once I've found a structure, then follow known length and offset words to find other structures. Then repeat this recursively and iteratively where it makes sense.

In my dreams, perhaps even automatically identify possible offsets and lengths based on what I've already told the system!

Fleisher

Not a direct answer to your question: Do you not have the executable files that work with these binary files with unknown formats? Using a ring3 debugger such as OllyDbg to reverse engineer that instead would be so much easier than to pretty much try to brute force file formats. – Pietra

Oh yes, one of the apps I should have added to the "which program did you ever want to write but never found the time to do it" question ;) – Ovalle

In some cases I do have executable files that process them to an extent. Sometimes the files are executable code (but not in a standard format) and may well contain their own decoding routines. We may have limited shards of partial documentation as a starting point. In other cases I have nothing. – Fleisher

Answer 1

Here are some tips that come to mind:

From my experience, interactive scripting languages (I use Python) can be a great help. You can write a simple framework to deal with binary streams and some simple algorithms. Then you can write scripts that will take your binary and check various things. For example:

Do some statistical analysis on various parts. Random data, for example, will tell you that this part is probably compressed/encrypted. Zeros may mean padding between parts. Scattered zeros may mean integer values or Unicode strings and so on. Try to spot various offsets. Try to convert parts of the binary into 2 or 4 byte integers or into floats, print them and see if they make sence. Write some functions that will search for repeating or very similar parts in the data, this way you can easily spot headers.

Try to find as many strings as possible, try different encodings (c strings, pascal strings, utf8/16, etc.). There are some good tools for that (I think that Hex Workshop has such a tool). Strings can tell you a lot.

Good luck!

Seroka

Hachoir from the answer below is exactly this kind of framework. It comes with predefined set of fields: different kinds of strings, dates, bits, floats, padding, etc. Built-in parsers can be used as examples along with the docs. – Pelion

Answer 2

For Mac OS X, there's a great tool that's even better than my iBored: Synalyze It! (http://www.synalysis.net/)

Compared to iBored, it is better suited for non-blocked files, while also giving full control over structures, including scriptability (with Lua). And it visualizes structures better, too.

Batrachian

Answer 3

Tupni; to my knowledge not directly available out of Microsoft Research, but there is a paper about this tool which can be of interest to someone wanting to write a similar program (perhaps open source):

Tupni: Automatic Reverse Engineering of Input Formats (@ ACM digital library)

Abstract

Recent work has established the importance of automatic reverse engineering of protocol or file format specifications. However, the formats reverse engineered by previous tools have missed important information that is critical for security applications. In this paper, we present Tupni, a tool that can reverse engineer an input format with a rich set of information, including record sequences, record types, and input constraints. Tupni can generalize the format specification over multiple inputs. We have implemented a prototype of Tupni and evaluated it on 10 different formats: five file formats (WMF, BMP, JPG, PNG and TIF) and five network protocols (DNS, RPC, TFTP, HTTP and FTP). Tupni identified all record sequences in the test inputs. We also show that, by aggregating over multiple WMF files, Tupni can derive a more complete format specification for WMF. Furthermore, we demonstrate the utility of Tupni by using the rich information it provides for zeroday vulnerability signature generation, which was not possible with previous reverse engineering tools.

Garda

Link to the paper: research.microsoft.com/en-us/um/people/wdcui/papers/… – Turnout

Answer 4

My own tool "iBored", which I released just recently, can do parts of this. I wrote the tool to visualize and debug file system formats (UDF, HFS, ISO9660, FAT etc.), and implemented search, copy and later even structure and templates support. The structure support is pretty straight-forward, and the templates are a way to identify structures dynamically.

The entire thing is programmable in a Visual BASIC dialect, allowing you to test values, read specific blocks, and all.

The tool is free, works on all platforms (Win, Mac, Linux), but as it's personal tool which I just released to the public to share it, it's not much documented.

However, if you want to give it a try, and like to give feedback, I might add more useful features.

I'd even open source it, but as it's written in REALbasic, I doubt many people will join such a project.

Link: iBored home page

Batrachian

Looks like the start of a nice binary file analysis tool, but is still very disk-centric (512-byte blocks is a bit of a give-away...) – Turnout

@SteveBennett: it likes to cluster the file into equal-sized blocks, that's true, but one can easily change the block size via the menu. And iBored can also handle the entire file as one block. The only disadvantage is that it'll have trouble with large files because it tries to show all data in a single scrollable block view then, as one block, which can lead to performance issues. – Batrachian

Answer 5

I still occasionally use an old hex editor called A.X.E., Advanced Hex Editor. It seems to have largely disappeared from the Internet now, though Google should still be able to find it for you. The last version I know of was version 3.4, but I've really only used the free-for-personal-use version 2.1.

Its most interesting feature, and the one I've had the most use for deciphering various game and graphics formats, is its graphical view mode. That basically just shows you the file with each byte turned into a color-coded pixel. And as simple as that sounds, it has made my reverse-engineering attempts a lot easier at times.

I suppose doing it by eye is quite the opposite of doing automatic analysis, though, and the graphical mode won't be much use for finding and following offsets...

The later version has some features that sound like they could fit your needs (scripts, regularity finder, grammar generator), but I have no idea how good they are.

Hoopoe

URL is dead, maybe it's here now: advanced-hex-editor-a-x-e.en.softonic.com – Turnout

@Steve Thanks for the heads-up. That's the one. Despite all the virus-free promises there, though, my virus scanner gave me a virus warning trying to download it, so I didn't bother. I've rephrased my answer. – Hoopoe

Answer 6

There is Hachoir which is a Python library for parsing any binary format into fields, and then browse the fields. It has lots of parsers for common formats, but you can also write own parsers for your files (eg. when working with code that reads or writes binary files, I usually write a Hachoir parser first to have a debugging aid). Looks like the project is pretty much inactive by now, though.

Theologize

Answer 7

Kaitai is an open-source language for describing binary structures in data streams. It comes with a translator that can output parsing code for many programming languages, for inclusion in your own program code.

Batrachian

Answer 8

My project icebuddha.com supports this using python to describe the format in the browser.

Scruggs

Answer 9

A cut'n'paste of my answer to a similar question:

One tool is WinOLS, which is designed for interpreting and editing vehicle engine managment computer binary images (mostly the numeric data in their lookup tables). It has support for various endian formats (though not PDP, I think) and viewing data at various widths and offsets, defining array areas (maps) and visualising them in 2D or 3D with all kinds of scaling and offset options. It also has a heuristic/statistical automatic map finder, which might work for you.

It's a commercial tool, but the free demo will let you do everything but save changes to the binary and use engine management features you don't need.

Bismuthinite