Support for decompilation of 64-bit x86 files?

[Manouchehri]

Since this question is going to get asked sooner or later, might as well ask it now: what needs to implemented for 64-bit support?

llvmir2hll was able to decompile a simple hello world LLVM IR file compiled by a 64-bit Linux and macOS host. I didn't test anything complex or McSema (which someone should definitely try and let us know!).

Is bin2llvmir the main road block?

[PeterMatula]

Hi,
I actually already hacked my local RetDec and tried a full x86-64 bit decompilation of a simple ack.c program (the one from retdec-regression-tests). It did pass the whole chain all the way to C. It even looked somehow OK-ish. The biggest problem was that x86-64 is using a different calling conventions that we currently do not handle at all. Also, even if this was not an issue, we still would not just add the support without at least a few regression tests.

When adding a new architecture, most of the work needs to be done in capstone2llvmir. This library already supports 64-bit for x86, mips, and ppc. However, other changes, like in this case, might be required to get some reasonable results.

So, what needs to be done to enable x86-64:

1. Disable RetDec's defences agains unsupported formats -- currently x86-64 won't even get through scripts.
2. Create x86 decoder set to 64-bit mode in decoder.
3. Add support for x86-64 calling conventions.
4. Test it a bit. If some other major problem occur, modify other parts.
5. Write some regression tests.

We would like to get to it pretty soon, but before that, some refactorizations are in order:

1. Refactor capstone2llvmir -- v1.0 is kind of a prototype.
2. Refactor decoder pass in bin2llvmir and merge it with control-flow pass -- this is a relic from before capstone2llvmir that was just forced to use it. It does not make sense to have these two parts separated. Once this is done, all decompilation results should be much better and adding x86-64 much easier.

[PeterMatula]

I have been asked what changes did I make in order to try x86-64 decompilation. Instead of listing them somewhere, I decided to create a branch where it is enabled. So if anyone wants to play with it, here you go. I was able to decompile the simplest hello world program (hello-x86_64.zip). You can see in the hello.c.frontend.dsm file that 64-bit instructions were indeed created. I did not try anything more complex. There is a good chance it would not work.

Keep in mind, that everything I wrote above still holds. This does not mean RetDec supports x86-64, or anything like that. All I did was let these files go through scripts into decompilation (commit). Much more work will be needed to properly support this.

Please do not report any issues related to this. But if while you are playing with it you fix/improve something, feel free to contribute.

[Mcilie]

Guys hows it coming along? when do you think it will be done?

[PeterMatula]

@Mcilie I'm spending more time than I thought on #116, so this did not really moved much.

[bannsec]

+1 on this. Given the majority of binaries are 64-bit now (especially linux elfs), not having 64bit decompilation support is a major issue for usability.