This is a write-up for the 0ctf 2016 quals “VM” reverse challenge worth 7 points.
The main process of solving this challenge contains 3 phase:
- Find the location of input check algorithm
- Reverse input transform algorithm
I wasted much life in the second phase. This function is flattend by a big switch structure and contains a lot of goto instruction. It’s hard to reverse such a function statically, So we use dynamic analysis and black box analysis. we make a few assumptions on the transform pattern. We can reverse the algorithm without understanding assembly or decompiled code.