WallShadow wrote:Care to explain your method?
Sure. There were a few issues with the algorithm itself. The way it rotated and performed bitwise operations, I noticed that it was cyclically rotating the input. Encryption was only 6 rotations of each 4 byte block, so rather than using 6 rotations in reverse that the decryption function used, one could continue two two more forward rotations to complete the cycle full circle again.
The Java code was very inefficient, using the same memory segments for each pass, so only one out of every 4 decryption runs would be accurate in a loop. Rather than mess with Java (which was running slowly anyway) I wrote a quick program in C++ to perform 2 forward rotations, then compare the end of the output to match against the expected padding and if the padding matches the algorithm, record the results. This was iterated within the keyspace of he 4 byte key and divided among separate forked process on a 24 core server. (The C++ variant was running several thousand times faster than the Java code)
I let it run for about a minute and then grepped the log for " to " just to see what I'd come up with. At that moment I had about 16 matches with recognizable English text. I stopped it there and didn't bother to examine the whole keyspace.
All in all, I think this was a nice small challenge.