It's much more likely that there is a buffer overflow vulnerability in the server-software, and they're using that to simply send some sort of exit code using it. Although possible, they probably aren't using a customised "game.exe". My best bet is that the game allows a script to be run client-side on connection to a server, and they call the exploit using that, or they connect to your server cleanly and launch an external program that takes advantage of the present vulnerability.
Most game servers run on a specific port, and it's always the same one across servers (by default anyway), so it'd be trivial for them to just write a short program that sends the server a packet that will take advantage of a buffer overflow vulnerability.
It's not who votes that counts, it's who counts the votes