Wednesday, June 5, 2019

Fixing stack overflow on older game by limiting exposed OpenGL extensions.

There's this GameHouse game, called AirStrike 3D. This game itself is released back in very old days, not accounting for the hardware development and new GPUs and new OpenGL extensions. Things were mostly fixed-size buffers back then. Until at one point when I decided to install it back, I can't run this game.

Trying to run the game simply crashes. I run the game in Windowed mode by editing their config.ini
At first I thought this was Windows 10 problem since running the game in VirtualBox with Windows XP seems fine. Until I have an idea to run the game with Mesa3D instead, but still crashes. I decided to start Visual Studio debugger and the crash point to some random location and access violation about can't execute piece of code of RAM (due to the permissions), so I thought "this is probably stack overflow" so I decided to check the stack register and what a surprise: I see all the OpenGL extensions string in the stack register, so this is caused because the extension string returned by my OpenGL driver is simply too long for the game to handle. Also my laptop is dual-GPU but both GPUs can't run the game because the extension string too long.

After bit of search, I found this Mesa3D page which describe how to limit the OpenGL extension string returned to workaround some game. It work great but I don't want to use Mesa3D because my laptop is not an AMD Threadripper which has dozen of threads, so I decide to roll my own OpenGL32 which forwards all OpenGL calls to Windows OpenGL32.dll but intercepts glGetString(GL_EXTENSIONS) call and limit the extension string returned.

First, I take a look on Mesa3D source code on their extension table list which I can use and I found this header file which is exactly what I'm looking for. Then I realize that Windows OpenGL32.dll has 360 functions and I don't want to write the forwarding functions by hand, who wants to do that. So instead of writing them by hand, I used my Lua programming power to parse the gl.h and WinGDI.h header file to create a file which forwards the GL function to original Windows OpenGL functions. Fixing any calling conventions and making sure the result function names aren't somewhat decorated by generating def file too, I finally have working program.

After putting the new DLL to the game folder and setting the necessary environment variable, I'm surprised the game finally runs. The game runs at about 500FPS in my HD Graphics 620 (got CPU bottleneck), but this game is fixed-function pipeline so I'm not surprised about the absurdly high FPS.

Game runs at ~540FPS. For anyone curious, here's my astrike.log.
The source code of the hooked OpenGL32.dll that I'm talking about is available in my GitHub including the build instructions (it's CMake) and if you too bother to compile and only want the 32-bit OpenGL32.dll, just go to the releases folder. One thing that I notice that it only able to handle at most 4048 string length or the stack overflow occur, so setting the extensions year to 2009 or earlier should work.

If some older game have same issue but you don't want to use Mesa3D, you can try my hooked OpenGL32.dll above and tell me how it performs.

5 comments:

  1. is it working? because i need it for nostalgia

    ReplyDelete
  2. Replies
    1. If you read it correctly, I linked it in the blog post.

      Just in case you're blind: https://github.com/MikuAuahDark/gl-ext-limit

      Delete
  3. This comment has been removed by a blog administrator.

    ReplyDelete
  4. hmm...on win10 it just gets stuck, using the release dll. setting real dll via environment variable to c:\windows\syswow64\opengl32.dll doesnt help.
    any chance of a 32 bit binary release for win10?
    was hoping this worked, as normally have to rename et.exe to quake3.exe (or others) to trick gpu driver. unfortunatelly it has some other consequences, including windows and gpu driver applying other compat settings. for instance, quake3.exe cannot run on single core...when you force it to, it goes into crazy super slow.

    ReplyDelete