[SOLVED] Hooked function crashes in 32 bit windows

c++ / delphi package - dll injection and api hooking
Post Reply
blackpaw
Posts: 33
Joined: Mon Nov 05, 2007 1:08 am

[SOLVED] Hooked function crashes in 32 bit windows

Post by blackpaw »

I have a Print Providor Hook - hooks various functions in the local print providor (localspl.dll), it gets the functions by trapping "InitializePrintProvidor" and calling HookCode on the function pointers passed to that.

Works perfectly in Windows 7 64 bit. Dies on its ass in Windows 7 32 bit. As far as I can tell from log statements, it happens immediately after a return from any of my hooked functions.

I've tried minimizing the foot print, reducing it to just one hooked function that is merely a stub - it returns immediately, calling the original function. Still crashes the print spooler.

If I only hook the InitializePrintProvidor, it works fine, apart from doing nothing useful :)

Feels like a mismatch in calling conventions or somesuch, differences in 32/64 code. I've examined all the declerations and compared with the originals in the windows header, they all match.

One thing - none of them are declared with WINAPI (__stdcall).

Any suggestions as to where to go next?

thanks,

Lindsay
Last edited by blackpaw on Wed May 07, 2014 9:56 am, edited 1 time in total.
madshi
Site Admin
Posts: 10754
Joined: Sun Mar 21, 2004 5:25 pm

Re: Hooked function crashes in 32 bit windows

Post by madshi »

In x64 there is only one calling convention. So whatever calling convention specifier you use doesn't matter. In x86 there really are all kinds of different calling conventions. So my best guess is that the calling convention is wrong. Simply try different ones. If everything goes to working correctly, then you know you found the correct one. :wink: Try WINAPI first, it's the most likely one...
blackpaw
Posts: 33
Joined: Mon Nov 05, 2007 1:08 am

Re: Hooked function crashes in 32 bit windows

Post by blackpaw »

Thanks madshi, but the actual prototypes declared in windows are just plain c.

I'll take a look at sample sdk code tomorrow and compare.
madshi
Site Admin
Posts: 10754
Joined: Sun Mar 21, 2004 5:25 pm

Re: Hooked function crashes in 32 bit windows

Post by madshi »

Don't hang yourself up on prototype declarations. C++ is so complicated with its defines and includes and headers and project options etc that it's hard to predict which calling convention is really used. Just use trial and error. It will cost you 5 minutes of your time and probably fix the issue.
blackpaw
Posts: 33
Joined: Mon Nov 05, 2007 1:08 am

Re: Hooked function crashes in 32 bit windows

Post by blackpaw »

Good point :) I'll give it a spin
blackpaw
Posts: 33
Joined: Mon Nov 05, 2007 1:08 am

Re: Hooked function crashes in 32 bit windows

Post by blackpaw »

It was calling conventions in the end (WINAPI), though I didn't have to edit my code.

For a start off I couldn't, I was hooking function pointers in the PRINTPROVIDOR structure and they had no calling conventions specified, and defaulted to cdecl.

What I need to do was to set __stdcall as the default in the project properties, so the windows functions pointers were stdcall as well. All worked after that.

Thanks - Lindsay
madshi
Site Admin
Posts: 10754
Joined: Sun Mar 21, 2004 5:25 pm

Re: [SOLVED] Hooked function crashes in 32 bit windows

Post by madshi »

Windows almost always uses WINAPI. It's very rare that cdecl is used. It's kinda strange, though, that when using the "official" Microsoft header, you're getting problems. Anyway, glad to hear you got it solved. If you want my opinion: I'd rather define the structure yourself, so you don't rely on the project options. Otherwise maybe at some point you recreate the project or something, and suddenly all hell breaks lose because the project reverts to cdecl. Just my 2 cents, of course...
Post Reply