Function hook getting unexpectedly called again

c++ / delphi package - dll injection and api hooking
Post Reply
ChrisHebert
Posts: 29
Joined: Thu Sep 04, 2008 5:28 pm

Function hook getting unexpectedly called again

Post by ChrisHebert »

So in our product on Windows we're hooking ReplaceFileW (along with many other functions). Everything has been working fine for years and still is everywhere I try, but on one customer's environment I'm seeing something weird. I can't interactively debug things so I'm limited to logging info to a file for later interpretation.

I'm at the point now where I log the address of myReplaceFileW and the value of the NextHook that HookAPI gives me right after invoking HookAPI. They look reasonable. I get into myReplaceFileW and log the value of the NextHook and it is still the same as it was. When I get to the point in myReplaceFileW where I call the function represented by NextHook, myReplaceFileW is being invoked, which is not expected. On all the machines where we've tried to reproduce it, it appears that invoking NextHook results in the real ReplaceFileW getting called as expected.

What techniques are available to try to understand what's happening?
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: Function hook getting unexpectedly called again

Post by madshi »

So you are getting endless recursion?

Could this be some sort of conflict with another hooking library? It's hard to say for me, with the limited information I have.

Some questions:

1) Do you use the latest madCodeHook version?
2) Does this happen with all applications, or just with some (or one)?
3) Does it happen with both 32bit and 64bit applications?
4) Do you use any flags when calling HookAPI()?
5) HookAPI() knows different API hooking methods, depending on the situation. The usual method is to overwrite the original API code. A secondary method is to leave the original API code untouched and to modify the import & export tables instead. Does your log file indicate which of the methods madCodeHook was using?
dcsoft
Posts: 380
Joined: Sat Dec 11, 2004 2:11 am
Location: San Francisco Bay Area, CA USA
Contact:

Re: Function hook getting unexpectedly called again

Post by dcsoft »

Assuming the hooking is being implemented correctly, could the original hooked function be calling itself recursively? Is it a problem if your hook function only calls NextHook() and returns? Does it cause infinite recursion as Madshi or is the behavior OK? If it is OK with the empty hook function, you may just need to adjust your hook code to expect recursion and handle it correctly.

Thanks,
David
ChrisHebert
Posts: 29
Joined: Thu Sep 04, 2008 5:28 pm

Re: Function hook getting unexpectedly called again

Post by ChrisHebert »

Seems like a hooking conflict. The customer has indicated that they're using some other software to "monitor" file activity. When that software is removed from the picture, things work fine. So it would seem that this other software is also doing function hooking and doing so in a less robust fashion.

Since the customer says they're planning to stop using that monitoring software, there is no urgency around this problem, so I will take the suggestions/questions under advisement for now.

Thanks
iconic
Site Admin
Posts: 1065
Joined: Wed Jun 08, 2005 5:08 am

Re: Function hook getting unexpectedly called again

Post by iconic »

You might ask that customer what software they were using and report back to Madshi. Otherwise, your same concern will pop back up eventually I'd imagine be it someone else

--Iconic
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: Function hook getting unexpectedly called again

Post by madshi »

As far as I'm aware, madCodeHook is very careful in the way it interacts with other hooking libraries. If madCodeHook detects that another software has already hooked an API, I'm even switching to a different hooking method automatically, in order to reduce the likelihood of problems. My best guess is that the other software is probably at fault, doing something strange. There are 2 options to improve things from our side, though:

1) There's a flag named "FOLLOW_JMP" which you can use in your HookAPI() call. That flag will change the way madCodeHook behaves if it detects that another hooking library has already hooked the API we want to hook. *However*, although I'm not sure I think that in this specific case our hook is probably getting installed first and the other hook then overwrites our hook. If that's the case, the "FOLLOW_JMP" flag won't make any difference.

2) In madCodeHook 4.0 I plan to replace my 6-byte absolute JMP instruction with a 5-byte relative JMP instruction, which is what most other hooking libraries are using. I hope that this will further improve stability when multiple hooking libraries have to work together...
Post Reply