Function hook getting unexpectedly called again

c++ / delphi package - dll injection and api hooking

Function hook getting unexpectedly called again

Postby ChrisHebert » Mon Nov 16, 2015 5:28 pm

So in our product on Windows we're hooking ReplaceFileW (along with many other functions). Everything has been working fine for years and still is everywhere I try, but on one customer's environment I'm seeing something weird. I can't interactively debug things so I'm limited to logging info to a file for later interpretation.

I'm at the point now where I log the address of myReplaceFileW and the value of the NextHook that HookAPI gives me right after invoking HookAPI. They look reasonable. I get into myReplaceFileW and log the value of the NextHook and it is still the same as it was. When I get to the point in myReplaceFileW where I call the function represented by NextHook, myReplaceFileW is being invoked, which is not expected. On all the machines where we've tried to reproduce it, it appears that invoking NextHook results in the real ReplaceFileW getting called as expected.

What techniques are available to try to understand what's happening?
ChrisHebert
 
Posts: 29
Joined: Thu Sep 04, 2008 5:28 pm

Re: Function hook getting unexpectedly called again

Postby madshi » Thu Nov 19, 2015 11:30 am

So you are getting endless recursion?

Could this be some sort of conflict with another hooking library? It's hard to say for me, with the limited information I have.

Some questions:

1) Do you use the latest madCodeHook version?
2) Does this happen with all applications, or just with some (or one)?
3) Does it happen with both 32bit and 64bit applications?
4) Do you use any flags when calling HookAPI()?
5) HookAPI() knows different API hooking methods, depending on the situation. The usual method is to overwrite the original API code. A secondary method is to leave the original API code untouched and to modify the import & export tables instead. Does your log file indicate which of the methods madCodeHook was using?
madshi
Site Admin
 
Posts: 10268
Joined: Sun Mar 21, 2004 5:25 pm

Re: Function hook getting unexpectedly called again

Postby dcsoft » Sun Nov 22, 2015 6:52 pm

Assuming the hooking is being implemented correctly, could the original hooked function be calling itself recursively? Is it a problem if your hook function only calls NextHook() and returns? Does it cause infinite recursion as Madshi or is the behavior OK? If it is OK with the empty hook function, you may just need to adjust your hook code to expect recursion and handle it correctly.

Thanks,
David
dcsoft
 
Posts: 380
Joined: Sat Dec 11, 2004 2:11 am
Location: San Francisco Bay Area, CA USA

Re: Function hook getting unexpectedly called again

Postby ChrisHebert » Mon Nov 23, 2015 1:53 pm

Seems like a hooking conflict. The customer has indicated that they're using some other software to "monitor" file activity. When that software is removed from the picture, things work fine. So it would seem that this other software is also doing function hooking and doing so in a less robust fashion.

Since the customer says they're planning to stop using that monitoring software, there is no urgency around this problem, so I will take the suggestions/questions under advisement for now.

Thanks
ChrisHebert
 
Posts: 29
Joined: Thu Sep 04, 2008 5:28 pm

Re: Function hook getting unexpectedly called again

Postby iconic » Tue Nov 24, 2015 2:28 am

You might ask that customer what software they were using and report back to Madshi. Otherwise, your same concern will pop back up eventually I'd imagine be it someone else

--Iconic
iconic
Site Admin
 
Posts: 967
Joined: Wed Jun 08, 2005 5:08 am

Re: Function hook getting unexpectedly called again

Postby madshi » Tue Nov 24, 2015 5:43 pm

As far as I'm aware, madCodeHook is very careful in the way it interacts with other hooking libraries. If madCodeHook detects that another software has already hooked an API, I'm even switching to a different hooking method automatically, in order to reduce the likelihood of problems. My best guess is that the other software is probably at fault, doing something strange. There are 2 options to improve things from our side, though:

1) There's a flag named "FOLLOW_JMP" which you can use in your HookAPI() call. That flag will change the way madCodeHook behaves if it detects that another hooking library has already hooked the API we want to hook. *However*, although I'm not sure I think that in this specific case our hook is probably getting installed first and the other hook then overwrites our hook. If that's the case, the "FOLLOW_JMP" flag won't make any difference.

2) In madCodeHook 4.0 I plan to replace my 6-byte absolute JMP instruction with a 5-byte relative JMP instruction, which is what most other hooking libraries are using. I hope that this will further improve stability when multiple hooking libraries have to work together...
madshi
Site Admin
 
Posts: 10268
Joined: Sun Mar 21, 2004 5:25 pm


Return to madCodeHook

Who is online

Users browsing this forum: No registered users and 21 guests