Page 1 of 1

Injection library problem with another hooking program

Posted: Thu Mar 09, 2017 1:17 pm
by wj_Lee
Hi! Madshi,

I've been using MadCHook 3.1.13 and found something strange

My program and Symantec DLP hooks same process and same GDI Print API.
I've used HookAPI() with no 'dwFlags' and returns true.
But, its behavior seems to skip hooking API
My callback function does nothing, like it never been called.

I've tested these things

- Symantec DLP hooks API first(It means to turn on the DLP). and my program succeeds to hook the same API. DLP works but mine is not working
- My program hooks API first. and Symantec DLP hooks the same API. Both hook functions are working well
- Symantec DLP hooks API first. and my program succeeds to hook the same API with FOLLOW_JMP option. Both hook functions are working well


So, I wondering,

Q1. Are there any cases skipping hook API but returning success?
I saw HookAPI() returns false, but I've never seen like this
Q2. If yes, follow jump option makes do not skipping hook?

Re: Injection library problem with another hooking program

Posted: Thu Mar 09, 2017 1:26 pm
by madshi
I'm not sure. Does HookAPI return false or true in the case where the hook doesn't work? Your comments are not clear about that.

Re: Injection library problem with another hooking program

Posted: Thu Mar 09, 2017 4:18 pm
by wj_Lee
Oh... sorry for my poor English.

- Normal -
1. Here is my pseudo code of callback function

Code: Select all

CB_StartDocW(...)
{
	DbgMsg("Start CB_StartDocW");
	MyCode(...); // 
	....
	ORIG_StartDocW(...);
}
2. inject library and hook API 'StartDocW()' by calling HookAPI().

Code: Select all

HookAPI("Gdi32.dll", "StartDocW", CB_StartDocW, (PVOID*)&ORIG_StartDocW);
and HookAPI() returns true and *pNextHook is not null.

3. Do 'Print' in notepad.exe, then program call the hooked 'StartDocW()'.
Consequently, CB_StartDocW() is called before original StartDocW() called.
It works well. I can see my dbgmsg, and mycode() is called before original StartDocW().
finally, original StartDocW() is called and printing is started.
------------

- Problem Found -
0. I turn on the DLP

1. same as the normal state

2. same as the normal state.

3. Do 'Print' in notepad.exe, then program call the 'StartDocW()'.
And CB_StartDocW() should be called before original StartDocW() called.
But, CB_StartDocW is does nothing.
I can not see my dbgmsg, and mycode() looks like it never been called.
But, printing is working normally. It means original StartDocW() has been called.
------------

Now, I hope that you understand what I've post before.

My question is, If another exist hook, is there case that MCH3 skip the hook?
Skip means that HookAPI returns true, but the program works like I didn't call HookAPI.
If no case MCH3 does not skip the hook, May the DLP does something for protect other hooks.
Is there way to check DLP protect MCH hooks?

Re: Injection library problem with another hooking program

Posted: Thu Mar 09, 2017 8:45 pm
by iconic
Maybe enabling DLP causes the application to use a different API?!? You might use a program like Rohitab's API monitor to see for yourself, this way you will know for sure. Secondly, it's possible that your hook might become unhooked, although it's a long shot given your circumstances. You might try calling RenewHook() after you call the original API inside your hook callback. Make this the last line of code you call. Outside of what I have mentioned I'm not sure what else it could be, sorry

--Iconic

Re: Injection library problem with another hooking program

Posted: Fri Mar 10, 2017 9:09 am
by madshi
Considering that FOLLOW_JMP "fixes" the issue, my best guess would be the following:

madCodeHook detects when an API is already hooked by someone else, and if you then try to hook it with madCodeHook. In this situation madCodeHook tries to avoid overwriting the other hooking library's hook another time, because doing so can in some cases produce instability. Instead madCodeHook switches to its secondary API hooking method. Which by default is the MIXTURE_MODE. In this mode madCodeHook patches the export table of the GDI dll, and also all import tables of all dlls in the process, to make it seems like the API is located somewhere else than it really is. And then that new address is hooked as usual by madCodeHook. This usually works fine. However, if a process has done GetProcAddress to get the API address and stored the API address in some variable, or if a process *directly* calls the API address with a hard coded address (will happen if GDI calls GDI), then the MIXTURE_MODE hook won't work.

FOLLOW_JMP uses a different approach: Instead of hooking the target API, it hooks the hook callback function of the other hooking library. That works surprisingly well, but comes with its own share of disadvantages. E.g. if the other hooking library uninstalls its own API hook, your's wlil stop working, too.

I've been thinking about making FOLLOW_JMP default, instead of MIXTURE_MODE, because it seems to work better for most people. If it works fine for you, I see no reason not to use it. It will only have any effect if the target API was already hooked by somebody else. Otherwise it has no effect.