Bug in MCH? CreateProcess hook problem under Win NT 4

c++ / delphi package - dll injection and api hooking
Post Reply
torstenv
Posts: 26
Joined: Sat Jun 26, 2004 5:13 pm

Bug in MCH? CreateProcess hook problem under Win NT 4

Post by torstenv »

I noticed a problem with hooking CreateProcess under Win NT 4. To reproduce, you can take the program "total commander", which is available at http://www.ghisler.com/download.htm. You can use the MCHDemos and inject HookProcessCreation.dll from "Various Dlls". Now if you start TotalCommander and try to start notepad by using the F4 key, you'll be asked 4 times if you want to allow TotalCommander to start notpad. You can always click OK, but it won't start. If you uninject HookProcessCreation.dll again, it works fine. That's a problem with the MCH, isn't it?

T.
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

I guess it's a problem with the HookProcessCreation.dll. But I'll check it.
torstenv
Posts: 26
Joined: Sat Jun 26, 2004 5:13 pm

Post by torstenv »

madshi wrote:I guess it's a problem with the HookProcessCreation.dll. But I'll check it.
I doubt that, but you probably know best where to look first. Anyway, I think the problem is more likely to be in the MCH stuff somewhere, because this problem arose when we were trying to hook CreateProcess and had troubles. I just pointed to the MCHDemos, because it's easily reproducible and it proves that the problem cannot be in one of our routines.

T.
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

Kind of found the problem. If I remove the MessageBoxW call from the HookProcessCreation.dll demo, everything works as expected. I suspect that MessageBoxW internally does some wide<->ansi conversion and thus violates hooking rule 7... :(

Anyway, it's not a bug in madCodeHook. This behaviour is exactly what you get when hooking rule 7 is violated.

http://help.madshi.net/HookingRules.htm

I wish I could work around this problem, but I don't know how. Would have to patch Windows itself to get this solved. You'd have the very same problem with any other API hooking package, too.
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

P.S: MessageBoxW internalls calls "RegQueryValueExA" in NT4. Don't ask me why, but this is the reason for the problems in the HookProcessCreation.dll (violation of hooking rule 7, as mentioned before).
torstenv
Posts: 26
Joined: Sat Jun 26, 2004 5:13 pm

Post by torstenv »

madshi wrote:P.S: MessageBoxW internalls calls "RegQueryValueExA" in NT4. Don't ask me why, but this is the reason for the problems in the HookProcessCreation.dll (violation of hooking rule 7, as mentioned before).
Thx, that seemed to have helped.
Post Reply