UNICODE weirdness

[dcsoft]

Hello, I have called HookAPI to intercept CreateWindowExW. My function tries to OutputDebugStringW() a UNICODE string, and it hangs the app (Internet Explorer). Here's my function:

Code: Select all

HWND WINAPI HookCreateWindowExW( IN DWORD dwExStyle,
				 IN LPCWSTR lpClassName,
				 IN LPCWSTR lpWindowName,
				 IN DWORD dwStyle,
				 IN int X,
				 IN int Y,
				 IN int nWidth,
				 IN int nHeight,
				 IN HWND hWndParent,
				 IN HMENU hMenu,
				 IN HINSTANCE hInstance,
				 IN LPVOID lpParam)
{

	// Call next hook
	HWND hwnd = NextCreateWindowExW( 
                    dwExStyle, lpClassName, lpWindowName,
                    dwStyle, X, Y, nWidth, nHeight, hWndParent,
                    hMenu, hInstance,	lpParam);

	// Log this function call:
                // The following line, which accesses an LPCWSTR, 
	// causes the hooked app (Internet Explorer) to hang.
	// If this line is commented out, Internet Explorer works fine.
	OutputDebugStringW(lpClassName);

	// And return the result
	return hwnd;
}

I have tried variations of the above, for example, instead of OutputDebugStringW, I have tried using lstrcmpiW() to compare lpClassName to some other string; this comparison also hangs Internet Explorer in the same way. So the problem is not in OutputDebugStringW.

Furthermore, I have also intercepted the ANSI version, CreateWindowExA, and following the same logic, OutputDebugStringA works fine. To test, I run the code in WinME (which uses the Ansi version of CreateWindowEx).

It seems the problem is inherent in the UNICODE version, when run on Win2K and XP. Whenever I touch the UNICODE string, Internet Explorer hangs.

Let me detail what I mean by "hang": I have broken into the above code using a debugger and stepped through each line; this is rapid response. No hangs at this point. So there is nothing inherently wrong with any of the code in the hook. But after my hook gets called, Internet Explorer freezes. When I break into the debugger during the freeze, the callstack is filled with NTDLL.DLL, GDI32.DLL, etc. as if the system is out to lunch. How could simply calling a function with a Unicode string parameter be causing this? Could I be messing up the callstack?

Does anyone have any ideas?

Thanks,
David

[madshi]

Does it work when simply removing OutputDebugStringW?

I guess you're stumbling over hooking rule 7 here (see documentation). I vaguely remember that OutputDebugStringW internally calls an Ansi API which violates hooking rule 7. Not sure then why you have the same problem when using lstrcmpiW, though.

The documentation sais this about the class name:

Pointer to a null-terminated string or a class atom created by a previous call to the RegisterClass or RegisterClassEx function. The atom must be in the low-order word of lpClassName; the high-order word must be zero.

Maybe the freezing occurs because you're dealing with an atom, not with a real string? I think you have to check for atoms in your code.

[dcsoft]

Does it work when simply removing OutputDebugStringW?

I guess you're stumbling over hooking rule 7 here (see documentation). I vaguely remember that OutputDebugStringW internally calls an Ansi API which violates hooking rule 7. Not sure then why you have the same problem when using lstrcmpiW, though.

The documentation sais this about the class name:

Pointer to a null-terminated string or a class atom created by a previous call to the RegisterClass or RegisterClassEx function. The atom must be in the low-order word of lpClassName; the high-order word must be zero.

Maybe the freezing occurs because you're dealing with an atom, not with a real string? I think you have to check for atoms in your code.

Thanks much Madshi, you saved the day (saved me many days! ). Checking for the atom (HIWORD is 0) of the class name, and not calling OutputDebugStringW or lstrcmpiW in that case removed the hang. It's working fine now. BTW, I still have OutputDebugStringW in there, and it works great, no problem with OutputDebugStringW in general.

Thanks also for reminding me to re-read the hooking rules. It's been a few weeks since I've used madCodeHook and had forgotten of the importance to call only wide functions from wide hooks.

Thanks again,
David

[madshi]

BTW, I still have OutputDebugStringW in there, and it works great, no problem with OutputDebugStringW in general.

Hmmmm... I think the problem only existed in NT4, not sure, though. I'm sure that there was a problem with OutputDebugStringW somewhere. But if you tested it all and got no problems, then probably you don't need to worry.