HookAPI on 64bit deadlocks

c++ / delphi package - dll injection and api hooking
Post Reply
power888
Posts: 54
Joined: Sat May 23, 2009 8:55 am

HookAPI on 64bit deadlocks

Post by power888 »

Hi. madshi..

I have some problem of madCHook (3.1.5)

When I execute app (32bit) on 64 bit O/S, sometimes, deadlock is occurred in certain PC..
** And I was call InjectLibrary in another place (seperate Exe) not DLLMain.

stack is follow..

(BTW, this stack is very similar following subject)
Subject name :: InjectLibrary on 64 bit deadlocks
by alanmorgan » Tue Jun 07, 2011 9:08 pm


0018df28 77c04b6c 000000f4 0018df68 00000000 ntdll_77bc0000!NtWaitForKeyedEvent+0x15
0018df8c 77c04da9 005cd650 000000f8 00000000 ntdll_77bc0000!TppWaitpSet+0x206
0018e000 77c0a8b5 005cd650 000000f8 00000000 ntdll_77bc0000!TpSetWait+0xf2
0018e048 77c0a7ad 00000001 005cae80 7d54abb4 ntdll_77bc0000!TppTimerpInitTimerQueueQueue+0xb0
0018e098 77c0e8d6 005cca0c 7d54abfc 005caa60 ntdll_77bc0000!TppTimerpAllocTimerQueue+0x198
0018e0d0 77c0e7f7 005cc100 005cc9c0 7d54aa0c ntdll_77bc0000!TppTimerpAcquirePoolTimerQueue+0x31
0018e120 77c0e70c 0018e1c0 7681fbbb 00000000 ntdll_77bc0000!TppTimerAlloc+0x177
0018e198 77007e8c 0018e1c0 7681fbbb 00000000 ntdll_77bc0000!TpAllocTimer+0x99
0018e1b0 76831cde 7681fbbb 00000000 00000000 KERNELBASE!CreateThreadpoolTimer+0x18
0018e1c8 76831cad 7681fbbb 00000000 0018e1e8 rpcrt4!RPC_THREAD_POOL::CreateTimer+0x21
0018e1ec 7681ad5b 00000001 00007530 005cab80 rpcrt4!GarbageCollectionNeeded+0x22
0018e21c 7681abc6 005cab80 76820a65 00000000 rpcrt4!LRPC_CASSOCIATION::RemoveReference+0xd3
0018e224 76820a65 00000000 005cab80 005cab80 rpcrt4!LRPC_BIND_CCALL::~LRPC_BIND_CCALL+0x2d
0018e238 76820a1a 005cab80 0018e278 7681b1ec rpcrt4!LRPC_CCALL::~LRPC_CCALL+0xb2
0018e244 7681b1ec 00000001 005cacb0 005cab80 rpcrt4!LRPC_CCALL_AVRF::`scalar deleting destructor'+0xd
0018e258 76817607 0018e2f4 005cab80 0018e6e4 rpcrt4!LRPC_BASE_CCALL::FreeObject+0xca
0018e268 7681ff77 0018e2c8 0018e2f4 0018e284 rpcrt4!LRPC_BASE_CCALL::FreeBuffer+0xd1
0018e278 768172ea 0018e2c8 0018e294 76817f95 rpcrt4!LRPC_CCALL::FreeBuffer+0x4e
0018e284 76817f95 0018e2c8 0018e3f4 0018e2a8 rpcrt4!I_RpcFreeBuffer+0x10
0018e294 76818068 0018e2f4 757d45e8 00000001 rpcrt4!NdrFreeBuffer+0x1f
0018e2a8 768b018c 00000000 768b016a 91fbaa28 rpcrt4!NdrpClientFinally+0x3b
0018e2b0 768b016a 91fbaa28 00000000 00000000 rpcrt4!NdrClientCall2+0x29c
0018e6c4 757e0d13 757d45e8 757d3f5e 0018e6e4 rpcrt4!NdrClientCall2+0x27f
0018e6dc 757e06d4 0018e720 91fbabec 00000000 sechost!LsaLookuprClose+0x19
0018e718 757e0315 00000000 00862860 721e7a03 sechost!LsaLookupClose+0x1f
0018e76c 757e05f8 005c8c40 00862828 0018e804 sechost!LookupAccountSidInternal+0x7e
0018e790 75454820 005c8c40 00862828 0018e804 sechost!LookupAccountSidLocalW+0x1e
0018e7b0 721e7900 00000000 005c8c40 00862828 advapi32!LookupAccountSidW+0x46
0018e82c 721e77ee 005c8c40 72205a08 0018ea50 winsta!CUtils::GetNameFromSid+0xa9
0018e850 721e7788 721e5d00 721e5c5c 721e3b68 winsta!CPublicBinding::CreateLocalSPNs+0x63
0018e858 721e5c5c 721e3b68 0000000d 00000000 winsta!CPublicBinding::GetLSMBinding+0x1e
0018e9fc 721e3b4c 00000000 00000001 00000006 winsta!Public_WinStationQueryInformationW+0x9e
0018ea28 76ba9a7a 00000000 ffffffff 00000006 winsta!WinStationQueryInformationW+0x10f
0018f350 76ba9b00 0018f360 00000000 00000000 user32!GetClientKeyboardType+0x88
0018f36c 74052800 00000000 00000002 740548fc user32!GetKeyboardType+0x3c
0018f39c 74053322 74054b23 00000000 74071e0b myHook!00000026+0x31e
0018f3f0 74071c06 74050000 00000001 00000000 myHook!00000048+0xde
0018f420 74073e7d 74071dfa 00000001 575c3a43 myHook!StaticLibHelper_Init+0x48
0018f938 74074f81 74050000 00000001 00000000 myHook!DllMain+0x19d [d:\processprotect-3264\23 protect kill\hookterminateapis.cpp @ 866]
*** This point is HookAPI call in DllMain.

0018f978 74075029 74050000 0018f9a4 77bf99a0 myHook!__DllMainCRTStartup+0x6c [f:\dd\vctools\crt_bld\self_x86\crt\src\dllcrt0.c @ 330]
0018f984 77bf99a0 74050000 00000001 00000000 myHook!_DllMainCRTStartup+0x1e [f:\dd\vctools\crt_bld\self_x86\crt\src\dllcrt0.c @ 293]
0018f9a4 77bfd939 7407500b 74050000 00000001 ntdll_77bc0000!LdrpCallInitRoutine+0x14
0018fa98 77bfd7fc 00000000 7d54b728 00000000 ntdll_77bc0000!LdrpRunInitializeRoutines+0x26f
0018fc04 77bfc558 71af001d 77cc01a0 00000000 ntdll_77bc0000!LdrpLoadDll+0x4d1
0018fc3c 71af046b 77cc01a0 00000000 71af001d ntdll_77bc0000!LdrLoadDll+0xaa
WARNING: Frame IP not in any known module. Following frames may be wrong.
0018fcb8 77bf9ed1 7d54b62c 00000000 00000000 0x71af046b
0018fd00 77bf9ef9 0018fd24 77bc0000 00000000 ntdll_77bc0000!_LdrpInitialize+0x1b2
0018fd10 00000000 0018fd24 77bc0000 00000000 ntdll_77bc0000!LdrInitializeThunk+0x10
power888
Posts: 54
Joined: Sat May 23, 2009 8:55 am

Re: HookAPI on 64bit deadlocks

Post by power888 »

and My Dll is like

DllMain()

if (fdwReason == DLL_PROCESS_ATTACH) {

if (startFlag==0) { <== startFlag is global variable.. (for setting policy)
startFlag=1;
bRet = GetHookInitialData(); <== for reading default data...
}
InitializeMadCHook();

HookAPI("kernel32.dll", "TerminateProcess", TerminateProcessCallback, (PVOID*) &TerminateProcessNext);
*** At this point, program is deadlock

HookAPI("ntdll.dll", "NtTerminateProcess", NtTerminateProcessCallback, (PVOID*) &NtTerminateProcessNext);
HookAPI("ntdll.dll", "NtSuspendProcess", NtSuspendProcessCallback, (PVOID*) &NtSuspendProcessNext);
HookAPI("ntdll.dll", "NtDebugActiveProcess", NtDebugActiveProcessCallback, (PVOID*) &NtDebugActiveProcessNext);
HookAPI("ntdll.dll", "NtTerminateThread", NtTerminateThreadCallback, (PVOID*) &NtTerminateThreadNext);
HookAPI("ntdll.dll", "NtSuspendThread", NtSuspendThreadCallback, (PVOID*) &NtSuspendThreadNext);
HookAPI("kernel32.dll", "TerminateThread", TerminateThreadCallback, (PVOID*) &TerminateThreadNext);
} else if (fdwReason == DLL_PROCESS_DETACH) {
// FinalizeMadCHook is needed only if you're using the static madCHook.lib
FinalizeMadCHook();
}

return true;
}
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: HookAPI on 64bit deadlocks

Post by madshi »

Can you please double check with 3.1.6, just to be safe?

A few questions:

(1) Does the deadlock occur in the moment when you inject, or some time later when a new process is started, while DLL injection is active?
(2) Does every process deadlock, or only some?
(3) Is it always the same process which deadlocks? Or is it random which one deadlocks?
(4) Once the deadlock occurs, does it then always occur?
(5) How often does the problem occur? Like 1 out of 10 times? Or more often?
(6) Can you reproduce this on your development PC?
(7) Which exact OS are we talking about? Or does this problem occur on multiple OSs? Which ones?
(8) Have you called InitializeMadCHook() before calling HookAPI()?
(9) Which exact API are you hooking when the deadlock occurs? (hookterminateapis.cpp @ 866)
power888
Posts: 54
Joined: Sat May 23, 2009 8:55 am

Re: HookAPI on 64bit deadlocks

Post by power888 »

Thanks for your reply..

Can you please double check with 3.1.6, just to be safe?
==> yes. (3.1.2), (3.1.3), (3.1.5), (3.1.6) have same status

A few questions:

(1) Does the deadlock occur in the moment when you inject, or some time later when a new process is started, while DLL injection is active?
==> happend when new process is started while Dll injection is activated... (when LoadLibrary, deadlock is not happened)

(2) Does every process deadlock, or only some?
==> only some process is deadlocked...

(3) Is it always the same process which deadlocks? Or is it random which one deadlocks?
==> only some specific process is deadlocked...
==> and mainly, this application is developed by Visual Basic 6.0 or VB.Net
==> Currently, I have 2 applications that cause deadlock. (and I had developed it by VB6.0)
** and have some problem with mstsc... (cant execute any local app)

(4) Once the deadlock occurs, does it then always occur?
==> not certain.. But with some process, if deadlock is occurred, then always happened deadlock when this process is started until Unload Driver.

(5) How often does the problem occur? Like 1 out of 10 times? Or more often?
==> not certain.. but 1 out of 100. (But if deadlock is occurred, then always happened deadlock when this process is started.)

(6) Can you reproduce this on your development PC?
==> not mine.. but I will try to reproduce it,.

(7) Which exact OS are we talking about? Or does this problem occur on multiple OSs? Which ones?
==> Mainly occur 32bit application in Win7-64Bit.
But sometimes, occur in Win7-32 bits.

(8) Have you called InitializeMadCHook() before calling HookAPI()?
==> of course.
in DLLMain()
InitializeMadCHook();
HookAPI(); <== and flag of HookAPI is 0.

(9) Which exact API are you hooking when the deadlock occurs? (hookterminateapis.cpp @ 866)
==> TerminateProcess API.. (Not NTTerminateProcess)

** and if you need it, I will send it with full source (dll & exe)
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: HookAPI on 64bit deadlocks

Post by madshi »

There are 3 more important things you could/should test:

(1) Can you reproduce it with a clean OS installation (e.g. in a virtual machine)? That is important to know because if you can reproduce it there, it means that the problem got nothing to do with other software (like anti-virus etc) being installed on the PC. And it also means it should be possible for me to reproduce the problem. If you can't reproduce the problem in a clean OS installation, then we're in trouble.

(2) Can you please try the HookProcessTermination and PrintMonitor demos?

http://madshi.net/HookProcessTermination.zip
http://madshi.net/PrintMonitor.zip

If the problem occurs with the HookProcessTermination demo, too, then I don't need any source code or test projects from you, which would make things easier. If the problem occurs with the HookProcessTermination demo, but not with the PrintMonitor demo, then that suggests that hooking the TerminateProcess() API might be problematic. If the problem occurs with the PrintMonitor demo, too, we know that it has nothing to do with hooking the TerminateProcess() API.

(3) Could you try reproducing the problem with an empty hook dll? Just comment out the HookAPI() calls. That will help us figure out whether the problem is caused by the DLL injection, or by the API hooking. If the problem still occurs after you commented out the HookAPI() calls, then please also try commenting out the Initialize/FinalizeMadCHook() calls. That will help figure out whether Initialize/FinalizeMadCHook() cause the problem or not.

Thanks!
power888
Posts: 54
Joined: Sat May 23, 2009 8:55 am

Re: HookAPI on 64bit deadlocks

Post by power888 »

(1) Can you reproduce it with a clean OS installation (e.g. in a virtual machine)? That is important to know because if you can reproduce it there, it means that the problem got nothing to do with other software (like anti-virus etc) being installed on the PC. And it also means it should be possible for me to reproduce the problem. If you can't reproduce the problem in a clean OS installation, then we're in trouble.
==> I think that deadlock happened with Anti-Virus app had installed on the PC.
In Clean PC, I had tested several times, but dont happened...

(2) Can you please try the HookProcessTermination and PrintMonitor demos?

http://madshi.net/HookProcessTermination.zip
http://madshi.net/PrintMonitor.zip

If the problem occurs with the HookProcessTermination demo, too, then I don't need any source code or test projects from you, which would make things easier. If the problem occurs with the HookProcessTermination demo, but not with the PrintMonitor demo, then that suggests that hooking the TerminateProcess() API might be problematic. If the problem occurs with the PrintMonitor demo, too, we know that it has nothing to do with hooking the TerminateProcess() API.
==> Already, I had tested your sample..
HoolProcessTermination have same issues. but PrintMonitor have not issues.
So, I think that deadlock is occurred with TerminateProcess API.

(3) Could you try reproducing the problem with an empty hook dll? Just comment out the HookAPI() calls. That will help us figure out whether the problem is caused by the DLL injection, or by the API hooking. If the problem still occurs after you commented out the HookAPI() calls, then please also try commenting out the Initialize/FinalizeMadCHook() calls. That will help figure out whether Initialize/FinalizeMadCHook() cause the problem or not.
==> I will test it.
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: HookAPI on 64bit deadlocks

Post by madshi »

Ok, good information. How does your TerminateProcess hook callback function look like? Try doing nothing in it except calling the original API ("return TerminateProcessNext(...);"). Do the deadlocks go away then?
power888
Posts: 54
Joined: Sat May 23, 2009 8:55 am

Re: HookAPI on 64bit deadlocks

Post by power888 »

I will test it with blank TerminateProcess callback like
(BUT, It's very difficult to reproduce,... actually, in my customer site, deadlock happened 1 or 2 time in a week)

Code: Select all

BOOL	WINAPI TerminateProcessCallback(HANDLE hProcess, UINT uExitCode)
{
     return TerminateProcessNext(hProcess, uExitCode);
}
And below is current callback module.

** My TerminateProcess Callback module

Code: Select all

BOOL	WINAPI TerminateProcessCallback(HANDLE hProcess, UINT uExitCode)
{

	if ((hProcess == 0) || (hProcess == INVALID_HANDLE_VALUE) ) {
		return TerminateProcessNext(hProcess, uExitCode);
	}

	if (!IsAllowed(hProcess)) {
	    SetLastError(ERROR_ACCESS_DENIED);
		return false;
	} else
		return TerminateProcessNext(hProcess, uExitCode);
}

[b]** IsAllowed function :: Check for terminate process[/b]
BOOL IsAllowed(HANDLE hProcess)
// ask the user whether the current process may terminate the specified process
{
TTerminationRequest tr;
DWORD session;
USHORT bufLenInChars;
WCHAR	curProcessName[1024];


	if ((hProcess == 0) || (hProcess == INVALID_HANDLE_VALUE)) 
		return true;

	if ((hProcess) && (hProcess != GetCurrentProcess())) {

		bufLenInChars = sizeof(curProcessName)/sizeof(WCHAR);
		ProcessIdToFileNameW(ProcessHandleToId(hProcess), curProcessName, bufLenInChars);

		// which terminal server (XP fast user switching) session shall we contact?
		if ((AmSystemProcess()) && (!GetCurrentSessionId()))
			// some system processes are independent of sessions
			// so let's contact the HookProcessTermination application instance
			// which is running in the current input session
			session = GetInputSessionId();
		else
			// we're an application running in a specific session
			// let's contact the HookProcessTermination application instance
			// which runs in the same session as we do
			session = GetCurrentSessionId();

		// contact our application, which then will ask the user for confirmation
		// hopefully there's an instance running in the specified session

		WCHAR	pc3[MAX_PATH_1024];
		memset(pc3, 0x00, sizeof(pc3));

		wcsncpy_s_Check(pc3, curProcessName, sizeof(pc3)/sizeof(WCHAR), NULL, __LINE__, __FILE__);
		wcsncpy_s(pc3, sizeof(pc3)/sizeof(WCHAR), curProcessName, sizeof(pc3)/sizeof(WCHAR)-1);

		if (wcslen(pc3)==0) 
			return true;

		SetUpper(pc3, pc3);

		if ( (wcsstr(pc3, L"TARGET01.EXE")!=NULL) ||
			(wcsstr(pc3, L"TARGET02.EXE")!=NULL) )
			return false;
		else
			return true;
	}
	return true;
}
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: HookAPI on 64bit deadlocks

Post by madshi »

One thing that could eventually be problematic is the use of big local var arrays (WCHAR [1024]), because if there's a thread short on stack space calling TerminateProcess(), it might get into trouble. But this is just a shot in the dark. Let's see what your test results will be.
power888
Posts: 54
Joined: Sat May 23, 2009 8:55 am

Re: HookAPI on 64bit deadlocks

Post by power888 »

hmm.. Evenif after remove HookAPI(TerminateProcess), sometimes (very rarely, 1 or 2 times in a week), some application's execution is failed... (Process is started, But suspened right now)..
(Above application is started and closed every 10 minutes)

So, I had set some application's process as Exclude Process, But have same status.....

which part will I checked?

BTW, I dont hook CreateProcessEXA(W), and you said that InjectLibrary will use CreateProcessEXA(W).. is it any relation with process suspend?
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: HookAPI on 64bit deadlocks

Post by madshi »

Injection into newly created processes is done by the kernel mode driver and this does not use CreateProcessEx, it also does not suspend the process.

I'm still waiting for your test results with an empty hook dll. Did you already try that?

One more thing: You say the problem did not occur in a clean VM. Did you try to install that Visual Basic 6.0 or VB.Net application in the clean VM? Maybe the problem occurs in the clean VM then, too? I'm asking because if we knew that the problem occurs in a clean VM with just that VB application installed, then we knew that it's not your OS installation which is problematic, nor any other software on that PC. If you can't reproduce the problem in the clean VM even after installing the VB application, then probably the problem occurs only in combination of that VB application with some other 3rd party software.
Post Reply