Debugging a system crash

[alfaunits]

I am trying to debug a very uncommon situation.
I inject the DLL (system wide), which works fine. When I try to uninject the DLL, the system crashes with D0000144 BSOD code, sometimes in ClassPnP sometimes in Ntfs, but mostly in ClassPnP.

Considering this is a system crash, the DLL would not be able to it, so there is some driver issue. I have set up Driver Verifier to check the iFileOp driver (madCodeHook driver for my DLL), but the exact same issue occurs, with no additional data from DV.

At the point when I get the crash (WinDBG connected to the VMWare test machine where the iFileOp driver is running), all I can see is an already bad state (no debug information before that point suggests any issues).

Any suggestions on how to proceed here? WinXP only, does not happen with W7.
EDIT: Actually it happens with W7 as well...

[madshi]

Does this occur every time, or just once in a while?

Could you please try the latest beta build, just to be safe?

http://madshi.net/madCollectionBeta.exe

[alfaunits]

It happens all the time, no exception.

I am trying v3.1.3. today.
EDIT: Same result.

[madshi]

Can you please try this demo:

http://madshi.net/HookProcessCreation.zip

Does this demo (with the files compiled by me) work on your PC? If so, please try to recompile the demo yourself, and configure and sign a new driver. Does it still work? If not, something goes wrong on your PC when compiling or when creating the driver. If it does work fine when you recompile it, you should probably try to find out what the difference is between the demo and your own project. E.g. you could comment out all HookAPI() calls in your dll to check whether that fixes the issue. Or copy your code to the demo hook dll to check whether that way the demo also produces a system crash.

[alfaunits]

No, only this particular injected DLL faults.
Even with no hooking the issue occurs. What I could pinpoint is that if I call one of my API (a function that allocates memory for communication with our driver) during DLL Init, the uninjection causes a system-wide crash. I commented out everything else.
The DllMain calls my AFP_RegisterSession API, which creates the AFP_Message thread. It seems as if the uninjection closes the hStopEvent handle before the WaitForSingleObject gets a chance to react. DLL_Unload does nothing at this time (and there are no hooks active nor was there any hook active - I just injected the DLLs). That scenario has no issues in a regular app (non-injected calls).
NOTE: There is no comm. with the driver in the thread, it sends no Flt messages during the test.

The thread code (skip to next code snippet for the important part):

Code: Select all

WORD __stdcall AFP_Message(PVOID ThreadData)
{
    PAFP_THREAD         lpData          = ThreadData;
    HANDLE              hPort           = lpData->hPort;
    PCALLBACK_DATA      lpMsg           = &lpData->lpMsg;
    PCALLBACK_RESULT    lpResult        = lpData->lpResult;
    PARS_RESULT_DATA    lpResultData    = &lpResult->Data;
    HANDLE              hStopEvent      = lpData->hStopEvent;
    PVOID               vaDataMsg       = (PVOID)((PCHAR)lpData - 4096),
                        vaDataRes       = (PVOID)((PCHAR)lpResult - 4096);
    HRESULT             hRes;
    ARS_CALLBACK        Callback;
    PVOID               lpCallbackData;
    DWORD               dwReserved;
    PVOID               lpReserved;
    OVERLAPPED          lpOverlap;
    AFF_STRING          lpFile, lpUser, lpDest, lpProc;

    lpOverlap.hEvent    = lpData->hEvent;
    while(1)
    {
MsgLoop:
        hRes    = FilterGetMessage(hPort,
                                   (PFILTER_MESSAGE_HEADER)lpMsg,
                                   ARS_MESSAGE_MAX_SIZE,
                                   &lpOverlap);
        switch(hRes)
        {
            case S_OK:
                goto GotMsg;
            case __HRESULT_FROM_WIN32(ERROR_IO_PENDING):
                while(1)
                {
                    switch(WaitForSingleObject(lpOverlap.hEvent,
                                               10))
                    {
                        case WAIT_TIMEOUT:
                            if(WaitForSingleObject(hStopEvent,
                                                   0) == WAIT_OBJECT_0)
                            {
                                goto Cleanup;
                            }
                            break;
                        case WAIT_OBJECT_0:
GotMsg:
/*
                            lpFile.Length   = lpMsg->Data.dwFileName;
                            lpFile.Name     = (PWCHAR)lpMsg->Data.Data;
                            lpUser.Length   = lpMsg->Data.dwUserName;
                            lpUser.Name     = (PWCHAR)((PCHAR)lpFile.Name +
                                                       lpFile.Length);
                            lpDest.Length   = lpMsg->Data.dwDestName;
                            lpDest.Name     = (PWCHAR)((PCHAR)lpUser.Name +
                                                       lpUser.Length);
                            dwReserved      = lpMsg->Data.dwReserved;
                            lpReserved      = (PWCHAR)((PCHAR)lpDest.Name +
                                                       lpDest.Length);
                            lpProc.Length   = lpMsg->Data.dwProcess;
                            lpProc.Name     = lpMsg->Data.szProcess;
                            Callback        = lpMsg->Data.Callback;
                            lpCallbackData  = lpMsg->Data.ClbData;
                            memcpy(&lpResult->Header,
                                   &lpMsg->Header,
                                   sizeof(lpResult->Header));
                            // Set up the default results for the callee.
                            lpResultData->dwTimeoutResult   = 
                                lpMsg->Data.dwTimeoutResult; // 0xC0000022;
                            lpResultData->dwARS_Result      =
                                lpMsg->Data.dwARS_Result; // ARS_RESULT_DENY
                            lpResultData->Parameters.Create.szFileName  =
                                (PVOID)&lpResult->Data.Data;
                            lpResultData->Parameters.Create.dwFileName  = 0;
                            __try
                            {
                                Callback(&lpFile,
                                         &lpDest,
                                         &lpUser,
                                         &lpProc,
                                         lpMsg->Data.ProcessID,
                                         lpMsg->Data.dwOp,
                                         lpResultData,
                                         lpCallbackData,
                                         dwReserved,
                                         lpReserved);
                            }
                            __except(EXCEPTION_EXECUTE_HANDLER)
                            {
                                OutputDebugStringW(L"AFP: Callback generated exception\n");
                            }
                            __try
                            {
                                hRes    = FilterReplyMessage(hPort,
                                                             &lpResult->Header,
                                                             sizeof(*lpResultData) +
                                                             lpResultData->Parameters.Create.dwFileName);
                            }
                            __except(EXCEPTION_EXECUTE_HANDLER)
                            {
                                OutputDebugStringW(L"AFP: FltMgr reply generated exception\n");
                            }
                            if(hRes != S_OK)
                                DbgPrint("AFF Thread: FRM returned error %u\n", hRes);
                            goto MsgLoop;
*/
                        default:
                            goto Cleanup;
                    }
                }
                break;
            default:
                goto Cleanup;
        }
    }
Cleanup:
    CloseHandle(hPort);
    CloseHandle(lpOverlap.hEvent);
    CloseHandle(hStopEvent);
    if(vaDataMsg)
        VirtualFree(vaDataMsg,
                    0,
                    MEM_RELEASE);
    if(vaDataRes)
        VirtualFree(vaDataRes,
                    0,
                    MEM_RELEASE);
    OutputDebugString(L"AFP: Closing callback session\n");
    return 0;
}

if I comment out this part:

Code: Select all

                            if(WaitForSingleObject(hStopEvent,
                                                   0) == WAIT_OBJECT_0)
                            {
                                goto Cleanup;
                            }

the crash does not occur.

[madshi]

Creating threads within a hook dll is generally a dangerous thing to do. See hooking rule number 9:

http://help.madshi.net/HookingRules.htm

So if I understand you correctly, you're creating a thread in DLL_PROCESS_ATTACH and the thread continues to run? Then in DLL_PROCESS_DETACH you're trying to shutdown the thread? The problem with this is that during DLL_PROCESS_DETACH your secondary thread might not get any CPU time, due to the loader lock, and thus the secondary thread might not have a chance to shut down. Then if uninjection succeeds, and your secondary thread is still running, it will try to execute code in your dll, but your dll was already freed. So basically the secondary thread will try to execute code in a memory area which is no longer allocated. As a result every process from which your dll is uninjected will crash with an access violation. If a crash like that happens in certain system processes, the OS will blue screen.

The best solution for this is to not create a secondary thread in your hook dll. It's generally a bad idea. A hook dll should try to be as low key as possible. Some applications are very breakable. They can be confused by the slightest difference in behaviour. If you create a secondary thread in your hook dll, some applications might be confused into thinking that your thread is actually theirs (all dlls are notified about the thread you created!) and they might behave incorrectly because of that. Furthermore secondary threads created in a hook dll have the problem that closing down those threads from DllMain(PROCESS_DETACH) is not really possible in a reliable way.

Is there no other way you can achieve your goals, without creating secondary threads in your hook dll?

[alfaunits]

I see Unfortunately, the thread is required, there is no other way to get messages from the driver.
I don't intend to let production code uninject then, I was hoping it is something different, so I could speed up debugging (no need to reboot a VM, just put the new DLL/SYS in).

I was debugging it to make sure it wasn't an actual bug/leak/memory corruption, which could bite me later.

Thanks.

[madshi]

And you do need to create that thread in every process your dll is injected to? The only way to make this stable with uninjection is if your driver (or your application) notifies every hook dll about the soon-to-come uninjection, so that the threads can be closed down before the dlls are actually uninjected. You'd need to leave enough time for the dlls to close the thread down before you initialize the uninjection.

Please be aware that creating a thread in DLL_PROCESS_ATTACH of a hook dll is not recommended. I can't (and won't even try to) stop you from doing that. But if you do that, please understand that I can't guarantee stability. Just so you know...

[alfaunits]

I can see why the thread termination (of any kind) from DLL_PROCESS_DETACH won't work, but why is just creating a thread from PROCESS_ATTACH bad? (it need not run right away).

[madshi]

Every time a thread is created, all dlls are notified (see DLL_THREAD_ATTACH). Not a problem for probably 99.99% of all dlls out there. But there might be the one weird dll which expects the application to create a specific secondary thread and the dll might then do specific stuff when the first DLL_THREAD_ATTACH event arrives, e.g. create a window or something. If you create a thread in your DllMain, you might confuse such a weird dll into thinking that your thread is actually the application's secondary thread the dll was waiting for. In that situation the weird dll might then create a window in the context of your secondary thread. The window would then be owned by the "wrong" thread which could result in all sorts of problems (e.g. when your thread dies, so does the window).

Granted, the problem I described is a rather weird and rare case, but I've seen weirder things than that. Anyway, by far more problematic is the clean shutdown of the threads. If you get that solved somehow then your solution should be stable for most applications that exist out there. For maximum compatability not creating any threads is still the recommended approach, though.

[iconic]

DLL_THREAD_XxX events are only received by a module if DisableThreadLibraryCalls() is not used within that loaded module. Anyhow, the chances of an issue are lessened if you know how to disable the notifications before you create a thread, reenabling it after of course. I don't condone the production use of this hack code in which I have written but I have had reasons to use this technique in the past before. The proper way would be to not just reenable this for all modules after disabling the notifications but instead to remember the module's flags on an independent basis and apply the same setting back to it. Of course modules can also be loading during this external call so you might miss those modules... Thought you might find it interesting or at least entertaining is all

Assume 32-bit code...

Code: Select all

const
   LDR_DISABLE_THREAD_CALLS  =  $00040000;

function DisableThreadLibraryCallsEx(const hProcess: ULONG; bDisable: BOOL; hMod: HMODULE = $FFFF): BOOL;
var
        bRet: ULONG;
     PebAddr: ULONG;
         Peb: TPeb;
         Ldr: TPebLdrData;
      LdrMod: TLdrModule;
      pFlink: Pointer;
       pHead: Pointer;
     dwFlags: ULONG;
    dwModule: ULONG;
     dwPatch: ULONG;
begin
       dwModule := 0;
       dwPatch := 0;
      if (hProcess > 0) then
       begin
         PebAddr := PebBaseAddress(hProcess);
      if NtReadVirtualMemory(hProcess, Ptr(PebAddr), @Peb, sizeof(Peb), @bRet) = 0 then
       begin
      if (Peb.Ldr <> nil) and (Peb.LoaderLock <> nil) and
         (NtReadVirtualMemory(hProcess, Peb.Ldr, @Ldr, sizeof(Ldr), @bRet) = 0)  then
       begin
         pFlink := Ldr.InLoadOrderModuleList.Flink;
         pHead := pFlink;
      while (NtReadVirtualMemory(hProcess, pFlink, @LdrMod, sizeof(LdrMod), @bRet) = 0) do
       begin
      if LdrMod.InLoadOrderModuleList.Flink = pHead then
         Break;
         if (hMod = $FFFF) or (hMod = LdrMod.BaseAddress) then
         begin
         if NtReadVirtualMemory(hProcess, Ptr(DWORD(pFlink) + $34), @dwFlags, sizeof(dwFlags), @bRet) = 0 then
         begin
         if (bDisable) then
         dwFlags := (dwFlags or LDR_DISABLE_THREAD_CALLS)
      else
         dwFlags := (dwFlags xor LDR_DISABLE_THREAD_CALLS);
         if WriteProcessMemory(hProcess, Ptr(DWORD(pFlink) + $34), @dwFlags, sizeof(dwFlags), bRet) then
         inc(dwPatch);
         inc(dwModule);
         end;
         if (hMod <> $FFFF) then
         Break;
         end;
         pFlink := LdrMod.InLoadOrderModuleList.Flink;
        end;
      end;
    end;
  end;
   result := (dwModule > 0) and (dwModule = dwPatch);
end;

Code: Select all

  DisableThreadLibraryCallsEx(hProcess, True);
  InjectLibrary(hProcess, DLL); <--- Thread is created inside DLL's DLL_PROCESS_ATTACH event
  Wait or Sleep() for thread to execute and send notifications
  DisableThreadLibraryCallsEx(hProcess, False);

--Iconic

[madshi]

I *think* that DisableThreadLibraryCalls() stops your own dll from receiving DLL_THREAD_??? events, but it does not stop other dlls in your process from receiving those events if you create threads in your dll. At least that's how I understand the documentation. If I'm correct, this doesn't really help at all.

IIRC, in win9x there was a special flag you could set to create a thread without notifying anyone about it. But I don't think this is still possible in the NT family. At least I don't know how...

[iconic]

I *think* that DisableThreadLibraryCalls() stops your own dll from receiving DLL_THREAD_??? events, but it does not stop other dlls in your process from receiving those events if you create threads in your dll.

Yes, DisableThreadLibraryCalls() only prevents your module from receiving DLL_THREAD_XxX events, not any other module from seeing them. The function that I have supplied however disables DLL_THREAD_XxX events for *any/all* loaded modules, they never receive these events so it is to what you describe in the Win9x method that you mentioned. This is why I named it DisableThreadLibraryCallsEx

--Iconic

[madshi]

Ooops, I only read your text but not your code. You're right, that might work, as ugly as it is...

[iconic]

It definitely works And you're 100% correct, it's UGLY!!!

--Iconic