InjectLibrary fails, GetLastError = ERROR_INSUFFICIENT_BUFFE

c++ / delphi package - dll injection and api hooking
Post Reply
pto
Posts: 15
Joined: Wed Oct 20, 2004 5:21 pm

InjectLibrary fails, GetLastError = ERROR_INSUFFICIENT_BUFFE

Post by pto »

I created an utility using dynamic dll hooking and it's been in use for over a year with no problem. But some reason, since last week it stops working on Windows 2000. It still works on XP.

In an attempt to fix the problem, I updated madCHook.dll to the one included in version 2.5.0.15. It did not help. I changed the code to add more debugging messages after calling InjectLibrary. GetLastError() returns 122 which is the code for ERROR_INSUFFICIENT_BUFFER. I need help on how to fix it.

My real problem is that my customed dll is supposed to be hooked into current and future instances of target application. It appears that it's only hooked for the current instances. When I started a new instance of the target application, the hook does not seem to be there. I think it's due to the error with InjectLibrary I mentioned above. A portion of the code is copied below

Code: Select all

#include "stdafx.h"

#define PRINTFILE_TEMP     "C:\\JTI\\PrintFile\\Temp"
#define PRINTFILE_INQUEUE  "C:\\JTi\\PrintFile\\InQueue\\"
#define ENDFILE            "end.end"         // Use to end the application
#define INJECTSERVICE      "NfuMonSvc"       // Name of injection service
#define INJECTDLL          "NfuMonApi.dll"   // Name of injection dll

typedef struct
  // this is the information record which we receive via Ipc
  TDllInjectRequest {
    CHAR  cAction;    			  // 'I' = InjectDLL; 'U' = UninjectDll; 'R' = Remove the service
    CHAR  szModuleName [MAX_PATH + 1];    // Name of DLL
    DWORD dwTimeOut;
    DWORD dwSession;

...
...


//-------------------------------------------------------------------------
// Purpose: Inject the DLL into system wide process
//-------------------------------------------------------------------------
BOOL Inject(CHAR cAction)
// (un)inject our dll system wide
{
  TDllInjectRequest dir;
  BOOL              res;
  BOOL              result;
  char cMsg[200];
  DWORD wError;

  // first let's try to inject the dlls without the help of the service
  if (cAction == 'I')
  {
       result =   InjectLibrary(ALL_SESSIONS | SYSTEM_PROCESSES, INJECTDLL, 10000);
	   wError = GetLastError ();
     sprintf (cMsg, "Injection result %d, error = %d", result, wError);
     
     // This shows result = -1 and error = 122
	   MessageBox (NULL, cMsg, "NfuDirMon", MB_OK);    
  }
  else result = UninjectLibrary(ALL_SESSIONS | SYSTEM_PROCESSES, INJECTDLL, 10000);
  if (!result && osvi.dwPlatformId == VER_PLATFORM_WIN32_NT) {
    // didn't work, so let's try to ask our service for help
    // first of all we wait until the service is ready to go
    MessageBox (NULL, "Asking service to inject/uninject", "NfuDirMon", MB_OK);
    WaitForService(INJECTSERVICE);
    
    // then we prepare a dll injection request record
    dir.cAction   = cAction;
    dir.dwTimeOut = 5000;
    dir.dwSession = GetCurrentSessionId();
    strcpy (dir.szModuleName, INJECTDLL);
    
    // now we try to contact our injection service
    result = SendIpcMessage(INJECTSERVICE, &dir, sizeof(dir), &res, sizeof(res), 15000, true) && res;
  }
  
  return result;
}

...
I'm very grateful if you can help.

Thanks,
Patrick
madshi
Site Admin
Posts: 10766
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

Does the service get contacted? Does the service get the request? Did you change anything on your w2k system? What happens if you try to inject an empty dll?
pto
Posts: 15
Joined: Wed Oct 20, 2004 5:21 pm

Post by pto »

Yes, the service gets contacted and receives the request. I got the same result when trying to inject an empty dll (empty.dll included in madCodeHook demos).

As for the W2K systems, there may be a few changes as most of them have auto update and also Symantec Anti Virus LiveUpdate. Our admin person thinks it is caused by Symantec Antivirus data definition 9/15/2005 ver. 23 and higher.

Since I put the debugging messages in, I got the same error message on my XP pro. The funny thing is the dll gets injected on all instances, current and future, of the target application on XP and not on W2K. So it could be the error was there all along, but somehow managed to work on all XP and W2K.

We also have a few Win98 systems, and again, the injection used to work on them before. With madCHook.dll (2.5.0.15), we got illegal operation error and blue screen of death. Reverted back to the old version, we got illegal operation error but no blue screen when the instance of target app was closed. The illegal operation happens in kernel32.dll (invalid page fault), which is the dll we tried to inject our own into.

Thanks,
Patrick
madshi
Site Admin
Posts: 10766
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

What I don't understand is how it suddenly can stop working. I mean something must have changed, right?
pto
Posts: 15
Joined: Wed Oct 20, 2004 5:21 pm

Post by pto »

That's puzzling to me too. The one thing that's consistent among them is the Symatec Antivirus definition file. Could that be the culprit?

I tried with Antivirus disabled on my XP and still received error 122.

Patrick
pto
Posts: 15
Joined: Wed Oct 20, 2004 5:21 pm

Post by pto »

One other thing I don't understand is the return value from InjectLibrary function. Why is it returning -1 for BOOL type? If you look through the code I posted, it does not execute the 'Asking service to inject/uninject' portion since -1 is still TRUE.

When I uninject the dll, that portion of the code gets called.

Patrick
madshi
Site Admin
Posts: 10766
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

0 = false
everything else (including -1) = true

Delphi returns -1 for long bool "true".
madshi
Site Admin
Posts: 10766
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

Right now I don't know what to suggest. You could do this:

(1) Do the precompiled madCodeHook demos work?
(2) Uninstall AV. Reboot. Retest.
pto
Posts: 15
Joined: Wed Oct 20, 2004 5:21 pm

Post by pto »

The precomipled HookProcessTermination.exe and InjectService.exe works on XP pro, but not on W2K. On W2K, InjectService.exe was started yet HookProcessTermination does not seem work. It always complains that InjectService must be started first, even though it's running.

As for uninstalling AV, I've not done that. I'll have to ask the admin person to do that for me.
madshi
Site Admin
Posts: 10766
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

Well, I think it can either be a AV update or some OS updates which break madCodeHook. If you found the reason, please let me know. Thanks!
pto
Posts: 15
Joined: Wed Oct 20, 2004 5:21 pm

Post by pto »

It's both AV and OS updates. OS update on W2k causes the problem while on Win98, we think it is AV since we don't do any Win98 OS update.
But I cannot be tottally sure since we did not rollback any OS update in order to test.

Instead I changed my code a bit and it seems to solve the problem. My injected dll used to hook just GetLogicalDrives api to trick target applications that A & B drives exist. It worked for a over a year, until last week it suddenly stops on W2k and Win98. It still worked for XP though.

In my many attempts, I tried to also hook 'GetDriveTypeA' and 'GetDriveTypeW' api, and lo and behold it works. I guess on W2k, the OS now checks for drive type as well in order to show them in its drive list. I've also added RenewHook call just in case AV removes it - since I saw the dll gets injected for current and not future instances of the apllication. I also updated madCHook.dll to version 2.5.0.15.

The updated madCHook.dll, however, fails on Win98. We get 'Illegal operation' error whenever we quit the target application. We changed the dll back to the old version and the error goes away. So right now, we are running the old version of madCHook.dll and updated version of our customed dll.

Just for good measure, I tried precompiled HookProcessTermination.exe and InjectService.exe on W2k again. It still fails. One thing I noticed is that in the Task Manager, InjectService.exe shows up as InjectService.e (with dot e at the end), while on XP it is InjectService (with no dot e). They work on XP though.

So I think the W2k OS update has tighten up security and I need to mask the drive type as well. On Win98, the new madCHook.dll does not work. It's causing Page Fault errors.

Thanks for help and your quick responses.
Patrick
madshi
Site Admin
Posts: 10766
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

pto wrote:It's both AV and OS updates. OS update on W2k causes the problem while on Win98, we think it is AV since we don't do any Win98 OS update.
But I cannot be tottally sure since we did not rollback any OS update in order to test.

Instead I changed my code a bit and it seems to solve the problem. My injected dll used to hook just GetLogicalDrives api to trick target applications that A & B drives exist. It worked for a over a year, until last week it suddenly stops on W2k and Win98. It still worked for XP though.

In my many attempts, I tried to also hook 'GetDriveTypeA' and 'GetDriveTypeW' api, and lo and behold it works. I guess on W2k, the OS now checks for drive type as well in order to show them in its drive list.
Does that mean the problem is fully solved for you?
pto wrote:I've also added RenewHook call just in case AV removes it - since I saw the dll gets injected for current and not future instances of the apllication.
RenewHook doesn't help for injection! So does injection still not work for future instances of the app?
pto wrote:I also updated madCHook.dll to version 2.5.0.15.

The updated madCHook.dll, however, fails on Win98. We get 'Illegal operation' error whenever we quit the target application. We changed the dll back to the old version and the error goes away. So right now, we are running the old version of madCHook.dll and updated version of our customed dll.

Just for good measure, I tried precompiled HookProcessTermination.exe and InjectService.exe on W2k again. It still fails. One thing I noticed is that in the Task Manager, InjectService.exe shows up as InjectService.e (with dot e at the end), while on XP it is InjectService (with no dot e). They work on XP though.
Interesting. I guess I have to recheck everything with all w2k OS updates.
pto wrote:So I think the W2k OS update has tighten up security and I need to mask the drive type as well. On Win98, the new madCHook.dll does not work. It's causing Page Fault errors.
Please try the latest beta build. It contains a fix for the Win98 finalization crash:

http://madshi.net/madCollectionBeta.exe
pto
Posts: 15
Joined: Wed Oct 20, 2004 5:21 pm

Post by pto »

Does that mean the problem is fully solved for you?
Yes, it is.
RenewHook doesn't help for injection! So does injection still not work for future instances of the app?
This part is not too clear to me too. Before I hooked 'GetDriveType*' api, I had MessageBox displayed whenever my dll gets called. I didn't get any messages for future instances of the app, even after I added RenewHook calls. That led me to think that injection failed. After I added the 'GetDriveType*' , unfortunately I also removed the messages. It's now working for future instances as well. So it could be that because of the missing 'GetDriveType*', my dll didn't get invoked for future instances of the app, or something else?

Remember earlier on I mentioned about the error code ERROR_INSUFFICIENT_BUFFER. Unfortunately I also removed that MessageBox too. So it could be the error is always there, but I just never check.
Please try the latest beta build. It contains a fix for the Win98 finalization crash
It works.

Thanks
Patrick
madshi
Site Admin
Posts: 10766
Joined: Sun Mar 21, 2004 5:25 pm

Post by madshi »

Ok, thanks for the feedback. I'm glad that you got it all working now.
Post Reply