StackHash AppCrash

c++ / delphi package - dll injection and api hooking
Post Reply
choochy2003
Posts: 88
Joined: Fri Mar 21, 2008 4:52 am
Location: Adelaide, South Australia
Contact:

StackHash AppCrash

Post by choochy2003 »

I seem to be getting a strange application crash lately within the application that the Hook DLL communicates with. It seems whenever an event is triggered that sends a message via IPC from the Hook DLL to the receiving application (TestClient.exe) the application crashes with the following details:

Problem Event Name: APPCRASH
Application Name: TestClient.exe
Application Version: 1.0.0.0
Application Timestamp: 553b142e
Fault Module Name: StackHash_2264
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 00000000

Even if I comment out all of the code within the Callback procedure, it still causes the crash.

However, the strange thing is if I recompile a brand new executable version modified version of the TestClient application, it sometimes goes away for a while. It seems somewhat random, although when it starts to happen it keeps happening.

Any ideas on what I should be looking into? I feel its something to do around the IPC stuff. I am using XE7 and the latest release of madCodeHook (3.1.9) the problem also happens with the version previous to this one.
choochy2003
Posts: 88
Joined: Fri Mar 21, 2008 4:52 am
Location: Adelaide, South Australia
Contact:

Re: StackHash AppCrash

Post by choochy2003 »

After some playing around I figured out under what situations it sometimes it works and sometimes it does. Its seems that if I build the Application directly from the IDE it works, but if I am building it via msbuild it doesn't work, like so:

msbuild "TestClient.dproj" /p:Config=Release;Platform=Win64 /t:Build

Is there something I need to account for when building a madCodeHook project using command line via msbuild?
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: StackHash AppCrash

Post by madshi »

Please double check that the calling convention, list of parameters and return type of your IPC callback function is correct. Also please make sure that you've called InitializeMadCHook() somewhere in your EXE's initialization. Maybe the InitializeMadCHook() was missing?
choochy2003
Posts: 88
Joined: Fri Mar 21, 2008 4:52 am
Location: Adelaide, South Australia
Contact:

Re: StackHash AppCrash

Post by choochy2003 »

I did include the InitializeMadCHook in my Delphi code but that made no difference.

I just think its strange that when compiled from the IDE it works fine, but when compiled with msbuild it doesn't. What could be different between the 2 different compilation methods? Does madCodeHook inject anything into the application post build? something that it might not do if using msbuild.
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: StackHash AppCrash

Post by madshi »

Ah, Delphi, sorry, I missed that. You can remove the InitializeMadCHook() call again, that's only for MSVC++.

Are you using madExcept, too? If so, please make sure that your msbuild compilation creates a proper mapfile.
choochy2003
Posts: 88
Joined: Fri Mar 21, 2008 4:52 am
Location: Adelaide, South Australia
Contact:

Re: StackHash AppCrash

Post by choochy2003 »

No, I have checked and madExcept is disabled in the project's madExcept Settings and there is no reference to the madExcept unit anywhere in the code.

I have checked the created binary under both scenarios, and the file size of the binary is exactly the same (to the byte) in both cases. So it seems to look like there shouldn't any difference in what code gets included in the compile, but just in how the code operates.

Also, I noticed today that even the IDE compiled binary causes the problem, however it doesn't happen anywhere as regularly as when it is copied through the msbuild method. And sometimes an a reboot solves the issue and other times it does not. Its all rather random to be honest.
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: StackHash AppCrash

Post by madshi »

Ok, then try enabling madExcept to see if it's able to catch the crash and report proper callstacks. Maybe that helps us figuring out what's going on.
choochy2003
Posts: 88
Joined: Fri Mar 21, 2008 4:52 am
Location: Adelaide, South Australia
Contact:

Re: StackHash AppCrash

Post by choochy2003 »

Yeah I tried that but it didn't seem to capture anything.
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: StackHash AppCrash

Post by madshi »

So the crash still occurs with madExcept enabled, but madExcept doesn't catch it, instead the OS reports the crash? That's bad...

Ok, next try: Try commenting out all the "HookAPI" calls, but leave the rest as it is. Does the crash still occur?
choochy2003
Posts: 88
Joined: Fri Mar 21, 2008 4:52 am
Location: Adelaide, South Australia
Contact:

Re: StackHash AppCrash

Post by choochy2003 »

I have been playing around with it and it seems that the issue goes away when I stop using MethodToProcedure when assigning the Callback with the CreateIpcQueue function.
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: StackHash AppCrash

Post by madshi »

Ok, that's progress. Please show me your IPC related code. The CreateIpcQueue code, your IPC callback function, and the way you're calling MethodToProcedure etc.
choochy2003
Posts: 88
Joined: Fri Mar 21, 2008 4:52 am
Location: Adelaide, South Australia
Contact:

Re: StackHash AppCrash

Post by choochy2003 »

This is running as Win64 application.

CreateIpcQueue & MethodToProcedure calls:

Code: Select all

CreateIpcQueue(
    NETWORKFILTER_IPCQUEUE_NAME, 
    MethodToProcedure(Self, @TNetworkFilter.FilterCallbackHandler)
    );
IPC Callback function:

Code: Select all

procedure TNetworkFilter.FilterCallbackHandler(
    AName          : PAnsiChar;
    AMessageBuffer : Pointer;
    AMessageLength : DWORD;
    AAnswerBuffer  : Pointer;
    AAnswerLength  : DWORD
    ); stdcall;
Keep in mind that the issue happens regardless if I am processing anything in the callback method or if the method is empty. In fact, the callback never gets entered
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: StackHash AppCrash

Post by madshi »

Looks ok to me on a quick glance. What happens if you remove the MethodToProcedure and try with an empty global IPC callback function without all the class hoopla? Just as a test, of course.
choochy2003
Posts: 88
Joined: Fri Mar 21, 2008 4:52 am
Location: Adelaide, South Australia
Contact:

Re: StackHash AppCrash

Post by choochy2003 »

If I stop using the MethodToProcedure and use an empty Global function or one with my processing logic in it. It all works fine.
madshi
Site Admin
Posts: 10753
Joined: Sun Mar 21, 2004 5:25 pm

Re: StackHash AppCrash

Post by madshi »

And you're absolutely sure that the "Self" pointer you're feeding to MethodToProcedure stays valid as long as you use the IPC queue? Maybe the object is destroyed and recreated at some point and you forgot to destroy and recreate the IPC queue in that situation?

To be honest, I've no clue what's going on here. Can you find a way to reproduce this in a simple test project? If so, send that test project to me and it'll probably be easy for me to find the cause of the problem.
Post Reply