MCH v3.1.8 x64 Crash when exception raised in hooked procedure

c++ / delphi package - dll injection and api hooking
Post Reply
thall
Posts: 5
Joined: Fri Aug 14, 2020 3:52 pm

MCH v3.1.8 x64 Crash when exception raised in hooked procedure

Post by thall »

Hi Madshi,

MadCodeHook: v3.1.8
Compiler: Delphi 10.3.3 (also tested XE3)
OS: Windows 21H1 64-bit

We have a problem with a 64-bit program crashing when an exception is raised inside a hooked procedure. When compiled as 32-bit everything works as expected, so we're not sure if we are doing something wrong or its an issue in madCodeHook.

Below is a trivial example that reproduces the issue only when compiled to 64-bit:

Code: Select all

unit FMain;

interface

uses
  Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
  Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls;

type
  TForm1 = class(TForm)
    Button1: TButton;
    Button2: TButton;
    procedure Button1Click(Sender: TObject);
    procedure Button2Click(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

uses
  madCodeHook;

var
  HookMeNext: procedure;

procedure HookMe;
begin
  raise Exception.Create('Hooked method');
end;

procedure HookMeCallBack;
begin
  HookMeNext;
end;

procedure DontHookMe;
begin
  raise Exception.Create('Not Hooked method');
end;

procedure TForm1.Button1Click(Sender: TObject);
begin
  HookMe;
end;

procedure TForm1.Button2Click(Sender: TObject);
begin
  DontHookMe;
end;

initialization
  HookCode(@HookMe, @HookMeCallBack, @HookMeNext);

end.
Many thanks,
Tony
iconic
Site Admin
Posts: 1019
Joined: Wed Jun 08, 2005 5:08 am

Re: MCH v3.1.8 x64 Crash when exception raised in hooked procedure

Post by iconic »

Is there any specific reason why you'd need(?) to hook a procedure that intentionally raises an exception?

***Update***
I've just tested and reproduced on x64 only when calling the "hooked" function, as you've said x86 seems to work fine. When commenting out the "raise exception" line it works fine here, so it's definitely related to the way the exception is being generated from within the VCL on x64 when the code hook is set. Hmmm strange :? I'll wait for Madshi to see this thread and see what he thinks, he knows much more than I when it comes to exception handling.


--Iconic
madshi
Site Admin
Posts: 10434
Joined: Sun Mar 21, 2004 5:25 pm

Re: MCH v3.1.8 x64 Crash when exception raised in hooked procedure

Post by madshi »

Exceptions are difficult to handle in x64 for the simple reason that exception handling works completely different in x64. Basically, in x86, exception handling works simply by pushing exception frames to the stack. In x64, it's required that any code section has additional exception related information stored in the EXE/DLL file the code belongs to, for exception handling to work correctly. The problem for madCodeHook is that API hooking requires the use of dynamically allocated code stubs, and for those exception handling won't work properly.

There are ways to solve it somehow, but it's complicated and I haven't managed to look into that yet.

However, you can probably work around it by adding a try..except block around your hook callback function.
madshi
Site Admin
Posts: 10434
Joined: Sun Mar 21, 2004 5:25 pm

Re: MCH v3.1.8 x64 Crash when exception raised in hooked procedure

Post by madshi »

We continued to discuss this topic via email. In case other users are listening here, here's a summary of what we found:

The problem occurs due to madCodeHook's "safe unhooking" feature. If you use the NO_SAFE_UNHOOKING flag when calling HookAPI() then the problem no longer appears. Most other API hooking libraries don't have madCodeHook's "safe unhooking" feature, so they don't have this problem with x64 exception handling.

Here are the technical details:

When using NO_SAFE_UNHOOKING, madCodeHook just writes a JMP instruction to the hooked API. Nothing else. And that jump instruction jumps to your hook callback function. It's a very simple solution. And most other API hooking libraries do the same thing.

However, when not using the NO_SAFE_UNHOOKING flag, madCodeHook has to add a lot of extra code, which basically "counts" how often a thread enters your hook callback function and how often a thread leaves your hook callback function. Only this way madCodeHook knows if your hook callback is still "in use" by a thread or not. Unfortunately this counting is very difficult to do and requires madCodeHook to install 2 separate dynamically created code stubs which are called when your hook callback function is entered and when it's left. And these 2 code stubs are what's causing problems with the exception handling in x64. It is probably possible to fix it somehow, but difficult.

If you don't ever unload your hook dll, then you can use the NO_SAFE_UNHOOKING flag safely. If you want to be able to unload your hook dll, then I don't recommend using NO_SAFE_UNHOOKING because safe unhooking is an important feature to achieve good stability (no crashes) during unhooking.
thall
Posts: 5
Joined: Fri Aug 14, 2020 3:52 pm

Re: MCH v3.1.8 x64 Crash when exception raised in hooked procedure

Post by thall »

Thank you madshi.

Our app hooks during initialization and the hooks last the lifetime of the process, so the NO_SAFE_UNHOOKING flag should fix the issue for us.
madshi
Site Admin
Posts: 10434
Joined: Sun Mar 21, 2004 5:25 pm

Re: MCH v3.1.8 x64 Crash when exception raised in hooked procedure

Post by madshi »

Agreed. You should be fine with that flag, you won't need the "safe unhooking" feature. I'd suggest to not call UnhookAPI() at all.
Post Reply