We have a problem with a 64-bit program crashing when an exception is raised inside a hooked procedure. When compiled as 32-bit everything works as expected, so we're not sure if we are doing something wrong or its an issue in madCodeHook.
Below is a trivial example that reproduces the issue only when compiled to 64-bit:
Is there any specific reason why you'd need(?) to hook a procedure that intentionally raises an exception?
***Update***
I've just tested and reproduced on x64 only when calling the "hooked" function, as you've said x86 seems to work fine. When commenting out the "raise exception" line it works fine here, so it's definitely related to the way the exception is being generated from within the VCL on x64 when the code hook is set. Hmmm strange I'll wait for Madshi to see this thread and see what he thinks, he knows much more than I when it comes to exception handling.
Exceptions are difficult to handle in x64 for the simple reason that exception handling works completely different in x64. Basically, in x86, exception handling works simply by pushing exception frames to the stack. In x64, it's required that any code section has additional exception related information stored in the EXE/DLL file the code belongs to, for exception handling to work correctly. The problem for madCodeHook is that API hooking requires the use of dynamically allocated code stubs, and for those exception handling won't work properly.
There are ways to solve it somehow, but it's complicated and I haven't managed to look into that yet.
However, you can probably work around it by adding a try..except block around your hook callback function.
We continued to discuss this topic via email. In case other users are listening here, here's a summary of what we found:
The problem occurs due to madCodeHook's "safe unhooking" feature. If you use the NO_SAFE_UNHOOKING flag when calling HookAPI() then the problem no longer appears. Most other API hooking libraries don't have madCodeHook's "safe unhooking" feature, so they don't have this problem with x64 exception handling.
Here are the technical details:
When using NO_SAFE_UNHOOKING, madCodeHook just writes a JMP instruction to the hooked API. Nothing else. And that jump instruction jumps to your hook callback function. It's a very simple solution. And most other API hooking libraries do the same thing.
However, when not using the NO_SAFE_UNHOOKING flag, madCodeHook has to add a lot of extra code, which basically "counts" how often a thread enters your hook callback function and how often a thread leaves your hook callback function. Only this way madCodeHook knows if your hook callback is still "in use" by a thread or not. Unfortunately this counting is very difficult to do and requires madCodeHook to install 2 separate dynamically created code stubs which are called when your hook callback function is entered and when it's left. And these 2 code stubs are what's causing problems with the exception handling in x64. It is probably possible to fix it somehow, but difficult.
If you don't ever unload your hook dll, then you can use the NO_SAFE_UNHOOKING flag safely. If you want to be able to unload your hook dll, then I don't recommend using NO_SAFE_UNHOOKING because safe unhooking is an important feature to achieve good stability (no crashes) during unhooking.