I can't say why this happens. Maybe the program which calls SetWindowsHookEx behaves somehow strangely and thus doesn't like the MessageBox call at all. Is it your own program which calls SetWindowsHookEx?
I suggest sending an IPC message to your application and let that show a message box and return the result back to the hook dll. You can see how that works in the HookProcessTermination demo.
procedure HandleProcessTerminationRequest(name : pchar;
messageBuf : pointer;
messageLen : dword;
answerBuf : pointer;
answerLen : dword); stdcall;
// this function is called by the ipc message whenever our dll contacts us
var s1, s2, s3 : string;
Form1: TForm1;
begin
Form1 := TForm1.Create(Application); // Process stop response on this line
...
I don't have the overview about what you're doing exactly. If you can create a small demo (please as small as possible) which reproduces the problems, I might have a look at it to see what's wrong.