Trouble hooking DirectX 7 endscene

Xwire · Post by **Xwire** » Tue Dec 14, 2004 7:00 pm

Hello I would like to first say that you have an EXCELLENT product which i will be purchasing shortly. I am having trouble hooking directX7(d3d) endscene event, i can hook other directx 7 events such as beginscene or clear but i am unable to hook directx 7 endscene. I have successfully hooked endscene for directX 6,8,9 but not 7. The hook returns 0(no error) but my endscene callback is not being called, i know that my test applications are calling endscene and i can hook beginscene successfully for the same applications.

const GUID IID_IDirectDraw7 ={0x15e65ec0,0x3b9c,0x11d2, { 0xb9,0x2f,0x00,0x60,0x97,0x97,0xea,0x5b }};
const GUID IID_IDirect3D7 = { 0xf5049e77,0x4861,0x11d2, { 0xa4,0x7,0x0,0xa0,0xc9,0x6,0x29,0xa8 } };

HRESULT (WINAPI *DirectDrawCreateExNext) (PVOID guid, IUnknown** directDraw, PVOID iid, PVOID unkOuter);
HRESULT (WINAPI *CreateDevice7Next) (PVOID self, PVOID classId, PVOID surface, PVOID* device);
HRESULT (WINAPI *SetCooperativeLevelNext)(PVOID self, HWND hwnd, DWORD flags);

HRESULT (WINAPI *EndScene7Next) (PVOID self);
HRESULT WINAPI EndScene7Callback(PVOID self)
{
MessageBox(0,"END SCENE 6/7","",0);
if (ShowUI == true) OSDDraw();
return EndScene7Next(self);
}

HRESULT WINAPI SetCooperativeLevelCallback(PVOID self, HWND hwnd, DWORD flags)
{

HRESULT hr = SetCooperativeLevelNext(self, hwnd, flags);
dXwnd = hwnd;
return hr;
}

HRESULT WINAPI CreateDevice7Callback(PVOID self, PVOID classId, PVOID surface, PVOID* device)
{

MessageBox(0,"CreateDevice7","",0);
HRESULT result = CreateDevice7Next(self, classId, surface, device);

if ( !result && device && *device && !EndScene7Next )
{
MessageBox(0,"HOOK ENDSCENE","",0);
HookCode(GetInterfaceMethod(*device, 6), EndScene7Callback, (PVOID*) &EndScene7Next);
}

return result;
}

HRESULT WINAPI DirectDrawCreateExCallback(PVOID guid, IUnknown** directDraw, PVOID iid, PVOID unkOuter)
{
MessageBox(0,"Create EX","",0);
HRESULT result = DirectDrawCreateExNext(guid, directDraw, iid, unkOuter);
IUnknown* d3d7;
IUnknown* dx7;

if ( !result ) {

if (!SetCooperativeLevelNext || !CreateDevice7Next)
{
if (!(*directDraw)->QueryInterface(IID_IDirectDraw7, (PVOID*) &dx7))
{
if (!SetCooperativeLevelNext) HookCode(GetInterfaceMethod(dx7, 20), SetCooperativeLevelCallback, (PVOID*) &SetCooperativeLevelNext);

if (!(dx7)->QueryInterface(IID_IDirect3D7, (PVOID*) &d3d7))
{
if (!CreateDevice7Next) HookCode(GetInterfaceMethod(d3d7, 4), CreateDevice7Callback, (PVOID*) &CreateDevice7Next);
d3d7->Release();
}

dx7->Release();
}
}
return result;
}

BOOL HookDirect3D7()
{
HookAPI("ddraw.dll", "DirectDrawCreateEx", DirectDrawCreateExCallback, (PVOID*) &DirectDrawCreateExNext);
return true;
}

Post by **madshi** » Wed Dec 15, 2004 9:07 am

The hook returns 0(no error)

No, the return value is a boolean. When HookCode returns 0, this means failure!

Xwire · Post by **Xwire** » Wed Dec 15, 2004 12:20 pm

Hello, right you are it is the hook itself that is failing -- returning 0
I cannot understand why though.

If i simply change the 6 (Interface Method EndScene) to a 5 (Interface Method BeginScene) everything is fine and the hook installs and works. EndScene and BeginScene share the same function signature to it is easy to test.

Changing this (EndScene):
HookCode(GetInterfaceMethod(*device, 6), EndScene7Callback, (PVOID*) &EndScene7Next);

To (BeginScene):
HookCode(GetInterfaceMethod(*device, 5), EndScene7Callback, (PVOID*) &EndScene7Next);

Works fine.

GetLastError(); on the failure returns 0.

The interface is IID_IDirect3DDevice7, using d3d.h (d3dim.dll and ddraw.dll).

GetInterfaceMethod(*device, 6) returns 7394BFFA (1939128314)

I have no idea why the hook would fail I have had great success up until this point, perhaps it is the structure of the file but i would not know how to check this.

Thank-you in advance for any responses.

Post by **madshi** » Wed Dec 15, 2004 12:57 pm

I guess that the binary structure of the EndScene function has a format which madCodeHook doesn't like. If madCodeHook fears that installing the hook could result in instability, it refuses to install the hook. Stability is the most important priority of madCodeHook.

Can you please do the following:

(1) Which address does "GetInterfaceMethod(*device, 6)" return?
(2) Check out the module list to see in which dll this address is located (you can use the ProcessMonitor from www.sysinternals.com do to so).
(3) Send that dll to me, and let me know which address was reported in (1).

This way I can analyze the assembler code of the EndScene method. Maybe I can see why madCodeHook refuses to hook it.

Post by **madshi** » Wed Dec 15, 2004 2:13 pm

Got your email.

I don't see any reason why hooking should fail! I've loaded the dll in a test project and asked madCodeHook whether it would like to hook that function, and it said to me "no problem".

Are you sure that your HookCode call really is called with 7394BFFA and that it really returns 0? That doesn't make much sense to me...

Xwire · Post by **Xwire** » Wed Dec 15, 2004 6:17 pm

Yes i'm sure of the address and that HookCode returns 0 and the GetLastError returns 0. I started a project from scratch, rebooted my development computer, shut down anything that might be interfering, to see if i missed something but the result is the same. I used your direct3d hook example and slightly modified it to hook endscene and the result is the same, I can hook BeginScene no problem whatsoever, but EndScene will not hook. I have Hooked EndScene in Direct3D versions 6,8,9 no problem but not 7.
The complete source I am using for this test follows, and simple changing the 6 (EndScene) to a 5 (BeginScene) works perfectly unfortunatally it is EndScene i must hook.

Code: Select all

// ***************************************************************
#include <stdio.h>
#include <windows.h>
#include "madCHook.h"
// ***************************************************************

const GUID IID_IDirect3D7 = { 0xf5049e77,0x4861,0x11d2, { 0xa4,0x7,0x0,0xa0,0xc9,0x6,0x29,0xa8 } };

// ***************************************************************

PVOID GetInterfaceMethod(PVOID intf, DWORD methodIndex)
{
  return *(PVOID*)(*(DWORD*)intf + methodIndex * 4);
}

// ***************************************************************
// Direct3D7

HRESULT (WINAPI *DirectDrawCreateExNext) (PVOID guid, IUnknown** directDraw, PVOID iid, PVOID unkOuter);
HRESULT (WINAPI      *CreateDevice7Next) (PVOID self, PVOID classId, PVOID surface, PVOID* device);
HRESULT (WINAPI    *SetRenderState7Next) (PVOID self, DWORD stateType, DWORD state);
HRESULT (WINAPI    *EndScene7Next) (PVOID self);


HRESULT WINAPI EndScene7Callback(PVOID self)
{
  MessageBox(0,"ENDSCENE 7 CALLBACK","",0);
  return EndScene7Next(self);
}

HRESULT WINAPI CreateDevice7Callback(PVOID self, PVOID classId, PVOID surface, PVOID* device)
{
  HRESULT result = CreateDevice7Next(self, classId, surface, device);
  HRESULT hr = HookCode(GetInterfaceMethod(*device, 6), EndScene7Callback, (PVOID*) &EndScene7Next);  
	
  DWORD dw = GetLastError();
  char buffer[50];

  sprintf(buffer,"%d",hr);
  MessageBox (0,buffer,"HookCode Result",0);

  sprintf(buffer,"%d",GetInterfaceMethod(*device, 6));
  MessageBox(0,buffer,"GetInterfaceMethod Address",0);

  sprintf(buffer,"%d",dw);
  MessageBox(0,buffer,"GetLastErrorResult",0);
  
  return result;
}

HRESULT WINAPI DirectDrawCreateExCallback(PVOID guid, IUnknown** directDraw, PVOID iid, PVOID unkOuter)
{
  HRESULT result = DirectDrawCreateExNext(guid, directDraw, iid, unkOuter);
  IUnknown* d3d7;
  if ( !result && !CreateDevice7Next && !(*directDraw)->QueryInterface(IID_IDirect3D7, (PVOID*) &d3d7) ) {
  HookCode(GetInterfaceMethod(d3d7, 4), CreateDevice7Callback, (PVOID*) &CreateDevice7Next);
  d3d7->Release();
  }
  return result;
}

BOOL HookDirect3D7()
{
  return HookAPI("ddraw.dll", "DirectDrawCreateEx", DirectDrawCreateExCallback, (PVOID*) &DirectDrawCreateExNext);
}

// ***************************************************************

BOOL WINAPI DllMain(HANDLE hModule, DWORD fdwReason, LPVOID lpReserved)
{
  if (fdwReason == DLL_PROCESS_ATTACH) {
    // InitializeMadCHook is needed only if you're using the static madCHook.lib
    InitializeMadCHook();

    HookDirect3D7();
   
  } else if (fdwReason == DLL_PROCESS_DETACH)
    // FinalizeMadCHook is needed only if you're using the static madCHook.lib
    FinalizeMadCHook();

  return true;
}

if you wish i can provide the project files, and a sample direct3d 7 application or two from the SDK, they are very small. Any help you can offer is greatly appreciated, thank you in advance.

Post by **madshi** » Wed Dec 15, 2004 7:17 pm

Yeah, please send me those files. Might take a day or two, until I find time to check it out, though.

Xwire · Post by **Xwire** » Sat Dec 18, 2004 6:46 pm

Hello, just wondering if you got the files I sent, not trying to rush you or anything, I have other work todo in the interm. When you find something please let me know.

Post by **madshi** » Sat Dec 18, 2004 8:58 pm

Got them. Just had no time to check them out yet.

Post by **madshi** » Sun Jan 02, 2005 11:47 am

Sorry for the delay in replying.

You found a bug in madCodeHook! Hook installation failed if the to be patched code was located 6 or 7 bytes before the next page boundary. The reason was that I called VirtualProtect with 6 bytes only, but I needed to call it with 8 bytes instead. Although madCodeHook only really changes 6 bytes, it needs write access to 8 bytes.

Anyway, it was an easy fix and the DX7 EndScene hook works fine now. How fast do you need a fixed build?

Xwire · Post by **Xwire** » Sun Feb 06, 2005 5:43 am

Everything works wonderfully, I am a very, very happy customer, thank-you Madshi.

Post by **madshi** » Mon Feb 07, 2005 7:59 am