I am trying to diagnose an elusive bug, that is triggered under a Windows 10 x64 system (but not on my Win10 x64 VM).
I tested madexcept works fine, by raising an exception at application start, to which madexcept reacts with an error window, as expected.
But the exception I am trying to catch, exception code 0x409, raised in ntdll.dll, is only caught by Windows Error Reporting (WER), not madexcept.
Any idea how I could track down the error source?
You could try run your application like this on your own PC/VM. Maybe it will raise a couple new exceptions? If so, you should fix them. If not, you could try running it this way on the target system which shows that elusive bug.
Whether or not the "instantly crash" feature will help you find the problem I can't say for sure. But often, such really bad crashes are caused by buffer overruns, or writing to an already freed buffer or things like that. All of those often go unnoticed and only produce crashes some time later. The "instantly crash" feature immediately produces a crash whenever such a bad thing happens, which makes it much easier to fix the true cause of buffer overruns etc.
One potential problem is that this feature consumes quite a lot of extra RAM, so there's a certain danger you might run into "out of memory" problems. If that happens, you might be out of luck. Or maybe you could then try to compile your EXE as 64bit, which usually fixes the "out of memory" problems. If you do compile your EXE as 64bit, I'd suggest to use the latest beta build, just to be safe:
Unfortunately, it still happens only on the test system, which isn't fully updated. I wonder if it might be this, but on the other hand, I'd like finding out what triggers the bug.
Interestingly, even adding a logging function to every function call from my own code, did not give any conclusive results, it just suddenly stops running. Very odd bug.
And I finally found the error. It was a cast of a window procedure pointer to a DWORD instead of LONG_PTR in some code that I missed to update fully to 64 bit.
I ended up using the minidump created by WER and loading it into Visual Studio to get a stack trace. (see https://docs.microsoft.com/en-us/visual ... dump-files ). For the functions in the Delphi compiled executable, the stack trace just showed memory addresses, of course, no names. But using Delphi's CPU view and goto address feature, I could find out the relevant assembler code, and the source code lines.
This gave me a hint there was a bug when calling DefFrameProc, and I worked my way back from there.
The reason the bug went unseen before, is that on this user's system memory was allocated on high virtual addresses, even for the very first allocations, which is unusual (also on a 64-bit system). MakeObjectInstance from Classes.pas, which calls VirtualAlloc, would always return addresses well below 4GiB, except on this user's system. I am not sure what triggers this behavior, since all the relevant settings in Windows Defender (ASLR etc.) were the same on my VM (which did not show the issue), and the user's system (which did trigger the bug). Yet, disabling ASLR fully "solved" it, and stopped VirtualAlloc from allocating on high memory addresses.
I have found no setting that forces this behavior, that is probably hidden somewhere in the registry. Adjusting the AllocationPreference registry key as shown in https://jazz.net/forum/questions/230876 ... preference might be an option, but this key did not exist on the user's system, so was not the cause.
While I'd like to recreate the settings that cause this high memory address preferring behavior, I implemented a quick solution: reserving 4GB of memory, at the very start of the process, allows to reproduce this issue on my machine. It's a bit slow, but it works. Maybe there is a better solution.
Edit: there is a similar approach which should be more performant here: https://randomascii.wordpress.com/2012/ ... made-easy/
Edit2: see also https://docs.microsoft.com/en-us/cpp/bu ... hentropyva
Just a summary for others who might have a similar problem.
Thanks for your suggestions.
Edit3: turns out the reason why I could not see this issue in madexcept (or any other exception catcher) is that the system called RaiseFailFastException.
However the documentation also states:
So there might be an opportunity to catch it with a mini debugger appended, and create a stack trace that way. Maybe a future extension for madexcept, but just an idea.If the WER service is disabled or cannot be started or there is no debugger attached to the process, the process will be terminated.
This function raises a second chance exception. If JIT debugging is enabled, a debugger will attach to the process.
Code: Select all
date/time : 2020-06-23, 15:31:58, 257ms computer name : user name : registered owner : operating system : Windows 10 x64 build 18363 system language : English system up time : 5 hours 51 minutes program up time : 21 seconds processors : 8x Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz physical memory : 8968/16338 MB (free/total) free disk space : (C:) 329.38 GB (L:) 453.05 GB display mode : 3072x1728, 32 bit process id : $e34 allocated memory : 871.48 MB largest free block : 47.55 MB command line : "L:\Bin_C10.tst\Impulse.exe" executable : Impulse.exe current module : madExcept32.dll module date/time : 2018-11-29 18:58 version : 18.104.22.168 compiled with : Delphi 10.3 Rio madExcept version : 5.0.0 contact name : contact email : callstack crc : $12fdf180, $a269bf50, $86e77e46 exception number : 1 exception class : Exception exception message : Out of memory. madExcept - HandleExceptionThread ($173c): <priority:2> 053e0f28 +390 madExcept32.dll madDebugMm 237 +91 GetMemDbg 053e57c8 +150 madExcept32.dll madExceptDbg 1935 +40 ReallocMemCallback 00404c49 +00d Impulse.exe System 57 +0 @ReallocMem 00409644 +028 Impulse.exe System 57 +0 @UStrSetLength 004191bb +127 Impulse.exe madStrings DecodeUtf8 00428cb5 +061 Impulse.exe madMapFile ReadPackedString 00429385 +0d1 Impulse.exe madMapFile Public_LoadFromStream 0042998f +2ef Impulse.exe madMapFile ImportStr 00429ae5 +03d Impulse.exe madMapFile GetMapFileFromResources 0042c84d +1d9 Impulse.exe madMapFile TMapFile.Create 0042dcf1 +12d Impulse.exe madMapFile FindMapFile 00445cac +064 Impulse.exe madExcept CheckExceptAddr 004730dc +058 Impulse.exe madExcept ReceiveHandleException 004736be +08a Impulse.exe madExcept HandleExceptionThread 77a66357 +017 KERNEL32.DLL BaseThreadInitThunk