Exception 0x409 in ntdll.dll in Windows 10

delphi package - automated exception handling

Exception 0x409 in ntdll.dll in Windows 10

Postby maelh » Thu Jun 04, 2020 1:43 am

Hello,

I am trying to diagnose an elusive bug, that is triggered under a Windows 10 x64 system (but not on my Win10 x64 VM).

I tested madexcept works fine, by raising an exception at application start, to which madexcept reacts with an error window, as expected.

But the exception I am trying to catch, exception code 0x409, raised in ntdll.dll, is only caught by Windows Error Reporting (WER), not madexcept.

Any idea how I could track down the error source?

Best regards,
Maël
maelh
 
Posts: 3
Joined: Sat May 30, 2020 9:28 pm

Re: Exception 0x409 in ntdll.dll in Windows 10

Postby madshi » Thu Jun 04, 2020 8:58 am

You could try activating "active error search -> instantly crash on buffer overrun" on the madExcept main settings page. This feature requires you to distribute the madExcept32/64.dll with your EXE file (same folder). If you don't distribute those dlls, the "instantly crash" feature will simply not become active.

You could try run your application like this on your own PC/VM. Maybe it will raise a couple new exceptions? If so, you should fix them. If not, you could try running it this way on the target system which shows that elusive bug.

Whether or not the "instantly crash" feature will help you find the problem I can't say for sure. But often, such really bad crashes are caused by buffer overruns, or writing to an already freed buffer or things like that. All of those often go unnoticed and only produce crashes some time later. The "instantly crash" feature immediately produces a crash whenever such a bad thing happens, which makes it much easier to fix the true cause of buffer overruns etc.

One potential problem is that this feature consumes quite a lot of extra RAM, so there's a certain danger you might run into "out of memory" problems. If that happens, you might be out of luck. Or maybe you could then try to compile your EXE as 64bit, which usually fixes the "out of memory" problems. If you do compile your EXE as 64bit, I'd suggest to use the latest beta build, just to be safe:

http://madshi.net/madCollectionBeta.exe
madshi
Site Admin
 
Posts: 10305
Joined: Sun Mar 21, 2004 5:25 pm

Re: Exception 0x409 in ntdll.dll in Windows 10

Postby maelh » Wed Jun 10, 2020 7:24 pm

Thanks a lot for your reply. I wanted to wait until I have done all and write my success story, including tips on how to solve it.

Unfortunately, it still happens only on the test system, which isn't fully updated. I wonder if it might be this, but on the other hand, I'd like finding out what triggers the bug.

Interestingly, even adding a logging function to every function call from my own code, did not give any conclusive results, it just suddenly stops running. Very odd bug.
maelh
 
Posts: 3
Joined: Sat May 30, 2020 9:28 pm

Re: Exception 0x409 in ntdll.dll in Windows 10

Postby madshi » Thu Jun 11, 2020 1:13 pm

Did you try the "instantly crash" feature, with distributing the madExcept32/64.dll with your EXE file? No change at all when doing that?
madshi
Site Admin
 
Posts: 10305
Joined: Sun Mar 21, 2004 5:25 pm

Re: Exception 0x409 in ntdll.dll in Windows 10

Postby maelh » Fri Jun 12, 2020 3:51 am

I did, but it had no effect.

And I finally found the error. It was a cast of a window procedure pointer to a DWORD instead of LONG_PTR in some code that I missed to update fully to 64 bit.

I ended up using the minidump created by WER and loading it into Visual Studio to get a stack trace. (see https://docs.microsoft.com/en-us/visual ... dump-files ). For the functions in the Delphi compiled executable, the stack trace just showed memory addresses, of course, no names. But using Delphi's CPU view and goto address feature, I could find out the relevant assembler code, and the source code lines.

This gave me a hint there was a bug when calling DefFrameProc, and I worked my way back from there.

The reason the bug went unseen before, is that on this user's system memory was allocated on high virtual addresses, even for the very first allocations, which is unusual (also on a 64-bit system). MakeObjectInstance from Classes.pas, which calls VirtualAlloc, would always return addresses well below 4GiB, except on this user's system. I am not sure what triggers this behavior, since all the relevant settings in Windows Defender (ASLR etc.) were the same on my VM (which did not show the issue), and the user's system (which did trigger the bug). Yet, disabling ASLR fully "solved" it, and stopped VirtualAlloc from allocating on high memory addresses.

I have found no setting that forces this behavior, that is probably hidden somewhere in the registry. Adjusting the AllocationPreference registry key as shown in https://jazz.net/forum/questions/230876 ... preference might be an option, but this key did not exist on the user's system, so was not the cause.

While I'd like to recreate the settings that cause this high memory address preferring behavior, I implemented a quick solution: reserving 4GB of memory, at the very start of the process, allows to reproduce this issue on my machine. It's a bit slow, but it works. Maybe there is a better solution.

Edit: there is a similar approach which should be more performant here: https://randomascii.wordpress.com/2012/ ... made-easy/
Edit2: see also https://docs.microsoft.com/en-us/cpp/bu ... hentropyva

Just a summary for others who might have a similar problem.

Thanks for your suggestions.

Edit3: turns out the reason why I could not see this issue in madexcept (or any other exception catcher) is that the system called RaiseFailFastException.

However the documentation also states:
If the WER service is disabled or cannot be started or there is no debugger attached to the process, the process will be terminated.

This function raises a second chance exception. If JIT debugging is enabled, a debugger will attach to the process.

So there might be an opportunity to catch it with a mini debugger appended, and create a stack trace that way. Maybe a future extension for madexcept, but just an idea.
maelh
 
Posts: 3
Joined: Sat May 30, 2020 9:28 pm

Re: Exception 0x409 in ntdll.dll in Windows 10

Postby madshi » Fri Jun 12, 2020 3:24 pm

That's quite interesting, thanks for letting me know!
madshi
Site Admin
 
Posts: 10305
Joined: Sun Mar 21, 2004 5:25 pm

Re: Exception 0x409 in ntdll.dll in Windows 10

Postby ZSL » Tue Jun 23, 2020 5:35 am

I too have a problem with ntdll.dll and switched on "instant crash" but I keep getting out of memory errors:

Code: Select all
date/time          : 2020-06-23, 15:31:58, 257ms
computer name      :
user name          :
registered owner   :
operating system   : Windows 10 x64 build 18363
system language    : English
system up time     : 5 hours 51 minutes
program up time    : 21 seconds
processors         : 8x Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
physical memory    : 8968/16338 MB (free/total)
free disk space    : (C:) 329.38 GB (L:) 453.05 GB
display mode       : 3072x1728, 32 bit
process id         : $e34
allocated memory   : 871.48 MB
largest free block : 47.55 MB
command line       : "L:\Bin_C10.tst\Impulse.exe"
executable         : Impulse.exe
current module     : madExcept32.dll
module date/time   : 2018-11-29 18:58
version            : 4.0.6.0
compiled with      : Delphi 10.3 Rio
madExcept version  : 5.0.0
contact name       :
contact email      :
callstack crc      : $12fdf180, $a269bf50, $86e77e46
exception number   : 1
exception class    : Exception
exception message  : Out of memory.

madExcept - HandleExceptionThread ($173c): <priority:2>
053e0f28 +390 madExcept32.dll madDebugMm    237 +91 GetMemDbg
053e57c8 +150 madExcept32.dll madExceptDbg 1935 +40 ReallocMemCallback
00404c49 +00d Impulse.exe     System         57  +0 @ReallocMem
00409644 +028 Impulse.exe     System         57  +0 @UStrSetLength
004191bb +127 Impulse.exe     madStrings            DecodeUtf8
00428cb5 +061 Impulse.exe     madMapFile            ReadPackedString
00429385 +0d1 Impulse.exe     madMapFile            Public_LoadFromStream
0042998f +2ef Impulse.exe     madMapFile            ImportStr
00429ae5 +03d Impulse.exe     madMapFile            GetMapFileFromResources
0042c84d +1d9 Impulse.exe     madMapFile            TMapFile.Create
0042dcf1 +12d Impulse.exe     madMapFile            FindMapFile
00445cac +064 Impulse.exe     madExcept             CheckExceptAddr
004730dc +058 Impulse.exe     madExcept             ReceiveHandleException
004736be +08a Impulse.exe     madExcept             HandleExceptionThread
77a66357 +017 KERNEL32.DLL                          BaseThreadInitThunk
ZSL
 
Posts: 54
Joined: Thu Mar 31, 2016 11:23 pm

Re: Exception 0x409 in ntdll.dll in Windows 10

Postby madshi » Wed Jun 24, 2020 10:45 am

The "instantly crash" feature consumes a lot of extra RAM, so bigger projects may run into "out of memory" problems, unfortunately. There's not so much I can do about that. If you haven't yet, you could try enabling the "large address aware" option to use 4GB instead of 2GB. If that doesn't help, either, the only other option would be to compile in 64bit. Not sure if that's easily possible for you?
madshi
Site Admin
 
Posts: 10305
Joined: Sun Mar 21, 2004 5:25 pm


Return to madExcept

Who is online

Users browsing this forum: No registered users and 16 guests