Poor performance of madExcept under x64?

[davidheffernan]

I'm using madExcept 4.0.2. My app, under certain circumstances, runs much slower under x64 than x86. I traced the onset of this regression to the point at which I added x64 madExcept. After a little investigation it seems that the very act of throwing and swallowing an exception is enough to highlight the huge performance difference between x86 and x64 madExcept. I attach a very simple program to highlight the issue.

Under x86 this runs, in Release build, in 0.087s on my machine.
Under x64 this runs, in Release build, in 50s on my machine.

So under x64, the cost of throwing and swallowing an exception is 500 times greater than under x86.

I'd very much appreciate it if you could take a look into this.

[madshi]

There is a fundamental difference in exception handling between x86 and x64. In x86 madExcept is not actually doing any stack tracing until an exception really proves to be unhandled. In x64 madExcept uses a different approach and does a full stack trace already at the moment when the exception is raised. Obviously that means that there's a performance loss, every time an exception is raised. Does that happen so often in your program? In my own programs exceptions are really only raised when something goes wrong, which is very rare. I understand that it is a valid programming concept to use exceptions to control the flow of a program, though.

I guess I could try to modify the x64 approach to be more similar to how madExcept behaves in x86. But that would come at the cost of stack trace quality. Right now x64 stack traces should be pretty much "perfect". The x86 method of stack tracing is much more "hit and miss". So, would you be willing to trade in stack trace quality to get better performance for *handled* exceptions?

[davidheffernan]

It's an interesting one. I think a lot of the time then that design choice doesn't matter for my code either. But sometimes it seems to. But in my program it's currently very tricky to really understand what's causing some of the slowdowns that I see. I know that madExcept x64 is implicated but I'm still not 100% sure that the performance cost of raising an exception is actually the source of the problem.

Stepping back a little, for me at least there would be an excellent way to avoid paying the stack trace penalty. A good deal of the exceptions that are raised in my app, even if they are unhandled, do not ever need a stack trace. For example, I have no desire to collect a stack trace for the silent exception EAbort. Similarly, I have an exception class hierarchy derived from an exception that I name EExpected. These exceptions never generate bug reports. So I could easily decide at the point of raising whether or not I wanted to pay the cost of a stack trace.

So, the ability to filter out stack tracing at the point where the exception is raised would be great. If I had that capability then the stack trace performance would simply not matter to me. Perhaps such functionality already exists, I've never really explored the full gory details of what's available in madExcept.

[davidheffernan]

Another thought. In the real application there are a **lot** of threads. Typically one per processor. I'm currently working on a 16 core machine but I have clients with 64 core machines.

Under x64, does the stack trace grabbing process interrogate the stacks for all threads, every time an exception is raised in any thread? If so then I could well imagine that being a significant problem.

[davidheffernan]

Just adding some threads to my test project confirms my suspicions. The presence of extra threads that are working does result in performance degradation. I think there is a real problem here.

[madshi]

madExcept performance should not vary depending on how many threads you have. When an exception is raised, a callstack is only calculated for the thread which crashed. The callstacks for the other threads are only calculated once the exception is really fully handled/shown by madExcept.

You could this:

(1) Copy madExcept.pas and mad.inc into your project folder.
(2) Edit mad.inc so that "{$D+}{$L+}".
(3) madExcept.SetLastException contains the code "stackInfo.preparedStack := PrepareStackTrace(...)". Replace this with "stackInfo.preparedStack := ''".

How does that change performance?

[davidheffernan]

Changing to stackInfo.preparedStack := '' does indeed restore performance to the levels that I would expect to see. I'm not sure why you wanted me to set {$D+}{$L+}. I didn't do that because I wanted to run a release build of my app which does not have debug info. Anyway, simply skipping that call to PrepareStackTrace has a big impact.

Since I'm already including the madExcept source into my code, I could quite easily add my own filter to skip that call to PrepareStackTrace for exceptions that I know in advance will not require stack traces. However, I'd much prefer it if the official madExcept code had such a filtering mechanism in place because for obvious reasons it's much preferable to use 3rd party code unmodified. So, if you do plan to add such a mechanism, please let me know, and I'll wait until you do it.

Thanks once again for your speedy and helpful support.

[madshi]

The "{$D+}{$L+}" allows you to set breakpoints in madExcept.pas. I thought that might be helpful in case you wanted to debug some more what madExcept.pas does.

Ok, at least it's good to know that the problem is caused only by the stack tracing and by nothing else. Let me think about how I could maybe solve that...

[davidheffernan]

Going back to your first comment, and a question that you asked:

So, would you be willing to trade in stack trace quality to get better performance for *handled* exceptions?

Yes I would be willing to trade that, not least because the quality of stack traces in x86 has never been an issue for me (the x86 stack traces are good). The more I look at this, the more I believe that calling StackWalk64 every time any exception is raised is simply untenable, at least in my app.

[davidheffernan]

What about using CaptureStackBackTrace?

If it's good enough for John Carmack......

[davidheffernan]

I'm now testing out a variant of your code that uses CaptureStackBackTrace instead of dbghelp/StackWalk64 in PrepareStackTrace. This gives good performance now. I think this is the simplest thing to do to your code to give a big boost to performance.

The next obvious area to tackle is collecting the stack traces for all the other threads, at the time at which the bug report is generated. I suspect that this is slow because of the repeated calls to SymInitialize. Surely that can be done just once in the lifetime of the process?

[madshi]

I'm not convinced that CaptureStackBackTrace provides the same quality stack traces that StackWalk64 does. After all, there must be a reason that StackWalk64 is so much slower. Anyway, I have this on my list of things to do look at.

The other threads? Why is it such a big problem if their stack tracing is slow? It's also relatively slow with madExcept 3 in x86. Those stack traces are only calculated if the exception is really shown on screen. So it shouldn't matter much how fast the code runs.

[davidheffernan]

I don't know why StackWalk64 is so slow. It does so much more than CaptureStackBackTrace and perhaps that explains why it is slower. My admittedly trivial tests show identical results. CaptureStackBackTrace is using stack frames

When PrepareStackTrace uses dbghelp/StackWalk64 there is also the dbghelp serialization mutex to factor in. If the process has a lot of threads that are raising and handling threads, then that mutex kills multi-threaded scaling. So for me there are huge benefits to be had with CaptureStackBackTrace.

As for stack tracing for the other threads that becomes an issue if the process has a lot of threads. For example, on a 64 core machine my app would not uncommonly have 140 threads, many of which would be blocking. Getting traces for all of those threads can take a long time, around 15 seconds say. Adding SYMOPT_DEFERRED_LOADS to the SymSetOptions call drops that down to 3 or 4 seconds.

Anyway, for now I've got a variant of madExcept that performs acceptably for me on x64, but I would reiterate that I surely won't be the only of your customers that hits problems of this nature. Thanks as always for your help.

[madshi]

CaptureStackBackTrace is using stack frames

Yes, exactly and using stack frames is an option, it's not mandatory. So every function which does not use stack frames will not show in the callstack. I believe StackWalk64 is more intelligent and also works for functions which don't have a stack frame. I'm only guessing here, though.

[davidheffernan]

using stack frames is an option, it's not mandatory

Isn't that where the x86 and x64 ABIs differ? My recollection is that the x64 ABI always uses stack frames. Doesn't the x64 exception model depend on that?