Threads getting "stuck"

delphi package - automated exception handling

Threads getting "stuck"

Postby anj_hm » Wed Nov 25, 2020 10:51 am

I've inherited a rather large web broker based web service. It uses the TIdHTTPWebBrokerBridge with a TWebModule handling both SOAP and RESTful like requests.

My first task has been to make it more reliable. My company was/is aware that it was having issues - to the degree that it's forcibly restarted every hour.

I've been knocking various issues off. One big issue I identified was that there was lots and lots of database locking and in the end I came to the conclusion that transactions were being left open. What's that got to do with madExcept?

After exhausting all of the obvious avenues and studying our internal logs I realised that there were threads (handling requests) that were never getting to end. The forced restart of the web service (above) released the locks. This added more evidence that threads weren't getting to end and so were holding locks.

Running the web service locally and purposefully causing a database issue (in this case a timeout) I've been able to recreate the scenario where the thread "hangs".

The web service is built using Delphi 10.1 Berlin (Update 1), FastDCC & madExcept 5.0, FireDAC is used for the database functionality.

I've attached a picture of where the debugging got me. To handle timeouts (as an example) FireDAC runs within a thread, so we've got a no. of levels of nested threads from a request. FireDAC raises an exception that bubbles up through the layers, this is then caught by madExcept as a "hidden" exception, it's handing of this then "hangs" and the standard application logic to handle the exception doesn't occur (hence no rollback/commit etc.)

As per the diagram. An AV is raised when trying to get the stack dump. This forces it into the local exception handler and it calls InternalError() at which point it falls into _DoneExcept() and "hangs" on TObject.Free().

Any ideas? I know it's not much to go on. I've tried to produce a "cut down" repro and although I have, the nature that this scenario doesn't occur all of the time means it's not a great repro :( Obviously in our LIVE environment the no. of requests coming in means that this issue occurs quite regulary.

So, I disabled madExcept. So far we haven't seen any sign of threads not getting to end or the database related issues.
Attachments
madExceptHanging.jpg
Debugging
madExceptHanging.jpg (214.06 KiB) Viewed 312 times
anj_hm
 
Posts: 2
Joined: Thu Nov 19, 2020 9:05 am

Re: Threads getting "stuck"

Postby madshi » Thu Nov 26, 2020 3:29 pm

Ugh, web stuff. I always get gray hair when trying to compile and test web stuff.

Anyway. Have you tried stepping into the TObject.Free call to understand why it hangs there? Must have a reason? And why does F7 go to TObject.Free? I don't think madExcept's exception handling uses TObject while doing exception handling. Maybe that TObject.Free call is originating from your original project?
madshi
Site Admin
 
Posts: 10319
Joined: Sun Mar 21, 2004 5:25 pm

Re: Threads getting "stuck"

Postby anj_hm » Thu Nov 26, 2020 5:04 pm

I'll see what I can do about tracing further in.

F7 doesn't jump directly to the TObject.Free. That's the line within _DoneExcept() that hangs.

I don't know enough about the low-level side of things. I assume _DoneExcept() is running as a side effect of implicit code inserted by the compiler based on an exception going out of scope, so being freed?
anj_hm
 
Posts: 2
Joined: Thu Nov 19, 2020 9:05 am

Re: Threads getting "stuck"

Postby madshi » Thu Nov 26, 2020 7:58 pm

Yes, I think so. Maybe if GetStackDump crashes, you can put a breakpoint on _DoneExcept's call to TObject.Free? IIRC, madExcept actually hooks into TObject.Free in some situations...
madshi
Site Admin
 
Posts: 10319
Joined: Sun Mar 21, 2004 5:25 pm


Return to madExcept

Who is online

Users browsing this forum: No registered users and 22 guests