Debugging fails with madExcept and msbuild

delphi package - automated exception handling
Post Reply
markk
Posts: 15
Joined: Fri Mar 04, 2016 12:07 am

Debugging fails with madExcept and msbuild

Post by markk »

I've just started using msbuild (use external msbuild in project settings) to reduce the number of 'out of memory' errors we get using the Delphi XE6 IDE with a large project.
When I first started using this setting I'm sure that debugging (breakpointing) worked but every build I do at the moment it doesn't.

The problem appears to be that the madExcept post build process alters the modification date of the .exe which puts it out of sync with the .rsm, compiles at the moment it show a 1 second difference in the modification dates.

Embarcadero state that this can happen at the end of the following page http://qc.embarcadero.com/wc/qcmain.aspx?d=125912

Problem occurs in 4.0.12 and 4.0.14
madshi
Site Admin
Posts: 10754
Joined: Sun Mar 21, 2004 5:25 pm

Re: Debugging fails with madExcept and msbuild

Post by madshi »

Does this build fix the problem?

http://madshi.net/madCollectionBeta.exe (installer 2.7.13.2)
markk
Posts: 15
Joined: Fri Mar 04, 2016 12:07 am

Re: Debugging fails with madExcept and msbuild

Post by markk »

Debugging still doesn't work

The only difference I can see with this build and 4.0.14 are some XE8 .inc fixes and dcu, bpl, bpi changes. The patch exe appears to be exactly the same.
madshi
Site Admin
Posts: 10754
Joined: Sun Mar 21, 2004 5:25 pm

Re: Debugging fails with madExcept and msbuild

Post by madshi »

That's unlikely. The madExceptPatch.exe is automatically recompiled, every time I build a new installer. I've just checked, if I compile the latest beta build, madExceptPatch.exe in the installation folder has a date of April, 19th. Can you please double check?
markk
Posts: 15
Joined: Fri Mar 04, 2016 12:07 am

Re: Debugging fails with madExcept and msbuild

Post by markk »

Looking into it more I've discovered I supplied some wrong information.
Last time after I installed the beta I then copied all the files to a subversion folder (over the top of an existing copy) without looking at them prior to the copy. Following this I uninstalled the beta as for us the IDE integration appears to make the IDE a lot more unstable.
This time I did the install and looked more carefully at the files now in C:\Program Files (x86)\madCollection
There is no madExcept folder at all, so no patch exe.

BTW, thanks for your quick responses.
markk
Posts: 15
Joined: Fri Mar 04, 2016 12:07 am

Re: Debugging fails with madExcept and msbuild

Post by markk »

Okay, I'm daft, I didn't realise that when the installer went through it hadn't selected madExcept, thought the colouring was related to freeware, non-freeware.
Now just got to go back and review my notes to get it to be commercial version
markk
Posts: 15
Joined: Fri Mar 04, 2016 12:07 am

Re: Debugging fails with madExcept and msbuild

Post by markk »

With the addition of /restoreFileTime to the post build event patcher command line (discovered from a quick look at the source changes when it didn't work at first) debugging is working again, unfortunately, when I did a test exception the upload of it failed, when I did a second one (in the same application session) I got a access violation or similar in TryRead called from line 1170 of madStackTrace.InitEbpFrames

Command line I'm now using is
"..\..\Library\Third party\madCollection\madExcept\Tools\madExceptPatch.exe" /restoreFileTime "$(OUTPUTPATH)" "$(PROJECTDIR)\$(PROJECTNAME).mes" "$(OUTPUTDIR)\$(OUTPUTNAME).map"
markk
Posts: 15
Joined: Fri Mar 04, 2016 12:07 am

Re: Debugging fails with madExcept and msbuild

Post by markk »

In 4.0.14 I had to alter GetStackDump to avoid an access violation and took a guess at the following.

Code: Select all

  function GetStackDump(const context: TContext) : UnicodeString;
  var esp      : TPAByte;
      i1, i2   : integer;
      mbi      : TMemoryBasicInformation;
      stackTop : NativeUInt;
  begin
    try
      result := '';
      NativeUInt(esp) := {$ifdef win64} context.Rsp {$else} context.Esp {$endif};
      if VirtualQuery(esp, mbi, sizeOf(mbi)) = sizeOf(mbi) then
           stackTop := NativeUInt(mbi.BaseAddress) + mbi.RegionSize
      else stackTop := $ffffffff;
      if assigned(esp) then
      begin
        for i1 := 0 to 19 do begin
          result := result + #$D#$A + RetDeleteW(IntToHexExW(NativeUInt(esp), 8), 1, 1) + '  ';
          for i2 := 0 to 15 do
            if NativeUInt(@esp[i2]) >= stackTop then begin
              if i2 = 8 then
                result := result + '  ';
              result := result + '   ';
            end else begin
              if i2 = 8 then
                result := result + '- ';
              result := result + RetDeleteW(IntToHexExW(NativeUInt(esp[i2]), 2), 1, 1) + ' ';
            end;
          result := result + ' ';
          for i2 := 0 to 15 do begin
            if NativeUInt(@esp[i2]) >= stackTop then
              break;
            if esp[i2] in [$20..$7e] then
                 result := result + WideChar(esp[i2])
            else result := result + '.';
          end;
          inc(NativeUInt(esp), 16);
          if NativeUInt(esp) >= stackTop then
            break;
        end;
        Delete(result, 1, 2);
      end
      else
      begin
        result := InternalError('GetStackDump', GetShowRelativeAddrs, GetShowRelativeLines);
        assert(assigned(esp));
      end;
    except
      result := InternalError('GetStackDump', GetShowRelativeAddrs, GetShowRelativeLines);
    end;
madshi
Site Admin
Posts: 10754
Joined: Sun Mar 21, 2004 5:25 pm

Re: Debugging fails with madExcept and msbuild

Post by madshi »

The try..except should really catch the crash. Of course that won't help if you run inside of the IDE. The debugger will then break on the exception, so you have to manually continue execution to get the same result as outside of the IDE.

I'm wondering *why* ESP is 0 in your case, though? That shouldn't normally be the case. I'm wondering whether it's worth it adding extra code for this special case, just to avoid the debugger stopping. Your code changes should make no difference at all outside of the IDE.
markk
Posts: 15
Joined: Fri Mar 04, 2016 12:07 am

Re: Debugging fails with madExcept and msbuild

Post by markk »

I added the extra bit of code initially to see about finding the reason for being nul and left it in as I hate getting avoidable exceptions when I'm debugging, but by automatic compulsion I'd added an assert as well which pretty much made it pointless for avoiding the debugger pause! It's also only been hit occasionally.

Thanks for formatting that bit of code in my last post.

The access violation hasn't happened again in madStackTrace.InitEbpFrames, just a one off so far.

I'm looking into the upload issue to see what I can do to get it working, it appears to be that the php custom script is not using a base64 decoded value of the MailFrom or MailBody fields which are now begin base64 encoded.
markk
Posts: 15
Joined: Fri Mar 04, 2016 12:07 am

Re: Debugging fails with madExcept and msbuild

Post by markk »

Looking into the uploading a little further it seems that PHP does not support Content-Transfer-Encoding in http messages. I came across a few (heated) discussions regarding whether this is correct or a bug and that implementation of the standard around that area is a bit of a mess.

from https://tools.ietf.org/html/rfc7578#section-4.4
4.7. Content-Transfer-Encoding Deprecated

Previously, it was recommended that senders use a Content-Transfer-
Encoding encoding (such as "quoted-printable") for each non-ASCII
part of a multipart/form-data body because that would allow use in
transports that only support a "7bit" encoding. This use is
deprecated for use in contexts that support binary data such as HTTP.
Senders SHOULD NOT generate any parts with a Content-Transfer-
Encoding header field.

Currently, no deployed implementations that send such bodies have
been discovered.

4.8. Other "Content-" Header Fields

The multipart/form-data media type does not support any MIME header
fields in parts other than Content-Type, Content-Disposition, and (in
limited circumstances) Content-Transfer-Encoding. Other header
fields MUST NOT be included and MUST be ignored.
I've also seen posts saying that auto decode works in PHP for POST requests but not PUT etc (doesn't on our site with 5.4 though)
madshi
Site Admin
Posts: 10754
Joined: Sun Mar 21, 2004 5:25 pm

Re: Debugging fails with madExcept and msbuild

Post by madshi »

Strange, my server doesn't have a problem with it.

So if I'm not allowed to use that, how should I send unicode strings? Currently I'm sending them as base64 encoded utf8. If I can't specify the Content-Transfer-Encoding, then I can't use base64. So how am I supposed to send that?

Anyway, the POST text is composed in "madExcept.TWinHttp.InternalPost". What happens if you simply remove the Content-Transfer-Encoding field there? Does the POST work then?
markk
Posts: 15
Joined: Fri Mar 04, 2016 12:07 am

Re: Debugging fails with madExcept and msbuild

Post by markk »

Removing the base64 encoding is what I did yesterday and it all works fine for us. There is currently only a small possibility that we might encounter non ascii data during our text/html exception notification.
phpmailer looks like it will base64 encode the message for you if asked to (I don't know if it auto does it for text/html). I would guess that the easiest safest option for the message body would be to tell phpmailer to always use utf8 and base64 encoding since ascii is a subset of utf8.

Code: Select all

    for i1 := 0 to fields.ItemCount - 1 do begin
      if ForceUtf8 or IsTrueUnicodeString(fields[fields.Items[i1]]) or (encodeEmail and (PosStr('<', fields[fields.Items[i1]]) > 0)) then begin
        charset := 'utf-8';
        // Kinetic Engineering M.Kendall 21-Apr-2016 do not send Content-Transfer-Encoding as
        // its not supposed to be used in HTTP content and PHP does not automatically decode
        // it causing our server custom bug report mailer script to fail
        as1 := EncodeUtf8(fields[fields.Items[i1]]);
      end else begin
        charset := 'iso-8859-1';
        as1 := AnsiString(fields[fields.Items[i1]]);
      end;
      data := data + #$D#$A +
              '--' + AnsiString(CBoundary) + #$D#$A +
              'Content-Disposition: form-data; name="' + UnicodeToAnsi(fields.Items[i1]) + '"' + #$D#$A +
              'Content-Type: text/plain;charset=' + charset + #$D#$A +
              #$D#$A +
              as1;
    end;
madshi
Site Admin
Posts: 10754
Joined: Sun Mar 21, 2004 5:25 pm

Re: Debugging fails with madExcept and msbuild

Post by madshi »

Ok, the next build will for HTTP Posts by default not use "Content-Transfer-Encoding". There'll be an undocumented boolean variable (named "DontBase64HttpUpload") exported that allows switching back to the old solution.
Post Reply