Page 1 of 1

Crazy strings leaking in simple operations

Posted: Mon Jul 03, 2017 7:40 pm
by an0rmal
hi everyone!

First thanks for this useful tool, I was using Fastmm4 for trying to fix this bug that is driving me crazy, but after seeing the demos... :crazy:

Well, I've just coding here a console program, i am using a SQLite wrapper to learn how to code for this little DB (another small and very good program)

So, everything is working nice. But, when i started importing a text file in the DB, it started to break to console again. So I started testing it, first with FastMM4, now with madExcept.

This little program only creates a Sqlite database, opens a text file, chess games, reads lines and insert lines inside a commit.

Madexcept points me to some lines in my code, examples:

> sTagValue:=Copy(sTagValue,2,Length(sTagValue)-2); // remove ""

> if sGame[Length(sGame)]<>' ' then sGame:=sGame+' ';

Variables are declared:
var
sGame,
sTagValue:RawUTF8; // RawUTF8 = type AnsiString(CP_UTF8);

Simple utf8 strings.

If I try to process a large file, leaked memory stacks up, breaking to console after it reaches 2GB... Something pretty bad is there.

I simply cannot understood this. This line

> if sGame[Length(sGame)]<>' ' then sGame:=sGame+' ';

is leaking everytime. Is there something i cannot see with my eyes? This line is leaking between 300-500bytes per execution.

Sorry if I sound too newbie, but I simply cannot see where the error could be. I am just trying to learn here.

If anyone can give a hand i'll be eternally happy. Thanks!

Example of one of this crazy string leaking:
allocation number: 4603
program up time: 362 ms
type: AnsiString
address: $6356720
size: 411
access rights: read/write
code page: 65001
reference counter: 1
string length: 398
contents: 1.e4 c5 2.Nf3 Nc6 3.d4 cxd4 4.Nxd4 e5 5.Nb5 a6 6.Nd6+ Bxd6 7.Qxd6 Qf6 8. Qd1 Qg6 9.Nc3 Nge7 10.h4 h5

main thread ($2bb0):
02cf8dea madExcept32.dll madExceptDbg 1603 GetMemCallback
004070a8 pgnpam.exe System 4614 @GetMem
0040b973 pgnpam.exe System 24129 @NewAnsiString
0040c5b6 pgnpam.exe System 28009 @LStrSetLength
0040c23f pgnpam.exe System 27086 @LStrCat
00602d8b pgnpam.exe pgnpam 958 initialization

memory dump:
06356720 e9 fd 01 00 01 00 00 00 - 8e 01 00 00 31 2e 65 34 ............1.e4
06356730 20 63 35 20 32 2e 4e 66 - 33 20 4e 63 36 20 33 2e c5 2.Nf3 Nc6 3.
06356740 64 34 20 63 78 64 34 20 - 34 2e 4e 78 64 34 20 65 d4 cxd4 4.Nxd4 e
06356750 35 20 35 2e 4e 62 35 20 - 61 36 20 36 2e 4e 64 36 5 5.Nb5 a6 6.Nd6
06356760 2b 20 42 78 64 36 20 37 - 2e 51 78 64 36 20 51 66 + Bxd6 7.Qxd6 Qf
06356770 36 20 38 2e 20 51 64 31 - 20 51 67 36 20 39 2e 4e 6 8. Qd1 Qg6 9.N
06356780 63 33 20 4e 67 65 37 20 - 31 30 2e 68 34 20 68 35 c3 Nge7 10.h4 h5
06356790 20 31 31 2e 42 67 35 20 - 64 35 20 31 32 2e 65 78 11.Bg5 d5 12.ex
063567a0 64 35 20 4e 62 34 20 31 - 33 2e 42 78 65 37 20 4e d5 Nb4 13.Bxe7 N
063567b0 78 63 32 2b 20 31 34 2e - 4b 64 32 20 4e 78 61 31 xc2+ 14.Kd2 Nxa1
063567c0 20 31 35 2e 42 61 33 20 - 4e 63 32 20 31 36 2e 51 15.Ba3 Nc2 16.Q
063567d0 78 63 32 20 42 66 35 20 - 31 37 2e 51 61 34 2b 20 xc2 Bf5 17.Qa4+
063567e0 42 64 37 20 31 38 2e 51 - 62 34 20 51 68 36 2b 20 Bd7 18.Qb4 Qh6+
063567f0 31 39 2e 4b 64 31 20 4f - 2d 4f 2d 4f 20 32 30 2e 19.Kd1 O-O-O 20.
06356800 51 63 35 2b 20 4b 62 38 - 20 32 31 2e 51 64 36 2b Qc5+ Kb8 21.Qd6+
06356810 20 4b 61 38 20 32 32 2e - 51 78 65 35 20 52 68 65 Ka8 22.Qxe5 Rhe
06356820 38 20 32 33 2e 51 64 34 - 20 51 67 36 20 32 34 2e 8 23.Qd4 Qg6 24.
06356830 66 33 20 42 66 35 20 32 - 35 2e 51 64 32 20 52 63 f3 Bf5 25.Qd2 Rc
06356840 38 20 32 36 2e 64 36 20 - 52 63 36 20 32 37 2e 20 8 26.d6 Rc6 27.
06356850 62 33 20 42 65 36 20 32 - 38 2e 4b 63 31 20 52 65 b3 Be6 28.Kc1 Re

Re: Crazy strings leaking in simple operations

Posted: Mon Jul 03, 2017 7:53 pm
by madshi
Just tried in Delphi Berlin 10.2, but it doesn't know the type "RawUTF8". What kind of type is that?

"type AnsiString(CP_UTF8)" doesn't compile for me.

Re: Crazy strings leaking in simple operations

Posted: Tue Jul 04, 2017 6:56 am
by an0rmal
Hi,

I am using Syncommons unit from Synopse Mormot ORM library.
RawUtf8 is defined as

{$ifdef HASCODEPAGE}
RawUTF8 = type AnsiString(CP_UTF8); // Codepage for an UTF8 string
{$else}
RawUTF8 = type AnsiString;
{$endif}

Just a Ansistring. I don't think the problem comes from this.

I've also tested replacing all rawutf8 strings with ansistring, or simply string, and keeps on leaking. It's driving me mad.

Re: Crazy strings leaking in simple operations

Posted: Tue Jul 04, 2017 7:24 am
by madshi
A leak can happen if you stop Delphi from freeing the string properly. Is it possible that the sGame and sTagValue variables are overwritten with zeroes by some code of yours? E.g. something like "integer(sGame) := 0" or "ZeroMemory(pointer(sGame))" would produce a leak because this will stop Delphi from properly releasing the string.

The madExcept leak reporting always shows you the line where a leaked resources was created. The cause of the leak might not be that line but it might be some other code, e.g. an "integer(sGame) := 0", as explained above. It could also be a buffer overrun somewhere which accidently clears your sGame or sTagValue variables. Have you tried activating the "instantly crash on buffer overrun" feature in madExcept? Do you get crashes if you do?

Re: Crazy strings leaking in simple operations

Posted: Tue Jul 04, 2017 10:09 am
by an0rmal
Thanks for tips, i must revise the code but i am nearly sure i am not filling the strings with direct memory manipulation. I'll check.
Also will try the crash buffer option.
Sir, you really have coded a great and useful tool!