memory leaks in wxSqlite3-3.0.0.1 & sqlite3 Topic is solved

Talk here about issues with one of the components hosted at wxCode, or suggest features for it.
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by Widgets »

After chasing memory leaks in my application which uses wxSqlite3-2.1.3 without much luck, I build a test case using the code for minimal from wxSqlite3-3.0.0.1 and found 1700++ instances of leaked blocks after instrumenting the code for leak detection using the recommended MSVC run-time instrumentation.

The majority of the blocks are 1184 bytes, with a few smaller ones, but I saw at least one block of 64,000 bytes. That is a lot of memory.

My question is: is the code intended to have no leaks or is some clean up code known to be missing??

These test we done using MSVC C++ Express 2010.

I very much would like to sort this out, but the number of code lines in sqlite3.c alone scares me off from digging in :-)

A typical line looks like:
sqlite3.c(15232) : {250881} normal block at 0x02760810, 1184 bytes long.
Data: < ^ > 0D 00 00 00 1F 00 5E 00 03 E2 03 C4 03 A6 03 88

And, FWIW, as far as I have checked, all the leaks originate at this same line in the sqlite3.c code
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by utelle »

Widgets wrote:After chasing memory leaks in my application which uses wxSqlite3-2.1.3 without much luck, I build a test case using the code for minimal from wxSqlite3-3.0.0.1 and found 1700++ instances of leaked blocks after instrumenting the code for leak detection using the recommended MSVC run-time instrumentation.

The majority of the blocks are 1184 bytes, with a few smaller ones, but I saw at least one block of 64,000 bytes. That is a lot of memory.
Which version of SQLite do you use? Do you use a precompiled version of SQlite or do you compile it yourself? In the latter case: which #defines do you use? Which modifications did you apply to the code of the minimal sample?

The handling of the internal SQLite data structures in wxSQLite3 was rewritten in version 3.0.0.1 to use reference counting. This should usually guarantee that all memory is cleaned up properly. There might be still bugs in wxSQlite3 related to this new feature, but ...
Widgets wrote:My question is: is the code intended to have no leaks or is some clean up code known to be missing??
... as far as I know the code of wxSQLite3 and SQLite exposes no memory leaks. Before I release a new version of wxSQLite3 I usually check at least the minimal sample and up to now I didn't detect memory leaks. I use wxSQLite3 and SQLite myself for many years in different projects and didn't experience memory leaks, neither in wxSQLite3 nor in SQLite.
Widgets wrote:These test we done using MSVC C++ Express 2010.
I use VC++ 2008 and VC++ 2010 for development and I usually also test with gcc, so I don't think it's specific to the compiler.
Widgets wrote:I very much would like to sort this out, but the number of code lines in sqlite3.c alone scares me off from digging in :-)

A typical line looks like:
sqlite3.c(15232) : {250881} normal block at 0x02760810, 1184 bytes long.
Data: < ^ > 0D 00 00 00 1F 00 5E 00 03 E2 03 C4 03 A6 03 88
A problem in debugging SQLite is that the SQLite amalgamation source file has too many lines for the VC++ debugger. For debugging SQLite one would have to divide the sources into smaller pieces.

But SQLite is very well tested. If you experience memory leaks it's probably due to your application's code not finalizing SQLite statements and/or other SQLite data structures.
Widgets wrote:And, FWIW, as far as I have checked, all the leaks originate at this same line in the sqlite3.c code
This is not surprising as SQLite encapsulates memory allocation in a small function which is located around that very line in the SQLite amalgamation source code.

Regards,

Ulrich
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by Widgets »

Thank you for your response, Ulrich.
utelle wrote: Which version of SQLite do you use? Do you use a precompiled version of SQlite or do you compile it yourself? In the latter case: which #defines do you use? Which modifications did you apply to the code of the minimal sample?
I took all the source code from wxSqlite3-3.0.0.1 and build a new MSVC project for the 'minimal' example, i.e. minimal.cpp, minmal.rc as well as the code for wxsqlite3.cpp and all of sqlite3.c & .h.
None of the 'minimal' code was modified.
In sqlite3.c I inserted:
at the very top:

Code: Select all

---------------------------
 #ifndef SQLITE_API
# define SQLITE_API
#endif
// start of my code +++++++++++++++++
if defined( _MSC_VER )
//-------------------------------------------------------------------
// this statement NEEDS to go BEFORE all headers
// see other part at line 10626 ++
#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>
// ------------------------------------------------------------------
#endif
// end of my code +++++++++++++++++++++++
/************** Begin file sqliteInt.h ***************************************/
---------------------------
near 12154:

Code: Select all

#endif /* _SQLITEINT_H_ */

/************** End of sqliteInt.h *******************************************/
// start of my code ++++++++++++++++++++++++++++++
#if defined( _MSC_VER )
// this block needs to go AFTER all headers
#include <crtdbg.h>
#ifdef _DEBUG
#define DEBUG_NEW new(_NORMAL_BLOCK, __FILE__, __LINE__)
#define new DEBUG_NEW
#endif
#endif
// end of my code ++++++++++++++++++++++++++++
/************** Begin file global.c ******************************************/
As far as #defines, I currently have defined:

Code: Select all

WIN32
WXUSINGDLL
_UNICODE
__WXDEBUG__
__WXMSW__
_CONSOLE
_DEBUG
I understand there are some sqlite specific #defines, but at present none are defined.
If there are any which ought to be defined, please let me know and I will set them up.
To have as much control over what is actually in the executable, I have also compiled it with static linking of all of the application's code, other then the wxWidgets 2.8.10 DLLs.
utelle wrote:... as far as I know the code of wxSQLite3 and SQLite exposes no memory leaks. Before I release a new version of wxSQLite3 I usually check at least the minimal sample and up to now I didn't detect memory leaks. I use wxSQLite3 and SQLite myself for many years in different projects and didn't experience memory leaks, neither in wxSQLite3 nor in SQLite.
I take this then that 'minimal' is intended to clean up after itself and ought to have no (known)memory leaks.
utelle wrote:I use VC++ 2008 and VC++ 2010 for development and I usually also test with gcc, so I don't think it's specific to the compiler.
That is encouraging, at least we have a (more or less) common compile environment
utelle wrote:A problem in debugging SQLite is that the SQLite amalgamation source file has too many lines for the VC++ debugger. For debugging SQLite one would have to divide the sources into smaller pieces.
I have suspected as much a number of times ;-)
utelle wrote: But SQLite is very well tested. If you experience memory leaks it's probably due to your application's code not finalizing SQLite statements and/or other SQLite data structures.
That is why I used the minimal example as 'application'. It removes all of my code and a lot of wxWidgets code as well.
Widgets wrote:And, FWIW, as far as I have checked, all the leaks originate at this same line in the sqlite3.c code
This is not surprising as SQLite encapsulates memory allocation in a small function which is located around that very line in the SQLite amalgamation source code.

Regards,

Ulrich[/quote]
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by utelle »

Widgets wrote:I took all the source code from wxSqlite3-3.0.0.1 and build a new MSVC project for the 'minimal' example, i.e. minimal.cpp, minmal.rc as well as the code for wxsqlite3.cpp and all of sqlite3.c & .h.
None of the 'minimal' code was modified.
Ok. Maybe you could provide your VC++ solution and project file for inspection.

One more question: do you use the ANSI build or the Unicode build of wxWidgets?
Widgets wrote:In sqlite3.c I inserted:
at the very top:

Code: Select all

---------------------------
 #ifndef SQLITE_API
# define SQLITE_API
#endif
// start of my code +++++++++++++++++
if defined( _MSC_VER )
//-------------------------------------------------------------------
// this statement NEEDS to go BEFORE all headers
// see other part at line 10626 ++
#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>
// ------------------------------------------------------------------
#endif
// end of my code +++++++++++++++++++++++
/************** Begin file sqliteInt.h ***************************************/
I assume you did not remove the defines

Code: Select all

#define SQLITE_CORE 1
#define SQLITE_AMALGAMATION 1
as they are very important.
Widgets wrote:As far as #defines, I currently have defined:

Code: Select all

WIN32
WXUSINGDLL
_UNICODE
__WXDEBUG__
__WXMSW__
_CONSOLE
_DEBUG
I understand there are some sqlite specific #defines, but at present none are defined.
If there are any which ought to be defined, please let me know and I will set them up.
By not defining some of the SQLite specific #defines you disable some SQLite features but that's not critical. All important #defines take reasonable default values.
Widgets wrote:To have as much control over what is actually in the executable, I have also compiled it with static linking of all of the application's code, other then the wxWidgets 2.8.10 DLLs.
This might impose problems if you compiled wxWidgets itself differently, i.e. with dynamic VC++ runtime. If parts of your application are compiled with static runtime while other parts with dynamic runtime you'd be mixing up memory management. This could be an explanation.
Widgets wrote:
utelle wrote:... as far as I know the code of wxSQLite3 and SQLite exposes no memory leaks. Before I release a new version of wxSQLite3 I usually check at least the minimal sample and up to now I didn't detect memory leaks. I use wxSQLite3 and SQLite myself for many years in different projects and didn't experience memory leaks, neither in wxSQLite3 nor in SQLite.
I take this then that 'minimal' is intended to clean up after itself and ought to have no (known)memory leaks.
Yes. In my development environment which is similar to yours I don't experience memory leaks.

One difference in project setup is that I build SQLite as a separate static library when I intend to include it statically, but this shouldn't make a difference.

Without seeing your project file I have no clue why you experience memory leaks.

Regards,

Ulrich
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by Widgets »

I do build a Unicode version.
I will attach the .sln file for your review and will shortly try to compile the whole thing with static wxWidgets lib.
Due to file size restrictions I had to remove the sqlite3.c file - the one I used was taken from the wxSqlite3.3.0.0.1 package and carries this notice

Code: Select all

** This file is an amalgamation of many separate C source files from SQLite
** version 3.7.10.  
PS: I did include the sqlite code statically; it was only the wxWidgets code which was as DLL

PPS: I just recompiled with static linking of all wxWidgets libs: wxbase28ud.lib & wxregexud.lib
and I still get very much the same results, though I did not do a 'diff' :-)
Attachments
wx.zip
(139.09 KiB) Downloaded 199 times
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by utelle »

Widgets wrote:I do build a Unicode version.
Good - as I do almost all my developing and testing with the Unicode version.
Widgets wrote:I will attach the .sln file for your review and will shortly try to compile the whole thing with static wxWidgets lib.
I'll try to take a look over the weekend.
Widgets wrote:Due to file size restrictions I had to remove the sqlite3.c file
No problem.

Regards,

Ulrich
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by utelle »

Using your VC++ project file I was able to reproduce the memory leaks.

Unfortunately I have to confirm that the minimal sample coming with wxSQLite3 indeed exposes memory leaks. I wondered why I didn't experienced them myself, but the explanation seems to be quite simple: I used a precompiled SQLite DLL for testing, precompiled in release mode that is, and I guess the VC++ debugger doesn't catch memory leaks in release mode DLLs.

The reason for the memory leaks is the sequence in which the database is closed and statements and result sets are finalized. The minimal sample doesn't explicitly finalize several statement and result set instances, so the destructors have to do the job. But they can't because the database is expplicitly closed before the destructors are invoked.

I have to think about how to fix the problem. I'll try to provide a wxSQLite3 bug fix release ASAP.

Until I can provide a fix you can avoid the memory leaks by not closing the database in the same code block in which any statement or result set instance is used, that is, all statements and result set instances have to be destructed before closing the database.

Regards,

Ulrich
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by utelle »

I modified the code for finalizing statements and result sets and for closing a database to avoid potential memory leaks.

The Finalize methods are now optimized to release the SQLite handles as soon as they are finalized instead of holding them until the dtor of the related statement or result set is called.

The Close method flags the database instance as closed, but the underlying SQLite database handle will be held open as long as there are references to it.

Finally I fixed a potential bug in the minimal sample by moving the call to ShutdownSQLite to a separate try-catch block.

At least in my test environment the minimal sample doesn't expose memory leaks any longer.

I changed the following 3 files of wxSQLite3:

include/wx/wxsqlite3.h
src/wxsqlite3.cpp
samples/minimal.cpp

The files can be downloaded from the wxCode SVN.

Please give them a try and let me know whether you experience any problems.

Regards,

Ulrich
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by Widgets »

Thank you very much, Ulrich, for the superfast response and fix.
Unfortunately I have experienced a hardware fault on one of my hard disks and will need to address that issue before I can spend a lot of time on the application to adapt my code and test the fix. The development machine is still up and running but a good bit of my time will be diverted ;-)
With any luck I should be back 'in business' by the start of the week after getting the disk replaced and the rest of the 'stuff' on it restored to a reasonable state
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by utelle »

Widgets wrote:Thank you very much, Ulrich, for the superfast response and fix.
You are welcome. Fortunately it wasn't that difficult to fix the issue.
Widgets wrote:Unfortunately I have experienced a hardware fault on one of my hard disks and will need to address that issue before I can spend a lot of time on the application to adapt my code and test the fix.
Good luck for getting your hardware up and running again. - Changes to your application code should be minor or not necessary at all.

Even if it takes a few days until you can dedicate some time to testing the fixed version I'd like to get some feedback before I make a new file release whether the wxSQLite3 code is now working as expected or whether you detect additional issues.

Regards,

Ulrich
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by Widgets »

Got the new disk up and running and most of the environment restored. :-)

I can confirm that the minimal sample now compiles and runs without memory leaks.

As for my app, I seem to have some problems when I replace the old include/wx/wxsqlite3.h & src/wxsqlite3.cpp with the new versions.
Looks like it might have something to do with header sequencing or possibly #define, though I haven't spent much time chasing it as yet.

Are there any #defines I should or should not have defined?
I'm hoping to get more time to look into this issue tomorrow and will report when it all works.
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by utelle »

Widgets wrote:As for my app, I seem to have some problems when I replace the old include/wx/wxsqlite3.h & src/wxsqlite3.cpp with the new versions.
Looks like it might have something to do with header sequencing or possibly #define, though I haven't spent much time chasing it as yet.

Are there any #defines I should or should not have defined?
In wxsqlite3.h I just had to add an additional member variable and in wxsqlite3.cpp there are no changes related to any #define. Therefore you shouldn't experience any compile time problems by just replacing the files.

Regards,

Ulrich
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by Widgets »

One of the simplest errors I get looks like this:

Code: Select all

1>wxsqlite3.cpp(772): error C2248: 'wxSQLite3StatementReference::m_stmt' : cannot access private member declared in class 'wxSQLite3StatementReference'
1>          wxsqlite3.cpp(313) : see declaration of 'wxSQLite3StatementReference::m_stmt'
1>          wxsqlite3.cpp(261) : see declaration of 'wxSQLite3StatementReference'
This occurs in the sub-project wxSqlite3, where I copied the old project and replaced the two files.
Other sub-projects which use wxSqlite3 give me errors along the line of:

Code: Select all

1>  wdChipPlugin.cpp
1>C:\pkg\wx\wxWd-now-newwxSql3\wxsqlite3\include\wx/wxsqlite3.h(523): error C2079: 'wxSQLite3Database' uses undefined class 'WXDLLIMPEXP_FWD_SQLITE3'
1>C:\pkg\wx\wxWd-now-newwxSql3\wxsqlite3\include\wx/wxsqlite3.h(587): error C2061: syntax error : identifier 'wxSQLite3Database'
1>C:\pkg\wx\wxWd-now-newwxSql3\wxsqlite3\include\wx/wxsqlite3.h(596): error C2143: syntax error : missing ';' before '*'
1>C:\pkg\wx\wxWd-now-newwxSql3\wxsqlite3\include\wx/wxsqlite3.h(596): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by utelle »

Widgets wrote:One of the simplest errors I get looks like this:

Code: Select all

1>wxsqlite3.cpp(772): error C2248: 'wxSQLite3StatementReference::m_stmt' : cannot access private member declared in class 'wxSQLite3StatementReference'
1>          wxsqlite3.cpp(313) : see declaration of 'wxSQLite3StatementReference::m_stmt'
1>          wxsqlite3.cpp(261) : see declaration of 'wxSQLite3StatementReference'
This occurs in the sub-project wxSqlite3, where I copied the old project and replaced the two files.
Other sub-projects which use wxSqlite3 give me errors along the line of:

Code: Select all

1>  wdChipPlugin.cpp
1>C:\pkg\wx\wxWd-now-newwxSql3\wxsqlite3\include\wx/wxsqlite3.h(523): error C2079: 'wxSQLite3Database' uses undefined class 'WXDLLIMPEXP_FWD_SQLITE3'
1>C:\pkg\wx\wxWd-now-newwxSql3\wxsqlite3\include\wx/wxsqlite3.h(587): error C2061: syntax error : identifier 'wxSQLite3Database'
1>C:\pkg\wx\wxWd-now-newwxSql3\wxsqlite3\include\wx/wxsqlite3.h(596): error C2143: syntax error : missing ';' before '*'
1>C:\pkg\wx\wxWd-now-newwxSql3\wxsqlite3\include\wx/wxsqlite3.h(596): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
I think all errors are related to the same problem:

For whatever reason the preprocessor symbol WXDLLIMPEXP_FWD_SQLITE3 is not defined. This symbol is defined in wxsqlite3def.h which is included by wxsqlite3.h. I have no clue why this symbol is not defined, unless file wxsqlite3def.h you use in your project is outdated. The symbol WXDLLIMPEXP_FWD_SQLITE3 was introduced in version 2.0.0 of wxSQLite3.

Regards,

Ulrich
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: memory leaks in wxSqlite3-3.0.0.1 & sqlite3

Post by Widgets »

Problem solved!
Thank you very much, Ulrich.

I had obviously copied the wrong header files. Getting the latest from the SVN repository cleared up all my problems and as far as I can tell, there are no more reported leaks - even after instrumenting the sqlite.c code. :-)

Again, loads of "Thank you"s for wxSqlite3 and your help.

PS: is there something I need to or can do to mark the topic "solved"?
As soon as I posted the question, I found the answer :-)
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
Post Reply