Is it an implementation problem of operator new in debug mode

Do you have a typical platform dependent issue you're battling with ? Ask it here. Make sure you mention your platform, compiler, and wxWidgets version.
Post Reply
Ronald
Filthy Rich wx Solver
Filthy Rich wx Solver
Posts: 235
Joined: Mon Mar 05, 2018 4:17 am

Is it an implementation problem of operator new in debug mode

Post by Ronald » Sun Jan 17, 2021 4:30 am

The codes below runs as expected, but reports memory leak in debug mode in VS2019 on Windows

Code: Select all

#define WXUSINGDLL
#include <wx/wx.h>

#include <libpq-fe.h>

int main()
{
    const char * keys[6] = { "host", "port", "user", "password", "dbname", nullptr };
    const char * values[6] = { "192.168.1.100", "3333", "test", "123456", "postgres", nullptr };
    PGconn * conn = PQconnectdbParams(keys, values, 0);
    PQfinish(conn);
    return 0;
}

Code: Select all

Detected memory leaks!
Dumping objects ->
{5677} normal block at 0x00000245185E85C0, 13 bytes long.
 Data: <libpq socket > 6C 69 62 70 71 20 73 6F 63 6B 65 74 00 
{5676} normal block at 0x00000245185B35B0, 96 bytes long.
 Data: <          ^ E   > 81 00 00 00 00 00 00 00 C0 85 5E 18 45 02 00 00 
{1594} normal block at 0x00000245185E19D0, 40 bytes long.
 Data: <                > FF FF FF FF FF FF FF FF FF FF FF FF 00 00 00 00 
Object dump complete.
Version Info:
OS: WIndows 10
IDE: VS2019
wxWidgets: 3.1.4

Here is another sample (using poppler) that reports leak.
viewtopic.php?f=1&t=47898

Without defining WXUSINGDLL, no leak is reported.

It seems to be an error report of memory leak.
Is it an implementation problem of operator new in debug mode?

User avatar
doublemax
Moderator
Moderator
Posts: 15866
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: Is it an implementation problem of operator new in debug mode

Post by doublemax » Sun Jan 17, 2021 8:56 am

There is no need to create multiple threads for the same topic. Apparently nobody here knows a solution.

Maybe try your luck on the wx-users group where the core wx developers can see it:
https://groups.google.com/g/wx-users
Use the source, Luke!

Ronald
Filthy Rich wx Solver
Filthy Rich wx Solver
Posts: 235
Joined: Mon Mar 05, 2018 4:17 am

Re: Is it an implementation problem of operator new in debug mode

Post by Ronald » Sun Jan 17, 2021 1:44 pm

doublemax wrote:
Sun Jan 17, 2021 8:56 am
There is no need to create multiple threads for the same topic. Apparently nobody here knows a solution.

Maybe try your luck on the wx-users group where the core wx developers can see it:
https://groups.google.com/g/wx-users
OK, thanks.

utelle
Moderator
Moderator
Posts: 1013
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: Is it an implementation problem of operator new in debug mode

Post by utelle » Sun Jan 17, 2021 5:28 pm

For quite a while you are presenting artifical examples here in the forum, which exhibit memory leaks for you under certain circumstances. Without full information which steps exactly you take, it is extremely difficult to give you useful advice.

As with any memory leaks you have to track down where exactly the memory is leaked. It is known that some external libraries tend to leak memory. I don't know whether poppler or postgresql client belong to this category. For some libraries you have to call special initialization/termination functions to clean up allocated resources.

AFAICT wxWidgets itself does not expose memory leak behaviour. Over time I have develop quite a few wxWidgets applications which made use of several other external libraries, but never ended up with memory leaking applications.

Simply defining (or not defining) WXUSINGDLL doesn't make sense and is certainly not enough to create a valid executable, because the linker has to reference different libraries:

If WXUSINGDLL is defined you have to link against the link libraries of a shared wxWidgets build; if WXUSINGDLL is not defined you have to link against the libraries of a static wxWidgets build.

So, if you define WXUSINGDLL, but link against the static wxWidgets build, there might be side effects of defining the WXUSINGDLL symbol causing memory leaks in the end. Since you are not showing the full compiler and linker commands, it is impossible to tell whether this could explain the effects you are seeing.

As said before, it is rather unlikely that wxWidgets is the primary cause of the memory leaks, unless you misuse it in your own application. If there are memory leaks in other external libraries your application makes use of, you have 2 options: either track down the root cause of the memory leak in the external library (what may be a time consuming task), or live with the memory leaks (quite often they do no real harm).

Ronald
Filthy Rich wx Solver
Filthy Rich wx Solver
Posts: 235
Joined: Mon Mar 05, 2018 4:17 am

Re: Is it an implementation problem of operator new in debug mode

Post by Ronald » Mon Jan 18, 2021 6:15 am

utelle wrote:
Sun Jan 17, 2021 5:28 pm
For quite a while you are presenting artifical examples here in the forum, which exhibit memory leaks for you under certain circumstances. Without full information which steps exactly you take, it is extremely difficult to give you useful advice.

As with any memory leaks you have to track down where exactly the memory is leaked. It is known that some external libraries tend to leak memory. I don't know whether poppler or postgresql client belong to this category. For some libraries you have to call special initialization/termination functions to clean up allocated resources.

AFAICT wxWidgets itself does not expose memory leak behaviour. Over time I have develop quite a few wxWidgets applications which made use of several other external libraries, but never ended up with memory leaking applications.

Simply defining (or not defining) WXUSINGDLL doesn't make sense and is certainly not enough to create a valid executable, because the linker has to reference different libraries:

If WXUSINGDLL is defined you have to link against the link libraries of a shared wxWidgets build; if WXUSINGDLL is not defined you have to link against the libraries of a static wxWidgets build.

So, if you define WXUSINGDLL, but link against the static wxWidgets build, there might be side effects of defining the WXUSINGDLL symbol causing memory leaks in the end. Since you are not showing the full compiler and linker commands, it is impossible to tell whether this could explain the effects you are seeing.

As said before, it is rather unlikely that wxWidgets is the primary cause of the memory leaks, unless you misuse it in your own application. If there are memory leaks in other external libraries your application makes use of, you have 2 options: either track down the root cause of the memory leak in the external library (what may be a time consuming task), or live with the memory leaks (quite often they do no real harm).
I've tried to reduce the code and locate the error.
If comment out m_strings = new wxString[count]; in the wx code below, the leak gone.

Code: Select all

// this class provides a temporary wxString* from a
// wxArrayString
class WXDLLIMPEXP_BASE wxCArrayString
{
public:
    wxCArrayString(const wxArrayString & array)
        : m_array(array), m_strings(NULL)
    {
    }
    ~wxCArrayString() { delete[] m_strings; }

    size_t GetCount() const { return m_array.GetCount(); }
    
    wxString * GetStrings()
    {
        if (m_strings) return m_strings;
        const size_t count = m_array.GetCount();
        m_strings = new wxString[count];
        for (size_t i = 0; i < count; ++i)
            m_strings[i] = m_array[i];
        return m_strings;
    }

    wxString * Release()
    {
        return nullptr;
        wxString * r = GetStrings();
        m_strings = NULL;
        return r;
    }

private:
    const wxArrayString & m_array;
    wxString * m_strings;
};

What's weird is the leak is about libpq

Code: Select all

Detected memory leaks!
Dumping objects ->
{5677} normal block at 0x000002C1C71A9760, 13 bytes long.
 Data: <libpq socket > 6C 69 62 70 71 20 73 6F 63 6B 65 74 00 
{5676} normal block at 0x000002C1C7177A40, 96 bytes long.
 Data: <        `       > 81 00 00 00 00 00 00 00 60 97 1A C7 C1 02 00 00 
{1594} normal block at 0x000002C1C71A1E60, 40 bytes long.
 Data: <                > FF FF FF FF FF FF FF FF FF FF FF FF 00 00 00 00 
Object dump complete.

Runnable code, which depends on pgsql, if no pgsql no leak

Code: Select all

///////////////////////////////////////////////////////////////////////////////
// Name:        wx/arrstr.h
// Purpose:     wxArrayString class
// Author:      Mattia Barbon and Vadim Zeitlin
// Modified by:
// Created:     07/07/03
// Copyright:   (c) 2003 Vadim Zeitlin <[email protected]>
// Licence:     wxWindows licence
///////////////////////////////////////////////////////////////////////////////

#ifndef _WX_ARRSTR_H
#define _WX_ARRSTR_H

#include "wx/defs.h"
#include "wx/string.h"
#include "wx/dynarray.h"

#if wxUSE_STD_CONTAINERS_COMPATIBLY
#include <vector>
#endif

// This comparison function ignores case when comparing strings differing not
// in case only, i.e. this ensures that "Aa" comes before "AB", unlike with
// wxStringSortAscending().
inline int wxCMPFUNC_CONV
wxDictionaryStringSortAscending(const wxString & s1, const wxString & s2)
{
    const int cmp = s1.CmpNoCase(s2);
    return cmp ? cmp : s1.Cmp(s2);
}


inline int wxCMPFUNC_CONV
wxDictionaryStringSortDescending(const wxString & s1, const wxString & s2)
{
    return wxDictionaryStringSortAscending(s2, s1);
}

WXDLLIMPEXP_BASE
int wxCMPFUNC_CONV wxCmpNatural(const wxString & s1, const wxString & s2);

WXDLLIMPEXP_BASE
int wxCMPFUNC_CONV wxCmpNaturalGeneric(const wxString & s1, const wxString & s2);

inline int wxCMPFUNC_CONV wxNaturalStringSortAscending(const wxString & s1, const wxString & s2)
{
    return wxCmpNatural(s1, s2);
}

inline int wxCMPFUNC_CONV wxNaturalStringSortDescending(const wxString & s1, const wxString & s2)
{
    return wxCmpNatural(s2, s1);
}

typedef int (wxCMPFUNC_CONV * CMPFUNCwxString)(wxString *, wxString *);
WX_DEFINE_USER_EXPORTED_TYPEARRAY(wxString, wxArrayStringBase,
                                  wxARRAY_DUMMY_BASE, WXDLLIMPEXP_BASE);

class WXDLLIMPEXP_BASE wxArrayString : public wxArrayStringBase
{
public:
    // type of function used by wxArrayString::Sort()
    typedef int (wxCMPFUNC_CONV * CompareFunction)(const wxString & first,
                                                   const wxString & second);

    wxArrayString() {}
    wxArrayString(size_t sz, const char ** a);
    wxArrayString(size_t sz, const wchar_t ** a);
    wxArrayString(size_t sz, const wxString * a);

    int Index(const wxString & str, bool bCase = true, bool bFromEnd = false) const;

    void Sort(bool reverseOrder = false);
    void Sort(CompareFunction function);
    void Sort(CMPFUNCwxString function) { wxArrayStringBase::Sort(function); }

    size_t Add(const wxString & string, size_t copies = 1)
    {
        wxArrayStringBase::Add(string, copies);
        return size() - copies;
    }
};

// Unlike all the other sorted arrays, this one uses a comparison function
// taking objects by reference rather than value, so define a special functor
// wrapping it.
class wxSortedArrayString_SortFunction
{
public:
    typedef int (wxCMPFUNC_CONV * CMPFUNC)(const wxString &, const wxString &);

    explicit wxSortedArrayString_SortFunction(CMPFUNC f) : m_f(f) {}

    bool operator()(const wxString & s1, const wxString & s2)
    {
        return m_f(s1, s2) < 0;
    }

private:
    CMPFUNC m_f;
};

// this class provides a temporary wxString* from a
// wxArrayString
class WXDLLIMPEXP_BASE wxCArrayString
{
public:
    wxCArrayString(const wxArrayString & array)
        : m_array(array), m_strings(NULL)
    {
    }
    ~wxCArrayString() { delete[] m_strings; }

    size_t GetCount() const { return m_array.GetCount(); }
    
    wxString * GetStrings()
    {
        if (m_strings) return m_strings;
        const size_t count = m_array.GetCount();
        m_strings = new wxString[count];
        for (size_t i = 0; i < count; ++i)
            m_strings[i] = m_array[i];
        return m_strings;
    }

    wxString * Release()
    {
        return nullptr;
        wxString * r = GetStrings();
        m_strings = NULL;
        return r;
    }

private:
    const wxArrayString & m_array;
    wxString * m_strings;
};


// ----------------------------------------------------------------------------
// helper functions for working with arrays
// ----------------------------------------------------------------------------

// by default, these functions use the escape character to escape the
// separators occurring inside the string to be joined, this can be disabled by
// passing '\0' as escape

WXDLLIMPEXP_BASE wxString wxJoin(const wxArrayString & arr,
                                 const wxChar sep,
                                 const wxChar escape = wxT('\\'));

WXDLLIMPEXP_BASE wxArrayString wxSplit(const wxString & str,
                                       const wxChar sep,
                                       const wxChar escape = wxT('\\'));


// ----------------------------------------------------------------------------
// This helper class allows to pass both C array of wxStrings or wxArrayString
// using the same interface.
//
// Use it when you have two methods taking wxArrayString or (int, wxString[]),
// that do the same thing. This class lets you iterate over input data in the
// same way whether it is a raw array of strings or wxArrayString.
//
// The object does not take ownership of the data -- internally it keeps
// pointers to the data, therefore the data must be disposed of by user
// and only after this object is destroyed. Usually it is not a problem as
// only temporary objects of this class are used.
// ----------------------------------------------------------------------------

class wxArrayStringsAdapter
{
public:
    // construct an adapter from a wxArrayString
    wxArrayStringsAdapter(const wxArrayString & strings)
        : m_type(wxSTRING_ARRAY), m_size(strings.size())
    {
        m_data.array = &strings;
    }

    // construct an adapter from a wxString[]
    wxArrayStringsAdapter(unsigned int n, const wxString * strings)
        : m_type(wxSTRING_POINTER), m_size(n)
    {
        m_data.ptr = strings;
    }

#if wxUSE_STD_CONTAINERS_COMPATIBLY
    // construct an adapter from a vector of strings
    wxArrayStringsAdapter(const std::vector<wxString> & strings)
        : m_type(wxSTRING_POINTER), m_size(strings.size())
    {
        m_data.ptr = m_size == 0 ? NULL : &strings[0];
    }
#endif // wxUSE_STD_CONTAINERS_COMPATIBLY

    // construct an adapter from a single wxString
    wxArrayStringsAdapter(const wxString & s)
        : m_type(wxSTRING_POINTER), m_size(1)
    {
        m_data.ptr = &s;
    }

    // default copy constructor is ok

    // iteration interface
    size_t GetCount() const { return m_size; }
    bool IsEmpty() const { return GetCount() == 0; }
    const wxString & operator[] (unsigned int i) const
    {
        wxASSERT_MSG(i < GetCount(), wxT("index out of bounds"));
        if (m_type == wxSTRING_POINTER)
            return m_data.ptr[i];
        return m_data.array->Item(i);
    }
    wxArrayString AsArrayString() const
    {
        if (m_type == wxSTRING_ARRAY)
            return *m_data.array;
        return wxArrayString(GetCount(), m_data.ptr);
    }

private:
    // type of the data being held
    enum wxStringContainerType
    {
        wxSTRING_ARRAY,  // wxArrayString
        wxSTRING_POINTER // wxString[]
    };

    wxStringContainerType m_type;
    size_t m_size;
    union
    {
        const wxString * ptr;
        const wxArrayString * array;
    } m_data;

    wxDECLARE_NO_ASSIGN_CLASS(wxArrayStringsAdapter);
};

#endif // _WX_ARRSTR_H


#include <libpq-fe.h>

int main()
{
    const char * keys[6] = { "host", "port", "user", "password", "dbname", nullptr };
    const char * values[6] = { "192.168.1.99", "4433", "stocklab_db", "SlSlSl34567#$%^&", "postgres", nullptr };
    PGconn * conn = PQconnectdbParams(keys, values, 0);
    PQfinish(conn);

    return 0;
}

utelle
Moderator
Moderator
Posts: 1013
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Re: Is it an implementation problem of operator new in debug mode

Post by utelle » Mon Jan 18, 2021 12:46 pm

Ronald wrote:
Mon Jan 18, 2021 6:15 am
I've tried to reduce the code and locate the error.
In principle, that is the right approach, but you need to do it in a reasonable way. Otherwise this will get you nowhere.

First of all, it is still not clear whether you are using the shared or static build of wxWidgets, and whether your compiler and linker command options are properly set accordingly.

If you want to test a wxWidgets application, then you should set up a proper wxWidgets main application. It doesn't make much sense to use a standard C main function without initializing wxWidgets.
Ronald wrote:
Mon Jan 18, 2021 6:15 am
If comment out m_strings = new wxString[count]; in the wx code below, the leak gone.
The class for which you showed code is a wxWidgets class. So, what exactly did you do? Did you modify the wxWidgets source coed and rebuild the wxWidgets library?

That the leak was gone, could be pure coincidence.
Ronald wrote:
Mon Jan 18, 2021 6:15 am
What's weird is the leak is about libpq

Code: Select all

Detected memory leaks!
Dumping objects ->
{5677} normal block at 0x000002C1C71A9760, 13 bytes long.
 Data: <libpq socket > 6C 69 62 70 71 20 73 6F 63 6B 65 74 00 
{5676} normal block at 0x000002C1C7177A40, 96 bytes long.
 Data: <        `       > 81 00 00 00 00 00 00 00 60 97 1A C7 C1 02 00 00 
{1594} normal block at 0x000002C1C71A1E60, 40 bytes long.
 Data: <                > FF FF FF FF FF FF FF FF FF FF FF FF 00 00 00 00 
Object dump complete.
Well, there is no guarantee that the memory leak is related to libpq, although it is likely. It could be also memory reallocated elsewhere, accidentally containing libpq content due to missing memory initialization.
Ronald wrote:
Mon Jan 18, 2021 6:15 am
Runnable code, which depends on pgsql, if no pgsql no leak
Again, the code you showed consists mostly of internal wxWidgets code. In what way is it linked to your problem? How did you compile and link it?

Unfortunately, the Visual Studio debugger only tells you, if it found memory leaks, but not in which module the memory was allocated. You should use tools like Visual Leak Detector to identify the root cause of memory leaks.

Ronald
Filthy Rich wx Solver
Filthy Rich wx Solver
Posts: 235
Joined: Mon Mar 05, 2018 4:17 am

Re: Is it an implementation problem of operator new in debug mode

Post by Ronald » Mon Jan 18, 2021 1:42 pm

utelle wrote:
Mon Jan 18, 2021 12:46 pm
First of all, it is still not clear whether you are using the shared or static build of wxWidgets, and whether your compiler and linker command options are properly set accordingly.
The application is using shared build of wxWidgets.

I think I've set everything properly.

Thanks for your detailed answer, however I think it might be fine to write a demo to test it.

Ronald
Filthy Rich wx Solver
Filthy Rich wx Solver
Posts: 235
Joined: Mon Mar 05, 2018 4:17 am

Re: Is it an implementation problem of operator new in debug mode

Post by Ronald » Sat Jan 23, 2021 9:10 am

Tested poppler in a default MFC application, it also reports leaks.

Totally 2 lines are added:

Code: Select all

#include <poppler/cpp/poppler-page.h>
...
delete (poppler::page *)nullptr;
It is likely a bug of the lib or a bug of memory leak reporter.

Ronald
Filthy Rich wx Solver
Filthy Rich wx Solver
Posts: 235
Joined: Mon Mar 05, 2018 4:17 am

Re: Is it an implementation problem of operator new in debug mode

Post by Ronald » Sat Jan 23, 2021 9:10 am

Tested poppler in a default MFC application, it also reports leaks.

Totally 2 lines are added:

Code: Select all

#include <poppler/cpp/poppler-page.h>
...
delete (poppler::page *)nullptr;
It is likely a bug of the lib or a bug of memory leak reporter.

ONEEYEMAN
Part Of The Furniture
Part Of The Furniture
Posts: 5032
Joined: Sat Apr 16, 2005 7:22 am
Location: USA, Ukraine

Re: Is it an implementation problem of operator new in debug mode

Post by ONEEYEMAN » Sat Jan 23, 2021 5:03 pm

Hi,
As said - many libraries reports memory leaks.

You just found that the poppler is one of them.

Thank you.

Ronald
Filthy Rich wx Solver
Filthy Rich wx Solver
Posts: 235
Joined: Mon Mar 05, 2018 4:17 am

Re: Is it an implementation problem of operator new in debug mode

Post by Ronald » Sun Jan 24, 2021 3:52 pm

ONEEYEMAN wrote:
Sat Jan 23, 2021 5:03 pm
As said - many libraries reports memory leaks.
The leaks in the libs tend to mix with leaks in my codes.
This is a problem in practice.
I'll debug it later.
ONEEYEMAN wrote:
Sat Jan 23, 2021 5:03 pm
You just found that the poppler is one of them.
postgresql lib also leaks. I think it should not.

Post Reply