MYSQL

NAVIGATION
CATEGORIES
REFERRENCE
LINKS
  • Query Object Memory Leak?

    13 answers - 638 bytes - related search similar search Add To My Delicious Add To My Stumble Upon Add To My Google Mark Add To My Facebook Add To My Digg Add To My Reddit

    Sorry, looks like I somehow put this in the wrong thread. (thread:
    CLIENT_FUND_RWS) Maybe a mod. can delete that post?
    I am developing code that needs to create query objects often. There
    seems to be a memory leak - memory not being released by the query
    object when it goes out of scope. Is this a known issue, or am I missing
    something? I have tried this with VC7.1 and VC8.
    code to recreate problem:
    // Begin Code
    mysqlpp::Connection conn = new mysqlpp::Connection();
    for(int i=0; i <1000000; i++)
    {
    mysqlpp::Query myQuery = conn.query();
    }
    // End Code
    Thanks!
    - Bill K.
  • No.1 | | 1204 bytes | |

    Bill K wrote:
    Maybe a mod. can delete that post?

    Ahahaha. This is a mail list. I can't delete emails from other
    people's inboxes. Theoretically I could remove it from the archives,
    but most of the damage has already been done. Live and learn, dude. :)

    I am developing code that needs to create query objects often. There
    seems to be a memory leak - memory not being released by the query
    object when it goes out of scope.

    I think you've misdiagnosed the problem. I ran your test code on a
    Linux box, and the amount of memory stays static throughout the
    program's run time. Between this and what I see in the call stack
    window when after it crashes, I think it's more likely that the problem
    is somewhere down in VC++'s runtime library.

    If you're motivated enough, an interesting test would be to revert Query
    back to the old style stream mechanism. (See the v2.0.0 ChangeLog
    entry.) We couldn't make that reversion permanent, because the old
    stream style is obsolete, but it would tell us whether I'm right about
    where the problem is. If the symptom changes, it's in the runtime library.
  • No.2 | | 2169 bytes | |

    Warren Young wrote:
    Bill K wrote:
    >Maybe a mod. can delete that post?
    >

    Ahahaha. This is a mail list. I can't delete emails from other
    people's inboxes. Theoretically I could remove it from the archives,
    but most of the damage has already been done. Live and learn, dude. :)

    I know this is a mailing list, I was referring to deleting from the
    archive. But ok, you can't; thanks for answering my question. =)
    >I am developing code that needs to create query objects often. There
    >seems to be a memory leak - memory not being released by the query
    >object when it goes out of scope.
    >

    I think you've misdiagnosed the problem.
    I never said I had diagnosed or otherwise narrowed down the problem.
    I ran your test code on a Linux box, and the amount of memory stays
    static throughout the program's run time. Between this and what I see
    in the call stack window when after it crashes, I think it's more
    likely that the problem is somewhere down in VC++'s runtime library.

    If you're motivated enough, an interesting test would be to revert
    Query back to the old style stream mechanism. (See the v2.0.0
    ChangeLog entry.) We couldn't make that reversion permanent, because
    the old stream style is obsolete, but it would tell us whether I'm
    right about where the problem is. If the symptom changes, it's in the
    runtime library.

    I built MySQL++ 1.7.40, and rebuilt my test project with that library.
    The query object has no leak, tested with the same code. So, it may be
    with MS's runtime library like you say.

    At first I thought the problem was with MS's known std::basic_iostream
    memory leaking problem. ( info here:

    ) However, I obtained a non-public hotfix from MS that supposedly fixed
    that problem, which it looks like it did, when I test out
    std::stringstream and std::iostream objects in a similar loop.

    I will look into it some more. Anyone else having similar issues?

    Thank you for your help.
    - Bill K.
  • No.3 | | 671 bytes | |

    Bill K wrote:

    I am saying that the MS hotfix does not fix the
    problem I am having with the query object, like I thought it would.

    , thanks for clearing that up.

    MS hotfix fixed the memory leak in basic_iostream (used in fstream and
    stringstream, and others) - which I thought may have fixed the query
    object since it inherits from ostream. But no.

    , but here's another data point for you: I just tried the test
    program again, and when it crashes it's only using 5 MB. It can't be a
    memory leak.

    Here, it's crashing when the CRT dereferences a null pointer. Why the
    CRT is doing that is the mystery.
  • No.4 | | 1579 bytes | |

    Bill K wrote:
    I know this is a mailing list, I was referring to deleting from the
    archive.

    Sorry if I annoyed you, but the very fact that you asked suggested that
    you are more used to web forums, corporate email systems, or other
    captive systems. a public mailing list, you have to assume that once
    the message is gone, it's gone.

    But ok, you can't

    Indeed. :) I "own" this project, but I don't have any special access to
    the mail servers. You'd have as much chance of success asking MySQL
    Inc.'s server admins directly to remove the message as I would.

    >I think you've misdiagnosed the problem.

    I never said I had diagnosed or otherwise narrowed down the problem.

    Uh, okay, I think you're arguing about semantics here. You offered a
    hypothesis as to where the problem is, and I told you that I thought you
    were wrong, giving evidence to back up my assertion. Is that
    distinction in phrasing really worth fighting for?

    At first I thought the problem was with MS's known std::basic_iostream
    memory leaking problem. ( info here:

    ) However, I obtained a non-public hotfix from MS that supposedly fixed
    that problem, which it looks like it did, when I test out
    std::stringstream and std::iostream objects in a similar loop.

    I'm not sure I understand. Are you saying that the problem is indeed
    fixed when you apply the patch, or that it only fixes the case of
    creating stream objects but not Query objects?
  • No.5 | | 2242 bytes | |

    Warren Young wrote:
    Bill K wrote:
    >I know this is a mailing list, I was referring to deleting from the
    >archive.
    >

    Sorry if I annoyed you, but the very fact that you asked suggested
    that you are more used to web forums, corporate email systems, or
    other captive systems. a public mailing list, you have to assume
    that once the message is gone, it's gone.

    But ok, you can't

    Indeed. :) I "own" this project, but I don't have any special access
    to the mail servers. You'd have as much chance of success asking
    MySQL Inc.'s server admins directly to remove the message as I would.

    I think you've misdiagnosed the problem.
    >I never said I had diagnosed or otherwise narrowed down the problem.
    >

    Uh, okay, I think you're arguing about semantics here. You offered a
    hypothesis as to where the problem is, and I told you that I thought
    you were wrong, giving evidence to back up my assertion. Is that
    distinction in phrasing really worth fighting for?
    >
    >At first I thought the problem was with MS's known
    >std::basic_iostream memory leaking problem. ( info here:
    >
    >) However, I obtained a non-public hotfix from MS that supposedly
    >fixed that problem, which it looks like it did, when I test out
    >std::stringstream and std::iostream objects in a similar loop.
    >

    I'm not sure I understand. Are you saying that the problem is indeed
    fixed when you apply the patch, or that it only fixes the case of
    creating stream objects but not Query objects?

    Sorry for the confusion. I am saying that the MS hotfix does not fix the
    problem I am having with the query object, like I thought it would. The
    MS hotfix fixed the memory leak in basic_iostream (used in fstream and
    stringstream, and others) - which I thought may have fixed the query
    object since it inherits from ostream. But no. I only mentioned the MS
    hotfix so anyone aware of the memory leak in MS's CRT would know I have
    a patch for it, so that shouldn't be the problem.

    Thanks
    - Bill K.
  • No.6 | | 1161 bytes | |

    Warren Young wrote:
    Bill K wrote:
    >>

    >I am saying that the MS hotfix does not fix the problem I am having
    >with the query object, like I thought it would.
    >

    , thanks for clearing that up.
    >
    >MS hotfix fixed the memory leak in basic_iostream (used in fstream
    >and stringstream, and others) - which I thought may have fixed the
    >query object since it inherits from ostream. But no.
    >

    , but here's another data point for you: I just tried the test
    program again, and when it crashes it's only using 5 MB. It can't be
    a memory leak.

    Here, it's crashing when the CRT dereferences a null pointer. Why the
    CRT is doing that is the mystery.

    Crashes, eh? It doesn't crash for me. Adding an "if( !(i%1000) ){
    std::cout << i << endl; }" in the loop, I can see it fly through and
    finish all million loops, using over 80MB of memory at the end,
    according to Windows Task Manager.

    I'll have a look at it more tomorrow.

    Thanks again.
    - Bill
  • No.7 | | 2892 bytes | |

    To put my penny's worth in, trying it on Visual Studio Express with mysql++
    2.0.7 shows the memory usage going up to about 65M as Bill describes (mine
    doesn't crash).

    However, turning on memory leak detection
    (
    ) shows no leaks. I even put a
    memory leak in the Query constructor and recompiled mysql My memory leak
    showed up, no other one did.

    I'm not sure why the memory usage goes up so high, but it might have
    something to do with the fact that the runtime and /S don't necessarily
    return any memory freed up to the general pool straight away (for efficiency
    reasons). I also tried increasing the loop size ten fold, and after the
    memory usage got up to about 300M on my machine, Windows started reclaiming
    the memory and it had dropped to 14M by the end.

    I've included the revised test prog below (note, you shouldn't be using new
    to assign to the conn object, as that does cause a memory leak).

    Regards,

    Matt.

    #include "stdafx.h"
    #include <stdlib.h>
    #include <crtdbg.h>
    #include "connection.h"
    #include "query.h"

    void test()
    {
    printf("starting test\r\n");

    mysqlpp::Connection conn; // = new mysqlpp::Connection();

    for(int i=0; i <10; i++)
    {
    mysqlpp::Query myQuery = conn.query();
    }

    printf("done test\r\n");
    }

    int _tmain(int argc, _TCHAR* argv[])
    {
    _CrtSetDbgFlag ( _CRTDBG_ALLC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );

    test();

    return 0;
    }

    Message
    From: Bill K [mailto:reply (AT) billism (DOT) com]
    Sent: 22 June 2006 02:41
    To: plusplus (AT) lists (DOT) mysql.com
    Subject: Re: Query Memory Leak?

    Warren Young wrote:
    Bill K wrote:
    >>

    >I am saying that the MS hotfix does not fix the problem I am having
    >with the query object, like I thought it would.
    >

    , thanks for clearing that up.
    >
    >MS hotfix fixed the memory leak in basic_iostream (used in fstream
    >and stringstream, and others) - which I thought may have fixed the
    >query object since it inherits from ostream. But no.
    >

    , but here's another data point for you: I just tried the test
    program again, and when it crashes it's only using 5 MB. It can't be
    a memory leak.

    Here, it's crashing when the CRT dereferences a null pointer. Why the
    CRT is doing that is the mystery.

    Crashes, eh? It doesn't crash for me. Adding an "if( !(i%1000) ){ std::cout
    << i << endl; }" in the loop, I can see it fly through and finish all
    million loops, using over 80MB of memory at the end, according to Windows
    Task Manager.

    I'll have a look at it more tomorrow.

    Thanks again.
    - Bill
  • No.8 | | 3599 bytes | |

    Matt Dargavel wrote:
    To put my penny's worth in, trying it on Visual Studio Express with mysql++
    2.0.7 shows the memory usage going up to about 65M as Bill describes (mine
    doesn't crash).

    However, turning on memory leak detection
    (
    ) shows no leaks. I even put a
    memory leak in the Query constructor and recompiled mysql My memory leak
    showed up, no other one did.

    I'm not sure why the memory usage goes up so high, but it might have
    something to do with the fact that the runtime and /S don't necessarily
    return any memory freed up to the general pool straight away (for efficiency
    reasons). I also tried increasing the loop size ten fold, and after the
    memory usage got up to about 300M on my machine, Windows started reclaiming
    the memory and it had dropped to 14M by the end.

    I did a similar test, with 100 million loops (no crash). By the time it
    got to the 50 millionth loop, its memory usage was up to 692M, but then
    started dropping, and it maintained below 25M for the rest of the
    iterations, however; the Windows Page File Usage never dropped until the
    program ended. PF Usage was up to 1.9G towards the end. When the program
    ended, the Page File Usage went back down to about 300M where it was
    before the test program started.
    (I am running the "release" executable, not the "debug")

    I've included the revised test prog below (note, you shouldn't be using new
    to assign to the conn object, as that does cause a memory leak).

    Regards,

    Matt.
    --
    #include "stdafx.h"
    #include <stdlib.h>
    #include <crtdbg.h>
    #include "connection.h"
    #include "query.h"

    void test()
    {
    printf("starting test\r\n");

    mysqlpp::Connection conn; // = new mysqlpp::Connection();

    for(int i=0; i <10; i++)
    {
    mysqlpp::Query myQuery = conn.query();
    }

    printf("done test\r\n");
    }

    int _tmain(int argc, _TCHAR* argv[])
    {
    _CrtSetDbgFlag ( _CRTDBG_ALLC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );

    test();

    return 0;
    }

    Message
    From: Bill K [mailto:reply (AT) billism (DOT) com]
    Sent: 22 June 2006 02:41
    To: plusplus (AT) lists (DOT) mysql.com
    Subject: Re: Query Memory Leak?

    Warren Young wrote:

    >Bill K wrote:
    >

    I am saying that the MS hotfix does not fix the problem I am having
    with the query object, like I thought it would.

    >, thanks for clearing that up.
    >>

    >

    MS hotfix fixed the memory leak in basic_iostream (used in fstream
    and stringstream, and others) - which I thought may have fixed the
    query object since it inherits from ostream. But no.

    >, but here's another data point for you: I just tried the test
    >program again, and when it crashes it's only using 5 MB. It can't be
    >a memory leak.
    >>

    >Here, it's crashing when the CRT dereferences a null pointer. Why the
    >CRT is doing that is the mystery.
    >>

    >
    >

    Crashes, eh? It doesn't crash for me. Adding an "if( !(i%1000) ){ std::cout
    << i << endl; }" in the loop, I can see it fly through and finish all
    million loops, using over 80MB of memory at the end, according to Windows
    Task Manager.

    I'll have a look at it more tomorrow.

    Thanks again.

    - Bill
    --
  • No.9 | | 4400 bytes | |

    Windows memory reporting tools generally can't be trusted to show actual
    usage.

    A good example of this is have some program running. Look at the memory
    usage in the task manager. Minimize it, then restore it and you'll see the
    memory usuage drop dramatically on a lot of occasions.

    I think that the task manager type memory tools just look at memory
    allocations and don't take into account releases. This is just a guess.

    But if you are counting on the memory usage stats in the task manager to
    give you an accurate indication of how much memory a program is using,
    you'll be very disappointed.

    6/22/06, Bill K <reply (AT) billism (DOT) comwrote:

    Matt Dargavel wrote:
    To put my penny's worth in, trying it on Visual Studio Express with
    mysql++
    2.0.7 shows the memory usage going up to about 65M as Bill describes
    (mine
    doesn't crash).

    However, turning on memory leak detection
    (

    ) shows no leaks. I even put a
    memory leak in the Query constructor and recompiled mysql My memory
    leak
    showed up, no other one did.

    I'm not sure why the memory usage goes up so high, but it might have
    something to do with the fact that the runtime and /S don't necessarily
    return any memory freed up to the general pool straight away (for
    efficiency
    reasons). I also tried increasing the loop size ten fold, and after the
    memory usage got up to about 300M on my machine, Windows started
    reclaiming
    the memory and it had dropped to 14M by the end.
    --
    I did a similar test, with 100 million loops (no crash). By the time it
    got to the 50 millionth loop, its memory usage was up to 692M, but then
    started dropping, and it maintained below 25M for the rest of the
    iterations, however; the Windows Page File Usage never dropped until the
    program ended. PF Usage was up to 1.9G towards the end. When the program
    ended, the Page File Usage went back down to about 300M where it was
    before the test program started.
    (I am running the "release" executable, not the "debug")

    I've included the revised test prog below (note, you shouldn't be using
    new
    to assign to the conn object, as that does cause a memory leak).

    Regards,

    Matt.
    --
    #include "stdafx.h"
    #include <stdlib.h>
    #include <crtdbg.h>
    #include "connection.h"
    #include "query.h"

    void test()
    {
    printf("starting test\r\n");

    mysqlpp::Connection conn; // = new mysqlpp::Connection();

    for(int i=0; i <10; i++)
    {
    mysqlpp::Query myQuery = conn.query();
    }

    printf("done test\r\n");
    }

    int _tmain(int argc, _TCHAR* argv[])
    {
    _CrtSetDbgFlag ( _CRTDBG_ALLC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );

    test();

    return 0;
    }

    Message
    From: Bill K [mailto:reply (AT) billism (DOT) com]
    Sent: 22 June 2006 02:41
    To: plusplus (AT) lists (DOT) mysql.com
    Subject: Re: Query Memory Leak?

    Warren Young wrote:
    >
    >Bill K wrote:
    >>

    I am saying that the MS hotfix does not fix the problem I am having
    with the query object, like I thought it would.

    >, thanks for clearing that up.
    >>
    >>

    MS hotfix fixed the memory leak in basic_iostream (used in fstream
    and stringstream, and others) - which I thought may have fixed the
    query object since it inherits from ostream. But no.

    >, but here's another data point for you: I just tried the test
    >program again, and when it crashes it's only using 5 MB. It can't be
    >a memory leak.
    >>

    >Here, it's crashing when the CRT dereferences a null pointer. Why the
    >CRT is doing that is the mystery.
    >>
    >>

    >

    Crashes, eh? It doesn't crash for me. Adding an "if( !(i%1000) ){
    std::cout
    << i << endl; }" in the loop, I can see it fly through and finish all
    million loops, using over 80MB of memory at the end, according to
    Windows
    Task Manager.

    I'll have a look at it more tomorrow.

    Thanks again.

    - Bill
  • No.10 | | 760 bytes | |

    Matt Dargavel wrote:
    To put my penny's worth in, trying it on Visual Studio Express with mysql++
    2.0.7 shows the memory usage going up to about 65M as Bill describes (mine
    doesn't crash).

    Thanks for the data point.

    I'm not sure why the memory usage goes up so high, but it might have
    something to do with the fact that the runtime and /S don't necessarily
    return any memory freed up to the general pool straight away (for efficiency
    reasons).

    Yes, memory pool management does behave like you describe, but I don't
    see why the memory manager is requesting that many pages in the first
    place. It should be able to create the query objects reusing the same
    memory block over and over again.
  • No.11 | | 671 bytes | |

    Warren Young wrote:
    I did something very similar, but mod 10,000 instead, and only got a few
    dots before it dereferenced a null pointer.

    further investigation, it was that I was only getting one dot -- when
    i == 0 -- because it was crashing on the first Query object destruction.
    And that was because I was using a mysqlpp.dll built with VC++ 2003
    with my test program built and running under VC++ 2005. D'oh!

    Running Matt Dargavel's test program, running against a consistent set
    of libraries, I also see a 66 MB heap after a million runs. This is
    with the latest repository version of MySQL++, MySQL 4.1.14, and VC++ 2005.
  • No.12 | | 1156 bytes | |

    Bill K wrote:
    Crashes, eh? It doesn't crash for me.

    Well, that's weird enough to deepen my resistance to the idea that the
    problem is in MySQL The version of MySQL++ is one of the few things
    that we absolutely know are common between our two systems.

    I'd be curious to know if:

    1. It behaves exactly the same when you use VC++ 2003 vs. 2005. Any
    difference at all?

    2. Can you try MySQL++ 2.0.0? That would help to pin down the issue of
    whether it's the Query base class change, and not something else.

    3. What version of MySQL are you building against? The problem could be
    down in the C API. I'm using 4.1.14 on my Windows machine. (No
    particular reason I'm on this older versionjust no good reason to
    upgrade yet.)

    Adding an "if( !(i%1000) ){
    std::cout << i << endl; }" in the loop, I can see it fly through and
    finish all million loops, using over 80MB of memory at the end,
    according to Windows Task Manager.

    I did something very similar, but mod 10,000 instead, and only got a few
    dots before it dereferenced a null pointer.
  • No.13 | | 1973 bytes | |

    For what it's worth, I took a memory snapshot using _CrtMemCheckpoint() at
    the start and the end of the test loop. I ran 1000000 cycles of the loop
    and included the test results below. Note the extra 1000001 CRT blocks,
    predictably that number is always 1 more than the number of loops. (For
    those that care I re-ran the test with a couple of loops and dumped out the
    allocated CRT blocks- apparently they were allocated in xiosbase line 514).
    Any way, I think the point is mysql++ isn't to blame- the memory that is
    still allocated after the loop is within the CRT.

    0 bytes in 0 Free Blocks.
    752 bytes in 32 Normal Blocks.
    17261 bytes in 135 CRT Blocks.
    0 bytes in 0 Ignore Blocks.
    0 bytes in 0 Client Blocks.
    Largest number used: 24431 bytes.
    Total allocations: 39657 bytes.
    starting test

    done test
    0 bytes in 0 Free Blocks.
    752 bytes in 32 Normal Blocks.
    4021357 bytes in 1000136 CRT Blocks.
    0 bytes in 0 Ignore Blocks.
    0 bytes in 0 Client Blocks.
    Largest number used: 4022201 bytes.
    Total allocations: 96043753 bytes.

    Message
    From: Warren Young [mailto:mysqlpp (AT) etr-usa (DOT) com]
    Sent: 23 June 2006 06:49
    To: plusplus (AT) lists (DOT) mysql.com
    Subject: Re: Query Memory Leak?

    Warren Young wrote:
    I did something very similar, but mod 10,000 instead, and only got a
    few dots before it dereferenced a null pointer.

    further investigation, it was that I was only getting one dot -- when i
    == 0 -- because it was crashing on the first Query object destruction.
    And that was because I was using a mysqlpp.dll built with VC++ 2003 with
    my test program built and running under VC++ 2005. D'oh!

    Running Matt Dargavel's test program, running against a consistent set of
    libraries, I also see a 66 MB heap after a million runs. This is with the
    latest repository version of MySQL++, MySQL 4.1.14, and VC++ 2005.

Re: Query Object Memory Leak?


max 4000 letters.
Your nickname that display:
In order to stop the spam: 5 + 4 =
QUESTION ON "MYSQL"

EMSDN.COM