Query Object Memory Leak?
13 answers - 638 bytes -

Sorry, looks like I somehow put this in the wrong thread. (thread:
CLIENT_FUND_RWS) Maybe a mod. can delete that post?
I am developing code that needs to create query objects often. There
seems to be a memory leak - memory not being released by the query
object when it goes out of scope. Is this a known issue, or am I missing
something? I have tried this with VC7.1 and VC8.
code to recreate problem:
// Begin Code
mysqlpp::Connection conn = new mysqlpp::Connection();
for(int i=0; i <1000000; i++)
{
mysqlpp::Query myQuery = conn.query();
}
// End Code
Thanks!
- Bill K.
No.1 | | 1204 bytes |
| 
Bill K wrote:
Maybe a mod. can delete that post?
Ahahaha. This is a mail list. I can't delete emails from other
people's inboxes. Theoretically I could remove it from the archives,
but most of the damage has already been done. Live and learn, dude. :)
I am developing code that needs to create query objects often. There
seems to be a memory leak - memory not being released by the query
object when it goes out of scope.
I think you've misdiagnosed the problem. I ran your test code on a
Linux box, and the amount of memory stays static throughout the
program's run time. Between this and what I see in the call stack
window when after it crashes, I think it's more likely that the problem
is somewhere down in VC++'s runtime library.
If you're motivated enough, an interesting test would be to revert Query
back to the old style stream mechanism. (See the v2.0.0 ChangeLog
entry.) We couldn't make that reversion permanent, because the old
stream style is obsolete, but it would tell us whether I'm right about
where the problem is. If the symptom changes, it's in the runtime library.
No.2 | | 2169 bytes |
| 
Warren Young wrote:
Bill K wrote:
>Maybe a mod. can delete that post?
>
Ahahaha. This is a mail list. I can't delete emails from other
people's inboxes. Theoretically I could remove it from the archives,
but most of the damage has already been done. Live and learn, dude. :)
I know this is a mailing list, I was referring to deleting from the
archive. But ok, you can't; thanks for answering my question. =)
>I am developing code that needs to create query objects often. There
>seems to be a memory leak - memory not being released by the query
>object when it goes out of scope.
>
I think you've misdiagnosed the problem.
I never said I had diagnosed or otherwise narrowed down the problem.
I ran your test code on a Linux box, and the amount of memory stays
static throughout the program's run time. Between this and what I see
in the call stack window when after it crashes, I think it's more
likely that the problem is somewhere down in VC++'s runtime library.
If you're motivated enough, an interesting test would be to revert
Query back to the old style stream mechanism. (See the v2.0.0
ChangeLog entry.) We couldn't make that reversion permanent, because
the old stream style is obsolete, but it would tell us whether I'm
right about where the problem is. If the symptom changes, it's in the
runtime library.
I built MySQL++ 1.7.40, and rebuilt my test project with that library.
The query object has no leak, tested with the same code. So, it may be
with MS's runtime library like you say.
At first I thought the problem was with MS's known std::basic_iostream
memory leaking problem. ( info here:
) However, I obtained a non-public hotfix from MS that supposedly fixed
that problem, which it looks like it did, when I test out
std::stringstream and std::iostream objects in a similar loop.
I will look into it some more. Anyone else having similar issues?
Thank you for your help.
- Bill K.
No.3 | | 671 bytes |
| 
Bill K wrote:
I am saying that the MS hotfix does not fix the
problem I am having with the query object, like I thought it would.
, thanks for clearing that up.
MS hotfix fixed the memory leak in basic_iostream (used in fstream and
stringstream, and others) - which I thought may have fixed the query
object since it inherits from ostream. But no.
, but here's another data point for you: I just tried the test
program again, and when it crashes it's only using 5 MB. It can't be a
memory leak.
Here, it's crashing when the CRT dereferences a null pointer. Why the
CRT is doing that is the mystery.
No.4 | | 1579 bytes |
| 
Bill K wrote:
I know this is a mailing list, I was referring to deleting from the
archive.
Sorry if I annoyed you, but the very fact that you asked suggested that
you are more used to web forums, corporate email systems, or other
captive systems. a public mailing list, you have to assume that once
the message is gone, it's gone.
But ok, you can't
Indeed. :) I "own" this project, but I don't have any special access to
the mail servers. You'd have as much chance of success asking MySQL
Inc.'s server admins directly to remove the message as I would.
>I think you've misdiagnosed the problem.
I never said I had diagnosed or otherwise narrowed down the problem.
Uh, okay, I think you're arguing about semantics here. You offered a
hypothesis as to where the problem is, and I told you that I thought you
were wrong, giving evidence to back up my assertion. Is that
distinction in phrasing really worth fighting for?
At first I thought the problem was with MS's known std::basic_iostream
memory leaking problem. ( info here:
) However, I obtained a non-public hotfix from MS that supposedly fixed
that problem, which it looks like it did, when I test out
std::stringstream and std::iostream objects in a similar loop.
I'm not sure I understand. Are you saying that the problem is indeed
fixed when you apply the patch, or that it only fixes the case of
creating stream objects but not Query objects?
No.5 | | 2242 bytes |
| 
Warren Young wrote:
Bill K wrote:
>I know this is a mailing list, I was referring to deleting from the
>archive.
>
Sorry if I annoyed you, but the very fact that you asked suggested
that you are more used to web forums, corporate email systems, or
other captive systems. a public mailing list, you have to assume
that once the message is gone, it's gone.
But ok, you can't
Indeed. :) I "own" this project, but I don't have any special access
to the mail servers. You'd have as much chance of success asking
MySQL Inc.'s server admins directly to remove the message as I would.
I think you've misdiagnosed the problem.
>I never said I had diagnosed or otherwise narrowed down the problem.
>
Uh, okay, I think you're arguing about semantics here. You offered a
hypothesis as to where the problem is, and I told you that I thought
you were wrong, giving evidence to back up my assertion. Is that
distinction in phrasing really worth fighting for?
>
>At first I thought the problem was with MS's known
>std::basic_iostream memory leaking problem. ( info here:
>
>) However, I obtained a non-public hotfix from MS that supposedly
>fixed that problem, which it looks like it did, when I test out
>std::stringstream and std::iostream objects in a similar loop.
>
I'm not sure I understand. Are you saying that the problem is indeed
fixed when you apply the patch, or that it only fixes the case of
creating stream objects but not Query objects?
Sorry for the confusion. I am saying that the MS hotfix does not fix the
problem I am having with the query object, like I thought it would. The
MS hotfix fixed the memory leak in basic_iostream (used in fstream and
stringstream, and others) - which I thought may have fixed the query
object since it inherits from ostream. But no. I only mentioned the MS
hotfix so anyone aware of the memory leak in MS's CRT would know I have
a patch for it, so that shouldn't be the problem.
Thanks
- Bill K.
No.6 | | 1161 bytes |
| 
Warren Young wrote:
Bill K wrote:
>>
>I am saying that the MS hotfix does not fix the problem I am having
>with the query object, like I thought it would.
>
, thanks for clearing that up.
>
>MS hotfix fixed the memory leak in basic_iostream (used in fstream
>and stringstream, and others) - which I thought may have fixed the
>query object since it inherits from ostream. But no.
>
, but here's another data point for you: I just tried the test
program again, and when it crashes it's only using 5 MB. It can't be
a memory leak.
Here, it's crashing when the CRT dereferences a null pointer. Why the
CRT is doing that is the mystery.
Crashes, eh? It doesn't crash for me. Adding an "if( !(i%1000) ){
std::cout << i << endl; }" in the loop, I can see it fly through and
finish all million loops, using over 80MB of memory at the end,
according to Windows Task Manager.
I'll have a look at it more tomorrow.
Thanks again.
- Bill
No.7 | | 2892 bytes |
| 
To put my penny's worth in, trying it on Visual Studio Express with mysql++
2.0.7 shows the memory usage going up to about 65M as Bill describes (mine
doesn't crash).
However, turning on memory leak detection
(
) shows no leaks. I even put a
memory leak in the Query constructor and recompiled mysql My memory leak
showed up, no other one did.
I'm not sure why the memory usage goes up so high, but it might have
something to do with the fact that the runtime and /S don't necessarily
return any memory freed up to the general pool straight away (for efficiency
reasons). I also tried increasing the loop size ten fold, and after the
memory usage got up to about 300M on my machine, Windows started reclaiming
the memory and it had dropped to 14M by the end.
I've included the revised test prog below (note, you shouldn't be using new
to assign to the conn object, as that does cause a memory leak).
Regards,
Matt.
#include "stdafx.h"
#include <stdlib.h>
#include <crtdbg.h>
#include "connection.h"
#include "query.h"
void test()
{
printf("starting test\r\n");
mysqlpp::Connection conn; // = new mysqlpp::Connection();
for(int i=0; i <10; i++)
{
mysqlpp::Query myQuery = conn.query();
}
printf("done test\r\n");
}
int _tmain(int argc, _TCHAR* argv[])
{
_CrtSetDbgFlag ( _CRTDBG_ALLC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );
test();
return 0;
}
Message
From: Bill K [mailto:reply (AT) billism (DOT) com]
Sent: 22 June 2006 02:41
To: plusplus (AT) lists (DOT) mysql.com
Subject: Re: Query Memory Leak?
Warren Young wrote:
Bill K wrote:
>>
>I am saying that the MS hotfix does not fix the problem I am having
>with the query object, like I thought it would.
>
, thanks for clearing that up.
>
>MS hotfix fixed the memory leak in basic_iostream (used in fstream
>and stringstream, and others) - which I thought may have fixed the
>query object since it inherits from ostream. But no.
>
, but here's another data point for you: I just tried the test
program again, and when it crashes it's only using 5 MB. It can't be
a memory leak.
Here, it's crashing when the CRT dereferences a null pointer. Why the
CRT is doing that is the mystery.
Crashes, eh? It doesn't crash for me. Adding an "if( !(i%1000) ){ std::cout
<< i << endl; }" in the loop, I can see it fly through and finish all
million loops, using over 80MB of memory at the end, according to Windows
Task Manager.
I'll have a look at it more tomorrow.
Thanks again.
- Bill
No.8 | | 3599 bytes |
| 
Matt Dargavel wrote:
To put my penny's worth in, trying it on Visual Studio Express with mysql++
2.0.7 shows the memory usage going up to about 65M as Bill describes (mine
doesn't crash).
However, turning on memory leak detection
(
) shows no leaks. I even put a
memory leak in the Query constructor and recompiled mysql My memory leak
showed up, no other one did.
I'm not sure why the memory usage goes up so high, but it might have
something to do with the fact that the runtime and /S don't necessarily
return any memory freed up to the general pool straight away (for efficiency
reasons). I also tried increasing the loop size ten fold, and after the
memory usage got up to about 300M on my machine, Windows started reclaiming
the memory and it had dropped to 14M by the end.
I did a similar test, with 100 million loops (no crash). By the time it
got to the 50 millionth loop, its memory usage was up to 692M, but then
started dropping, and it maintained below 25M for the rest of the
iterations, however; the Windows Page File Usage never dropped until the
program ended. PF Usage was up to 1.9G towards the end. When the program
ended, the Page File Usage went back down to about 300M where it was
before the test program started.
(I am running the "release" executable, not the "debug")
I've included the revised test prog below (note, you shouldn't be using new
to assign to the conn object, as that does cause a memory leak).
Regards,
Matt.
--
#include "stdafx.h"
#include <stdlib.h>
#include <crtdbg.h>
#include "connection.h"
#include "query.h"
void test()
{
printf("starting test\r\n");
mysqlpp::Connection conn; // = new mysqlpp::Connection();
for(int i=0; i <10; i++)
{
mysqlpp::Query myQuery = conn.query();
}
printf("done test\r\n");
}
int _tmain(int argc, _TCHAR* argv[])
{
_CrtSetDbgFlag ( _CRTDBG_ALLC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );
test();
return 0;
}
Message
From: Bill K [mailto:reply (AT) billism (DOT) com]
Sent: 22 June 2006 02:41
To: plusplus (AT) lists (DOT) mysql.com
Subject: Re: Query Memory Leak?
Warren Young wrote:
>Bill K wrote:
>
I am saying that the MS hotfix does not fix the problem I am having
with the query object, like I thought it would.
>, thanks for clearing that up.
>>
>
MS hotfix fixed the memory leak in basic_iostream (used in fstream
and stringstream, and others) - which I thought may have fixed the
query object since it inherits from ostream. But no.
>, but here's another data point for you: I just tried the test
>program again, and when it crashes it's only using 5 MB. It can't be
>a memory leak.
>>
>Here, it's crashing when the CRT dereferences a null pointer. Why the
>CRT is doing that is the mystery.
>>
>
>
Crashes, eh? It doesn't crash for me. Adding an "if( !(i%1000) ){ std::cout
<< i << endl; }" in the loop, I can see it fly through and finish all
million loops, using over 80MB of memory at the end, according to Windows
Task Manager.
I'll have a look at it more tomorrow.
Thanks again.
- Bill
--
No.9 | | 4400 bytes |
| 
Windows memory reporting tools generally can't be trusted to show actual
usage.
A good example of this is have some program running. Look at the memory
usage in the task manager. Minimize it, then restore it and you'll see the
memory usuage drop dramatically on a lot of occasions.
I think that the task manager type memory tools just look at memory
allocations and don't take into account releases. This is just a guess.
But if you are counting on the memory usage stats in the task manager to
give you an accurate indication of how much memory a program is using,
you'll be very disappointed.
6/22/06, Bill K <reply (AT) billism (DOT) comwrote:
Matt Dargavel wrote:
To put my penny's worth in, trying it on Visual Studio Express with
mysql++
2.0.7 shows the memory usage going up to about 65M as Bill describes
(mine
doesn't crash).
However, turning on memory leak detection
(
) shows no leaks. I even put a
memory leak in the Query constructor and recompiled mysql My memory
leak
showed up, no other one did.
I'm not sure why the memory usage goes up so high, but it might have
something to do with the fact that the runtime and /S don't necessarily
return any memory freed up to the general pool straight away (for
efficiency
reasons). I also tried increasing the loop size ten fold, and after the
memory usage got up to about 300M on my machine, Windows started
reclaiming
the memory and it had dropped to 14M by the end.
--
I did a similar test, with 100 million loops (no crash). By the time it
got to the 50 millionth loop, its memory usage was up to 692M, but then
started dropping, and it maintained below 25M for the rest of the
iterations, however; the Windows Page File Usage never dropped until the
program ended. PF Usage was up to 1.9G towards the end. When the program
ended, the Page File Usage went back down to about 300M where it was
before the test program started.
(I am running the "release" executable, not the "debug")
I've included the revised test prog below (note, you shouldn't be using
new
to assign to the conn object, as that does cause a memory leak).
Regards,
Matt.
--
#include "stdafx.h"
#include <stdlib.h>
#include <crtdbg.h>
#include "connection.h"
#include "query.h"
void test()
{
printf("starting test\r\n");
mysqlpp::Connection conn; // = new mysqlpp::Connection();
for(int i=0; i <10; i++)
{
mysqlpp::Query myQuery = conn.query();
}
printf("done test\r\n");
}
int _tmain(int argc, _TCHAR* argv[])
{
_CrtSetDbgFlag ( _CRTDBG_ALLC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );
test();
return 0;
}
Message
From: Bill K [mailto:reply (AT) billism (DOT) com]
Sent: 22 June 2006 02:41
To: plusplus (AT) lists (DOT) mysql.com
Subject: Re: Query Memory Leak?
Warren Young wrote:
>
>Bill K wrote:
>>
I am saying that the MS hotfix does not fix the problem I am having
with the query object, like I thought it would.
>, thanks for clearing that up.
>>
>>
MS hotfix fixed the memory leak in basic_iostream (used in fstream
and stringstream, and others) - which I thought may have fixed the
query object since it inherits from ostream. But no.
>, but here's another data point for you: I just tried the test
>program again, and when it crashes it's only using 5 MB. It can't be
>a memory leak.
>>
>Here, it's crashing when the CRT dereferences a null pointer. Why the
>CRT is doing that is the mystery.
>>
>>
>
Crashes, eh? It doesn't crash for me. Adding an "if( !(i%1000) ){
std::cout
<< i << endl; }" in the loop, I can see it fly through and finish all
million loops, using over 80MB of memory at the end, according to
Windows
Task Manager.
I'll have a look at it more tomorrow.
Thanks again.
- Bill
No.10 | | 760 bytes |
| 
Matt Dargavel wrote:
To put my penny's worth in, trying it on Visual Studio Express with mysql++
2.0.7 shows the memory usage going up to about 65M as Bill describes (mine
doesn't crash).
Thanks for the data point.
I'm not sure why the memory usage goes up so high, but it might have
something to do with the fact that the runtime and /S don't necessarily
return any memory freed up to the general pool straight away (for efficiency
reasons).
Yes, memory pool management does behave like you describe, but I don't
see why the memory manager is requesting that many pages in the first
place. It should be able to create the query objects reusing the same
memory block over and over again.
No.11 | | 671 bytes |
| 
Warren Young wrote:
I did something very similar, but mod 10,000 instead, and only got a few
dots before it dereferenced a null pointer.
further investigation, it was that I was only getting one dot -- when
i == 0 -- because it was crashing on the first Query object destruction.
And that was because I was using a mysqlpp.dll built with VC++ 2003
with my test program built and running under VC++ 2005. D'oh!
Running Matt Dargavel's test program, running against a consistent set
of libraries, I also see a 66 MB heap after a million runs. This is
with the latest repository version of MySQL++, MySQL 4.1.14, and VC++ 2005.
No.12 | | 1156 bytes |
| 
Bill K wrote:
Crashes, eh? It doesn't crash for me.
Well, that's weird enough to deepen my resistance to the idea that the
problem is in MySQL The version of MySQL++ is one of the few things
that we absolutely know are common between our two systems.
I'd be curious to know if:
1. It behaves exactly the same when you use VC++ 2003 vs. 2005. Any
difference at all?
2. Can you try MySQL++ 2.0.0? That would help to pin down the issue of
whether it's the Query base class change, and not something else.
3. What version of MySQL are you building against? The problem could be
down in the C API. I'm using 4.1.14 on my Windows machine. (No
particular reason I'm on this older versionjust no good reason to
upgrade yet.)
Adding an "if( !(i%1000) ){
std::cout << i << endl; }" in the loop, I can see it fly through and
finish all million loops, using over 80MB of memory at the end,
according to Windows Task Manager.
I did something very similar, but mod 10,000 instead, and only got a few
dots before it dereferenced a null pointer.
No.13 | | 1973 bytes |
| 
For what it's worth, I took a memory snapshot using _CrtMemCheckpoint() at
the start and the end of the test loop. I ran 1000000 cycles of the loop
and included the test results below. Note the extra 1000001 CRT blocks,
predictably that number is always 1 more than the number of loops. (For
those that care I re-ran the test with a couple of loops and dumped out the
allocated CRT blocks- apparently they were allocated in xiosbase line 514).
Any way, I think the point is mysql++ isn't to blame- the memory that is
still allocated after the loop is within the CRT.
0 bytes in 0 Free Blocks.
752 bytes in 32 Normal Blocks.
17261 bytes in 135 CRT Blocks.
0 bytes in 0 Ignore Blocks.
0 bytes in 0 Client Blocks.
Largest number used: 24431 bytes.
Total allocations: 39657 bytes.
starting test
done test
0 bytes in 0 Free Blocks.
752 bytes in 32 Normal Blocks.
4021357 bytes in 1000136 CRT Blocks.
0 bytes in 0 Ignore Blocks.
0 bytes in 0 Client Blocks.
Largest number used: 4022201 bytes.
Total allocations: 96043753 bytes.
Message
From: Warren Young [mailto:mysqlpp (AT) etr-usa (DOT) com]
Sent: 23 June 2006 06:49
To: plusplus (AT) lists (DOT) mysql.com
Subject: Re: Query Memory Leak?
Warren Young wrote:
I did something very similar, but mod 10,000 instead, and only got a
few dots before it dereferenced a null pointer.
further investigation, it was that I was only getting one dot -- when i
== 0 -- because it was crashing on the first Query object destruction.
And that was because I was using a mysqlpp.dll built with VC++ 2003 with
my test program built and running under VC++ 2005. D'oh!
Running Matt Dargavel's test program, running against a consistent set of
libraries, I also see a 66 MB heap after a million runs. This is with the
latest repository version of MySQL++, MySQL 4.1.14, and VC++ 2005.