Comments on: malloc replacements? http://blog.pavlov.net/2007/11/21/malloc-replacements/ Ramblings from the mind of Stuart Parmenter Thu, 21 Aug 2008 07:18:48 +0000 http://wordpress.org/?v=MU By: Steve Chapel http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-763 Steve Chapel Mon, 26 Nov 2007 15:57:22 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-763 It's not that Firefox never gets above 200 MB. I regularly see it reach 500 MB when I have many tabs open with pages that use different plugins. I usually close most of my tabs on a regular basis, so memory use then drops to under 200 MB. If I run the Browser Mem Buster Test, which always keeps ten tabs open, memory use stays under 200 MB the entire time and drops to about 100 MB after closing all tabs but one. In my experience, fragmentation and memory leaks combined never seem to be taking more than around 100 MB total, even after a week of use. I suppose I need to ask: How many tabs are we talking about? Dozens? Hundreds? I suppose I also need to ask: On what OS are you seeing this extreme fragmentation, because I never see it with Windows XP. Let's get specific. It’s not that Firefox never gets above 200 MB. I regularly see it reach 500 MB when I have many tabs open with pages that use different plugins. I usually close most of my tabs on a regular basis, so memory use then drops to under 200 MB. If I run the Browser Mem Buster Test, which always keeps ten tabs open, memory use stays under 200 MB the entire time and drops to about 100 MB after closing all tabs but one. In my experience, fragmentation and memory leaks combined never seem to be taking more than around 100 MB total, even after a week of use.

I suppose I need to ask: How many tabs are we talking about? Dozens? Hundreds? I suppose I also need to ask: On what OS are you seeing this extreme fragmentation, because I never see it with Windows XP. Let’s get specific.

]]> By: pavlov http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-753 pavlov Mon, 26 Nov 2007 05:55:42 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-753 Steve: I disagree. Given the growth patterns of our fragmentation the app is going to keep taking up more and more and more memory over time without actually leaking. If you never get Firefox 2 over 200mb then I suspect you aren't loading very many windows or tabs at once. I can get Firefox 2 over 200mb _really_ quickly. Steve: I disagree. Given the growth patterns of our fragmentation the app is going to keep taking up more and more and more memory over time without actually leaking. If you never get Firefox 2 over 200mb then I suspect you aren’t loading very many windows or tabs at once. I can get Firefox 2 over 200mb _really_ quickly.

]]> By: Steve Chapel http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-750 Steve Chapel Mon, 26 Nov 2007 03:40:17 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-750 mizra says "For example, people are crying that Firefox, open for 2 days and doing nothing, eats 1GB or more RAM. If you change alloc, that number might be, say, 500MB (albeit I don’t think so) … but thats still bad!" It sounds like those people are suffering from extreme memory leaks. After I've been using Firefox 2 for a week, it uses only 200 MB of RAM. After three days of using Firefox 3 beta 1, it's still using only 126 MB. I can't see how changing the allocator is going to reduce memory usage by hundreds of megabytes. It won't do anything noticeable for people having the most severe problems. mizra says “For example, people are crying that Firefox, open for 2 days and doing nothing, eats 1GB or more RAM. If you change alloc, that number might be, say, 500MB (albeit I don’t think so) … but thats still bad!”

It sounds like those people are suffering from extreme memory leaks. After I’ve been using Firefox 2 for a week, it uses only 200 MB of RAM. After three days of using Firefox 3 beta 1, it’s still using only 126 MB. I can’t see how changing the allocator is going to reduce memory usage by hundreds of megabytes. It won’t do anything noticeable for people having the most severe problems.

]]> By: Yusuf Goolamabbas http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-674 Yusuf Goolamabbas Fri, 23 Nov 2007 13:46:18 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-674 Stuart, Forgot to mention that Andrew Tridgell of rsync/samba fame has also written talloc http://talloc.samba.org/ http://arjen-lentz.livejournal.com/60924.html http://www.algorithm.com.au/blog/files/jan-2006.php http://ozlabs.org/~rusty/index.cgi/tech/2006-02-17.html Stuart, Forgot to mention that Andrew Tridgell of rsync/samba fame has also written talloc

http://talloc.samba.org/
http://arjen-lentz.livejournal.com/60924.html
http://www.algorithm.com.au/blog/files/jan-2006.php
http://ozlabs.org/~rusty/index.cgi/tech/2006-02-17.html

]]> By: mirza http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-672 mirza Fri, 23 Nov 2007 13:26:47 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-672 hm, html distroyed my example, I will change braces: "std::vector{MyClass} is one big heap alloc, no fragmentation. std::vector{MyClass*} is many little allocs (fragmentation alert!)." hm, html distroyed my example, I will change braces:

“std::vector{MyClass} is one big heap alloc, no fragmentation. std::vector{MyClass*} is many little allocs (fragmentation alert!).”

]]> By: mirza http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-671 mirza Fri, 23 Nov 2007 13:22:32 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-671 Each alloc algorithm is trying to get best deal between low fragmentation and speed. Native c++ lib on each platform choose one that should be best for general usage. I think that solution to fragmented memory should be in elimination of allocations of *single* objects on heap. For example: std::vector is one big heap alloc, no fragmentation. std::vector is many little allocs (fragmentation alert!). If MyClass contains string attribute(s), fragmentation alert again! In such case MyClass should contain only (int) handle to string and strings should be stored together in string container (one implementation of which I sent to mike2leonard@netscape.com, but there are several available on internet). String container is, again, only two allocs (if done right), no mather how many strings, and therefore no fragmentation. So, what I am trying to say that if C++ is used in fragmentation-aware way, you don't need to change default alloc algorithm. If you allocate lots of individual objects on heap, changing alloc will not help you *much*. For example, people are crying that Firefox, open for 2 days and doing nothing, eats 1GB or more RAM. If you change alloc, that number might be, say, 500MB (albeit I don't think so) ... but thats still bad! Not to mention that less-fragmenting alloc will be slower by definition. Each alloc algorithm is trying to get best deal between low fragmentation and speed. Native c++ lib on each platform choose one that should be best for general usage. I think that solution to fragmented memory should be in elimination of allocations of *single* objects on heap. For example: std::vector is one big heap alloc, no fragmentation. std::vector is many little allocs (fragmentation alert!). If MyClass contains string attribute(s), fragmentation alert again! In such case MyClass should contain only (int) handle to string and strings should be stored together in string container (one implementation of which I sent to mike2leonard@netscape.com, but there are several available on internet). String container is, again, only two allocs (if done right), no mather how many strings, and therefore no fragmentation.

So, what I am trying to say that if C++ is used in fragmentation-aware way, you don’t need to change default alloc algorithm. If you allocate lots of individual objects on heap, changing alloc will not help you *much*. For example, people are crying that Firefox, open for 2 days and doing nothing, eats 1GB or more RAM. If you change alloc, that number might be, say, 500MB (albeit I don’t think so) … but thats still bad! Not to mention that less-fragmenting alloc will be slower by definition.

]]> By: adamo http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-634 adamo Thu, 22 Nov 2007 09:16:49 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-634 <a href="http://www.research.att.com/~gsf/download/" rel="nofollow">vmalloc</a> vmalloc

]]> By: Kelly http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-627 Kelly Thu, 22 Nov 2007 05:45:32 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-627 This isn't a drop-in replacement for malloc, but could probably be adapted to do that relatively easily though the algorithms sound similar to tcmalloc's: http://library.gnome.org/devel/glib/unstable/glib-Memory-Slices.html http://mail.gnome.org/archives/gtk-devel-list/2005-December/msg00031.html Also based on some of the comments here: http://live.gnome.org/MemoryReduction it might be possible to improve tcmalloc to better release mmapped blocks at least on linux and other unix systems. Right now it uses madvise(MADV_DONTNEED) which based on manpages for madvise probably just means the OS can swap it out rather than free it permanently, but I could be wrong. This isn’t a drop-in replacement for malloc, but could probably be adapted to do that relatively easily though the algorithms sound similar to tcmalloc’s:

http://library.gnome.org/devel/glib/unstable/glib-Memory-Slices.html
http://mail.gnome.org/archives/gtk-devel-list/2005-December/msg00031.html

Also based on some of the comments here: http://live.gnome.org/MemoryReduction
it might be possible to improve tcmalloc to better release mmapped blocks at least on linux and other unix systems. Right now it uses madvise(MADV_DONTNEED) which based on manpages for madvise probably just means the OS can swap it out rather than free it permanently, but I could be wrong.

]]> By: pd http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-617 pd Thu, 22 Nov 2007 02:04:01 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-617 No idea if these are any good, just thought I'd try to help with a bit of Googling: A Memory Allocator http://g.oswego.edu/dl/html/malloc.html A Comparison of Memory Allocators in Multiprocessors http://developers.sun.com/solaris/articles/multiproc/multiproc.html Hoard http://www.hoard.org/ http://www.cs.umass.edu/~emery/hoard/ mmap http://en.wikipedia.org/wiki/Mmap The Memory Fragmentation Problem: Solved? (1997) http://citeseer.ist.psu.edu/johnstone97memory.html Keep up the great work No idea if these are any good, just thought I’d try to help with a bit of Googling:

A Memory Allocator
http://g.oswego.edu/dl/html/malloc.html

A Comparison of Memory Allocators in Multiprocessors
http://developers.sun.com/solaris/articles/multiproc/multiproc.html

Hoard
http://www.hoard.org/
http://www.cs.umass.edu/~emery/hoard/

mmap
http://en.wikipedia.org/wiki/Mmap

The Memory Fragmentation Problem: Solved? (1997)
http://citeseer.ist.psu.edu/johnstone97memory.html

Keep up the great work

]]> By: Yusuf Goolamabbas http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-614 Yusuf Goolamabbas Thu, 22 Nov 2007 01:41:53 +0000 http://blog.pavlov.net/2007/11/21/malloc-replacements/#comment-614 portable libumem: https://labs.omniti.com/trac/portableumem portable libumem:

https://labs.omniti.com/trac/portableumem

]]>