Comments on: Windows Low Fragmentation Heap

By: “Vlad and analysis of dtrace was used” « pavlov.net

“Vlad and analysis of dtrace was used” « pavlov.net — Tue, 04 Dec 2007 22:41:47 +0000

[...] compare the Windows standard heap, the Windows Low-Fragmentation Heap, nedmalloc and jemalloc. I’ve posted pictures from the two Windows allocators before, but [...]

By: malloc replacements? « pavlov.net

malloc replacements? « pavlov.net — Wed, 21 Nov 2007 21:22:36 +0000

[...] such as tcmalloc, nedmalloc, Hoard, and jemalloc. Also native platform specific ones such as the Windows low-fragmentation heap. I’m having to dig in to some of their internals to pull out the data that we need which is [...]

By: pavlov

pavlov — Wed, 14 Nov 2007 16:56:08 +0000

Peter: I’ve tried using HeapCompact() but I don’t see a lot of big wins from it. I should probably do some more tests though.

By: Peter

Peter — Wed, 14 Nov 2007 16:04:03 +0000

Did you try to see if _heapmin() had any effect? I made some experiments on OS/2 last night, adding that to the cycle collector and it had at least a bit of an effect.

I also experimented with OS/2 system APIs (DosAllocMem/DosFreeMem) instead of malloc for allocations of large buffers (e.g. the pixel buffers in cairo). This basically allocates memory outside the C heaps and when freed gets released to the system immediately. Made a huge difference! Not sure if there is something like that on the Win or the other platforms.

By: Stuart Parmenter

Stuart Parmenter — Wed, 14 Nov 2007 00:02:53 +0000

I’ve posted some Windows Minefield beta 1 builds on my blog with the low-fragmentation heap enabled. Please try them out and give feedback on that post.

By: Lerc

Lerc — Tue, 13 Nov 2007 23:28:24 +0000

Is there any facility to tag memory allocations with info from the call stack so you see where they came from?

combining something like that with these pretty heapmap images so you could hover and see what caused the allocation would help shed light on the issue.

There is the possibilty of heap defragmentation ofr any objects that allocate memory for their own use and don’t pass pointers to others. Only one reference would have to be maintained. You’d need to have those objects select an alternate allocation system. or just add a ReallocatePrivateMemory method to base objects.

The lifetime hint comment by max in the previous blogpost is another good idea.

both of those will involve delving into the actual allocating code, but I suspect the quick fix of an alternate heap manager won’t be the best solution.

By: Anonymous

Anonymous — Tue, 13 Nov 2007 23:16:24 +0000

take a look at these papers on pools and fragmentation:

http://slashdot.org/comments.pl?sid=120623&cid=10160124

http://www.cs.umass.edu/~emery/pubs/berger-oopsla2002.pdf

By: Anonymous

Anonymous — Tue, 13 Nov 2007 23:07:15 +0000

Use Doug Lea’s malloc. It’s far better than Windows Low Frag heap.

http://g.oswego.edu/

By: ThomasK

ThomasK — Tue, 13 Nov 2007 16:00:43 +0000

Good work, Stuart.
This could help a lot of other projects (longer running processes in non-GC environments), so please keep your analysis abstract enough that things != FF could profit too.

By: Mirza

Mirza — Tue, 13 Nov 2007 10:37:18 +0000

Hello, I had the same problem: I needed to load millions of street names (as std::string) to memory at once. Albeit size of combined strings is some 100 MB, program easily used all of my 1GB of RAM and crashed. Therefore, I created simple class “StringContainer” (300 lines of C++ code) that stores all strings as one huge string plus additional binary tree for fast access. You can StrHandle h = container.put(std:string) and later use this handle to get string back std::string s = container.get(h). Now all strings use as little memory as combined size of all strings (also, multiple identical strings are stored only once!). Get method is using special binary tree to retrieve string quickly, but tree nodes are stored in std::vector, so tree itself also occupy single chunk of memory! Removing single string from this container is not possible, but it is not problem in way I am using it. For example you can setup one container for each HTML page or each Java Script so container lives as long as its domain is alive. If you want this class, send me your mail to mirza@seznam.cz.