Comments on: More allocator data — tcmalloc edition http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/ Ramblings from the mind of Stuart Parmenter Sat, 19 Jul 2008 11:13:50 +0000 http://wordpress.org/?v=MU By: Choco http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-2840 Choco Sat, 09 Feb 2008 16:20:50 +0000 http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-2840 Keep up the good research work pavlov, you have no idea how much i'm waiting for a memory update. Even with the newest version I'm struggling. While other browsers aren't coping better, I would love to see the day I can keep my mouse-rage in check. :D Keep up the good research work pavlov, you have no idea how much i’m waiting for a memory update. Even with the newest version I’m struggling. While other browsers aren’t coping better, I would love to see the day I can keep my mouse-rage in check. :D

]]> By: some http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-2796 some Wed, 30 Jan 2008 13:51:58 +0000 http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-2796 Some more naive questions, I hope you don't mind. AIUI, one cannot easily know what code allocated a piece of memory or is responsible for deallocating it. Thus it is difficult to alter allocations. But isn't there a list of allocations used to prevent double allocation? Can't that be used to copy memory from one location to another then change what is pointed to in the list of allocations? Doesn't the responsible code have a pointer to an item in the the list of allocations rather than the object directly such that this would be transparent? If not, what stops it happening this way? If memory can be moved about, can't unallocatable regions be greatly reduced? Some more naive questions, I hope you don’t mind.
AIUI, one cannot easily know what code allocated a piece of memory or is responsible for deallocating it. Thus it is difficult to alter allocations. But isn’t there a list of allocations used to prevent double allocation? Can’t that be used to copy memory from one location to another then change what is pointed to in the list of allocations? Doesn’t the responsible code have a pointer to an item in the the list of allocations rather than the object directly such that this would be transparent? If not, what stops it happening this way?

If memory can be moved about, can’t unallocatable regions be greatly reduced?

]]> By: DrkShadow http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-2757 DrkShadow Tue, 22 Jan 2008 07:29:01 +0000 http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-2757 Perhaps this is just a show of my naivety, but.. how would a bitmap allocator work? My thought is: store memory as a bitmap. First, it would have zero size. Then, when you allocate, it'll need to grab some memory, and start a linked list -- one item, pointing to the beginning of the allocated memory. When you deallocate that, the item will stay, and the next allocation, if smaller, will use the start of the space and a second item will be added to the linked list pointing to the byte after the end of the first. Free space is broken up as needed, more space is appended to the end, as needed. Surely, the linked list would grow, with varying size fragments and fragments that need to be combine. As things are freed, just throw them on the end of the LL -- later, when the system's not doing anything, sort the list. When grabbing a piece of space, do a binary search to find the closest match to the exact size you need. I envision it working like a filesystem, but with appendable data to the end. ... I'm curious, do you think this is too slow? Surely it would result in minimal fragmentation, though still some. As I said, I've never done anything this low-level, so I'm curious about your thoughts :-) -DrkShadow Perhaps this is just a show of my naivety, but.. how would a bitmap allocator work?

My thought is: store memory as a bitmap. First, it would have zero size. Then, when you allocate, it’ll need to grab some memory, and start a linked list — one item, pointing to the beginning of the allocated memory. When you deallocate that, the item will stay, and the next allocation, if smaller, will use the start of the space and a second item will be added to the linked list pointing to the byte after the end of the first. Free space is broken up as needed, more space is appended to the end, as needed.

Surely, the linked list would grow, with varying size fragments and fragments that need to be combine. As things are freed, just throw them on the end of the LL — later, when the system’s not doing anything, sort the list. When grabbing a piece of space, do a binary search to find the closest match to the exact size you need. I envision it working like a filesystem, but with appendable data to the end.

… I’m curious, do you think this is too slow? Surely it would result in minimal fragmentation, though still some. As I said, I’ve never done anything this low-level, so I’m curious about your thoughts :-)

-DrkShadow

]]> By: pavlov http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-1653 pavlov Wed, 19 Dec 2007 02:16:56 +0000 http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-1653 Peter: We built a tool to log and replay allocations. We're then able to easily plug in new allocators and see the results. We're unable to get good multi-threaded allocation performance results, but we can easily get good fragmentation results. It also lets us test each allocator against the same set of allocations so that we can get comparable results. Peter: We built a tool to log and replay allocations. We’re then able to easily plug in new allocators and see the results. We’re unable to get good multi-threaded allocation performance results, but we can easily get good fragmentation results. It also lets us test each allocator against the same set of allocations so that we can get comparable results.

]]> By: Peter http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-1034 Peter Fri, 07 Dec 2007 17:03:15 +0000 http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-1034 Could you give a few details on how you hook the allocators into the build, so that they are used for the various different ways to allocate memory that seem to be used in the codebase? Could you give a few details on how you hook the allocators into the build, so that they are used for the various different ways to allocate memory that seem to be used in the codebase?

]]> By: pd http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-1009 pd Fri, 07 Dec 2007 10:01:37 +0000 http://blog.pavlov.net/2007/12/06/more-allocator-data-tcmalloc-edition/#comment-1009 thanks for your continued diligence in your work it would be nice if your blog wasn't stuck in the crazy wordpress vertically narrow theme because then we might be able to compare diagrams side by side. Oh well. thanks for your continued diligence in your work

it would be nice if your blog wasn’t stuck in the crazy wordpress vertically narrow theme because then we might be able to compare diagrams side by side. Oh well.

]]>