Comments on: Allocation Data http://blog.pavlov.net/2007/11/13/allocation-data/ Ramblings from the mind of Stuart Parmenter Thu, 21 Aug 2008 07:15:19 +0000 http://wordpress.org/?v=MU By: Steve http://blog.pavlov.net/2007/11/13/allocation-data/#comment-415 Steve Thu, 15 Nov 2007 13:45:26 +0000 http://pavlovdotnet.wordpress.com/2007/11/13/allocation-data/#comment-415 Heavy use of the stack has it's own problems. If a given function uses more than 4096 bytes of stack space, Microsoft's compiler will add code to ensure that subsequent pages of memory are actually present. This makes the code larger and slower. I wouldn't be surprised to see other compilers generating similar code. https://bugzilla.mozilla.org/show_bug.cgi?id=359453 Heavy use of the stack has it’s own problems.

If a given function uses more than 4096 bytes of stack space, Microsoft’s compiler will add code to ensure that subsequent pages of memory are actually present. This makes the code larger and slower.

I wouldn’t be surprised to see other compilers generating similar code.

https://bugzilla.mozilla.org/show_bug.cgi?id=359453

]]> By: Leaks? Memory? We never forgot about you. « pavlov.net http://blog.pavlov.net/2007/11/13/allocation-data/#comment-411 Leaks? Memory? We never forgot about you. « pavlov.net Thu, 15 Nov 2007 09:37:19 +0000 http://pavlovdotnet.wordpress.com/2007/11/13/allocation-data/#comment-411 [...] got a great community and people eager to solve these problems. We’re now equipped with data and ready to fight this [...] [...] got a great community and people eager to solve these problems. We’re now equipped with data and ready to fight this [...]

]]> By: Brendan Eich http://blog.pavlov.net/2007/11/13/allocation-data/#comment-410 Brendan Eich Thu, 15 Nov 2007 06:18:09 +0000 http://pavlovdotnet.wordpress.com/2007/11/13/allocation-data/#comment-410 Benjamin: Using a single large allocation for a double hashtable instead of lots of little ones for a hashtable with chaining is the right answer if the entry size is small enough. See the big comment near the top of jsdhash.h. If there is a user of {js,pl}dhash configuring too large an entry size, you get a warning in DEBUG builds. But you didn't mention entry size, so I'm wondering why you think a single large allocation would fragment worse than a bunch of small ones? The table growth uses malloc and free, not realloc. Ignoring overlarge entry size, a double hashtable is strictly less fragmenting than an open table with chaining. /be Benjamin: Using a single large allocation for a double hashtable instead of lots of little ones for a hashtable with chaining is the right answer if the entry size is small enough. See the big comment near the top of jsdhash.h. If there is a user of {js,pl}dhash configuring too large an entry size, you get a warning in DEBUG builds.

But you didn’t mention entry size, so I’m wondering why you think a single large allocation would fragment worse than a bunch of small ones? The table growth uses malloc and free, not realloc. Ignoring overlarge entry size, a double hashtable is strictly less fragmenting than an open table with chaining.

/be

]]> By: Boris http://blog.pavlov.net/2007/11/13/allocation-data/#comment-390 Boris Wed, 14 Nov 2007 06:15:24 +0000 http://pavlovdotnet.wordpress.com/2007/11/13/allocation-data/#comment-390 COW is not the problem per se, really. The main callers of MutatePrep I see in the logs there, inclusing startup, are: * CDATA section parsing (1500) * AppendASCIItoUTF16 (~1000) * nsCookieService::GetCookieInternal (~1500) * nsHttpHeaderArray::Flatten (~1200) * AppendUTF16toUTF8 (~2200) * AppendUTF8toUTF16 (~800) * nsStandardURL::BuildNormalizedSpec (~3000) * nsHttpHeaderArray::SetHeader (~8600) * nsStandardURL::BuildNormalizedSpec (~3500) * nsCacheService::CreateRequest (~2500) * nsStandardURL::SetRef (~6000) * nsStandardURL::Resolve (~700) * nsStandardURL::GetPath (~400) I sort of have to stop, but unfortunately I've mostly looked at pretty big allocations there (a lot of the URI stuff does things in the 256+ byte range). For strings, it might be nice to do another log focusing on that 8-32 range. The SetRef calls are from LoadBindingDocumentInfo. I wonder why we even hit that code 6000 times... I guess we have a lot of bindings. :( Could we only allocate once in BuildNormalizedSpec somehow? nsCacheService::CreateRequest should at the very least preallocate the right string size. Another option would be to use a key with two strings in it, not just a concatenation of the two strings as a single string key (and hope that if we use two strings they can both share their buffers with caller). We should really make cookies output nsACString, not a char** and see how much of a difference that makes. I expect some. I also wonder whether it's worth precomputing the length we'll need and allocating it all at once instead of going through and doing one append per cookie. Depends on the typical cookie count, I guess... Maybe we should do 9-frame stacks to see the callers of the string conversion functions? COW is not the problem per se, really. The main callers of MutatePrep I see in the logs there, inclusing startup, are:

* CDATA section parsing (1500)
* AppendASCIItoUTF16 (~1000)
* nsCookieService::GetCookieInternal (~1500)
* nsHttpHeaderArray::Flatten (~1200)
* AppendUTF16toUTF8 (~2200)
* AppendUTF8toUTF16 (~800)
* nsStandardURL::BuildNormalizedSpec (~3000)
* nsHttpHeaderArray::SetHeader (~8600)
* nsStandardURL::BuildNormalizedSpec (~3500)
* nsCacheService::CreateRequest (~2500)
* nsStandardURL::SetRef (~6000)
* nsStandardURL::Resolve (~700)
* nsStandardURL::GetPath (~400)

I sort of have to stop, but unfortunately I’ve mostly looked at pretty big allocations there (a lot of the URI stuff does things in the 256+ byte range). For strings, it might be nice to do another log focusing on that 8-32 range.

The SetRef calls are from LoadBindingDocumentInfo. I wonder why we even hit that code 6000 times… I guess we have a lot of bindings. :(
Could we only allocate once in BuildNormalizedSpec somehow?

nsCacheService::CreateRequest should at the very least preallocate the right string size. Another option would be to use a key with two strings in it, not just a concatenation of the two strings as a single string key (and hope that if we use two strings they can both share their buffers with caller).

We should really make cookies output nsACString, not a char** and see how much of a difference that makes. I expect some. I also wonder whether it’s worth precomputing the length we’ll need and allocating it all at once instead of going through and doing one append per cookie. Depends on the typical cookie count, I guess…

Maybe we should do 9-frame stacks to see the callers of the string conversion functions?

]]> By: Stuart Parmenter http://blog.pavlov.net/2007/11/13/allocation-data/#comment-389 Stuart Parmenter Wed, 14 Nov 2007 05:30:41 +0000 http://pavlovdotnet.wordpress.com/2007/11/13/allocation-data/#comment-389 Benjamin: I can get data for larger allocations. I'll do that tomorrow. Benjamin: I can get data for larger allocations. I’ll do that tomorrow.

]]> By: Benjamin Smedberg http://blog.pavlov.net/2007/11/13/allocation-data/#comment-387 Benjamin Smedberg Wed, 14 Nov 2007 04:38:47 +0000 http://pavlovdotnet.wordpress.com/2007/11/13/allocation-data/#comment-387 Pav, is it possible to get the information for large allocs also? At the moz2 meetup today we were pondering whether pldhash/jsdhashtable allocations (which can be very large blocks of memory) could be causing unnecessary fragmentation, and whether a plhash (chaining) system might be better for certain allocation types. Pav, is it possible to get the information for large allocs also? At the moz2 meetup today we were pondering whether pldhash/jsdhashtable allocations (which can be very large blocks of memory) could be causing unnecessary fragmentation, and whether a plhash (chaining) system might be better for certain allocation types.

]]> By: darin http://blog.pavlov.net/2007/11/13/allocation-data/#comment-386 darin Wed, 14 Nov 2007 02:50:32 +0000 http://pavlovdotnet.wordpress.com/2007/11/13/allocation-data/#comment-386 perhaps CoW strings are causing more harm than good. perhaps CoW strings are causing more harm than good.

]]>