jemalloc builds

Since just before the holidays, Jason and I have been working on getting jemalloc ported to Windows, Mac and Linux as well as integrated in to our build system.  Each platform has its own set of challenges, with Windows being the most exciting.  I tried several approaches including dynamically patching over the C runtime allocation functions (malloc, free, etc) in memory, but kept running in to issues with the allocations the CRT does during startup.  Things like putenv() realloc and free allocations that were done before we had the chance to patch in.  I talked to Benjamin about replacing the CRT and he did a bunch of work to get a CRT building with jemalloc.  It turned out not to be very difficult — Microsoft includes the source as part of Visual Studio Professional (sadly, you can’t redistribute it).  I had to make various changes to the init functions to make sure we could use CRITICAL_SETIONs and thread local storage during malloc initialization.  Was a bit of a pain to make sure the malloc initialization code didn’t call things that needed to allocate but eventually I got everything working.  The result: A Windows jemalloc build.  The sunspider JS test looks to run about 5% faster.  You’ll notice that these builds take up a bit more memory initially but they should level out around 80mb.  There is plenty of tuning left to do, but I’m curious to hear how these run for people after running for a long time.

Linux builds are stable but I don’t have one handy.  Mac builds run for a little while before they crash, but the crash looks like one I fixed on Windows so I expect to have those early next week.

17 thoughts on “jemalloc builds

  1. Ferdinand

    I tried your build and it feels faster but doesn’t seem to make a real difference in memory usage. I have loaded heavy pages like Google Reader, Gmail and Digg to get the memory usage as high as possible. Right now Windows Taskmanager(vista 32bit) reports 151.652K with only this page open.

    Reply
  2. Nils

    For Windows: Can’t you just specify your own entry point which will do the patching and afterwards call the CRT entry-point?
    Something along the lines of:
    void __cdecl mozmainCRTStartup( void ) {
    /* … patching … */
    mainCRTStartUp(); // or WinMainCRTStartup ?
    }
    linking it with /ENTRY:”mozmainCRTStartup”

    Reply
  3. pavlov Post author

    Ferdinand: Thanks for the data point. We’ll keep tuning it.

    sdwilsh: We can distribute a script to patch things I think. It might just be a little delicate.

    Nils: I tried doing that but in order to start up the allocator we need a couple things that don’t allocate to be initialized in the CRT. Bad ordering problem ;(

    Reply
  4. Schrep

    On Vista loading/closing the same 50 web pages:

    Today’s trunk: 67MB
    JEMalloc: 64.5MB

    So a definite small improvement. Note these tests arn’t deterministic since ad content can vary. Once you get me mac builds I can run longer and more repeatable tests.

    5% on SunSpider is nothing to sneeze at neither!

    Reply
  5. Roger

    It keeps crashing with some MSVC++ error dialog after some time (several hours), even when I’m not using it at the moment (crashed a while ago while I was watching a movie).
    I’m using my FF3beta profile, so take with a grain of salt.

    Reply
  6. Pingback: jemalloc now on the trunk « pavlov.net

  7. Pola

    I’ve tried to compile te mozdev19.dll.
    After fixing the missing file unhandld.obj; I am still unable to compile it.

    I got
    fatal error LNK1181: cannot open input file ‘build\intel\dll_obj\asctime.obj’
    and
    NMAKE : fatal error U1073: don’t know how to make ‘build\intel\dll_obj\_setargv.obj’

    Do you have any sugestion.

    Thanks in advance.

    Reply
  8. Pingback: jemalloc on trunk — linux edition « pavlov.net

  9. Pingback: Just Browsing » Browsers and Commoditization

  10. Pingback: Firefox 3 Memory Usage « pavlov.net

  11. Pingback: Mark Finkle’s Weblog » XULRunner 1.9 RC3

  12. Pingback: Mark Finkle’s Weblog » Fennec Windows Mobile - Update

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s