The Short Tourist's Guide to Memory Allocaters

There's ancient malloc().

And then there is glibc's malloc():

 (illustration not to scale)

General:

A very good place to start is Mr. Golick's intro: Memory Allocators 101, then there is Mr. Luu's C tutorial, or even Doug Lea's dlmalloc.  But, that's only the beginning of a very good, very long story.

Strategies (the very meat of this post):

There is an interesting survey on the different ways of writing a memory allocator, it's very informative (and well-written),  and useful to know in general (even though it's a little old).

More Specific:

For even more pleasing (and productive) confusion, there's TCMalloc, Lockless and a short paper that compares glibc to the former and the latter.

Also notable is jemalloc, and they have a nice video too.  Here's a neat synopsis of this toy.

Profiling:

TCMalloc has profiling inbuilt (via perftools), Lockless has some hooks and jemalloc has a heap profiler.  If you want to check what glibc malloc gets up to, there's malloc_count, or, you can knit your own, using malloc hooks and struct mallinfo.

The Lockless site has some neat info on how to profile without a profiler, using gcc. (their article collection is also quite a good read [with a very special recommendation for the rather nifty 'Obfuscated Rogue' proggy <3 !]).

Happy alloc()-ing!

Ps.: Stuff I idly wonder about:

None of the above seem to take performance profiles of stuff as they run and then automate the conclusions.

There is probably a very good reason for it (there always is :-) --- but, I wonder what would happen if we had a wrapper that that 'kept score' and did test runs on the inputs it knows about and then swaps in different memory allocators on the next run, depending on what profiles match that particular usage pattern the best.

In other news, here's my sure-fire way of how to beat the backgammon world champion:   you roll a double-6 to run home, and with the final checker, you put their 15th checker on the bar.  EASY! ;-)

+_+_+_+_+_+_+_+_+_+_+_+_+_+_+_+_+

Obligatory New Years' Eve video... Ms Sophie is celebrating her birthday party.  The only problem is that all her guests deceased 25 years ago, and so, the butler has to stand in... (scroll forward to 2:24 to start the actual sketch, but, in full length with the German intro, it still makes a nice ambience, even if you do not speak German ;-)  the rest is, as they say, without words!

Happy New year everyone!!!

A gcc joke

$ gcc --help=common

...

  -pedantic                   This switch lacks documentation

  -pedantic-errors            Like -pedantic but issue them as errors

SVN Book diff idea

It would be nice to have SVN book versions on the website that have a differing portions of the text in blue for changes, and green for new additions and red for now obsolete parts.

It would make re-reading the book for a new version much easier for old hands, you could immediately spot everything you need to reconsider.

So, if I upgrade from 1.6 to 1.7, I would like to have the normal 1.7 book, the 1.7/1.6 'blue/green' version and the 1.6/1.7 'red' version.

(blue & green might not be possible to automate, but even just the blue version would be very useful, or perhaps it's even possible to have this all in one version.)

The svn diff Project: First part done, review passed!

The --invoke-diff-cmd part of the svn diff project has been completed (bar one possible improvement).

The BRANCH-README file explains it all.

Of course, there has to be a fitting video for this post:



Now for the second part --- the invoke-diff3-cmd section!

I found find. Now I'm lost.

I used find before quite a lot, but it never occurred to me to type 'man find'.

Well, I just did and found the software equivalent to this:




30 pages of man file.  Is there anything find doesn't do?

I'm going to have to grow a second brain I think.

I'm having a bash

I am currently working on my BRANCH_README file and I'm finding that my bash skills are seriously lacking :(

Sadly, I'm no longer a mud wizard, so the trusty old

   patch Gabriela barbarian_bash_skill +MAX
                                                             
spell is not longer an option and I will have to read a tutorial along with the bash reference instead.

UPDATE:  The above tuts are good, but this tutorial is amazing and has many examples, and in-depth explanations as to why and how things work.  
                                                          
However, it's sort of traditional that one has to 'bash with a bonecrushing sound':

  ... and this album is about as Heavy Metal as Cool Jazz can get.

The SVN_DBG story

Whilst talking on IRC about the exotic design motivation for the --invoke-diff-cmd that I offered, it was realised that I was blissfully ignorant as to how (and why) printf() works, how to escape '%' characters, and what a variadic function is.

As is happens, SVN_DBG which is a test suite friendly SVN debugging tool needed documenting, and it happens to be a very good demonstration of how variadic functions work(esp, if you take time to read the GNU C Preprocessor manual) ,  and so, danielsh kindly tasked me with writing a small precis that introduces the SVN_DBG tool to new devs in the HACKING GUIDE.

As a nice bonus, it turns out that some time ago, I read over a small, but vital clue of how the ANSI C OO stuff works.  I tend to read 'over' some stuff I do not understand in order to keep the universe of confusion minimised, and filed 'va_list' as some funky OO type linked list.   So I was quite chuffed at finally figuring out how the 'and here, a miracle occurs' part works -- it uses a variadic function.

This of course calls for an appropriate video:

Diff Project part 1 --invoke-diff-cmd -- Take 2 (interlude)

Because I couldn't decide what to do about the delimiter-prefix issue, where %f1 could potentially cause issues for script users, I made the shape of the switches OS dependent (%f1...%fn for unix, ---f1... for windows) and also added a feature that allows people to change that prefix, so you could say svn diff --invoke-diff-cmd="# diff -u #f1 #f2" if you want, and use '$', '-' or even '---' etc.

Not such a good idea(it was fun to code though) because changing the user interface across platforms is disorienting. 

I also rewrote svn_io_create_custom_diff_cmd() to use svn_string_buf_t instead of the string.h functions I used previously.

However, for now, I'm feeling somewhat strfry(Gabriela) and so I'll take a little break from --invoke-diff-cmd to let the concepts I've learned settle, and instead have a go at writing a short introduction to the SVN_DBG tool for the HACKING guide -- watch this space :)

If at first you don't succeed...

I tidied up the "comments" which are actually defines (what was I thinking?) in this file because, well, they looked messy :)

However, in this case, the position of the defines actually reflected the structure of the data -- something I have never seen before, but it's indeed very handy, once you know what you're looking at.

So, the patch then morphed from 'fixing' what was not broken to adding information about what is going on :)

Since we didn't want Doxygen to pick up that comment since it's only useful when you view the code, I used a /* plain comment */ here.

Completing the Gtest Build System OPW Assignment

Most important and useful knowledge I gained:

Explicit debugging  -- keeping a list of ideas, testing theories etc, keep your mind free of clutter for thinking and enabling you to casually return to even the most complex problem after a break without loss of continuity.

Most useful tool I learned about:

The 'remake' debugger for Makefiles, it provides a useful calling stack trace.
 

Things I relearned:

'make', how to document, shell scripting, simple C and C++.

New things I learned a little bit about: 

m4 macros, autoconf, python, configure, C++ and C compilation directives, gtest, libtool and the concept of shared libraries.
  

So what did I build for the Subversion OPW project?

Gtest Addition: 

1.  a patch for adding gtest as an option to the build system
2.  documentation for the gtest addition 
3.  an example test for the gtest addition 

And since I had spent all this time trying to understand what is happening in the build system, I documented what I think is going on, with the happy result that the Subversion project gets some documentation for the build system and I get my understanding double checked:

Documentation for the Subversion Build System: 

  4. Overview
  5. 3 walkthroughs for adding various types of components
  6. list of files and their purpose for the build system
  7. Walkthrough of the main files' contents

Next up: Diff and Freeze!

PS.: I know we're 8 weeks into the OPW project, but, 2 weeks of that I was ill, and so, I'm actually really in week 6.

My mouse battery lasted exactly for the duration of the project!  (and my kitteh looks just like that guy too :) )

Just another lazy Sunday

Today is dedicated to reading all about svn properties, and because it's cold and I want a fire, I'm in the garden, cooking pound chicken on the grill, copying this gentleman's excellent example:

The svn dev mailing list is a lot of fun at the moment, the [RFC] Build System Documentation discussion yields a lot of surprises, some neat puzzles and many 'aha!' experiences, and I'm learning a lot :)