It Seemed Like a Good Idea at the Time

My latest idea for the svn --diff-cmd can be seen here.

Let's hope it does better than this guys' great idea!

SVN --invoke-diff-cmd branch shipped

So what happened that took the time?

1. I forgot *everything* (and had to re-re-re-read the manuals)

2. My main laptop died (RIP frogburg, it croaked it's last) and so I had to build everything back from scratch on my remaining machine.  I also have a new desktop that I had so far avoided to set up, which I did at the same time.

3.  Because the trunk had changed quite a bit, my feeble attempts to merge initially produced a veritable mess that added to the general confusion.  It didn't help that this was my first merge that required actual work, nor did it help that I had a complete new install and for a few days didn't realise that kdiff3 wasn't installed, and thought that my code was broken (which it was, but not in this place) and so, I of course kept looking in the illuminated place for the door keys.

4. Somewhere in the new trunk code, a miracle happens.   That is, it picks up the actual command line input en passant 3 functions deep whilst it reads the config file, and I had to break out gdb to search the haystack.

5.  Every time I get stuck (or bored) I read a page of the TCP/IP book.  I read a lot of TCP/IP book.

Anyway, here is the new branch, this time as a patch.

Rewriting my SVN branch

My invoke-diff-cmd branch cannot simply be merged, and it looks like I need to rewire things quite a bit.

So, apart from learning python for my new project (more of that in a soon to be typed blogpost!) and playing around with pylint and reading up on HTML5 after I had some great suggestions on how to improve my 'Fail to Plan, Plan to Fail' application,  I'm still trying to figure out what is actually going on in the new svn diff code, and the way I'm currently feeling about this all is probably best described as in this video below:

Then again, I'm pretty sure I'll be dining on curried chicken quite soon ;-)

SVN Book diff idea

It would be nice to have SVN book versions on the website that have a differing portions of the text in blue for changes, and green for new additions and red for now obsolete parts.

It would make re-reading the book for a new version much easier for old hands, you could immediately spot everything you need to reconsider.

So, if I upgrade from 1.6 to 1.7, I would like to have the normal 1.7 book, the 1.7/1.6 'blue/green' version and the 1.6/1.7 'red' version.

(blue & green might not be possible to automate, but even just the blue version would be very useful, or perhaps it's even possible to have this all in one version.)

--invoke-diff3-cmd branch ready for review

Finally the feature --invoke-diff3-cmd for Subversion is ready -- 23 files in 8 directories, 86 code snippets for a total 782 lines of code, 53 log message components plus a BRANCH-README file.

Making sure I stay lucky

After nearly losing all my work yesterday, I've made this little script to run in the Emacs compile buffer before the make command.  This runs svn diff every time I compile and puts the result into a separate directory.

#!/bin/sh
CURRENT_DIR=`pwd` 
DATE_MARKER=$(date +%Y%m%d%T) 
SECURITY_PATCH_DIR=/home/g/security_patches/ 
WHAT=`echo "${CURRENT_DIR##*/}"`
EXT=.patch
SECURITY_DIFF=${SECURITY_PATCH_DIR}${WHAT}${DATE_MARKER}${EXT}
SVN=/home/g/InstalledSVN1.9/subversion
$SVN diff > $SECURITY_DIFF 

Why checkpoint commits are a good idea...

[12:11] <gbb> idea:  [c/sh]ould merge automagically take a security diff before merging, just to protect the clumsy from their foolish mistakes?  (I merged my working copy instead of my test dir in gdb, oww, repairable but ugh...)
[12:12] <stsp> gbb, you mean something like merge --dry-run?
[12:13] <stsp> with such things, it's tricky: if we do a check by default, one set of people will complain about the additional overhead, while the other group will complain about the need to specify an option to get the safe behaviour
[12:13] <gbb> hrm dry-run would have helped but not protected me from merging over all my work.  ;(  (luckily my emacs is my handbag and most of those files were still open)
[12:14] <stsp> i think it's the type of mistake that you just need to learn to avoid over time
[12:15] <stsp> if we had checkpoints implemented, you could checkpoint your working copy before a merge, to keep a record of the pre-merge state
[12:15] <stsp> i think that would be a nice solution
[12:16] <stsp> merge could automatically create such a checkpoint, in fact!
[12:16] <gbb> not sure what check points are.  but currently you can only revert to pristine, not to pre-merge
[12:16] <stsp> see http://subversion.tigris.org/issues/show_bug.cgi?id=3626
[12:17] <stsp> or shelving, even
[12:19] <gbb> that would be the ticket :)
[13:20] <ASFBot> gbg * r1540299 (subversion/branches/invoke-diff-merge-feature/ (14 files in 6 directories)) http://svn.apache.org/r1540299 : On the invoke-diff-cmd-feature branch: Checkpoint commit (ignore, trivial).
[13:32] <gbb> as Mr. Twain remarked: "A man who carries a cat by the tail learns something he can learn in no other way."
[13:32] <stsp> :)

My day/week/month is made!

Julian Foad said:

"Your BRANCH-README file <http://svn.apache.org/repos/asf/subversion/branches/invoke-diff-cmd-feature/BRANCH-README> is a fantastic window into the branch, much more detailed and helpful than the rest of us ever write. I can easily see exactly what the whole feature does, how it's structured, and what your plans are, without having been paying close attention. I'm mentioning this mainly so anybody else contemplating taking a look at this knows this is a good place to start reading. And I'm pleased to see you have taken care to adhere to our coding style. "

It's the most wonderful compliment I've heard in years :)

{And, to be honest, writing this BRANCH-README helped me a lot in getting things done and keep me focused on what I was (and am) trying to do.  It was danielsh who suggested that I should write one (ironically I initially omitted it, despite fixing a part of the hacking guide entry for this most excellent advice.)}

Don't delay, write _your_ BRANCH-README today!

New invoke-diff-cmd feature: --svn-cfg-file

Wrote this feature yesterday, all in one go.

Had the idea whilst pondering what to do with the --invoke-diff3-cmd interactive user dialogue.

Maybe the new agate necklace magic really works?  Heh.

How to use svn diff to create a list of visitable files with line numbers in the Emacs compile buffer

Here is how to use the svn diff output within the emacs compile buffer to generate a handy list of files with line numbers that you can click on to visit every bit of code you've changed:

M-x compile

svn diff |\
perl -ne '/Index: (.*)$/ && ($name = $1);\
/^@@.*\+(\d+),/ && print "./$name:$1:diff\n";'  

The svn diff Project: First part done, review passed!

The --invoke-diff-cmd part of the svn diff project has been completed (bar one possible improvement).

The BRANCH-README file explains it all.

Of course, there has to be a fitting video for this post:



Now for the second part --- the invoke-diff3-cmd section!

Merging the trunk to branch, the lazy way

#!/bin/sh                                                                                                                                 
${SVN_INSTALLATION?"Please set the environment variable SVN_INSTALLATION."}
SVN=$SVN_INSTALLATION
BRANCH_DIR=`pwd`
tmp=`$SVN info 2>&1 | grep SVN_ERR_WC_NOT_WORKING_COPY`
if [ -n "$tmp" ]; then
    echo "Error: The branch directory does not contain a SVN repository." >&2;
    exit;
fi
TRUNK_DIR=$1
REL_DIR=$2
if [ -z "$TRUNK_DIR" ]; then
    echo "Usage: mergeBranch <trunk directory>" >&2;
    exit;
fi
if [ ! -d "$TRUNK_DIR" ]; then
    echo "Error: Trunk directory not found." >&2;
    exit;
fi
cd $TRUNK_DIR
tmp='$SVN info 2>&1 | grep SVN_ERR_WC_NOT_WORKING_COPY'
if [ -z "$tmp" ]; then
    echo "Error: The trunk directory does not contain a SVN repository." 1>&2;
    cd $BRANCH_DIR
    exit;
fi
cd $TRUNK_DIR;
$SVN up
tmp=`$SVN info | grep "Last Changed Rev:"`
TRUNK=${tmp##*: }
cd $BRANCH_DIR
tmp=`$SVN info | grep "Last Changed Rev:"`
BRANCH=${tmp##*: }
$SVN up
WHAT=${BRANCH_DIR##*/}
echo "On the $WHAT branch: trunk revision $TRUNK merged into branch revision $BRANCH." > log.mesg
$SVN merge $REL_DIR

The handrolled version of *_create_custom_diff_cmd

This may or may not be a good idea, but if you enjoy this sort of thing, my latest effort at code comedy is here.

This is probably not the final version because copying Mel the Programmer's strategies is almost never a good idea, but, it just had to be done at least once ;-)



Update 30mins later:  Hah, already busted.  I take away any special end characters but if what I'm handling is not a delimiter, then I end up stealing user's chars.   Added a new test for this, now I have to fix it =)

(and why is it that every time, no matter how long I stare at my effort beforehand, the moment I commit code, I find a whopping show stopper a few minutes later? :-D )

Update on update:  Fixed.  And I should have coded that in the first place, the repetition should have made me suspicious, and I did think about it at the time, but thought that it can't be done differently for some reason.   Next time, I'll lay off the catnip before I commit.

Even left over lunches don't come free

I keep meaning to write something about the weird experience that re-learning to code can be.  I wrote the following to a friend of mine, and I think I could rewrite it, but well, it says all that needs to be explained I guess.

My svn jutsu is pitiful, because I literally had no time yet to learn it and also, no cause (as in need-it-right-now project I need to do-right-now).

See, if you know more than a beginner but less than a newbie, the question is, what part of the universe of confusion to you de-entropy first?

If you want to do A but suddenly codus interruptus hits and you first need to figure out B, which depends on knowing C that requires you check out D... E... Zzzz(repeat ad nauseum) the best decision is to patz and just get the job done without getting sidetracked into studying properly. 

The OPW placement was great, but put a lot of pressure on me to produce something semi-decent(and, I was (and am) slooooow) and so, I didn't have much time to read into things too deeply, plus, what can I say, coding is a habit and restarting after 10 years isn't all that easy -- you need to rebuild your coding short term memory space/riffs to be reliable, fluid and have enough slots, and that takes time.   

They say you never forget how to ride a bike, that's true, but when you try after a long time, you get doms in your muscles, wobble the wheels and possibly fall off at least once. 

Is it easier to re-learn than to start from scratch?  I don't think so, it's harder in quite a few ways, because you tend to be overly critical without having the matching sound judgment.  Even left over lunches don't come free :)

Reading through a manual for general understanding is very different to needing to get from A-B, and I had a lot of A-B'ing to do.  

In fact, I didn't use svn at all (other than to stash code) until I put a lot of code in the wrong tree and had to use diff to get my work out *blush* , so I managed to contribute to svn (the gtest project) using svn the first time when I made the branch where I stashed the completed job.  Plus I had commit access and no idea how to test this before I make a (potential) mess, so it had to work the first time.  Ostrich egg omlette, anyone?

I had kittens over the merge -- I let my branch rot somewhat.  So, I somehow ended up using kdiff at first and got spammed by 30 unresolved merges I had noooo idea what they were.  In my defense, making a large svn branch with lots of code snippets in many files and letting rot (I had no idea this could be a problem btw) is perhaps not the optimal introduction to basic svn usage... 

Eventually... I r.e.m.e.m.b.e.r.e.d that svn itself has a merge program and that kdiff is actually trying to do the merge instead of just showing me the code (I'm slow I know, LOL)  and it did it all bar a couple of simple line hierarchy queries.  

And, to top it all, this one segment of code that drives kdiff is actually the *one{and only}* thing I understand in svn (because I coded it, plus I tried to code a fix in for the merge label bug which died because it broke something else, and there could only be one), but did I make the connection?  Of course not, doh.   

Maybe I should've bought a dunce cap instead of a Stetson.

----  Well, this is what she wrote. ----

The SVN_DBG story

Whilst talking on IRC about the exotic design motivation for the --invoke-diff-cmd that I offered, it was realised that I was blissfully ignorant as to how (and why) printf() works, how to escape '%' characters, and what a variadic function is.

As is happens, SVN_DBG which is a test suite friendly SVN debugging tool needed documenting, and it happens to be a very good demonstration of how variadic functions work(esp, if you take time to read the GNU C Preprocessor manual) ,  and so, danielsh kindly tasked me with writing a small precis that introduces the SVN_DBG tool to new devs in the HACKING GUIDE.

As a nice bonus, it turns out that some time ago, I read over a small, but vital clue of how the ANSI C OO stuff works.  I tend to read 'over' some stuff I do not understand in order to keep the universe of confusion minimised, and filed 'va_list' as some funky OO type linked list.   So I was quite chuffed at finally figuring out how the 'and here, a miracle occurs' part works -- it uses a variadic function.

This of course calls for an appropriate video:

Diff Project part 1 --invoke-diff-cmd -- Take 2 (interlude)

Because I couldn't decide what to do about the delimiter-prefix issue, where %f1 could potentially cause issues for script users, I made the shape of the switches OS dependent (%f1...%fn for unix, ---f1... for windows) and also added a feature that allows people to change that prefix, so you could say svn diff --invoke-diff-cmd="# diff -u #f1 #f2" if you want, and use '$', '-' or even '---' etc.

Not such a good idea(it was fun to code though) because changing the user interface across platforms is disorienting. 

I also rewrote svn_io_create_custom_diff_cmd() to use svn_string_buf_t instead of the string.h functions I used previously.

However, for now, I'm feeling somewhat strfry(Gabriela) and so I'll take a little break from --invoke-diff-cmd to let the concepts I've learned settle, and instead have a go at writing a short introduction to the SVN_DBG tool for the HACKING guide -- watch this space :)

Yesterday was my last OPW day...

...and do take note that the door I opened in the previous post was a way-in and not a way-out :)

Without OPW I wouldn't have taken up coding again --- In fact I'm very surprised at myself that I am (nearly almost!) a geekess again.  But so it is indeed, I have be given partial commit access to Subversion, and some of the work I produced was good enough to be added to the project.

What's more, with almost every failure I learned a lot -- the Subversion devs are a fount of knowledge and very generous with useful advice and constructive, direct feedback.

I plan to continue to contribute to Subversion --- I only just started to make inroads into producing functional code, so now I can finally get to actually be productive.

I would like to thank the OPW team and Elego for providing me with both opportunity and motivation.

A Map of the Diff Problem

was gifted to me yesterday by Julian Foad, and curiously so,  it is making the difficult much easier by making it really big :)

Actually, none of those things are 'difficult' as such --- what made it hard was that I had no idea of the true dimension of the problem.  And so...:

Diff is difficult.

After much staring, stepping and grepping, I finally produced a solution for the partial problem defined in (Issue 2074, "--extensions '' doesn't work.) which is part of (Issue 2044, Fully customizable external diff invocations).

It's not bad(it works for very small values of functional), but also not very good, since it is rather agricultural and I'm not sure how to handle deprecation, but, it's a start.

However it only deals with one part of the problem and not with the issue of giving the user a wormhole to call any diff function they want to use, however they want to.

To be continued... ;-)

Completing the Gtest Build System OPW Assignment

Most important and useful knowledge I gained:

Explicit debugging  -- keeping a list of ideas, testing theories etc, keep your mind free of clutter for thinking and enabling you to casually return to even the most complex problem after a break without loss of continuity.

Most useful tool I learned about:

The 'remake' debugger for Makefiles, it provides a useful calling stack trace.
 

Things I relearned:

'make', how to document, shell scripting, simple C and C++.

New things I learned a little bit about: 

m4 macros, autoconf, python, configure, C++ and C compilation directives, gtest, libtool and the concept of shared libraries.
  

So what did I build for the Subversion OPW project?

Gtest Addition: 

1.  a patch for adding gtest as an option to the build system
2.  documentation for the gtest addition 
3.  an example test for the gtest addition 

And since I had spent all this time trying to understand what is happening in the build system, I documented what I think is going on, with the happy result that the Subversion project gets some documentation for the build system and I get my understanding double checked:

Documentation for the Subversion Build System: 

  4. Overview
  5. 3 walkthroughs for adding various types of components
  6. list of files and their purpose for the build system
  7. Walkthrough of the main files' contents

Next up: Diff and Freeze!

PS.: I know we're 8 weeks into the OPW project, but, 2 weeks of that I was ill, and so, I'm actually really in week 6.

My mouse battery lasted exactly for the duration of the project!  (and my kitteh looks just like that guy too :) )