--invoke-diff3-cmd branch ready for review

Finally the feature --invoke-diff3-cmd for Subversion is ready -- 23 files in 8 directories, 86 code snippets for a total 782 lines of code, 53 log message components plus a BRANCH-README file.

The svn diff Project: First part done, review passed!

The --invoke-diff-cmd part of the svn diff project has been completed (bar one possible improvement).

The BRANCH-README file explains it all.

Of course, there has to be a fitting video for this post:



Now for the second part --- the invoke-diff3-cmd section!

Even left over lunches don't come free

I keep meaning to write something about the weird experience that re-learning to code can be.  I wrote the following to a friend of mine, and I think I could rewrite it, but well, it says all that needs to be explained I guess.

My svn jutsu is pitiful, because I literally had no time yet to learn it and also, no cause (as in need-it-right-now project I need to do-right-now).

See, if you know more than a beginner but less than a newbie, the question is, what part of the universe of confusion to you de-entropy first?

If you want to do A but suddenly codus interruptus hits and you first need to figure out B, which depends on knowing C that requires you check out D... E... Zzzz(repeat ad nauseum) the best decision is to patz and just get the job done without getting sidetracked into studying properly. 

The OPW placement was great, but put a lot of pressure on me to produce something semi-decent(and, I was (and am) slooooow) and so, I didn't have much time to read into things too deeply, plus, what can I say, coding is a habit and restarting after 10 years isn't all that easy -- you need to rebuild your coding short term memory space/riffs to be reliable, fluid and have enough slots, and that takes time.   

They say you never forget how to ride a bike, that's true, but when you try after a long time, you get doms in your muscles, wobble the wheels and possibly fall off at least once. 

Is it easier to re-learn than to start from scratch?  I don't think so, it's harder in quite a few ways, because you tend to be overly critical without having the matching sound judgment.  Even left over lunches don't come free :)

Reading through a manual for general understanding is very different to needing to get from A-B, and I had a lot of A-B'ing to do.  

In fact, I didn't use svn at all (other than to stash code) until I put a lot of code in the wrong tree and had to use diff to get my work out *blush* , so I managed to contribute to svn (the gtest project) using svn the first time when I made the branch where I stashed the completed job.  Plus I had commit access and no idea how to test this before I make a (potential) mess, so it had to work the first time.  Ostrich egg omlette, anyone?

I had kittens over the merge -- I let my branch rot somewhat.  So, I somehow ended up using kdiff at first and got spammed by 30 unresolved merges I had noooo idea what they were.  In my defense, making a large svn branch with lots of code snippets in many files and letting rot (I had no idea this could be a problem btw) is perhaps not the optimal introduction to basic svn usage... 

Eventually... I r.e.m.e.m.b.e.r.e.d that svn itself has a merge program and that kdiff is actually trying to do the merge instead of just showing me the code (I'm slow I know, LOL)  and it did it all bar a couple of simple line hierarchy queries.  

And, to top it all, this one segment of code that drives kdiff is actually the *one{and only}* thing I understand in svn (because I coded it, plus I tried to code a fix in for the merge label bug which died because it broke something else, and there could only be one), but did I make the connection?  Of course not, doh.   

Maybe I should've bought a dunce cap instead of a Stetson.

----  Well, this is what she wrote. ----

Diff Project part 1 --invoke-diff-cmd -- Take 2 (interlude)

Because I couldn't decide what to do about the delimiter-prefix issue, where %f1 could potentially cause issues for script users, I made the shape of the switches OS dependent (%f1...%fn for unix, ---f1... for windows) and also added a feature that allows people to change that prefix, so you could say svn diff --invoke-diff-cmd="# diff -u #f1 #f2" if you want, and use '$', '-' or even '---' etc.

Not such a good idea(it was fun to code though) because changing the user interface across platforms is disorienting. 

I also rewrote svn_io_create_custom_diff_cmd() to use svn_string_buf_t instead of the string.h functions I used previously.

However, for now, I'm feeling somewhat strfry(Gabriela) and so I'll take a little break from --invoke-diff-cmd to let the concepts I've learned settle, and instead have a go at writing a short introduction to the SVN_DBG tool for the HACKING guide -- watch this space :)

Yesterday was my last OPW day...

...and do take note that the door I opened in the previous post was a way-in and not a way-out :)

Without OPW I wouldn't have taken up coding again --- In fact I'm very surprised at myself that I am (nearly almost!) a geekess again.  But so it is indeed, I have be given partial commit access to Subversion, and some of the work I produced was good enough to be added to the project.

What's more, with almost every failure I learned a lot -- the Subversion devs are a fount of knowledge and very generous with useful advice and constructive, direct feedback.

I plan to continue to contribute to Subversion --- I only just started to make inroads into producing functional code, so now I can finally get to actually be productive.

I would like to thank the OPW team and Elego for providing me with both opportunity and motivation.

Diff is difficult.

After much staring, stepping and grepping, I finally produced a solution for the partial problem defined in (Issue 2074, "--extensions '' doesn't work.) which is part of (Issue 2044, Fully customizable external diff invocations).

It's not bad(it works for very small values of functional), but also not very good, since it is rather agricultural and I'm not sure how to handle deprecation, but, it's a start.

However it only deals with one part of the problem and not with the issue of giving the user a wormhole to call any diff function they want to use, however they want to.

To be continued... ;-)

Completing the Gtest Build System OPW Assignment

Most important and useful knowledge I gained:

Explicit debugging  -- keeping a list of ideas, testing theories etc, keep your mind free of clutter for thinking and enabling you to casually return to even the most complex problem after a break without loss of continuity.

Most useful tool I learned about:

The 'remake' debugger for Makefiles, it provides a useful calling stack trace.
 

Things I relearned:

'make', how to document, shell scripting, simple C and C++.

New things I learned a little bit about: 

m4 macros, autoconf, python, configure, C++ and C compilation directives, gtest, libtool and the concept of shared libraries.
  

So what did I build for the Subversion OPW project?

Gtest Addition: 

1.  a patch for adding gtest as an option to the build system
2.  documentation for the gtest addition 
3.  an example test for the gtest addition 

And since I had spent all this time trying to understand what is happening in the build system, I documented what I think is going on, with the happy result that the Subversion project gets some documentation for the build system and I get my understanding double checked:

Documentation for the Subversion Build System: 

  4. Overview
  5. 3 walkthroughs for adding various types of components
  6. list of files and their purpose for the build system
  7. Walkthrough of the main files' contents

Next up: Diff and Freeze!

PS.: I know we're 8 weeks into the OPW project, but, 2 weeks of that I was ill, and so, I'm actually really in week 6.

My mouse battery lasted exactly for the duration of the project!  (and my kitteh looks just like that guy too :) )

Build System Gtest addition, next attempt

Resistance was futile... :)

Without the excellent tip to keep a log of debug activities, that I picked up from the Udacity debugging course, I probably would have given up or would still be mud wresting this thing.  (I will blog about this course next!)

The man who teaches this most excellent course is totally correct: the moment I started keeping notes instead of jamming my short term memory full of  Vogon poetry, my brain was  freed up to do some thinking, and I was able to take a break  to snatch a few moments of real life without losing my spot.

It's amazing how much work a few lines of trivial looking code can make, and mind you, whilst it 'works' currently I had to cheat a little :)

Here is my (partial, since I forgot to keep it up at some points) log, in all it's gory:

AM_M macros do not work in autconf, try autoconf --force and that will
reset (something) and make it work once, but no more.  Removed.

configure: WARNING: unrecognized options: --enable-maintainer-mode, --enable-gtest

1 Identify how many spurious warnings are generated

2 Examine Makefile for gbg_autoconf spoor

*make optional note in configure output to inform user this error
 message is spurious* it is generated in generator/gen-py and almost
 looks like code rot, is there a bug catergory for this, should this
 be queued?

Note: build-outputs.mk still have references to gtest_LDFLAGS etc
gtest/gtest-1.la: $(gtest_DEPS)
        cd gtest && $(LINK_CXX)) $(gtest_LDFLAGS) -o gtest-1.la $(LT_NO_UNDEFINED) 

        $(gtest_OBJECTS) $(LIBS)
WHY

Considering target file `all'.
 File `all' does not exist.
 Finished prerequisites of target file `all'.
Must remake target `all'.
g++ -Igtest_hack/include -Igtest_hack -c gtest_hack/src/gtest-all.cc
Putting child 0x090799a0 (all) PID 24007 on the chain.
Live child 0x090799a0 (all) PID 24007
Reaping winning child 0x090799a0 PID 24007
ar -rv libgtest.a gtest-all.o

Currently, the build system is "improving" our link command in two
ways:

1.  it is stepping into gtest (not necessarily horrible)
2.  it is adding all the objects fromthe compilation to the command line
FIX: Remove the extra compilation targets?
DONE: Change to build.conf

idea: make gtest a "exe"
Result: no change

Removed the headers from private-includes in build.conf
        gtest/include/gtest/internal/*.h
        gtest/include/gtest/*.h

This does not affect build behaviour.  Also, it can't stay there
because gtest is optional and the script crashes when it cannot find
required headers.

Did this instead in Makefile.in: GTEST_INCLUDES = -Igtest -Igtest/include

Also added a lot of stuff into Makefile.in here is the current selection:

COMPILE_GTEST_CXX = $(COMPILE_CXX) $(GTEST_INCLUDES) -o $@ -c
LT_COMPILE = $(LIBTOOL) $(LTFLAGS) --mode=compile $(COMPILE) $(LT_CFLAGS)
LT_COMPILE_CXX = $(LIBTOOL) $(LTCXXFLAGS) --mode=compile $(COMPILE_CXX) $(LT_CFLAGS)
## I don't understand why, but I need to provide "-o ... -c" in order to get the 

  object file in the correct directory                                               
LT_COMPILE_GTEST_CXX = $(LIBTOOL) $(LTCXXFLAGS) 

  --mode=compile $(COMPILE_CXX) $(LT_FLAGS) $(GTEST_INCLUDES) -o lib$@ -c

and also

LINK_GTEST_CXX = $(LIBTOOL) $(LTCXXFLAGS) --mode=link $(CXX) $(LT_LDFLAGS) 

  $(CXXFLAGS) $(LDFLAGS) -rpath $(libdir)

(forgot to document and prompty got lost and forgot what I did...)


idea: following the thread of the remake stack trace
------ completed all in apr-util

##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
g++ -std=c++98  -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE  -Wall -Wpointer-arith 

-Wwrite-strings -Wshadow -Wunused -Wunreachable-code -Wno-system-headers   \
-g3 -fno-omit-frame-pointer -fno-inline   -DSVN_DEBUG -DAP_DEBUG  -I./subversion/include 

-I./subversion -I/home/g/trunk/apr/include   -I/home/g/trunk/apr-util/incl\
ude -I/home/g/trunk/sqlite-amalgamation  -Igtest -Igtest/include 

-o gtest/src/gtest-all.lo -c  gtest/src/gtest-all.cc
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
##>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
cd gtest && /bin/bash /home/g/trunk/libtool --tag=CXX --silent --mode=link g++ 

-g3 -fno-omit-frame-pointer -fno-inline   -DSVN_DEBUG -DAP_DEBUG   -rpath /tmp/foo/\
lib  -o gtest-1.la  src/gtest-all.lo
##<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
libtool: link: `src/gtest-all.lo' is not a valid libtool object
build-outputs.mk:479: *** [gtest/gtest-1.la] Error 1

#0  gtest/gtest-1.la at /home/g/trunk/build-outputs.mk:479
#1  gtest at /home/g/trunk/build-outputs.mk:1472
#2  local-all at /home/g/trunk/Makefile:390
#3  all at /home/g/trunk/Makefile:382
Command-line invocation:

Problem:
by including "-o $@", the output is named ".lo" instead of ".o".  The reason
we added -o ... is to get the object file in the correct directory

TODO
should be building in the gtest directory
Ruby error : already at newest ruby, spurious warning?
get-deps.sh is giving an error message. Need to svn up at some point

Libtool won't link to a library not called libtool
Do we need to use libtool?
No, not necessarily, BUT the rest of SVN does and we would like to follow
their style

Ideas:
1.  Avoid libtool
Pluses: might work
minus: might be harder than libtool

2.  Hack gen-make.py
Pluses: ...
Minuses: serpents  will issue forth from the CPU

3.  Rename tree to libgtest
Pluses: Will work
Minuses: Kitten with ball of angora wool

4.  libtool is actually a svn shell script.  Is the restriction on
name inside the shell system?  Why was it put in the first place?
Hack it if it exists?

Pluses: it might just work
Minuses: it might not

5.  Get libtool compile command to add lib-prefix
Plus: right thing to do
test:
    Yes it could work, although it puts the lib in trunk
        TODO: Work out why/ask why it drops it in trunk
This involves changing the name of the targets all the way through the make
file

Plan
6: Rename target to libgtest
1.  update configure.ac, build.conf

removed getst from aclocl.m4 in orig trunk as a test...
test if svn still compiles w/o gtest.m4 .. yes it does, removed.

PLAN: patch & ship

patched copy, didn't work, failed with:

/bin/bash /home/g/projects/svn/trunk/libtool --tag=CXX --silent --mode=compile g++ -

std=c++98  -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE   -g -O2  -I./subve\
rsion/include -I./subversion -I/home/g/projects/svn/trunk/apr/include 

-I/home/g/projects/svn/trunk/apr-util/include 

-I/home/g/projects/svn/trunk/sqlite-amalgamat\
ion   -Igtest -Igtest/include -o libgtest/src/gtest-all.lo -c  gtest/src/gtest-all.cc

cd gtest && /bin/bash /home/g/projects/svn/trunk/libtool --tag=CXX --silent 

--mode=link g++  -g -O2   -rpath /dev/null/lib  -o libgtest-1.la  src/gtest-all.lo
libtool: link: `src/gtest-all.lo' is not a valid libtool object
make: *** [gtest/libgtest-1.la] Error 1
g@musashi:~/projects/svn/trunk$ cd -

patch copy of faulty compile is the same as output of working one.

changed get_dep.sh to move unzipped gtest into libgtest instead

Added in dep-get.sh: warnings about the spurious warning about the
headers in autogen.sh and the repeated spurous warnings in ./configure
--enable-gtest,

*IT COMPILES* :)

cd libgtest && /bin/bash /home/g/projects/svn/trunk/libtool 

 --tag=CXX --silent --mode=link g++  -g -O2 

 -rpath /dev/null/lib  -o libgtest-1.la

compile svn with gtest present but without flags test:
make clean
./configure --enable-maintaner-mode --prefix=/dev/null
make
PASS

compile svn without gtest present and without flags test:
make clean
./configure --enable-maintaner-mode --prefix=/dev/null
make
PASS

compile svn without gtest present and with flag test:
make clean
./configure --enable-maintaner-mode --prefix=/dev/null
make
PASS

roll patch, log & ship it.

Somewhere in there...

I has surrounded it!

Many thanks to Ben for making my post with the clunky attachment with tons of questions readable  :)

[PATCH] OPW 2013: Build System Gtest Addition

Details can be found here:
[PATCH] OPW 2013: Build System Gtest Addition

The original project proposals can be found here:

My First Blog post (Hello World reloaded)

My First Blog post (Hello World reloaded)

The Apache Foundation Subversion team has accepted me as their Gnome Outreach Program for Women intern for the next 3 month.

My current project list is as follows:
Community Guide: complete the 3 patches that are due, 2 minor changes, one section on how to make changes to the website ETA : Week 1

Add gtest to the build system for C++ Bindings. ETA : week 3
The svn diff project: Bug 2044 and Bug 2447 ETA : week 6

The svnadmin freeze project ETA : week 12

+ other bugfixes as directed
+ projects discovered by grepping for 'FIXMEs' etc
+ another project if I happen to run out of work :)

Those are very conservative estimates -- I have not coded anything in 10 years and so, I have no sensible way of estimating how long I will need to complete this.

My goal is to code as much as I possibly can and see just how much I manage (re)learn in 3 month -- I've never worked with coders as good as are on the Subversion team, so this is as close to 'Supersize Me' as one's coding hobby can get.

Maybe it can be done, maybe it can't. I'm going to try!