Re: git-gui clone WTFery

From: Shawn O. Pearce <spearce@spearce.org>
Date: Feb 3, 2010, 19:52 UTC
Message-ID: <20100203195215.GE14799@spearce.org>
In-Reply-To: <7vwryuhzsn.fsf@alter.siamese.dyndns.org>

Junio C Hamano <gitster@pobox.com> wrote:

Show 9 quoted lines

> Michele Ballabio <barra_cuda@katamail.com> writes:
> > In this case, git-gui is counting objects during a clone; would it be OK
> > to consider "buckets" as synonim of "objects" or "items" or something
> > else?  Then I would translate accordingly.
> 
> A cursory read of the codepath with that message tells me that Shawn is
> calling one [0-9a-f][0-9a-f] directory under .git/objects/ a "bucket" and
> counting the number of them while copying them to implement "git clone"
> inside git-gui (why?---that is a separate issue).

I just had an argument with someone in a private thread about why commit messages should be more than 30 characters long. Lets see if the git-gui history stands up to that and sufficiently explains why I rewrote clone in Tcl:

  $ git blame lib/choose_repository.tcl
  ...
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  633) 
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  634)           $o_cons start \
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  635)                   [mc "Counting objects"] \
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  636)                   [mc "buckets"]
  ...
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  673)           update
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  674) 
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  675)           file mkdir [file join .git objects pack]
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  676)           foreach i [glob -tails -nocomplain \
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  677)                   -directory [file join $objdir pack] *] {
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  678)                   lappend tolink [file join pack $i]
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  679)           }
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  680)           $o_cons update [incr bcur] $bcnt
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  681)           update
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  682) 
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  683)           foreach i $buckets {
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  684)                   file mkdir [file join .git objects $i]
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  685)                   foreach j [glob -tails -nocomplain \
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  686)                           -directory [file join $objdir $i] *] {
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  687)                           lappend tolink [file join $i $j]
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  688)                   }
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  689)                   $o_cons update [incr bcur] $bcnt
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  690)                   update
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  691)           }

Hmmph. 81d4d3dd and ab08b363 are the relevant commits:

  commit 81d4d3dddc5e96aea45a2623c9b1840491348b92
  Author: Shawn O. Pearce <spearce@spearce.org>
  Date:   Mon Sep 24 08:40:44 2007 -0400

    git-gui: Keep the UI responsive while counting objects in clone
    
    If we are doing a "standard" clone by way of hardlinking the
    objects (or copying them if hardlinks are not available) the
    UI can freeze up for a good few seconds while Tcl scans all
    of the object directories.  This is espeically noticed on a
    Windows system when you are working off network shares and
    need to wait for both the NT overheads and the network.
    
    We now show a progress bar as we count the objects and build
    our list of things to copy.  This keeps the user amused and
    also makes sure we run the Tk event loop often enough that
    the window can still be dragged around the desktop.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

  commit ab08b3630414dfb867825c4a5828438e1c69199d
  Author: Shawn O. Pearce <spearce@spearce.org>
  Date:   Sat Sep 22 03:47:43 2007 -0400

    git-gui: Allow users to choose/create/clone a repository
  ...
    Rather than relying on the git-clone Porcelain that ships with
    git we build the new repository ourselves and then obtain content
    by git-fetch.  This technique simplifies the entire clone process
    to roughly: `git init && git fetch && git pull`.  Today we use
    three passes with git-fetch; the first pass gets us the bulk of
    the objects and the branches, the second pass gets us the tags,
    and the final pass gets us the current value of HEAD to initialize
    the default branch.
    
    If the source repository is on the local disk we try to use a
    hardlink to connect the objects into the new clone as this can
    be many times faster than copying the objects or packing them and
    passing the data through a pipe to index-pack.  Unlike git-clone
    we stick to pure Tcl [file link -hard] operation thus avoiding the
    need to fork a cpio process to setup the hardlinks.  If hardlinks
    do not appear to be supported (e.g. filesystem doesn't allow them or
    we are crossing filesystem boundaries) we use file copying instead.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

My guess is, around this time period (Sept. 2007) git-clone was actually git-clone.sh (Yup, it was built-in in 8434c2f1afed "Build in clone" Apr 2008). Clone on Cygwin through git-clone.sh was very slow compared to clone in Tcl using a native Win32 wish process...

Today with git-clone in C, this is a WTF. But at the time, it was a nice performance boost on the Windows platform. And nobody has tried to clean this up yet in git-gui.

Yay.  My point about commit messages is still valid.  :-)

-- 
Shawn.