Re: [ANNOUNCE] Cogito-0.12

PB> It won't happen. Or rather, I hope the HTTP pulls become more efficient PB> soon. Actually, perhaps Linus has something done already, my workstation PB> is a bit derailed now so I couldn't pull from him in the last few days PB> (hopefully will sort that out today).

PB> Hmm, yes, I guess Linus won't be touching the HTTP backend at all. ;-) I PB> suggest you to check the last development in Linus' branch and sync with PB> Daniel Barkalow, who promised improving the pull tools as well.

If this weekend is not too late, I have been brewing what is called an "efficient pull from dumb servers" suite, which would hopefully fill this gap. I am still in the process of finishing the details, but basically it already seems to work.

Linus, please drop the patch I sent you earlier, privately by mistake not CCing the list, that implemented only the server end. I've changed some file formats already from that one.

The outline of how it works is like this.

 * I assume a dumb transport (read: static files only HTTP
   server) and no on-request server side processing.  All the
   smarts must go in the client.  The server side X.git being an
   ordinary GIT archive (no need for files in the work tree),
   plus:

   - X.git/objects/pack can have packed GIT archives.  I
     envision that this will be a series of 5 to 20 MB packs,
     occasionally adding a new incremental pack when
     X.git/objects/??/ directories accumulate enough standalone
     SHA1 files.  It is not necessary to have X.git/objects/??/
     files if an object is contained in one of the packs.

   - X.git/info/ has three extra files.

     - "inventory" lists all the branches stored in X.git/refs
       and looks like this (contents and path):

          ff83c8f3554ceb444b413beaeb49b4a781dae944 snap/0
          013e7c7ff498aae82d799f80da37fbd395545456 snap/10
          ff83c8f3554ceb444b413beaeb49b4a781dae944 heads/master
          dd7ba8b4949535c24e604a37709db0e3be9ccbbc heads/linus

       This is to facilitate discovery from a transport that is
       not so "ls" friendly, like HTTP.

     - "pack" lists available packs under X.git/objects/pack and
       looks like this (size and name):

          432495 pk-65fe69e9bc2e8a3e0881e008dde182522156ba7c.pack

       The file is there for discovery.  The size is used by the
       client to discover optimum set of packs to slurp.

     - "rev-cache" is a binary file that describes commit
       ancestry information in a dense format.  It lists all
       commits available from this repository along with who
       its parents are for each of the commit.  This file is
       produced append-only, so that the server side can use
       rsync based mirroring scheme.

   A new command "git-update-dumb-server" is used to prepare
   these three files.  There may need a helper script that uses
   git-pack-objects and friends to prepare packs partitioned to
   allow pulling a popular branch efficiently.

 * The client side is called "git-dumb-pull-script".  This
   downloads the above three files, and .idx files associated
   with packs described in "pack".  With the information in
   "inventory" about desired branch to pull from along with
   "rev-cache" ancestry information, it discovers the set of
   commits that is lacking from its local store.  By comparing
   that list with downloaded .idx files, along with size
   information for each pack, it comes up a list of packs to
   download to cover the most commits that it wants to obtain,
   and downloads them, verifies them and stores them in its
   .git/objects/pack/ directory.

   The above process of downloading packs would typically not
   cover all the things lacking, because some new commits may
   not be in any of the packs.  After this point, the usual
   commit-walking git-http-pull can be used to fill the rest,
   and it does not have to pull that many objects.  Dan's
   http-pull parallelism improvement would be very useful
   independently here.