Re: git pack/unpack over bittorrent - works!

On Mon, Sep 6, 2010 at 12:52 AM, Nicolas Pitre <nico@fluxnic.net> wrote:

 ack :)

 i kinda get it - but i realise that's not good enough: i need to be
able to _say_ i get it, in a way that satisfies you.

 ack.

 mmm packs not being to do with object enumeration i get.  i
understand that .idx files contain "lists of objects" which isn't the
same thing (and also happen to contain pointers/offsets to the objects
of its associated .pack)

 at some point i'd really like to know what the object list is (not
the objects themselves) that comes out of "git pack-objects --thin"
but my curiosity can wait.

 ok - this tells me (and it's confirmed, below) that you're describing
the situation based on what can be found in .git - _not_ what comes
out of "git pack-objects".  i wouldn't _dream_ of digging around in a
.git/ location looking for packs or idx files, but because i have
mentioned them _without_ prefixing every mention with "the
custom-generated .idx and/or .pack as generated by git pack-objects",
you may have got the wrong impression, for which i apologise.

 yep :)

 ok.  all of this makes sense - but it's enough for me to be able to
ask questions, rather than "do", if you know what i mean.

 yes.  ahh... that's what i've been doing: using "git pack-objects
--thin".  and the reason for that is because i've seen it used in the
http implementation of "git fetch".

 so, my questions up until now regarding .pack and .idx have all been
targetted at that, and based on that context, _not_ the packs+idx
files that are in .git/

 no, i want commits separated and individual and "compoundable".  the plan is:

* to get the ref associated with refs/heads/master
* to get the list of all commits associated with that master ref
* to work out how far local deviates from remote along that list of commits
* to get the objects which will make up the missing commits (if they
aren't already in the local store)
* to apply those commits in the correct order

in other words, the plan is to follow what git http fetch and/org git git:// fetch does as much as possible (ok, perhaps not).

the reason for getting the objects individually (blobs etc.) should be clear: prior commits _could_ have resulted in that exact object having been obtained already.

so far i have implemented:

* get the master ref using git for-each-ref
* get the list of all commits using git rev-list
* enumerate the list of objects associated with an individual commit by:
    i) creating a CUSTOM pack+idx using git pack-objects {ref}
    ii) *parsing* the idx file using gitdb's FileIndex to get the list
of objects
    iii) transferring that list to the local machine
* requesting *individual* objects from the enumerated list out of the idx file
   by using a CUSTOM "git pack-objects --thin {ref} < {ref}" command

that's as far as i've got, before you mentioned that it would be better to use "git rev-list --objects commit1..commit2" and to use "git cat-file" to obtain the actual object [what's not clear in this plan is how to store that cat'ed file at the local end, hence the continued use of git pack-objects --thin {ref} < {ref}]

the prior implementation was to treat the custom pack-object as if it was "the atomic leaf-node operation" instead of individual objects (blobs, trees).

 soorree.  i believe the source of the confusion is that you believed
that i intend to "blindly use a pack index file" as in "blindly go
rummaging around in .git/ at the remote end" when i have absolutely no
intention of doing so.

 what i _have_ been doing however is custom-generating pack-objects
and associated pack-indexes (just like git http fetch) _including_
using the --thin option because that's what git http fetch does.

 i believe that this results in the concerns that you raised (about
having access to unauthorised data) being dealt with.

 ... because it uses, from what i can gather, git pack-objects --thin

 and that's handled by git pack-objects --thin (am i right?)

 ok.

 so.  we have a hierarchical plan: get the commit list, get a
per-commit object-list, get the objects (if needed), store the
objects.

 problem: despite looking through virtually every single builtin/*.c
file which uses write_sha1_file (which i believe i have correctly
identified, from examining git unpack-objects, as being the function
which stores actual objects, including their type), i do not see a git
command (yet) which performs the reverse operation of "git cat-file".

builtin/apply.c - that's for patches builtin/checkout.c - that's for the merge result. builtin/notes.c - creating a note builtin/tag.c - creating a tag builtin/mktree.c - creating a tree object but *only* from a text listing

ok - maybe this is one of the ones that i need, but only if i use "git cat-file -p" to pretty-print the output of tree objects but i don't think that's a good idea.

what else...

cache_tree.c - nope. commit.c - nope. read-cache.c - beh? nope. has blank args "", 0

so... um... unless i actually manually create a pack object (perhaps using python-gitdb to construct it) out of the data obtained by "git cat-file" i don't see how this would work.

l.