threads / discuss / 18476

Summer of Code - Cached Packs/Object Lists

Subject: Summer of Code - Cached Packs/Object Lists

## tl;dr

5 messages between Mar 23, 2009 and Mar 23, 2009.

replies: 4people: 3as markdown or json

thisnukes4u@gmail.com· Mar 23, 2009, 01:53 UTC · lore

Hey list,

     My name is Thomas Coppi, and I am a junior studying Computer Science at New Mexico Tech.  I've been using git for a few years now for managing coursework and personal projects, but have never looked much into the git source code or internals, so I figure that Summer of Code is a good way to get started.

     I am particularly interested in the packfile caching project mentioned on the wiki, but I have a couple of questions:

1.  Would it be possible to implement both the packfile and object list caching mechanisms, or might would one interfere with the other in some way?
2.  With just a quick perusal of the daemon source, I noticed that it shells out to the upload-pack command. Where would it be appropriate to implement such a caching mechanism, in the daemon proper, the upload-pack code, or would both need to be updated?

Thanks in advance, and sorry if my questions seem a little bit newbish, as I mentioned this is my first time diving into git internals.

-- 
Thomas Coppi

Shawn O. Pearce· Mar 23, 2009, 01:59 UTC · re: thisnukes4u@gmail.com · lore

Re: Summer of Code - Cached Packs/Object Lists

Please line wrap your email at something useful to others when quoting, like 70-72 characters per line.

thisnukes4u@gmail.com wrote:

Show 6 quoted lines

> I am particularly interested in the packfile caching project
> mentioned on the wiki, but I have a couple of questions:
>
> 1.  Would it be possible to implement both the packfile and
> object list caching mechanisms, or might would one interfere with
> the other in some way?

You could do both. But I think most people on the list will argue that doing both is overkill and only one is necessary, and further, that only the one that offers the "biggest bank for the buck" should be implemented.

Whole pack file caching has been discussed on list a few times as a nice feature to have, but it raises some issues of cache management, not to mention the issue I posed about it being relatively useless on frequently changing repositories.

> 2.  With just a quick perusal of the daemon source, I noticed
> that it shells out to the upload-pack command. Where would it be
> appropriate to implement such a caching mechanism, in the daemon
> proper, the upload-pack code, or would both need to be updated?

The daemon doesn't get enough data from the client in order to perform any sort of caching.

So the caching has to happen in upload-pack, and/or pack-objects. (upload-pack forks out to pack-objects to create the pack file to send to the client)

-- 
Shawn.

Thomas Coppi· Mar 23, 2009, 02:49 UTC · re: Shawn O. Pearce · lore

Re: Summer of Code - Cached Packs/Object Lists

On Sun, Mar 22, 2009 at 7:59 PM, Shawn O. Pearce <spearce@spearce.org> wrote:

> Please line wrap your email at something useful to others when
> quoting, like 70-72 characters per line.

Sorry about that.

> You could do both.  But I think most people on the list will argue
> that doing both is overkill and only one is necessary, and further,
> that only the one that offers the "biggest bank for the buck"
> should be implemented.

Alright, that seems reasonable. Given that I think I would lean towards implementing an object list caching mechanism, since that seems to be more generally applicable. The logic for this would then need to be in the rev-list code(as mentioned in the JGit discussion), correct?

Oh, and I forgot to mention in my previous email that I am also available on freenode IRC under the nick tcoppi.

Thanks again,

-- Thomas Coppi

Shawn O. Pearce· Mar 23, 2009, 02:52 UTC · re: Thomas Coppi · lore

Re: Summer of Code - Cached Packs/Object Lists

Thomas Coppi <thisnukes4u@gmail.com> wrote:

Show 10 quoted lines

> On Sun, Mar 22, 2009 at 7:59 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
>> You could do both. ??But I think most people on the list will argue
>> that doing both is overkill and only one is necessary, and further,
>> that only the one that offers the "biggest bank for the buck"
>> should be implemented.
>
> Alright, that seems reasonable.  Given that I think I would lean
> towards implementing an object list caching mechanism, since that seems
> to be more generally applicable.  The logic for this would then need to
> be in the rev-list code(as mentioned in the JGit discussion), correct?

Probably. IIRC upload-pack forks a rev-list to produce the object list, and pipes that into the forked pack-objects' stdin. Thus rev-list is probably what would need to know how to include the cached list to its output.

-- 
Shawn.

Nicolas Pitre· Mar 23, 2009, 14:41 UTC · re: Shawn O. Pearce · lore

Re: Summer of Code - Cached Packs/Object Lists

On Sun, 22 Mar 2009, Shawn O. Pearce wrote:

Show 16 quoted lines

> Thomas Coppi <thisnukes4u@gmail.com> wrote:
> > On Sun, Mar 22, 2009 at 7:59 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
> >> You could do both. ??But I think most people on the list will argue
> >> that doing both is overkill and only one is necessary, and further,
> >> that only the one that offers the "biggest bank for the buck"
> >> should be implemented.
> >
> > Alright, that seems reasonable.  Given that I think I would lean
> > towards implementing an object list caching mechanism, since that seems
> > to be more generally applicable.  The logic for this would then need to
> > be in the rev-list code(as mentioned in the JGit discussion), correct?
> 
> Probably.  IIRC upload-pack forks a rev-list to produce the
> object list, and pipes that into the forked pack-objects' stdin.
> Thus rev-list is probably what would need to know how to include
> the cached list to its output.

Related to this, the first optimization is probably to avoid the fork altogether. The pack-objects code knows how to list objects by itself already, and that is used by git-repack. At the moment, packed tree objects during a fetch are probably accessed one extra time needlessly.

Nicolas