{"thread":{"id":"15554","subject":"upload-pack packfile caching","startedAt":"2008-09-16T17:52:51Z","lastAt":"2008-09-16T20:59:31Z","messageCount":2,"participants":["Scott Chacon","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"90853","messageId":"d411cc4a0809161052h43b1be7dh9b322c8ec7d49add@mail.gmail.com","threadId":"15554","inReplyTo":null,"subject":"upload-pack packfile caching","fromName":"Scott Chacon","fromEmail":"schacon@gmail.com","sentAt":"2008-09-16T17:52:51Z","receivedAt":"2008-09-16T17:52:51Z","isPatch":false,"sender":{"key":"schacon@gmail.com","avatar":"https://gravatar.com/avatar/9b13a8a078e1dcf8588c4eea9554445d51ebed6c41b51f56f4d96738130b05c6?d=mp&s=160"},"body":"I was wondering if it would be of general interest to have upload-pack\ntake an option to cache packfiles.  Unless I am mistaken, every clone\non a git server will recreate the same packfile until something new is\npushed into it, correct?  I thought it might be a good idea to pass an\noption to have it cache the packfile that is created if\ncreate_full_pack is set and re-use it until the repository is updated.\n If I patched upload-pack to do this, would there be any interest in\nit?\n\nScott\n"},{"id":"90863","messageId":"alpine.LFD.1.10.0809161549240.6279@xanadu.home","threadId":"15554","inReplyTo":"d411cc4a0809161052h43b1be7dh9b322c8ec7d49add@mail.gmail.com","subject":"Re: upload-pack packfile caching","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-09-16T20:59:31Z","receivedAt":"2008-09-16T20:59:31Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 16 Sep 2008, Scott Chacon wrote:\n\n> I was wondering if it would be of general interest to have upload-pack\n> take an option to cache packfiles.  Unless I am mistaken, every clone\n> on a git server will recreate the same packfile until something new is\n> pushed into it, correct?  I thought it might be a good idea to pass an\n> option to have it cache the packfile that is created if\n> create_full_pack is set and re-use it until the repository is updated.\n>  If I patched upload-pack to do this, would there be any interest in\n> it?\n\nWell, if you do that there are a few things to be careful about.\n\nFirst, having a server process able to write files is a security hazard.  \nIf you want to create a pack cache then it is best if created manually \nby the repository owner.  You don't want someone cloning a repository \nactually messing with such cache.\n\nSecondly, the dynamic creation of a pack currently take into account the \ncapabilities of the client so not to produce a pack with features that \nthe client does not support.  So in order not to have to cache pack with \nmany feature combinations, this cache should probably only take effect \nif pack capabilities of the server are also supported by the client.\n\nNow, the _only_ advantage of a cached pack file is in avoiding execution \nof rev-list.  Otherwise creation of a pack for streaming is almost \nidentical to straight copying of data from disk due to pack data reuse.  \nThe rev-list can be made faster by having the pack-objects process do \nthe object listing itself instead of piping the output from rev-list \ninto it ('git repack' does that but 'git-upload-pack' doesn't).  And I \nbelieve that rev-list could be made much much faster with pack v4.\n\nThat been said...\n\nWhat you could have is a simple file with 2 SHA1s: the first \ncorresponding to the output of 'git for-each-ref' and the second one \ncorresponding to the list of all objects reachable from those refs.\n\nFor example:\n\n1) git for-each-ref --format=\"%(objectname)\" --sort=objectname | sha1sum\n\n2) git for-each-ref --format=\"%(objectname)\" | \\\n   xargs git rev-list --objects | cut -c -40 | sort | sha1sum\n\nSo, if you do the above in a freshly cloned repository, you'll find that \nthe SHA1 in 2) corresponds to this:\n\n3) git show-index < .git/objects/pack/pack-*.idx | cut -f2 -d' ' | sha1sum\n\nwhich means that all objects reachable from all refs are found in the \nonly pack you have.\n\nNow, if the SHA1 in 2) is computed over the binary representation of all \nthose object names, you'll find out that it corresponds to the actual \npack name in the .git/objects/pack/ directory.\n\nSo what upload-pack could do is look for a special file with those 2 \nSHA1s, and if it exists then see if the first SHA1 matches the list of \nvalues for all refs, if so then the name of the pack to send out \ncorresponds to the second SHA1.  If that pack is found in the repository \nthen you just have to stream it out.\n\nCreating that file is then just a matter of doing the equivalent of the \nabove commands and repacking your repository \ninto a single pack.\n\n\nNicolas\n"}]}