{"thread":{"id":"22134","subject":"Idea: global git object cache","startedAt":"2010-01-08T21:05:15Z","lastAt":"2010-01-08T21:05:15Z","messageCount":1,"participants":["Avery Pennarun"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"131129","messageId":"32541b131001081305nc25a811i73c96d1d252b9246@mail.gmail.com","threadId":"22134","inReplyTo":null,"subject":"Idea: global git object cache","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-01-08T21:05:15Z","receivedAt":"2010-01-08T21:05:15Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"Hi all,\n\nOne thing I find curious about git is how objects mostly aren't shared\nbetween multiple repositories on the local system.  For example, if I\ndo:\n\n   git clone git://git.kernel.org/pub/scm/git/git.git  git1\n   git clone git://git.kernel.org/pub/scm/git/git.git  git2\n\nThen I end up downloading the same objects from kernel.org *twice*.\nIf I use --reference on the second clone, then I can avoid\nre-downloading all the objects, and it's much faster.\n\nUnfortunately, I have to provide that option by hand, which is a\nproblem for git-submodule: it goes out to clone someone else's\nrepository automatically and doesn't know how to guess a value for\n--reference.  Another thing I commonly want to do with submodules is\nto rm -rf the submodule's files, eg. because I change branches and git\ndoesn't clean it automatically.  But then when I switch branches back\nto the one with the submodule, git wants to go re-download the\nsubmodule *again*.  Redoing the checkout makes sense to me (just as\ngit deletes/recreates files when I normally switch branches) but\nre-downloading seems silly.\n\nSo here's my suggestion to minimize downloads in a pretty easy way:\n\n- whenever git creates a packfile in any repo (eg. during git gc or\ngit fetch), make an *extra* hardlink of it into\n~/.gitcache/objects/pack.\n\n- whenever git is considering which objects it does/doesn't currently\nhave, also consider the packs in ~/.gitcache/objects/pack (ie. using\nthe git/objects/alternates mechanism).  If one of the packs qualifies,\nhardlink it into the current repo.  Maybe give it a .keep file to\nindicate that it's counterproductive to repack this pack.\n\n- after git deletes a packfile in any repo (eg. during git gc), check\nthe link count of that pack in ~/.gitcache/objects/pack; if it's now\ndown to just 1, there are no other users of the pack, so delete it\nthere too.  You would also need to prune the cachedir occasionally to\ndeal with repositories that were deleted in other ways (eg. rm -rf).\n\n- share the list of refs in a similar way (noticing that you probably\nhave different refs in multiple repos that are named\n\"refs/heads/master\" of course) so that fetches will be efficient.\n\n- extra improvement to submodule behaviour: hardlink packs from the\nsubmodule into the supermodule's objects/pack directory (or use a\ndifferent directory like .git/submodules/pack to keep things\nseparate).  Also, submodules should use the superproject's pack\ndirectory as an alternate, in case (as often happens for me) the\nsupermodule already contains a bunch of objects from the submodule,\nbecause the modules were split at some point.\n\nI believe this would be quite easy to implement and would give an\nimmediate efficiency improvement.  The ~/.gitcache feature could be\nenabled/disabled by a config option.  Is there any reason not to do\nit?\n\nThanks,\n\nAvery\n\nP.S. I've been testing git's behaviour with lots of very large packs -\nI'm currently using about 58 packs of about 1 GB each - as part of my\n'bup' git-based backup tool (http://apenwarr.ca/log/?m=201001#04).\nRepacking and fsck are obviously horrendously slow with that much\ndata, but bup avoids those operations as much as possible, and a\n~/.gitcache wouldn't need to worry about them either (since each repo\nis still responsible for repacking its own packs).  Overall\nperformance for other git operations seems to be fine, though.  And\nsearching the cache as a last restore can be optimized by always\nsearching packs in MRU order, in case git doesn't already do this.\n"}]}