{"thread":{"id":"12109","subject":"[RFC] git-clone should create packed refs","startedAt":"2008-02-15T00:33:19Z","lastAt":"2008-02-15T01:13:38Z","messageCount":3,"participants":["Johan Herland","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"68790","messageId":"200802150133.19247.johan@herland.net","threadId":"12109","inReplyTo":null,"subject":"[RFC] git-clone should create packed refs","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-02-15T00:33:19Z","receivedAt":"2008-02-15T00:33:19Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Hi,\n\nI'm experimenting with converting deep (lots of history) CVS repos to Git, \nand I notice that cloning the resulting Git repos is _slow_. E.g. an \nexample repo with 10000 tags and 1000 branches will take ~24 seconds to \nclone. Debugging shows that >95% of that time is spent by calling \"git \nupdate-ref\" for each of the 11000 refs. I can easily get the total runtime \ndown to ~4 seconds by replacing the \"git update-ref ...\" with something \nlike \"echo $sha1 $destname >> $GIT_DIR/packed-refs\". Some more \ninvestigation shows that what's actually taking so long is not writing all \nthese 40-bytes ref files and their corresponding reflogs, but rather the \noverhead of creating the \"git update-ref\" process 11000 times (echo is a \nshell builtin, I presume, so doesn't have the same overhead). My conclusion \nis therefore that making \"git clone\" a builtin will solve my performance \nproblems (since the update-ref is now a function call, rather than a \nsubprocess).\n\nSearching the list, I find that - lo and behold - someone (CCed) is actually \nalready working on this. :)\n(BTW, a progress report on this work would be nice...)\n\n\nSo the only niggle I have left, is that when git-clone is cloning repos with \nthousands of refs, it makes sense to create a packed-refs file directly in \nthe clone, instead of having to run \"git pack-refs\" (or \"git gc\") \nafterwards to (re)pack the refs. This has pretty much the same reasoning as \ntransferring and storing the objects in packs instead of exploding them \ninto loose objects.\n\nIn my case, the upstream repo already has packed refs, so it just seems \nstupid to explode them into \"loose\" refs when cloning, and make me re-pack \nthem afterwards.\n\nLooking at git-clone.sh, I even find that when cloning, the refs are \ntransferred in a format similar (but not identical) to the packed-refs file \nformat (see CLONE_HEAD in git-clone.sh).\n\nAFAICS, the only complication with this proposal is how to deal with the \nreflogs. Right now, for each ref created, a corresponding reflog with a \nsingle entry is written. Therefore - in my example repo above - the \ncurrent \"git clone\" writes ~22000 files, and my proposal offers only a net \nreduction in #files written by ~50%, instead of ~100%. For reference, the \nreflog entries written by \"git clone\" look like this:\n\t\"000... $sha1 A U Thor $timestamp clone: from $repo\"\nIMHO, these entries don't carry much value:\n- The $sha1 is self-evident (and if later changed, will still be mentioned\n in the next reflog entry).\n- The author name and email would probably be self-evident/uninteresting in\n most cases.\n- The timestamp might be marginally useful, as I can't immediately document\n another way of getting the time of cloning.\n- The $repo would also be self-evident in many cases, and would in any case\n also be listed in the config file in the \"origin\" remote section.\nI'd therefore suggest to make reflog creation in \"git clone\" optional, in \norder to avoid having the number of files written be proportional to the \nnumber of refs.\n\nI would imagine that even though the time used on Linux for writing \nthousands of files might be negligible, this is not the case on certain \nother OSes...\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"68791","messageId":"alpine.LSU.1.00.0802150051320.30505@racer.site","threadId":"12109","inReplyTo":"200802150133.19247.johan@herland.net","subject":"Re: [RFC] git-clone should create packed refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-02-15T00:53:17Z","receivedAt":"2008-02-15T00:53:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 15 Feb 2008, Johan Herland wrote:\n\n> when git-clone is cloning repos with thousands of refs, it makes sense \n> to create a packed-refs file directly in the clone, instead of having to \n> run \"git pack-refs\" (or \"git gc\") afterwards to (re)pack the refs.\n\nSure, and it's easy, too. The format of the packed-refs file is exactly \nthe same as the output of \"git ls-remote \".\n\nCiao,\nDscho\n"},{"id":"68793","messageId":"200802150213.38532.johan@herland.net","threadId":"12109","inReplyTo":"alpine.LSU.1.00.0802150051320.30505@racer.site","subject":"Re: [RFC] git-clone should create packed refs","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-02-15T01:13:38Z","receivedAt":"2008-02-15T01:13:38Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Friday 15 February 2008, Johannes Schindelin wrote:\n> Hi,\n> \n> On Fri, 15 Feb 2008, Johan Herland wrote:\n> \n> > when git-clone is cloning repos with thousands of refs, it makes sense \n> > to create a packed-refs file directly in the clone, instead of having to \n> > run \"git pack-refs\" (or \"git gc\") afterwards to (re)pack the refs.\n> \n> Sure, and it's easy, too. The format of the packed-refs file is exactly \n> the same as the output of \"git ls-remote \".\n\nAs I said in my first email: similar, but not identical.\n\nContrast:\n\n$ git ls-remote origin\nec09f49905e94f3bbf04bf3521f1fc59b1345cbd HEAD\nec09f49905e94f3bbf04bf3521f1fc59b1345cbd refs/heads/another_branch\nec09f49905e94f3bbf04bf3521f1fc59b1345cbd refs/heads/master\nd6f7eacba9f07aa382a113e129866266c8d60642 refs/tags/complex_tag\nec09f49905e94f3bbf04bf3521f1fc59b1345cbd refs/tags/complex_tag^{}\nec09f49905e94f3bbf04bf3521f1fc59b1345cbd refs/tags/simple_tag\n$ cat .git/packed-refs\n# pack-refs with: peeled\nec09f49905e94f3bbf04bf3521f1fc59b1345cbd refs/heads/master\nec09f49905e94f3bbf04bf3521f1fc59b1345cbd refs/remotes/origin/another_branch\nec09f49905e94f3bbf04bf3521f1fc59b1345cbd refs/remotes/origin/master\nd6f7eacba9f07aa382a113e129866266c8d60642 refs/tags/complex_tag\n^ec09f49905e94f3bbf04bf3521f1fc59b1345cbd\nec09f49905e94f3bbf04bf3521f1fc59b1345cbd refs/tags/simple_tag\n$\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"}]}