{"thread":{"id":"50760","subject":"git-fast-import yields huge packfile","startedAt":"2019-03-16T20:31:07Z","lastAt":"2019-03-21T14:23:42Z","messageCount":6,"participants":["Richard Hipp","Linus Torvalds","Mike Hommey","Johannes Schindelin","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"371670","messageId":"CALwJ=MzrqPUNw=jc0NRtaJaJG+ErXNb577JNSN66GiGY4UFtRw@mail.gmail.com","threadId":"50760","inReplyTo":null,"subject":"git-fast-import yields huge packfile","fromName":"Richard Hipp","fromEmail":"drh@sqlite.org","sentAt":"2019-03-16T20:31:04Z","receivedAt":"2019-03-16T20:31:07Z","isPatch":false,"sender":{"key":"drh@sqlite.org","avatar":null},"body":"I'm trying to transform a repository from another VCS into a Git\nrepository using \"git fast-import\". It appears to work, but the\nresulting Git repository is huge relative to the original - 18 times\nlarger. Most of the space seems to be taken up by a single large\npackfile. That packfile is about 967 MB which is about 1/4th the\ntotal uncompressed size of all 41785 distinct Blobs in the original\nrepository. The source VCS is able to compress this down to 52 MB by\ncomparison.\n\nMaybe I'm doing something wrong with the fast-import stream that is\ndefeating Git's attempts at delta compression....\n\nAre there any utility programs available for analyzing packfiles so\nthat I try to figure out where the inefficiencies are cropping up, so\nthat I can try to address them?\n\nAnybody have any suggestions on what I should be looking for?\n\nIf anyone would care to see this oversized packfile and perhaps offer\nsuggestions on how I can make it more space-efficient, it can be\ncloned from https://github.com/drhsqlite/fossil-mirror.git - at least\nfor now - surely I will delete that repo and regenerate it once I\nfigure out this problem.\n\n-- \nD. Richard Hipp\ndrh@sqlite.org\n"},{"id":"371671","messageId":"CAADWXX81agAg1B9+FM5c4JsWzANKochNTcv2ShsRzAsZpHAVWw@mail.gmail.com","threadId":"50760","inReplyTo":"CALwJ=MzrqPUNw=jc0NRtaJaJG+ErXNb577JNSN66GiGY4UFtRw@mail.gmail.com","subject":"Re: git-fast-import yields huge packfile","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2019-03-16T21:04:33Z","receivedAt":"2019-03-16T21:04:46Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Sat, Mar 16, 2019 at 1:31 PM Richard Hipp wrote:\n>\n> Maybe I'm doing something wrong with the fast-import stream that is\n> defeating Git's attempts at delta compression....\n\nfast-import doesn't do fancy delta compression becayse that would\ndefeat the \"fast\" part of fast-import.\n\nJust do a git repack after the import to do the proper repacking. I\nget a 41Mb packfile when I try that on your repo.\n\nSo a simple\n\n git repack -adf\n\nshould fix things up for you (the \"-f\" to make sure it doesn't try to\nre-use things from the silly bad pack). Alternatively, use \"git gc\n--aggressive\", which will do that forced repack too.\n\n Linus\n"},{"id":"371675","messageId":"20190316221232.m6mphdnwy6mys4j2@glandium.org","threadId":"50760","inReplyTo":"CAADWXX81agAg1B9+FM5c4JsWzANKochNTcv2ShsRzAsZpHAVWw@mail.gmail.com","subject":"Re: git-fast-import yields huge packfile","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2019-03-16T22:12:32Z","receivedAt":"2019-03-16T22:12:42Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Sat, Mar 16, 2019 at 02:04:33PM -0700, Linus Torvalds wrote:\n> On Sat, Mar 16, 2019 at 1:31 PM Richard Hipp wrote:\n> >\n> > Maybe I'm doing something wrong with the fast-import stream that is\n> > defeating Git's attempts at delta compression....\n> \n> fast-import doesn't do fancy delta compression becayse that would\n> defeat the \"fast\" part of fast-import.\n\nfast-import however does try to do delta compression of blobs against the\nlast blob that was imported, so if you put your blobs in an order where\nthey can be delta-ed, you can win without a git repack.\n\nFor one-shot conversions, you can just rely on git repack.\n\nMike\n"},{"id":"371678","messageId":"CALwJ=Mw0A7XbB12K-X7OGT=kBXPKC4EN0zxtnz4gEz0+yHtEMQ@mail.gmail.com","threadId":"50760","inReplyTo":"CAADWXX81agAg1B9+FM5c4JsWzANKochNTcv2ShsRzAsZpHAVWw@mail.gmail.com","subject":"Re: git-fast-import yields huge packfile","fromName":"Richard Hipp","fromEmail":"drh@sqlite.org","sentAt":"2019-03-16T23:22:03Z","receivedAt":"2019-03-16T23:22:09Z","isPatch":false,"sender":{"key":"drh@sqlite.org","avatar":null},"body":"On 3/16/19, Linus Torvalds wrote:\n>\n> git repack -adf\n>\n\nThanks for the tip!\n\n-- \nD. Richard Hipp\ndrh@sqlite.org\n"},{"id":"372127","messageId":"nycvar.QRO.7.76.6.1903211503030.41@tvgsbejvaqbjf.bet","threadId":"50760","inReplyTo":"CALwJ=MzrqPUNw=jc0NRtaJaJG+ErXNb577JNSN66GiGY4UFtRw@mail.gmail.com","subject":"Re: git-fast-import yields huge packfile","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-03-21T14:09:40Z","receivedAt":"2019-03-21T14:10:00Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Richard,\n\nOn Sat, 16 Mar 2019, Richard Hipp wrote:\n\n> I'm trying to transform a repository from another VCS into a Git\n> repository using \"git fast-import\". It appears to work, but the\n> resulting Git repository is huge relative to the original - 18 times\n> larger. Most of the space seems to be taken up by a single large\n> packfile. That packfile is about 967 MB which is about 1/4th the\n> total uncompressed size of all 41785 distinct Blobs in the original\n> repository. The source VCS is able to compress this down to 52 MB by\n> comparison.\n\nI feel your pain, as I had the same problem back in the day. My use case\nwas mirroring an upstream Mercurial repository to a Git repository. This\nuse case went away, so I do not do that anymore (and there are more, less\nhappy reasons why I would no longer work on that git-remote-hg project,\nbut that's off topic). As one of the last rem(a)inders, Git for Windows\ncarries this patch:\n\nhttps://github.com/git-for-windows/git/commit/b91911ff8d3e2cf279b4708be89de2e3bc8e9e87\n\nEssentially, it *always* runs `git gc --auto` after running `fast-import`.\n\nWhich is a lot more high-level advice than the rather low-level `git\nrepack` hint given elsewhere in this thread.\n\nNow, I wonder whether we should integrate this into `fast-import` proper\n(with a knob to turn it off), maybe even offer to run `git gc --auto`\nevery imported commits?\n\nCiao,\nJohannes\n\n> Maybe I'm doing something wrong with the fast-import stream that is\n> defeating Git's attempts at delta compression....\n>\n> Are there any utility programs available for analyzing packfiles so\n> that I try to figure out where the inefficiencies are cropping up, so\n> that I can try to address them?\n>\n> Anybody have any suggestions on what I should be looking for?\n>\n> If anyone would care to see this oversized packfile and perhaps offer\n> suggestions on how I can make it more space-efficient, it can be\n> cloned from https://github.com/drhsqlite/fossil-mirror.git - at least\n> for now - surely I will delete that repo and regenerate it once I\n> figure out this problem.\n>\n> --\n> D. Richard Hipp\n> drh@sqlite.org\n>\n"},{"id":"372131","messageId":"87o964cnn0.fsf@evledraar.gmail.com","threadId":"50760","inReplyTo":"nycvar.QRO.7.76.6.1903211503030.41@tvgsbejvaqbjf.bet","subject":"Re: git-fast-import yields huge packfile","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-21T14:23:15Z","receivedAt":"2019-03-21T14:23:42Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 21 2019, Johannes Schindelin wrote:\n\n> Hi Richard,\n>\n> On Sat, 16 Mar 2019, Richard Hipp wrote:\n>\n>> I'm trying to transform a repository from another VCS into a Git\n>> repository using \"git fast-import\". It appears to work, but the\n>> resulting Git repository is huge relative to the original - 18 times\n>> larger. Most of the space seems to be taken up by a single large\n>> packfile. That packfile is about 967 MB which is about 1/4th the\n>> total uncompressed size of all 41785 distinct Blobs in the original\n>> repository. The source VCS is able to compress this down to 52 MB by\n>> comparison.\n>\n> I feel your pain, as I had the same problem back in the day. My use case\n> was mirroring an upstream Mercurial repository to a Git repository. This\n> use case went away, so I do not do that anymore (and there are more, less\n> happy reasons why I would no longer work on that git-remote-hg project,\n> but that's off topic). As one of the last rem(a)inders, Git for Windows\n> carries this patch:\n>\n> https://github.com/git-for-windows/git/commit/b91911ff8d3e2cf279b4708be89de2e3bc8e9e87\n>\n> Essentially, it *always* runs `git gc --auto` after running `fast-import`.\n>\n> Which is a lot more high-level advice than the rather low-level `git\n> repack` hint given elsewhere in this thread.\n>\n> Now, I wonder whether we should integrate this into `fast-import` proper\n> (with a knob to turn it off), maybe even offer to run `git gc --auto`\n> every imported commits?\n\nMy reading of the combination of Linus's & Mike Hommey's E-Mails is that\nthis just happened to work for you because the blob import order you\nused was such that you didn't get any on-the-fly deltas.\n\nBut as Linus notes you need to pass \"-f\" aka. \"--no-reuse-delta\" down to\npack-objects for this to work in the general case, so a plain \"git gc\"\nin that GFW patch won't do the right thing *unless* you didn't end up\nwith any deltas at all (or close enough for it not to matter).\n\nSo in the general case you need to run \"git gc --aggressive\" after a\n\"fast-import\". I'll add some docs about this in my re-roll of my\nconcurrent gc doc series:\nhttps://public-inbox.org/git/20190318161502.7979-1-avarab@gmail.com/\n\nI wonder if we should just leave it at that. The fast-import command is\nplumbing, and e.g. someone running N number of those now and doing a\n\"git gc --aggressive\" afterwards would have their use broken by this,\ntheir \"gc\" would abort if the \"--aggressive\" we spawned after the 1st\nfast-import invocation was still running.\n\nI was thinking of introducing some sub-mode for --aggressive that\ndoesn't tweak the window size, but just passes down \"-f\". It would more\ngenerally cover these cases, and eta less CPU than the increased window\nsize (although \"--no-reuse-delta\" by itself is very expensive).\n\n\n>> Maybe I'm doing something wrong with the fast-import stream that is\n>> defeating Git's attempts at delta compression....\n>>\n>> Are there any utility programs available for analyzing packfiles so\n>> that I try to figure out where the inefficiencies are cropping up, so\n>> that I can try to address them?\n>>\n>> Anybody have any suggestions on what I should be looking for?\n>>\n>> If anyone would care to see this oversized packfile and perhaps offer\n>> suggestions on how I can make it more space-efficient, it can be\n>> cloned from https://github.com/drhsqlite/fossil-mirror.git - at least\n>> for now - surely I will delete that repo and regenerate it once I\n>> figure out this problem.\n>>\n>> --\n>> D. Richard Hipp\n>> drh@sqlite.org\n>>\n"}]}