{"thread":{"id":"48915","subject":"How to speedup git clone for big binary files (disable delta compression)","startedAt":"2018-07-18T22:05:07Z","lastAt":"2018-07-19T05:34:01Z","messageCount":2,"participants":["René Scheibe","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"352992","messageId":"43b401ec-31fc-59dc-17c0-8dd7359726da@gmail.com","threadId":"48915","inReplyTo":null,"subject":"How to speedup git clone for big binary files (disable delta compression)","fromName":"René Scheibe","fromEmail":"rene.scheibe@gmail.com","sentAt":"2018-07-18T22:05:00Z","receivedAt":"2018-07-18T22:05:07Z","isPatch":false,"sender":{"key":"rene.scheibe@gmail.com","avatar":null},"body":"Hi,\n\nI was wondering why \"git clone\" seems to not respect \"-delta\" in .gitattributes.\n\n\n*Reproduction*\n\nI prepared a test repository with:\n\n- git v2.17.1\n- .gitattributes containing \"*.bin binary -delta\"\n- 10 commits with a 10 MB random binary file\n\nCode:\n---------------------------------------------------------------------\n#!/bin/bash\n\n# setup repository\ngit init --quiet repo\ncd repo\n\necho '*.bin binary -delta' > .gitattributes\ngit add .gitattributes\ngit commit --quiet -m 'attributes'\n\nfor i in $(seq 10); do\n    dd if=/dev/urandom of=data.bin bs=1MB count=10 status=none\n    git add data.bin\n    git commit --quiet -m \"data $i\"\ndone\ncd ..\n\n# create clone repository\ntime git clone --no-local repo clone\n\n# repack original repository\ncd repo\ntime git repack -a -d\n---------------------------------------------------------------------\n\nOutput:\n---------------------------------------------------------------------\nCloning into 'clone'...\nremote: Counting objects: 33, done.\nremote: Compressing objects: 100% (31/31), done.\nremote: Total 33 (delta 0), reused 0 (delta 0)\nReceiving objects: 100% (33/33), 95.40 MiB | 19.94 MiB/s, done.\n\nreal    0m25,085s\nuser    0m22,749s\nsys     0m0,948s\n\nCounting objects: 33, done.\nDelta compression using up to 2 threads.\nCompressing objects: 100% (21/21), done.\nWriting objects: 100% (33/33), done.\nTotal 33 (delta 0), reused 0 (delta 0)\n\nreal    0m5,652s\nuser    0m4,173s\nsys     0m0,178s\n---------------------------------------------------------------------\n\n\n*Observations*\n\n_time_\n\n- Cloning: \"clone\" takes always 25s\n- Optimizing: \"repack\" takes 25s with and 5s without delta compression\n\n_compressed objects_\n\n- Cloning: \"clone\" compresses always 31 objects\n- Optimizing: \"repack\" compresses 31 objects with and 21 objects without delta compression\n\n\n*Expectations*\n\nBoth operations (\"repack\" and \"clone\") are using \"pack-objects\".\n\nTherefore my expectation is that \"clone\" should respect \"-delta\" and be about as fast as \"repack\".\n\n\nCheers,\n  René\n"},{"id":"353007","messageId":"20180719053357.GA23884@sigill.intra.peff.net","threadId":"48915","inReplyTo":"43b401ec-31fc-59dc-17c0-8dd7359726da@gmail.com","subject":"Re: How to speedup git clone for big binary files (disable delta compression)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-07-19T05:33:58Z","receivedAt":"2018-07-19T05:34:01Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 19, 2018 at 12:05:00AM +0200, René Scheibe wrote:\n\n> Code:\n> ---------------------------------------------------------------------\n> #!/bin/bash\n> \n> # setup repository\n> git init --quiet repo\n> cd repo\n> \n> echo '*.bin binary -delta' > .gitattributes\n> git add .gitattributes\n> git commit --quiet -m 'attributes'\n> \n> for i in $(seq 10); do\n>     dd if=/dev/urandom of=data.bin bs=1MB count=10 status=none\n>     git add data.bin\n>     git commit --quiet -m \"data $i\"\n> done\n> cd ..\n> \n> # create clone repository\n> time git clone --no-local repo clone\n\nThis clone won't respect those attributes, because we don't dig into\nin-repo attributes. There's actually some inconsistency in how Git\nhandles attribute locations. Usually they're just read from the top of\nthe working tree, but in some instances we read them from the tree\nitself (e.g., git-archive respects some attributes from the tree it's\narchiving).\n\nIf you do:\n\n  echo \"*.bin binary -delta\" >repo/.git/info/attributes\n\nthen that does work (we always respect repo-level attributes like that).\n\n> # repack original repository\n> cd repo\n> time git repack -a -d\n\nIn this case we're reading the attributes from the working tree, and it\ndoes work. In theory the clone case could do so, too, but git-upload-pack,\nthe server side of the clone, avoids looking at the working tree at all.\nThat's something we _could_ address, but it doesn't really fix the\ngeneral case, since most clones will be from a bare repository anyway.\n\nSo in summary:\n\n  1. Depending on what you're trying to do, the .git/info/attributes\n     trick might be enough for you.\n\n  2. I do think it would be nice for more places to respect attributes\n     from in trees. There's a question of which tree, but I think in\n     general reading them from HEAD in a bare repository would do what\n     people want (it's a little funny if you're fetching branch \"foo\",\n     but HEAD points to \"bar\", but it's at least consistent with the\n     non-bare case). There's some prior art in the way we treat mailmaps\n     (in a bare repo, we read HEAD:.mailmap).\n\n     I suspect the patch may not be trivial, as I don't know how ready\n     the attributes code is to handle in-tree lookups (remember that it\n     is not just HEAD:.gitattributes we must care about, but other files\n     sprinkled through the repository, like \"HEAD:subdir/.gitattributes\".\n\n-Peff\n"}]}