{"thread":{"id":"40288","subject":"Questions about git-push for huge repositories","startedAt":"2015-09-06T08:16:31Z","lastAt":"2015-09-08T21:54:57Z","messageCount":9,"participants":["Levin Du","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"269526","messageId":"CAN6cQGPcGpaXUGu_7aaeJtMbruMocte-5po97vG5r39f=YdTZQ@mail.gmail.com","threadId":"40288","inReplyTo":null,"subject":"Questions about git-push for huge repositories","fromName":"Levin Du","fromEmail":"zslevin@gmail.com","sentAt":"2015-09-06T08:16:31Z","receivedAt":"2015-09-06T08:16:31Z","isPatch":false,"sender":{"key":"zslevin@gmail.com","avatar":null},"body":"Hi all,\n\nI meet with a strange problem:\n\nI've two repositories, with sizes:\n  - A:  6.1G\n  - B:  6G\n\nBoth A & B have been 'git gc' with:\n  git reflog expire --expire=now --all\n  git gc --prune=now --aggressive\n\nSince A & B share many common files, to save disk space, I'd like to merge them:\n(note: branch of A & B are independent, i.e. have no common ancestor.)\n   git clone --bare A  C\n   (cd B; git push ../C master:master_b)\n\nRepo C's size has grown to 12G. Doing a 'git gc' again, it drops to 6.2G.\n\nI expect that 'git push' push only new files and commits, which will\nsave lots of space.\nYet it turns out I'm wrong. Since Repo A has been published, pushing branch of B\nwill double the repo size, which is impossible for the storage limit.\n\nAny suggestions? Thanks in advance.\n\nBest Regards,\nLevin Du\n"},{"id":"269540","messageId":"xmqq4mj7bfsf.fsf@gitster.mtv.corp.google.com","threadId":"40288","inReplyTo":"CAN6cQGPcGpaXUGu_7aaeJtMbruMocte-5po97vG5r39f=YdTZQ@mail.gmail.com","subject":"Re: Questions about git-push for huge repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-09-06T17:48:00Z","receivedAt":"2015-09-06T17:48:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Levin Du <zslevin@gmail.com> writes:\n\n> Since A & B share many common files, to save disk space, I'd like to merge them:\n> (note: branch of A & B are independent, i.e. have no common ancestor.)\n\nNot having any shared history is exactly the cause.  If the\noptimization were to exchange list of all the commits, blobs and\ntrees each side has and sending only the ones that the receiving end\nlacks, you would get the result you seem to be expecting, but that\napproach is not taken because it is impractically expensive.\n\nInstead, the object transfer is optimized by comparing what commits\neach side has and sending trees and blobs that are reachable from\nthe commits that the receiving side does not have.  This approach\ndoes not have to exchange the list of trees and blobs at all, and in\na pair of repositories for the same project, it does not even have\nto send the list of all commits, because traversing from the tips of\nhistories and exchanging more recent ones iteratively is expected to\nfind commits common to both and because of the history graph is a\nDAG, we know what is behind commits that are common exist on both\nends.\n"},{"id":"269549","messageId":"CAN6cQGMf089ERn2kZbFpHJ6vyJ4BnjCm-m-E+hQsduH55XFvKw@mail.gmail.com","threadId":"40288","inReplyTo":"xmqq4mj7bfsf.fsf@gitster.mtv.corp.google.com","subject":"Re: Questions about git-push for huge repositories","fromName":"Levin Du","fromEmail":"zslevin@gmail.com","sentAt":"2015-09-07T01:05:41Z","receivedAt":"2015-09-07T01:05:41Z","isPatch":false,"sender":{"key":"zslevin@gmail.com","avatar":null},"body":"> Instead, the object transfer is optimized by comparing what commits\n> each side has and sending trees and blobs that are reachable from\n> the commits that the receiving side does not have.\n\nThe sender A sends all the commits that the receiver B does not have.\nThe commits contains trees and blobs. In my situation, branch in A has\nonly one commit. It seems that B has received lots of duplicate blobs,\nconcluded from the GC result.\n\nWhat I do not understand is, how duplicate blobs happen in a git repository?\nGit repository is famous for its content addressing storage system.\nI guess that A sends its packed file to B directly, no matter what are\nalready in\nB.\n"},{"id":"269550","messageId":"CAN6cQGNGP+n3L=tuRCymOTWFCnFNsq-tFHkaNm+W=o726mjmmw@mail.gmail.com","threadId":"40288","inReplyTo":"CAN6cQGMf089ERn2kZbFpHJ6vyJ4BnjCm-m-E+hQsduH55XFvKw@mail.gmail.com","subject":"Re: Questions about git-push for huge repositories","fromName":"Levin Du","fromEmail":"zslevin@gmail.com","sentAt":"2015-09-07T03:51:52Z","receivedAt":"2015-09-07T03:51:52Z","isPatch":false,"sender":{"key":"zslevin@gmail.com","avatar":null},"body":"I try to use 'git replace --graft' to work aroud this. Here's the process:\n  cd A\n  fetch ../B master:master_b\n  git replace --graft master_b master_a\n  # now master_b's parent is master_a\n\n  # do a filter-branch to make the stone solid\n  git filter-branch --tag-name-filter cat -- master_a..master_b\n\n  # prune all the old refs and do gc\n  git replace -d <origin_commit_of_master_b>\n  git update-ref -d refs/original/refs/heads/master_b\n  git reflog expire --expire=now --all\n  git gc --prune=now --aggressive\n\nAnd I'd like to make master_b look orphan, so using 'git replace' again:\n   git replace --graft master_b\n   git log master_b\n   # only show one commit, fine\n   git push /path/to/public/A master_b\n   # small amount of data pushed\n   du -hs  /path/to/public/A\n   # 6.2 GiB\n\nAll are fine, except when I want to push the replace ref:\n   git push /path/to/public/A 'refs/replace/*'\n\nIt pushes 6 GiB data again.\n\nSo right now, 'git replace --graft master_b' needs to run by users\nif they need a tidy history view.\n"},{"id":"269586","messageId":"CAN6cQGOO540FV9bTQPks+1nHS1xO10Rv8iNzAj8-cBihQ4_kEw@mail.gmail.com","threadId":"40288","inReplyTo":"CAN6cQGNGP+n3L=tuRCymOTWFCnFNsq-tFHkaNm+W=o726mjmmw@mail.gmail.com","subject":"Re: Questions about git-push for huge repositories","fromName":"Levin Du","fromEmail":"zslevin@gmail.com","sentAt":"2015-09-08T01:30:09Z","receivedAt":"2015-09-08T01:30:09Z","isPatch":false,"sender":{"key":"zslevin@gmail.com","avatar":null},"body":"I consider 'git push' need further optimization.\n\nTake kernel source code for example:\n\n# Clone the kernel to A and B\n$ git --version\ngit version 2.3.2\n$ git clone --bare  ../kernel/ A\n$ git clone --bare  ../kernel/ B\n\n# Create the orphan commit and check\n$ cd A\n$ git branch test\nSwitched to a new branch 'test'\n$ git replace --graft test\n$ git rev-parse test\ncbbae6741c60c9e09f87521e3a79810abd6a2fda\n$ git rev-parse test^{tree}\n929bdce0b48ca6079ad281a9d8ba24de3e49881a\n$ git rev-parse replace/cbbae6741c60c9e09f87521e3a79810abd6a2fda\n82d3e9ce1ca062c219f1209c5291ccd5603e5302\n$ git rev-parse 82d3e9ce1ca062c219f1209c5291ccd5603e5302^{tree}\n929bdce0b48ca6079ad281a9d8ba24de3e49881a\n$ git log --pretty=oneline 82d3e9ce1ca062c219f1209c5291ccd5603e5302 | wc -l\n1\n\nWe can see that commit 82d3e9ce1ca062c219f1209c5291ccd5603e5302 (root commit)\nis meant to replace for commit cbbae6741c60c9e09f87521e3a79810abd6a2fda .\nThey both contain the same tree 929bdce0b48ca6079ad281a9d8ba24de3e49881a .\n\n$ du -hs ../B\n1.6G ../B\n$ git push ../B 'refs/replace/*'\nCounting objects: 51216, done.\nDelta compression using up to 8 threads.\nCompressing objects: 100% (48963/48963), done.\nWriting objects: 100% (51216/51216), 139.61 MiB | 17.88 MiB/s, done.\nTotal 51216 (delta 3647), reused 34580 (delta 1641)\nTo ../B\n* [new branch]\nrefs/replace/cbbae6741c60c9e09f87521e3a79810abd6a2fda ->\nrefs/replace/cbbae6741c60c9e09f87521e3a79810abd6a2fda\n$ du -hs ../B\n1.7G ../B\n\nIt takes some time for 'git push' to compress the objects and B has\nfinally increased 0.1G,\nwhich is for the newly commit whose tree is already in the repository.\n"},{"id":"269590","messageId":"20150908050027.GB26331@sigill.intra.peff.net","threadId":"40288","inReplyTo":"CAN6cQGMf089ERn2kZbFpHJ6vyJ4BnjCm-m-E+hQsduH55XFvKw@mail.gmail.com","subject":"Re: Questions about git-push for huge repositories","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-09-08T05:00:27Z","receivedAt":"2015-09-08T05:00:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 07, 2015 at 09:05:41AM +0800, Levin Du wrote:\n\n> > Instead, the object transfer is optimized by comparing what commits\n> > each side has and sending trees and blobs that are reachable from\n> > the commits that the receiving side does not have.\n> \n> The sender A sends all the commits that the receiver B does not have.\n> The commits contains trees and blobs. In my situation, branch in A has\n> only one commit. It seems that B has received lots of duplicate blobs,\n> concluded from the GC result.\n\nRight. B tells A \"I already have this commit\", but A does not already\nhave it, so that information is not helpful. It cannot make any\nassumptions about what B has, and must send all trees and blobs\nreferenced by its commit.\n\n> What I do not understand is, how duplicate blobs happen in a git repository?\n> Git repository is famous for its content addressing storage system.\n> I guess that A sends its packed file to B directly, no matter what are\n> already in B.\n\nNot exactly.  During a push, git may or may not keep the packfile sent\nover the wire, depending on the number of objects in it and the\nreceive.unpackLimit config setting. The same object can exist in two\nseparate packfiles. One of the effects of \"git gc\" is to remove such\nduplicates.\n\nSo A effectively does send its whole pack in this case, but only because\nit cannot find any shared history with B (and B keeps it as-is until the\nnext gc because it is over the unpackLimit).\n\n-Peff\n"},{"id":"269593","messageId":"20150908054432.GC26331@sigill.intra.peff.net","threadId":"40288","inReplyTo":"CAN6cQGOO540FV9bTQPks+1nHS1xO10Rv8iNzAj8-cBihQ4_kEw@mail.gmail.com","subject":"Re: Questions about git-push for huge repositories","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-09-08T05:44:33Z","receivedAt":"2015-09-08T05:44:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 08, 2015 at 09:30:09AM +0800, Levin Du wrote:\n\n> Take kernel source code for example:\n> \n> # Clone the kernel to A and B\n> $ git --version\n> git version 2.3.2\n> $ git clone --bare  ../kernel/ A\n> $ git clone --bare  ../kernel/ B\n\nOK, two repos with the same source.\n\n> # Create the orphan commit and check\n> $ cd A\n> $ git branch test\n> Switched to a new branch 'test'\n> $ git replace --graft test\n> $ git rev-parse test\n> cbbae6741c60c9e09f87521e3a79810abd6a2fda\n> $ git rev-parse test^{tree}\n> 929bdce0b48ca6079ad281a9d8ba24de3e49881a\n> $ git rev-parse replace/cbbae6741c60c9e09f87521e3a79810abd6a2fda\n> 82d3e9ce1ca062c219f1209c5291ccd5603e5302\n> $ git rev-parse 82d3e9ce1ca062c219f1209c5291ccd5603e5302^{tree}\n> 929bdce0b48ca6079ad281a9d8ba24de3e49881a\n> $ git log --pretty=oneline 82d3e9ce1ca062c219f1209c5291ccd5603e5302 | wc -l\n> 1\n\nSo you've created a new commit object, 82d3e9ce1, which has the same\ntree as the original branch, but no parents.\n\nNote that fetch and push do not respect the \"replace\" mechanism. They\ncan't, because we have no idea if the other side of the connection\nshares our \"replace\" view of the world. So if I use \"replace\" to say\nthat commit X has parent Y, I cannot assume that pushing to some _other_\nrepository with X means that they also have all of Y.\n\nBut it should be OK, of course, to push the new orphan commit. I.e., if\nwe are pushing the object itself, not caring that it is part of a\n\"replace\" mechanism, that should be no different than pushing any other\ncommit.\n\n> $ du -hs ../B\n> 1.6G ../B\n> $ git push ../B 'refs/replace/*'\n> Counting objects: 51216, done.\n> Delta compression using up to 8 threads.\n> Compressing objects: 100% (48963/48963), done.\n> Writing objects: 100% (51216/51216), 139.61 MiB | 17.88 MiB/s, done.\n> Total 51216 (delta 3647), reused 34580 (delta 1641)\n> To ../B\n> * [new branch]\n> refs/replace/cbbae6741c60c9e09f87521e3a79810abd6a2fda ->\n> refs/replace/cbbae6741c60c9e09f87521e3a79810abd6a2fda\n> $ du -hs ../B\n> 1.7G ../B\n> \n> It takes some time for 'git push' to compress the objects and B has\n> finally increased 0.1G,\n> which is for the newly commit whose tree is already in the repository.\n\nRight, this is due to the commit-walking that Junio explained earlier.\nWe walk the commits only, and then expand the positive side (things the\nother side wants) into trees and blobs. Even though we know about a\ncommit that the other side has that points to the tree, we don't make\nthe connection.\n\nYou can get a more thorough answer by expanding and marking all trees\nand blobs, taking the set difference between all of the objects you want\nto send, and all of the objects you know the other side has. I.e.,\nbasically:\n\n  # what we want to send\n  git rev-list --objects 82d3e9ce1ca062c219f1209c5291ccd5603e5302 | sort >want\n\n  # what we know the other side has; turn off replacements, since we\n  # want the real value, not with our fake replace overlaid\n  git --no-replace-objects rev-list --objects refs/heads/master | sort >have\n\n  # set difference\n  comm -23 want have\n\nwhich should consist of only the one commit. But if you actually ran\nthat, you may notice that the second rev-list takes a long time to run.\nIn your exact case, one can get lucky by progressively drilling down\ninto commits and their trees (since the tip commit of \"master\" happens\nto share the identical tree with our new fake commit). But that is\nrather an uncommon example, and in more normal cases of fetching from\nsomebody, building on top, and then pushing back up, it is much more\nexpensive. In those cases it is much more efficient to walk the small\nnumber of new commits and then expand only their newly-added objects.\n\nIf you turn on reachability bitmaps, git _will_ do the thorough set\ndifference, because it becomes much cheaper to do so. E.g., try:\n\n    git repack -adb\n\nin repo A to build a single pack with bitmaps enabled. Then a subsequent\npush should send only a single object (the new commit).\n\nOf course the time spent building the bitmaps is larger than a single\npush, so this is not a good strategy if you are just trying to send one\ntree.\n\n-Peff\n"},{"id":"269619","messageId":"xmqq4mj493cp.fsf@gitster.mtv.corp.google.com","threadId":"40288","inReplyTo":"20150908054432.GC26331@sigill.intra.peff.net","subject":"Re: Questions about git-push for huge repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-09-08T18:24:06Z","receivedAt":"2015-09-08T18:24:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> If you turn on reachability bitmaps, git _will_ do the thorough set\n> difference, because it becomes much cheaper to do so. E.g., try:\n>\n>     git repack -adb\n>\n> in repo A to build a single pack with bitmaps enabled. Then a subsequent\n> push should send only a single object (the new commit).\n\nHmph, A has the tip of B, and has a new commit B hasn't seen but A\nknows that new commit's tree matches the tree of the tip of B.\n\nWouldn't --thin transfer from A to B know to send only that new\ncommit object without sending anything below the tree in such a\ncase, even without the bitmap?\n"},{"id":"269630","messageId":"20150908215457.GC24159@sigill.intra.peff.net","threadId":"40288","inReplyTo":"xmqq4mj493cp.fsf@gitster.mtv.corp.google.com","subject":"Re: Questions about git-push for huge repositories","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-09-08T21:54:57Z","receivedAt":"2015-09-08T21:54:57Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 08, 2015 at 11:24:06AM -0700, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> > If you turn on reachability bitmaps, git _will_ do the thorough set\n> > difference, because it becomes much cheaper to do so. E.g., try:\n> >\n> >     git repack -adb\n> >\n> > in repo A to build a single pack with bitmaps enabled. Then a subsequent\n> > push should send only a single object (the new commit).\n> \n> Hmph, A has the tip of B, and has a new commit B hasn't seen but A\n> knows that new commit's tree matches the tree of the tip of B.\n> \n> Wouldn't --thin transfer from A to B know to send only that new\n> commit object without sending anything below the tree in such a\n> case, even without the bitmap?\n\nI started to write about that in my analysis, but it gets confusing\nquickly. There are actually many tip trees, because A and B also share\nall of their tags. We do not mark every blob of every tip tree as a\npreferred base, because it is expensive to do so (and it just clogs our\nobject array).  Plus this only helps in the narrow circumstance that we\nhave the exact same tree as the tip (and not, say, the same tree as\nmaster^, which I think it would be unreasonable to expect git to find).\n\nBut if we do:\n\n  (cd ../B && git tag | git tag -d)\n\nto delete all of the other tips besides master, leaving only the one\nthat we know has the same tree, I'd expect git to figure it out.\n\nCertainly I would not expect it to save all of the delta compression,\nin the sense that we may throw away on-disk delta bases to older objects\n(because we don't realize the other side has those older objects). But I\nwould have thought before we even hit that phase, adding those objects\nas \"preferred bases\" would have marked them as \"do not send\" in the\nfirst place.\n\nThere is code in have_duplicate_entry() to handle this. I wonder why it\ndoesn't kick in.\n\n-Peff\n"}]}