{"thread":{"id":"24355","subject":"incremental push/pull for large repositories","startedAt":"2010-07-10T03:12:59Z","lastAt":"2010-07-12T15:17:45Z","messageCount":2,"participants":["Enrico Weigelt","Avery Pennarun"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"145258","messageId":"20100710031258.GA554@nibiru.local","threadId":"24355","inReplyTo":null,"subject":"incremental push/pull for large repositories","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-07-10T03:12:59Z","receivedAt":"2010-07-10T03:12:59Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"\nHi folks,\n\n\nI often have situations where I've rebased branches with large files \n(10th of megabytes per file) and pushing them to the remote. Normally \nthese files themselves stay untouched, but the history changes (eg. \ncommits reordered, several changes in smaller files,etc).\n\nIt seem that on each push, the whole branch is transferred, including\nall the large files, which already exist on the remote site. Is there\nany way to prevent this ?\n\nIMHO it would be enough having a negotiation on which objects really\nhave to be transferred before creating and transmitting the actual\npack file. That would save _much_ traffic (and transmit time) for\nthose situations. \n\nThis could also be used to remotely repair broken repositories: \njust repack leaving off the broken objects and fetch the missing\nones from remotes.\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone:  +49 36207 519931  email: weigelt@metux.de\n mobile: +49 151 27565287  icq:   210169427         skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"145369","messageId":"AANLkTin_syCNhoLFsMA8qwWkGBmW2kOZ-y7UrZkKZN_L@mail.gmail.com","threadId":"24355","inReplyTo":"20100710031258.GA554@nibiru.local","subject":"Re: incremental push/pull for large repositories","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-12T15:17:45Z","receivedAt":"2010-07-12T15:17:45Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Fri, Jul 9, 2010 at 11:12 PM, Enrico Weigelt <weigelt@metux.de> wrote:\n> I often have situations where I've rebased branches with large files\n> (10th of megabytes per file) and pushing them to the remote. Normally\n> these files themselves stay untouched, but the history changes (eg.\n> commits reordered, several changes in smaller files,etc).\n>\n> It seem that on each push, the whole branch is transferred, including\n> all the large files, which already exist on the remote site. Is there\n> any way to prevent this ?\n\nI was hoping someone else would have replied to you with a brilliant\nsolution to this by now, but I guess not, so I'll try with my limited\nknowledge.  I've seen this behaviour as well.\n\n>From what I understand, git uses an algorithm something like this to\ndetermine which objects need to be transmitted by a push:\n\n- find the latest commit T on the remote side that is also in the\nbranch you want to push.  (This part isn't an exhaustive search, and\nmight be off by a few commits if both ends have new changes, but this\nproblem usually happens only with fetch/pull, not push.)\n\n- on the client doing the push, get a list of all objects in all new\ncommits that weren't in commit T and generate and send the pack.\n\nAs you can imagine, this is terribly non-optimal.  For example, if you\nuse 'git revert', it uploads all the objects you reverted, even though\nthey obviously already existed in the remote repo.  Example:\n\n    #!/bin/sh\n    set -e\n    cd /tmp\n    mkdir repo\n    cd repo\n    git init --bare\n    cd ..\n    git clone repo worktree\n    cd worktree\n    for i in $(seq 1000); do echo $i >$i; done\n    git add .\n    git commit -m orig\n    git push  # sends about 1000 objects\n    echo\n    echo\n    for i in $(seq 1000); do echo $i >>$i; done\n    git commit -a -m doubled\n    git push  # sends about 1000 objects\n    echo\n    echo\n    git revert --no-edit HEAD\n    git push  # sends about 1000 objects (again!)\n\nThe promising looking \"--thin\" option to git-push doesn't help this at\nall.  I don't really know what it does, but whatever it does seems to\nbe relatively ineffective.  (I guess that's why it's not the default.)\n\nYou can imagine lots of ways to improve this, of course.  There's a\ntradeoff between searching the history for old objects (which can be\nslow in a huge repo) vs. just sending them and discarding duplicates\non the remote server.  For many projects, the tradeoff is an easy one:\njust send the files, since they're tiny anyway, and sending them is\nmuch faster than exhaustively searching the history.  But as soon as\nhuge files or huge numbers of files start to get involved, the\nsituation changes fast.\n\nbup (http://github.com/apenwarr/bup) uses a totally different method:\nthe server sends you its pack .idx files, and you never push any\nobject that matches anything in those .idx files.  That works fine in\nbup, because bup repositories are virtually never repacked.  (You pay\na time/bandwidth cost when you first talk to a repo, but it's worth it\nto potentially avoid re-sending gigabytes worth of data.)  But in git,\nwhere repacking happens frequently, this wouldn't fly, because the\nindexes change every time someone runs \"git gc\".\n\nIf you came up with a patch to do improved packing/negotiation, I bet\nit would be accepted.  Of course, it would have to either be optional\nor have a decent heuristic for when to enable itself, because *most*\nof the time, the default git behaviour is probably as fast as\npossible.\n\nHave fun,\n\nAvery\n"}]}