{"thread":{"id":"38844","subject":"Fwd: Seems to be pushing more than necessary","startedAt":"2015-03-18T10:55:26Z","lastAt":"2015-03-23T10:56:59Z","messageCount":17,"participants":["Graham Hay","Duy Nguyen","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"257939","messageId":"CAABECY1_L34sq0VPmD9UwRcwb3Fuh95OFcF26LM2eX1z-+8vkQ@mail.gmail.com","threadId":"38844","inReplyTo":"CAABECY3HbZ4q3uo82outUmCyQLXO39H+Fd2m8bLwkaubE9gJCw@mail.gmail.com","subject":"Fwd: Seems to be pushing more than necessary","fromName":"Graham Hay","fromEmail":"grahamrhay@gmail.com","sentAt":"2015-03-18T10:55:26Z","receivedAt":"2015-03-18T10:55:26Z","isPatch":false,"sender":{"key":"grahamrhay@gmail.com","avatar":"https://gravatar.com/avatar/235d8aff572de97e6f9a16a99563aa55058a8ed1b1bf315c4285e565361a6429?d=mp&s=160"},"body":"We have a fairly large repo (~2.4GB), mainly due to binary resources\n(for an ios app). I know this can generally be a problem, but I have a\nspecific question.\n\nIf I cut a branch, and edit a few (non-binary) files, and push, what\nshould be uploaded? I assumed it was just the diff (I know whole\ncompressed files are used, I mean the differences between my branch\nand where I cut it from). Is that correct?\n\nBecause when I push, it grinds to a halt at the 20% mark, and feels\nlike it's trying to push the entire repo. If I run \"git diff --stat\n--cached origin/foo\" I see the files I would expect (i.e. just those\nthat have changed). If I run \"git format-patch origin/foo..foo\" the\npatch files total 1.7MB, which should upload in just a few seconds,\nbut I've had pushes take over an hour. I'm using git 2.2.2 on Mac OS X\n(Mavericks), and ssh (git@github.com).\n\nAm I \"doing it wrong\"? Is this the expected behaviour? If not, is\nthere anything I can do to debug it?\n\nAny help gratefully received.\n\nThanks,\n\nGraham\n"},{"id":"257940","messageId":"CACsJy8Au48iFoC=Eo4T-fm6r6tA7VffRJyH2JbuC==wa61=G5A@mail.gmail.com","threadId":"38844","inReplyTo":"CAABECY1_L34sq0VPmD9UwRcwb3Fuh95OFcF26LM2eX1z-+8vkQ@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-18T11:14:11Z","receivedAt":"2015-03-18T11:14:11Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 5:55 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n> We have a fairly large repo (~2.4GB), mainly due to binary resources\n> (for an ios app). I know this can generally be a problem, but I have a\n> specific question.\n>\n> If I cut a branch, and edit a few (non-binary) files, and push, what\n> should be uploaded? I assumed it was just the diff (I know whole\n> compressed files are used, I mean the differences between my branch\n> and where I cut it from). Is that correct?\n>\n> Because when I push, it grinds to a halt at the 20% mark, and feels\n> like it's trying to push the entire repo. If I run \"git diff --stat\n> --cached origin/foo\" I see the files I would expect (i.e. just those\n> that have changed). If I run \"git format-patch origin/foo..foo\" the\n> patch files total 1.7MB, which should upload in just a few seconds,\n> but I've had pushes take over an hour. I'm using git 2.2.2 on Mac OS X\n> (Mavericks), and ssh (git@github.com).\n>\n> Am I \"doing it wrong\"? Is this the expected behaviour? If not, is\n> there anything I can do to debug it?\n\nIt would help if you pasted the push output. For example, does it stop\nat 20% at the \"compressing objects\" line or \"writing objects\". How\nmany total objects does it say?\n\nAnother question is how big are these binary files on average? Git\nconsiders a file is \"big\" if its size is 512MB or more (see\ncore.bigFileThreshold). If your binary files are are mostly under this\nlimit, but still big enough, then git may still try to compare new\nobjects with these to find the smallest \"diff\" to send. If it's the\ncase, you could set core.bigFileThreshold to cover these binary files.\n-- \nDuy\n"},{"id":"257941","messageId":"CAABECY2_MQWAkOu=6rR=mN=wKz-Yfzsu8o2+wb6F+h4FjBe9Mw@mail.gmail.com","threadId":"38844","inReplyTo":"CACsJy8Au48iFoC=Eo4T-fm6r6tA7VffRJyH2JbuC==wa61=G5A@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Graham Hay","fromEmail":"grahamrhay@gmail.com","sentAt":"2015-03-18T11:26:58Z","receivedAt":"2015-03-18T11:26:58Z","isPatch":false,"sender":{"key":"grahamrhay@gmail.com","avatar":"https://gravatar.com/avatar/235d8aff572de97e6f9a16a99563aa55058a8ed1b1bf315c4285e565361a6429?d=mp&s=160"},"body":"> It would help if you pasted the push output. For example, does it stop\n> at 20% at the \"compressing objects\" line or \"writing objects\". How\n> many total objects does it say?\n\nIt rattles through \"compressing objects\", and the first 20% of\n\"writing objects\", then slows to a crawl.\n\n    Writing objects:  33% (3647/10804), 80.00 MiB | 112.00 KiB/s\n\n>\n> Another question is how big are these binary files on average? Git\n> considers a file is \"big\" if its size is 512MB or more (see\n> core.bigFileThreshold). If your binary files are are mostly under this\n> limit, but still big enough, then git may still try to compare new\n> objects with these to find the smallest \"diff\" to send. If it's the\n> case, you could set core.bigFileThreshold to cover these binary files.\n\nNone of the files are very big (KB rather than MB), but there's a lot\nof them. I'll try setting the threshold to something lower, thanks.\n"},{"id":"257943","messageId":"CACsJy8DdjDnacVGU83iUDj5Na1xxS6kNDbK2D0Apfv0LAuzDvA@mail.gmail.com","threadId":"38844","inReplyTo":"CAABECY2_MQWAkOu=6rR=mN=wKz-Yfzsu8o2+wb6F+h4FjBe9Mw@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-18T11:42:24Z","receivedAt":"2015-03-18T11:42:24Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 6:26 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n>> It would help if you pasted the push output. For example, does it stop\n>> at 20% at the \"compressing objects\" line or \"writing objects\". How\n>> many total objects does it say?\n>\n> It rattles through \"compressing objects\", and the first 20% of\n> \"writing objects\", then slows to a crawl.\n>\n>     Writing objects:  33% (3647/10804), 80.00 MiB | 112.00 KiB/s\n\nThis 10804 looks wrong (i.e. sending that many compressed objects).\nAlso \"80 MiB\" sent at that point. If you modify just a couple files,\nsomething is really wrong because the number of new objects may be\nhundreds at most, not thousands.\n\nv2.2.2 supports \"git fast-export --anonymize\" [1] to create an\nanonymized \"clone\" of your repo that you can share, which might help\nus understand the problem.\n\nThere's also the environment variable GIT_TRACE_PACKET that can help\nsee what's going on at the protocol level, but I think you're on your\nown because without access to this repo, SHA-1s from that trace may\nnot make much sense.\n\n[1] https://github.com/git/git/commit/a8722750985a53cc502a66ae3d68a9e42c7fdb98\n-- \nDuy\n"},{"id":"257945","messageId":"CAABECY2_A83tWaqb_6YzXwr0Tv5AYVq0KW70j+EHqrx69k2P_A@mail.gmail.com","threadId":"38844","inReplyTo":"CACsJy8DdjDnacVGU83iUDj5Na1xxS6kNDbK2D0Apfv0LAuzDvA@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Graham Hay","fromEmail":"grahamrhay@gmail.com","sentAt":"2015-03-18T12:03:38Z","receivedAt":"2015-03-18T12:03:38Z","isPatch":false,"sender":{"key":"grahamrhay@gmail.com","avatar":"https://gravatar.com/avatar/235d8aff572de97e6f9a16a99563aa55058a8ed1b1bf315c4285e565361a6429?d=mp&s=160"},"body":"Are there any commands that I can use to show exactly what it is trying to push?\n\nI'll see if I can create a (public) repo that has the same problem.\nThanks for your help.\n\n\n> This 10804 looks wrong (i.e. sending that many compressed objects).\n> Also \"80 MiB\" sent at that point. If you modify just a couple files,\n> something is really wrong because the number of new objects may be\n> hundreds at most, not thousands.\n>\n> v2.2.2 supports \"git fast-export --anonymize\" [1] to create an\n> anonymized \"clone\" of your repo that you can share, which might help\n> us understand the problem.\n>\n> There's also the environment variable GIT_TRACE_PACKET that can help\n> see what's going on at the protocol level, but I think you're on your\n> own because without access to this repo, SHA-1s from that trace may\n> not make much sense.\n>\n> [1] https://github.com/git/git/commit/a8722750985a53cc502a66ae3d68a9e42c7fdb98\n> --\n> Duy\n"},{"id":"257946","messageId":"CACsJy8AFi8-EgUJNr+KTOngeUaGfi_O5f_=qba08qJs1jcEj=w@mail.gmail.com","threadId":"38844","inReplyTo":"CAABECY2_A83tWaqb_6YzXwr0Tv5AYVq0KW70j+EHqrx69k2P_A@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-18T12:26:38Z","receivedAt":"2015-03-18T12:26:38Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 7:03 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n> Are there any commands that I can use to show exactly what it is trying to push?\n\nIt's a bit more than a command. If you push when GIT_TRACE is set to\n2, you'll see it executes \"git pack-objects\" command with all its\narguments. This command expects some input from stdin. If you can\ncapture that, you can run it by yourself to create the exact pack that\nis transferred over network. Run that pack through \"git index-pack\n--verify-stat\" will show you SHA-1 of all sent objects.\n\nIt's quite a lot of work :) I created this script named \"git\" and put\nit in $PATH to capture input for pack-objects. You'll need to update\n\"/path/to/real/git\" to point to the real binary then you'll get\n/tmp/stdin\n\n-- 8< --\n#!/bin/sh\n\nif [ \"$1\" = pack-objects ]; then\nexec tee /tmp/stdin | /path/to/real/git \"$@\"\nelse\nexec /path/to/real/git \"$@\"\nfi\n-- 8< --\n\nThe remaining steps may be this (may need tweaking)\n\ngit pack-objects '--all-progress-implied' '--revs' '--stdout' '--thin'\n'--delta-base-offset' '--progress' < /tmp/stdin | git index-pack\n--fix-thin --stdin\npack    708538afeda8eb331858680e227f7713228ce782 <-- new pack\ngit verify-pack --verbose\n.git/objects/pack/pack-708538afeda8eb331858680e227f7713228ce782.pack\nd75631bd83ebdf03d4b0d925ff6734380f801fc6 commit 567 377 12\ndd44100a7cdad113b23d31876e469b74fbe21e1b tree   15069 10492 389\n8f4bbccea759d7a47616e29bd55b3f205b3615c2 tree   3869 2831 10881\n3db0460935bc843a2a70a0e087222eec61a0ff0d blob   12379 3529 13712\n\nHere we can see this push of mine sends four objects, 1 commit, 2\ntrees and 1 blob.\n-- \nDuy\n"},{"id":"257949","messageId":"CACsJy8BqA2dm8DpDrrFN4FxHG+4gnG3Q7c3kM=YmBrkuHftvog@mail.gmail.com","threadId":"38844","inReplyTo":"CACsJy8AFi8-EgUJNr+KTOngeUaGfi_O5f_=qba08qJs1jcEj=w@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-18T12:34:44Z","receivedAt":"2015-03-18T12:34:44Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 7:26 PM, Duy Nguyen <pclouds@gmail.com> wrote:\n> It's quite a lot of work :) I created this script named \"git\" and put\n> it in $PATH to capture input for pack-objects. You'll need to update\n> \"/path/to/real/git\" to point to the real binary then you'll get\n> /tmp/stdin\n\nForgot one important sentence: You need to push again using this fake\n\"git\" program to save data in /tmp/stdin. Also you can stop the push\nwhen it goes to \"compressing objects\" phase.\n-- \nDuy\n"},{"id":"257951","messageId":"CAABECY13XT3+rEGGtKfe3Xwg4RtzcWtpL=Nm7aivv4_35TvL0w@mail.gmail.com","threadId":"38844","inReplyTo":"CACsJy8BqA2dm8DpDrrFN4FxHG+4gnG3Q7c3kM=YmBrkuHftvog@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Graham Hay","fromEmail":"grahamrhay@gmail.com","sentAt":"2015-03-18T13:16:09Z","receivedAt":"2015-03-18T13:16:09Z","isPatch":false,"sender":{"key":"grahamrhay@gmail.com","avatar":"https://gravatar.com/avatar/235d8aff572de97e6f9a16a99563aa55058a8ed1b1bf315c4285e565361a6429?d=mp&s=160"},"body":"I created a repo with over 1GB of images, but it works as expected\n(only pushed 3 objects).\n\nSorry, I must have done something wrong. I put that script in\n~/Applications, and checked it worked. Then I ran this:\n\n$ GIT_TRACE=2 PATH=~/Applications:$PATH git push --set-upstream origin git-wtf\n12:48:28.839026 git.c:349               trace: built-in: git 'push'\n'--set-upstream' 'origin' 'git-wtf'\n12:48:28.907605 run-command.c:351       trace: run_command: 'ssh'\n'git@github.com' 'git-receive-pack\n'\\''grahamrhay/bornlucky-ios.git'\\'''\n12:48:30.137410 run-command.c:351       trace: run_command:\n'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin'\n'--delta-base-offset' '--progress'\n12:48:30.138246 exec_cmd.c:130          trace: exec: 'git'\n'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin'\n'--delta-base-offset' '--progress'\n12:48:30.144783 git.c:349               trace: built-in: git\n'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin'\n'--delta-base-offset' '--progress'\nCounting objects: 10837, done.\nDelta compression using up to 4 threads.\nCompressing objects: 100% (9301/9301), done.\nWriting objects:  21% (2276/10837)\n\nbut there was nothing in /tmp/stdin. Have I missed a step? I tried\nchanging the tee to point to ~ in case it was permissions related.\n\nI fear this is some Mac nonsense. I added an echo in the script, but\nit only gets called for the first git incantation.\n\n\nOn 18 March 2015 at 12:34, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Wed, Mar 18, 2015 at 7:26 PM, Duy Nguyen <pclouds@gmail.com> wrote:\n>> It's quite a lot of work :) I created this script named \"git\" and put\n>> it in $PATH to capture input for pack-objects. You'll need to update\n>> \"/path/to/real/git\" to point to the real binary then you'll get\n>> /tmp/stdin\n>\n> Forgot one important sentence: You need to push again using this fake\n> \"git\" program to save data in /tmp/stdin. Also you can stop the push\n> when it goes to \"compressing objects\" phase.\n> --\n> Duy\n"},{"id":"257952","messageId":"CACsJy8D16ERGH=P4etGRpZM_3BR9iYZBsUnWBGAbayVqmEhhXw@mail.gmail.com","threadId":"38844","inReplyTo":"CAABECY13XT3+rEGGtKfe3Xwg4RtzcWtpL=Nm7aivv4_35TvL0w@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-18T13:33:48Z","receivedAt":"2015-03-18T13:33:48Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 8:16 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n> I created a repo with over 1GB of images, but it works as expected\n> (only pushed 3 objects).\n>\n> Sorry, I must have done something wrong. I put that script in\n> ~/Applications, and checked it worked. Then I ran this:\n>\n> $ GIT_TRACE=2 PATH=~/Applications:$PATH git push --set-upstream origin git-wtf\n\nI think I encountered the same problem. Inserting\n--exec-path=$HOME/Applications between \"git\" and \"push\" was probably\nwhat made it work for me. Haven't investigated the reason yet. We\nreally should have an easier way to get this info without jumping\nthrough hoops like this.\n-- \nDuy\n"},{"id":"257960","messageId":"CAABECY2x=G+nYdSyxpz8MvE4G6RSwcGSXu1LzwAKYG7Mqwk1kA@mail.gmail.com","threadId":"38844","inReplyTo":"CACsJy8D16ERGH=P4etGRpZM_3BR9iYZBsUnWBGAbayVqmEhhXw@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Graham Hay","fromEmail":"grahamrhay@gmail.com","sentAt":"2015-03-18T15:14:31Z","receivedAt":"2015-03-18T15:14:31Z","isPatch":false,"sender":{"key":"grahamrhay@gmail.com","avatar":"https://gravatar.com/avatar/235d8aff572de97e6f9a16a99563aa55058a8ed1b1bf315c4285e565361a6429?d=mp&s=160"},"body":"Got there eventually!\n\n$ git verify-pack --verbose bar.pack\ne13e21a1f49704ed35ddc3b15b6111a5f9b34702 commit 220 152 12\n03691863451ef9db6c69493da1fa556f9338a01d commit 334 227 164\n... snip ...\nchain length = 50: 2 objects\nbar.pack: ok\n\nNow what do I do with it :)\n\nOn 18 March 2015 at 13:33, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Wed, Mar 18, 2015 at 8:16 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n>> I created a repo with over 1GB of images, but it works as expected\n>> (only pushed 3 objects).\n>>\n>> Sorry, I must have done something wrong. I put that script in\n>> ~/Applications, and checked it worked. Then I ran this:\n>>\n>> $ GIT_TRACE=2 PATH=~/Applications:$PATH git push --set-upstream origin git-wtf\n>\n> I think I encountered the same problem. Inserting\n> --exec-path=$HOME/Applications between \"git\" and \"push\" was probably\n> what made it work for me. Haven't investigated the reason yet. We\n> really should have an easier way to get this info without jumping\n> through hoops like this.\n> --\n> Duy\n"},{"id":"258020","messageId":"CACsJy8B=Dx2+OWch1zx_xg91de9yhttVMwgE1SxDOuSTns2b_w@mail.gmail.com","threadId":"38844","inReplyTo":"CAABECY2x=G+nYdSyxpz8MvE4G6RSwcGSXu1LzwAKYG7Mqwk1kA@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-19T10:59:19Z","receivedAt":"2015-03-19T10:59:19Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 10:14 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n> Got there eventually!\n>\n> $ git verify-pack --verbose bar.pack\n> e13e21a1f49704ed35ddc3b15b6111a5f9b34702 commit 220 152 12\n> 03691863451ef9db6c69493da1fa556f9338a01d commit 334 227 164\n> ... snip ...\n> chain length = 50: 2 objects\n> bar.pack: ok\n>\n> Now what do I do with it :)\n\nTry \"fast-export --anonymize\" as that would help us understand this.\nOr you can try to see if these commits exist in the remote repo. If\nyes, that only confirms that push sends more that it should, but it's\nhard to know why. Maybe if you fire up gitk and mark them commits,\nyou'll figure out a connection. There are actually objects in this\npack that are expected to exist in remote repo, but it's hard to\ntell..\n-- \nDuy\n"},{"id":"258038","messageId":"xmqq3850kf6d.fsf@gitster.dls.corp.google.com","threadId":"38844","inReplyTo":"CAABECY1_L34sq0VPmD9UwRcwb3Fuh95OFcF26LM2eX1z-+8vkQ@mail.gmail.com","subject":"Re: Fwd: Seems to be pushing more than necessary","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-03-19T18:36:26Z","receivedAt":"2015-03-19T18:36:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Graham Hay <grahamrhay@gmail.com> writes:\n\n> We have a fairly large repo (~2.4GB), mainly due to binary resources\n> (for an ios app). I know this can generally be a problem, but I have a\n> specific question.\n>\n> If I cut a branch, and edit a few (non-binary) files, and push, what\n> should be uploaded? I assumed it was just the diff (I know whole\n> compressed files are used, I mean the differences between my branch\n> and where I cut it from). Is that correct?\n\nIf you start from this state:\n\n (the 'origin')                    (you)\n    ---Z---A         clone ->    ---Z---A\n\nand edit a few files, say, a/b, a/c and d/e/f, and committed to make\nthe history look like this:\n\n (the 'origin')                    (you)\n    ---Z---A                     ---Z---A---B\n\ni.e. \"git diff --name-only A B\" would show these three files, then\nthe next push from you to the origin, i.e.\n\n (the 'origin')                    (you)\n    ---Z---A---B    <- push      ---Z---A---B\n\nwould involve transferring from you to the origin of the following:\n\n * The commit object that holds the message, authorship, etc. for B\n * The top-level tree object of commit B (as that is different from\n   that of A)\n * The tree object for 'a', 'd', 'd/e' and the blob object for\n   'a/b', 'a/c', and 'd/e/f'.\n\nHowever, that assumes that nothing is happening on the 'origin'\nside.\n\nIf the 'origin', for example, rewound its head to Z before you\nattempt to push your B, then you may end up sending objects that do\nnot exist in Z that are reachable from B.  Just like the above\nbullet points enumerated what is different between A and B, you\ncan enumerate what is different between Z and A and add that to the\nabove set.  That would be what will be sent.\n\nIf the 'origin' updated its tip to a commit you do not even know\nabout, normally you will be prevented from pushing B because we\nwould not want you to lose somebody else's work.  If you forced such\npush, then you may end up sending a lot more.\n"},{"id":"258096","messageId":"CAABECY1vOre+RbXKCnZqHr4rZSHHpYA=tnRK6fSnkjeyFv2ZAg@mail.gmail.com","threadId":"38844","inReplyTo":"xmqq3850kf6d.fsf@gitster.dls.corp.google.com","subject":"Re: Fwd: Seems to be pushing more than necessary","fromName":"Graham Hay","fromEmail":"grahamrhay@gmail.com","sentAt":"2015-03-20T09:20:18Z","receivedAt":"2015-03-20T09:20:18Z","isPatch":false,"sender":{"key":"grahamrhay@gmail.com","avatar":"https://gravatar.com/avatar/235d8aff572de97e6f9a16a99563aa55058a8ed1b1bf315c4285e565361a6429?d=mp&s=160"},"body":"That all seems quite reasonable, and is what I would expect to happen.\n\nHowever at the moment, if I create a branch from master and edit one\nline in one file,\nwith no other changes on the remote, it takes me over an hour to push\nthe new branch.\n\nOn 19 March 2015 at 18:36, Junio C Hamano <gitster@pobox.com> wrote:\n> Graham Hay <grahamrhay@gmail.com> writes:\n>\n>> We have a fairly large repo (~2.4GB), mainly due to binary resources\n>> (for an ios app). I know this can generally be a problem, but I have a\n>> specific question.\n>>\n>> If I cut a branch, and edit a few (non-binary) files, and push, what\n>> should be uploaded? I assumed it was just the diff (I know whole\n>> compressed files are used, I mean the differences between my branch\n>> and where I cut it from). Is that correct?\n>\n> If you start from this state:\n>\n>  (the 'origin')                    (you)\n>     ---Z---A         clone ->    ---Z---A\n>\n> and edit a few files, say, a/b, a/c and d/e/f, and committed to make\n> the history look like this:\n>\n>  (the 'origin')                    (you)\n>     ---Z---A                     ---Z---A---B\n>\n> i.e. \"git diff --name-only A B\" would show these three files, then\n> the next push from you to the origin, i.e.\n>\n>  (the 'origin')                    (you)\n>     ---Z---A---B    <- push      ---Z---A---B\n>\n> would involve transferring from you to the origin of the following:\n>\n>  * The commit object that holds the message, authorship, etc. for B\n>  * The top-level tree object of commit B (as that is different from\n>    that of A)\n>  * The tree object for 'a', 'd', 'd/e' and the blob object for\n>    'a/b', 'a/c', and 'd/e/f'.\n>\n> However, that assumes that nothing is happening on the 'origin'\n> side.\n>\n> If the 'origin', for example, rewound its head to Z before you\n> attempt to push your B, then you may end up sending objects that do\n> not exist in Z that are reachable from B.  Just like the above\n> bullet points enumerated what is different between A and B, you\n> can enumerate what is different between Z and A and add that to the\n> above set.  That would be what will be sent.\n>\n> If the 'origin' updated its tip to a commit you do not even know\n> about, normally you will be prevented from pushing B because we\n> would not want you to lose somebody else's work.  If you forced such\n> push, then you may end up sending a lot more.\n"},{"id":"258353","messageId":"CACsJy8AqrBc4HKEKSHmkSq2wKwUgUEWrZu3=9-WGsM+txR8S-g@mail.gmail.com","threadId":"38844","inReplyTo":"CAABECY063__YDW2jXPSnuvxp0kj=jB+k09mOEW3suEn5syAyXQ@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-23T10:05:58Z","receivedAt":"2015-03-23T10:05:58Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 19, 2015 at 6:11 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n>> Try \"fast-export --anonymize\" as that would help us understand this.\n>\n> Attached.\n\nThe bad news is it seems to be working for me (I recreated the remote\nrepo from this dump). I notice that you have two remotes, one shares\nmany refs (the remote \"ref39\"). The other, \"ref2\", does not share any\nSHA-1 with refs in \".git/refs/heads/\". Any chance you push to a\nno-share remote, which results in a lot of objects to be sent?\n-- \nDuy\n"},{"id":"258356","messageId":"CAABECY2xqH9ya3WGO_VR54c4Yi4CJ2355Wdh=cggpUV348D6cg@mail.gmail.com","threadId":"38844","inReplyTo":"CACsJy8AqrBc4HKEKSHmkSq2wKwUgUEWrZu3=9-WGsM+txR8S-g@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Graham Hay","fromEmail":"grahamrhay@gmail.com","sentAt":"2015-03-23T10:35:42Z","receivedAt":"2015-03-23T10:35:42Z","isPatch":false,"sender":{"key":"grahamrhay@gmail.com","avatar":"https://gravatar.com/avatar/235d8aff572de97e6f9a16a99563aa55058a8ed1b1bf315c4285e565361a6429?d=mp&s=160"},"body":"Hmm. I'm using a \"private\" fork of a repo, I pull from one and push to\nthe other, e.g.\n\ngit fetch foo\ngit rebase foo/master\ngit push --set-upstream origin bar\n\nIt's quite possible my workflow is causing the problem, but I'm not\nsure what I could do differently. What do you mean by a \"no-share\nremote\"?\n\nOn 23 March 2015 at 10:05, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Thu, Mar 19, 2015 at 6:11 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n>>> Try \"fast-export --anonymize\" as that would help us understand this.\n>>\n>> Attached.\n>\n> The bad news is it seems to be working for me (I recreated the remote\n> repo from this dump). I notice that you have two remotes, one shares\n> many refs (the remote \"ref39\"). The other, \"ref2\", does not share any\n> SHA-1 with refs in \".git/refs/heads/\". Any chance you push to a\n> no-share remote, which results in a lot of objects to be sent?\n> --\n> Duy\n"},{"id":"258358","messageId":"CACsJy8Cddt7S0NLcdz1ji_dDWUn_wCxie4+zxs5Akx1ObAvNEw@mail.gmail.com","threadId":"38844","inReplyTo":"CAABECY2xqH9ya3WGO_VR54c4Yi4CJ2355Wdh=cggpUV348D6cg@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-23T10:41:30Z","receivedAt":"2015-03-23T10:41:30Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 23, 2015 at 5:35 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n> Hmm. I'm using a \"private\" fork of a repo, I pull from one and push to\n> the other, e.g.\n>\n> git fetch foo\n> git rebase foo/master\n> git push --set-upstream origin bar\n>\n> It's quite possible my workflow is causing the problem, but I'm not\n> sure what I could do differently. What do you mean by a \"no-share\n> remote\"?\n\nI mean the refs (and associated objects) that are available on \"foo\"\nmay be not available on \"bar\" so when you push to \"origin\" you just\nneed to send more. That rebase could generate lots of new objects to\npush out too, I think.\n-- \nDuy\n"},{"id":"258359","messageId":"CAABECY3xLHr1CWsezF6EjSWKkYpAW9VPFk31zznzn12_GVgAeA@mail.gmail.com","threadId":"38844","inReplyTo":"CACsJy8Cddt7S0NLcdz1ji_dDWUn_wCxie4+zxs5Akx1ObAvNEw@mail.gmail.com","subject":"Re: Seems to be pushing more than necessary","fromName":"Graham Hay","fromEmail":"grahamrhay@gmail.com","sentAt":"2015-03-23T10:56:59Z","receivedAt":"2015-03-23T10:56:59Z","isPatch":false,"sender":{"key":"grahamrhay@gmail.com","avatar":"https://gravatar.com/avatar/235d8aff572de97e6f9a16a99563aa55058a8ed1b1bf315c4285e565361a6429?d=mp&s=160"},"body":"If I push straight to the other repo, it only pushes the 3 objects I'd\nexpect (instead of 10,000+). So it looks like that is the problem, but\nI don't really understand why.\n\n>From my point of view, there should be no difference, but I clearly\ndon't understand how it actually works. How does git decide what refs\nand/or objects are the same?\n\nFor a bit of background, the reason I have 2 remotes is to try and\navoid pushing to master. We work in a highly regulated industry, and\nour code needs to be reviewed before hitting the mainline. So I push\nto my fork and create a PR to the blessed repo, that way if I\naccidentally commit to master (I have form!) then I have an extra\nchance to catch it and don't have to back it out.\n\nThe two repos started out the same though, the only differences should\nbe the new work I have done. Is there any way I can continue to work\nlike this, or do I have to choose between slow pushes and safety?\n\nOn 23 March 2015 at 10:41, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Mon, Mar 23, 2015 at 5:35 PM, Graham Hay <grahamrhay@gmail.com> wrote:\n>> Hmm. I'm using a \"private\" fork of a repo, I pull from one and push to\n>> the other, e.g.\n>>\n>> git fetch foo\n>> git rebase foo/master\n>> git push --set-upstream origin bar\n>>\n>> It's quite possible my workflow is causing the problem, but I'm not\n>> sure what I could do differently. What do you mean by a \"no-share\n>> remote\"?\n>\n> I mean the refs (and associated objects) that are available on \"foo\"\n> may be not available on \"bar\" so when you push to \"origin\" you just\n> need to send more. That rebase could generate lots of new objects to\n> push out too, I think.\n> --\n> Duy\n"}]}