{"thread":{"id":"27956","subject":"refs/replace advice","startedAt":"2011-07-29T15:31:22Z","lastAt":"2011-08-02T21:54:15Z","messageCount":4,"participants":["Pete Wyckoff","Johannes Sixt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"172348","messageId":"20110729153122.GA4535@padd.com","threadId":"27956","inReplyTo":null,"subject":"refs/replace advice","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2011-07-29T15:31:22Z","receivedAt":"2011-07-29T15:31:22Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"I've got two near-identical git repos, both imported from\ngigantic upstream p4 repos.  They started at slightly different\ntimes so have different commit SHA1s, even though the tree\ncontents are the same.  I can't filter-branch either of them; too\nmany users already.\n\nI'm trying to use \"git replace\" to avoid cloning the entire set\nof duplicate commits across a slow inter-site link.  Like this:\n\n    ...---A----B----C   site1/top\n                     \\\n                      D---E---F  site1/proj\n\n    ...---A'---B'---C'  site2/top\n\nIt is true that \"git diff C C'\" is empty:  they are identical.\n\nThis set of commands, run from site2, clones most of the repo\nlocally (up to C'), then grabs the few changes D..F from the\nfaraway site1:\n\n    git clone /path/to/site2.git repo\n    cd repo\n    git remote add -f site1 /path/to/faraway/site1.git\n\nBut it causes an entire fetch of all commits because C != C'.\nI'd prefer it just to fetch D, E and F.  So I try:\n\n    git refs replace A' A\n\nbut it still fetches everything.  I toyed with grafting\nsite1's A on top of the parent of our local A':\n\n    echo A A'^ > .git/info/grafts\n\nno luck.\n\nI thought maybe I could \"git fetch --depth=N\" where N would cover\nthe range A'..site2/top, then replace.  But testing with \"git\nfetch --depth=3\" still wants to fetch 100k objects.\n\nAny ideas?\n\n                -- Pete\n"},{"id":"172349","messageId":"4E32D6A1.8020304@viscovery.net","threadId":"27956","inReplyTo":"20110729153122.GA4535@padd.com","subject":"Re: refs/replace advice","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2011-07-29T15:49:53Z","receivedAt":"2011-07-29T15:49:53Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 7/29/2011 17:31, schrieb Pete Wyckoff:\n> I'm trying to use \"git replace\" to avoid cloning the entire set\n> of duplicate commits across a slow inter-site link.  Like this:\n> \n>     ...---A----B----C   site1/top\n>                      \\\n>                       D---E---F  site1/proj\n> \n>     ...---A'---B'---C'  site2/top\n> \n> It is true that \"git diff C C'\" is empty:  they are identical.\n...\n> I thought maybe I could \"git fetch --depth=N\" where N would cover\n> the range A'..site2/top, then replace.  But testing with \"git\n> fetch --depth=3\" still wants to fetch 100k objects.\n\nOn site2, don't you want to 'git fetch --depth=N site1' such that F down\nto at least C (but not much more) is fetched, and then apply the graft or\nreplacement on site2?\n\n-- Hannes\n"},{"id":"172364","messageId":"20110729224633.GA21355@arf.padd.com","threadId":"27956","inReplyTo":"4E32D6A1.8020304@viscovery.net","subject":"Re: refs/replace advice","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2011-07-29T22:46:33Z","receivedAt":"2011-07-29T22:46:33Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"j.sixt@viscovery.net wrote on Fri, 29 Jul 2011 17:49 +0200:\n> Am 7/29/2011 17:31, schrieb Pete Wyckoff:\n> > I'm trying to use \"git replace\" to avoid cloning the entire set\n> > of duplicate commits across a slow inter-site link.  Like this:\n> > \n> >     ...---A----B----C   site1/top\n> >                      \\\n> >                       D---E---F  site1/proj\n> > \n> >     ...---A'---B'---C'  site2/top\n> > \n> > It is true that \"git diff C C'\" is empty:  they are identical.\n> ...\n> > I thought maybe I could \"git fetch --depth=N\" where N would cover\n> > the range A'..site2/top, then replace.  But testing with \"git\n> > fetch --depth=3\" still wants to fetch 100k objects.\n> \n> On site2, don't you want to 'git fetch --depth=N site1' such that F down\n> to at least C (but not much more) is fetched, and then apply the graft or\n> replacement on site2?\n\nYes, that makes sense, shallow clone needs to pull the entire tree.\n\nOn site1 (bare .git repo):\n\n    $ du -sm .\n    542     .\n    $ git merge-base site1/proj site1/top\n    ff016f956ccae7878a1b322ba950a0088c6e2ded  ;# this is A\n    $ git rev-list ff016f956ccae7878a1b322ba950a0088c6e2ded | wc\n\t566     566   23206\n\nOn site2:\n\n    $ du -sm .git\n    649     .git\n    $ git rev-parse :/1384557\n    0f95d91c37bc870d610b7bd45b316ab219750d31  ;# this is A'\n    $ git rev-list 0f95d91c37bc870d610b7bd45b316ab219750d31 | wc\n\t566     566   23206\n\nSame number of commits all the way back to the beginning of time,\nbut the timestamp in the root commit is different, so all the SHA1s\nare different.\n\nOn site2:\n\n    $ time git fetch git://site1/repo\n    warning: no common commits\n    remote: Counting objects: 124166, done.\n    remote: Compressing objects: 100% (64472/64472), done.\n    remote: Total 124166 (delta 59815), reused 121350 (delta 57062)\n    Receiving objects: 100% (124166/124166), 462.31 MiB | 5.31 MiB/s, done.\n    Resolving deltas: 100% (59815/59815), done.\n    From git://site1/repo\n     * branch            HEAD       -> FETCH_HEAD\n    0m56.25s user 0m5.18s sys 2m29.45s elapsed 41.11 %CPU\n\nA brand new repo on site2, cloning this time with a teensy depth:\n\n    $ time git fetch --depth=3 git://site1/repo\n    warning: no common commits\n    remote: Counting objects: 96440, done.\n    remote: Compressing objects: 100% (58844/58844), done.\n    remote: Total 96440 (delta 36454), reused 92650 (delta 35169)\n    Receiving objects: 100% (96440/96440), 415.87 MiB | 7.38 MiB/s, done.\n    Resolving deltas: 100% (36454/36454), done.\n    From git://site1/repo\n     * branch            HEAD       -> FETCH_HEAD\n    0m40.40s user 0m5.27s sys 1m50.29s elapsed 41.41 %CPU\n\nNo savings in data transport.\n\nWas hoping it would be possible to get just the changes, but walking\nback to FETCH_HEAD~3 shows that it imports all the files.  That makes\nsense given the use case for shallow clone.  But I want to tell the\nfetch machinery that I already have one of the commits it is going to\nsee.\n\nI'll just tell people to put up with the full copy, and try to fix\nthings so that only one site creates the git repo from p4 in the future.\nThanks for looking,\n\n\t\t-- Pete\n"},{"id":"172701","messageId":"20110802215415.GA13327@arf.padd.com","threadId":"27956","inReplyTo":"20110729153122.GA4535@padd.com","subject":"Re: refs/replace advice","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2011-08-02T21:54:15Z","receivedAt":"2011-08-02T21:54:15Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"pw@padd.com wrote on Fri, 29 Jul 2011 08:31 -0700:\n> I've got two near-identical git repos, both imported from\n> gigantic upstream p4 repos.  They started at slightly different\n> times so have different commit SHA1s, even though the tree\n> contents are the same.  I can't filter-branch either of them; too\n> many users already.\n>\n> I'm trying to use \"git replace\" to avoid cloning the entire set\n> of duplicate commits across a slow inter-site link.  Like this:\n\nTo follow-up, I decided this was not the right approach.\nInstead, for future git-p4 repo creation, I've patched git-p4\nto make sure the timestamps and hence the initial SHA1s are\nidentical.\n\nThanks, Hannes, for your suggestion, though.\n\n\t\t-- Pete\n"}]}