{"thread":{"id":"13581","subject":"git-svn pulling down duplicate revisions","startedAt":"2008-05-20T00:26:58Z","lastAt":"2008-06-02T17:59:53Z","messageCount":8,"participants":["Kevin Ballard","Eric Wong","Björn Steinbrink"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"77300","messageId":"1AD7D1A1-EC3B-450D-A648-04ADB8180E46@sb.org","threadId":"13581","inReplyTo":null,"subject":"git-svn pulling down duplicate revisions","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-05-20T00:26:58Z","receivedAt":"2008-05-20T00:26:58Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"I started a git-svn clone on a large svn repository, and I noticed \nthat for various branches, it kept pulling down the exact same \nrevisions (starting at r1). In other words, if I had 4 branches that \nshared common history, their common history all got pulled down 4 \ntimes. I double-checked, and the created commit objects were identical.\n\nWhy was git-svn pulling down the same revisions over and over, when it \nalready knows it has a commit object for those revisions?\n\n-Kevin Ballard\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n"},{"id":"78329","messageId":"20080602050050.GC9904@hand.yhbt.net","threadId":"13581","inReplyTo":"1AD7D1A1-EC3B-450D-A648-04ADB8180E46@sb.org","subject":"Re: git-svn pulling down duplicate revisions","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-06-02T05:00:51Z","receivedAt":"2008-06-02T05:00:51Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Kevin Ballard wrote:\n> I started a git-svn clone on a large svn repository, and I noticed \n> that for various branches, it kept pulling down the exact same \n> revisions (starting at r1). In other words, if I had 4 branches that \n> shared common history, their common history all got pulled down 4 \n> times. I double-checked, and the created commit objects were identical.\n> \n> Why was git-svn pulling down the same revisions over and over, when it \n> already knows it has a commit object for those revisions?\n\nCan you give me an example if a repository and command-line you used\nthat does this? Did you use 'git svn clone -s' or did you manually\nspecify the branch locations in the repo?\n\nIt could even be a lack of read permissions to the repository root\nthat would cause things like this.\n\nThanks,\n\n-- \nEric Wong\n"},{"id":"78330","messageId":"5FB2F28F-4C9A-422E-BF7D-B271471154F9@sb.org","threadId":"13581","inReplyTo":"20080602050050.GC9904@hand.yhbt.net","subject":"Re: git-svn pulling down duplicate revisions","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-06-02T05:06:49Z","receivedAt":"2008-06-02T05:06:49Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Jun 1, 2008, at 10:00 PM, Eric Wong wrote:\n\n> Kevin Ballard wrote:\n>> I started a git-svn clone on a large svn repository, and I noticed\n>> that for various branches, it kept pulling down the exact same\n>> revisions (starting at r1). In other words, if I had 4 branches that\n>> shared common history, their common history all got pulled down 4\n>> times. I double-checked, and the created commit objects were \n>> identical.\n>>\n>> Why was git-svn pulling down the same revisions over and over, when \n>> it\n>> already knows it has a commit object for those revisions?\n>\n> Can you give me an example if a repository and command-line you used\n> that does this? Did you use 'git svn clone -s' or did you manually\n> specify the branch locations in the repo?\n>\n> It could even be a lack of read permissions to the repository root\n> that would cause things like this.\n\nThe repository is, unfortunately, a private repo so I can't share it. \nI used `git svn clone -s` to clone it. I have the SVN perl bindings \nv1.4.4 (according to git svn --version).\n\nI definitely have read permissions to the repo root. If I specify to \nonly fetch -r 12000:HEAD (there's 14000-odd revisions), it doesn't \npull down any duplicates, but when I let it start from the root, it \npulls down hundreds of duplicates for multiple branches.\n\n-Kevin Ballard\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n"},{"id":"78334","messageId":"20080602054034.GA8366@untitled","threadId":"13581","inReplyTo":"5FB2F28F-4C9A-422E-BF7D-B271471154F9@sb.org","subject":"Re: git-svn pulling down duplicate revisions","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-06-02T05:40:34Z","receivedAt":"2008-06-02T05:40:34Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Kevin Ballard wrote:\n> On Jun 1, 2008, at 10:00 PM, Eric Wong wrote:\n> \n> >Kevin Ballard wrote:\n> >>I started a git-svn clone on a large svn repository, and I noticed\n> >>that for various branches, it kept pulling down the exact same\n> >>revisions (starting at r1). In other words, if I had 4 branches that\n> >>shared common history, their common history all got pulled down 4\n> >>times. I double-checked, and the created commit objects were \n> >>identical.\n> >>\n> >>Why was git-svn pulling down the same revisions over and over, when \n> >>it\n> >>already knows it has a commit object for those revisions?\n> >\n> >Can you give me an example if a repository and command-line you used\n> >that does this? Did you use 'git svn clone -s' or did you manually\n> >specify the branch locations in the repo?\n> >\n> >It could even be a lack of read permissions to the repository root\n> >that would cause things like this.\n> \n> The repository is, unfortunately, a private repo so I can't share it. \n> I used `git svn clone -s` to clone it. I have the SVN perl bindings \n> v1.4.4 (according to git svn --version).\n> \n> I definitely have read permissions to the repo root. If I specify to \n> only fetch -r 12000:HEAD (there's 14000-odd revisions), it doesn't \n> pull down any duplicates, but when I let it start from the root, it \n> pulls down hundreds of duplicates for multiple branches.\n\nCan you at least send me the 'svn log -v' output for that repo?\nFeel free to leave out the actual log messages and munge the path\nnames if you can't expose that information.\n\n-- \nEric Wong\n"},{"id":"78335","messageId":"0E759330-1A0A-489D-ADA3-B71A49951227@sb.org","threadId":"13581","inReplyTo":"20080602054034.GA8366@untitled","subject":"Re: git-svn pulling down duplicate revisions","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-06-02T05:55:00Z","receivedAt":"2008-06-02T05:55:00Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Jun 1, 2008, at 10:40 PM, Eric Wong wrote:\n\n> Kevin Ballard wrote:\n>> On Jun 1, 2008, at 10:00 PM, Eric Wong wrote:\n>>\n>>> Kevin Ballard wrote:\n>>>> I started a git-svn clone on a large svn repository, and I noticed\n>>>> that for various branches, it kept pulling down the exact same\n>>>> revisions (starting at r1). In other words, if I had 4 branches \n>>>> that\n>>>> shared common history, their common history all got pulled down 4\n>>>> times. I double-checked, and the created commit objects were\n>>>> identical.\n>>>>\n>>>> Why was git-svn pulling down the same revisions over and over, when\n>>>> it\n>>>> already knows it has a commit object for those revisions?\n>>>\n>>> Can you give me an example if a repository and command-line you used\n>>> that does this? Did you use 'git svn clone -s' or did you manually\n>>> specify the branch locations in the repo?\n>>>\n>>> It could even be a lack of read permissions to the repository root\n>>> that would cause things like this.\n>>\n>> The repository is, unfortunately, a private repo so I can't share it.\n>> I used `git svn clone -s` to clone it. I have the SVN perl bindings\n>> v1.4.4 (according to git svn --version).\n>>\n>> I definitely have read permissions to the repo root. If I specify to\n>> only fetch -r 12000:HEAD (there's 14000-odd revisions), it doesn't\n>> pull down any duplicates, but when I let it start from the root, it\n>> pulls down hundreds of duplicates for multiple branches.\n>\n> Can you at least send me the 'svn log -v' output for that repo?\n> Feel free to leave out the actual log messages and munge the path\n> names if you can't expose that information.\n\nI'll have to do it tomorrow when I'm at the office. How much log info \ndo you need? I can let it run until I see duplicate revisions (it's \npretty obvious, it starts over again from r1).\n\n-Kevin\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n"},{"id":"78352","messageId":"20080602104225.GA8401@untitled","threadId":"13581","inReplyTo":"0E759330-1A0A-489D-ADA3-B71A49951227@sb.org","subject":"Re: git-svn pulling down duplicate revisions","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-06-02T10:42:25Z","receivedAt":"2008-06-02T10:42:25Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Kevin Ballard wrote:\n> On Jun 1, 2008, at 10:40 PM, Eric Wong wrote:\n> \n> >Kevin Ballard wrote:\n> >>On Jun 1, 2008, at 10:00 PM, Eric Wong wrote:\n> >>\n> >>>Kevin Ballard wrote:\n> >>>>I started a git-svn clone on a large svn repository, and I noticed\n> >>>>that for various branches, it kept pulling down the exact same\n> >>>>revisions (starting at r1). In other words, if I had 4 branches \n> >>>>that\n> >>>>shared common history, their common history all got pulled down 4\n> >>>>times. I double-checked, and the created commit objects were\n> >>>>identical.\n> >>>>\n> >>>>Why was git-svn pulling down the same revisions over and over, when\n> >>>>it\n> >>>>already knows it has a commit object for those revisions?\n> >>>\n> >>>Can you give me an example if a repository and command-line you used\n> >>>that does this? Did you use 'git svn clone -s' or did you manually\n> >>>specify the branch locations in the repo?\n> >>>\n> >>>It could even be a lack of read permissions to the repository root\n> >>>that would cause things like this.\n> >>\n> >>The repository is, unfortunately, a private repo so I can't share it.\n> >>I used `git svn clone -s` to clone it. I have the SVN perl bindings\n> >>v1.4.4 (according to git svn --version).\n> >>\n> >>I definitely have read permissions to the repo root. If I specify to\n> >>only fetch -r 12000:HEAD (there's 14000-odd revisions), it doesn't\n> >>pull down any duplicates, but when I let it start from the root, it\n> >>pulls down hundreds of duplicates for multiple branches.\n> >\n> >Can you at least send me the 'svn log -v' output for that repo?\n> >Feel free to leave out the actual log messages and munge the path\n> >names if you can't expose that information.\n> \n> I'll have to do it tomorrow when I'm at the office. How much log info \n> do you need? I can let it run until I see duplicate revisions (it's \n> pretty obvious, it starts over again from r1).\n\nI'll need the revisions where branches were created from\nthe common ancestor (presumably trunk) and some revisions\nbefore it.\n\nFor debugging problems with restricted repositories, it may be worth it\nto create a repository skeleton cloning tool that just reads the output\nof 'svn log --xml -v' and recreates a new SVN repository with:\n\n * all log messages stripped\n\n * all new files are created with just a random string in them (to\n throw off rename detection on the git side)\n (except symlinks, see below)\n\n * all path components tokenized and each token replaced with\n a dictionary value. Something like:\n\n @tmp = map { $tok{$_} ||= ++$i; $tok{$_} } split(/\\//, $old_path);\n $new_path = join('/', @tmp);\n\n This way all copy history can be preserved\n\n * all modified files will just get a random byte appended to them\n\n * all committer names replaced with a dictionary value (similar to\n what is done to path components).\n\n\n-- \nEric Wong\n"},{"id":"78388","messageId":"C520B70F-2E61-42E6-AF20-E24E15A33CDD@sb.org","threadId":"13581","inReplyTo":"20080602104225.GA8401@untitled","subject":"Re: git-svn pulling down duplicate revisions","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2008-06-02T17:45:04Z","receivedAt":"2008-06-02T17:45:04Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Jun 2, 2008, at 3:42 AM, Eric Wong wrote:\n\n> Kevin Ballard wrote:\n>> On Jun 1, 2008, at 10:40 PM, Eric Wong wrote:\n>>\n>>> Kevin Ballard wrote:\n>>>> On Jun 1, 2008, at 10:00 PM, Eric Wong wrote:\n>>>>\n>>>>> Kevin Ballard wrote:\n>>>>>> I started a git-svn clone on a large svn repository, and I \n>>>>>> noticed\n>>>>>> that for various branches, it kept pulling down the exact same\n>>>>>> revisions (starting at r1). In other words, if I had 4 branches\n>>>>>> that\n>>>>>> shared common history, their common history all got pulled down 4\n>>>>>> times. I double-checked, and the created commit objects were\n>>>>>> identical.\n>>>>>>\n>>>>>> Why was git-svn pulling down the same revisions over and over, \n>>>>>> when\n>>>>>> it\n>>>>>> already knows it has a commit object for those revisions?\n>>>>>\n>>>>> Can you give me an example if a repository and command-line you \n>>>>> used\n>>>>> that does this? Did you use 'git svn clone -s' or did you \n>>>>> manually\n>>>>> specify the branch locations in the repo?\n>>>>>\n>>>>> It could even be a lack of read permissions to the repository root\n>>>>> that would cause things like this.\n>>>>\n>>>> The repository is, unfortunately, a private repo so I can't share \n>>>> it.\n>>>> I used `git svn clone -s` to clone it. I have the SVN perl bindings\n>>>> v1.4.4 (according to git svn --version).\n>>>>\n>>>> I definitely have read permissions to the repo root. If I specify \n>>>> to\n>>>> only fetch -r 12000:HEAD (there's 14000-odd revisions), it doesn't\n>>>> pull down any duplicates, but when I let it start from the root, it\n>>>> pulls down hundreds of duplicates for multiple branches.\n>>>\n>>> Can you at least send me the 'svn log -v' output for that repo?\n>>> Feel free to leave out the actual log messages and munge the path\n>>> names if you can't expose that information.\n>>\n>> I'll have to do it tomorrow when I'm at the office. How much log info\n>> do you need? I can let it run until I see duplicate revisions (it's\n>> pretty obvious, it starts over again from r1).\n>\n> I'll need the revisions where branches were created from\n> the common ancestor (presumably trunk) and some revisions\n> before it.\n>\n> For debugging problems with restricted repositories, it may be worth \n> it\n> to create a repository skeleton cloning tool that just reads the \n> output\n> of 'svn log --xml -v' and recreates a new SVN repository with:\n>\n> * all log messages stripped\n>\n> * all new files are created with just a random string in them (to\n> throw off rename detection on the git side)\n> (except symlinks, see below)\n>\n> * all path components tokenized and each token replaced with\n> a dictionary value. Something like:\n>\n> @tmp = map { $tok{$_} ||= ++$i; $tok{$_} } split(/\\//, $old_path);\n> $new_path = join('/', @tmp);\n>\n> This way all copy history can be preserved\n>\n> * all modified files will just get a random byte appended to them\n>\n> * all committer names replaced with a dictionary value (similar to\n> what is done to path components).\n\nIsn't there a script somewhere that's supposed to do this? Do you know \nwhere it is?\n\nIncidentally, I just checked and when I start the git-svn clone, it \nstarts pulling down revisions for the branch 'css_refactor@1559' (odd \nbranch name, but it claimed to find multiple branch points for this \n'css_refactor' branch). My guess is when it starts working on the next \nbranch, it doesn't view it as related to css_refactor and starts \npulling down the revisions again even though those revisions actually \nbelonged to trunk.\n\n-Kevin\n\n-- \nKevin Ballard\nhttp://kevin.sb.org\nkevin@sb.org\nhttp://www.tildesoft.com\n"},{"id":"78390","messageId":"20080602175953.GA31685@atjola.homenet","threadId":"13581","inReplyTo":"C520B70F-2E61-42E6-AF20-E24E15A33CDD@sb.org","subject":"Re: git-svn pulling down duplicate revisions","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-06-02T17:59:53Z","receivedAt":"2008-06-02T17:59:53Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.06.02 10:45:04 -0700, Kevin Ballard wrote:\n> Incidentally, I just checked and when I start the git-svn clone, it\n> starts pulling down revisions for the branch 'css_refactor@1559' (odd\n> branch name, but it claimed to find multiple branch points for this\n> 'css_refactor' branch). My guess is when it starts working on the next\n> branch, it doesn't view it as related to css_refactor and starts\n> pulling down the revisions again even though those revisions actually\n> belonged to trunk.\n\nHm, you could probably test that theory at least for branches that\nstarted from trunk I guess. First, clone trunk only, and then add the\nbranches/tags config entries and fetch the rest. If it holds, then the\nduplication should be gone I guess.\n\nBjörn\n"}]}