{"thread":{"id":"16625","subject":"help needed: Splitting a git repository after subversion migration","startedAt":"2008-12-07T17:41:01Z","lastAt":"2008-12-12T14:49:29Z","messageCount":8,"participants":["Thomas Jarosch","Michael J Gruber","Björn Steinbrink"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"97307","messageId":"493C0AAD.1040208@intra2net.com","threadId":"16625","inReplyTo":null,"subject":"help needed: Splitting a git repository after subversion migration","fromName":"Thomas Jarosch","fromEmail":"thomas.jarosch@intra2net.com","sentAt":"2008-12-07T17:41:01Z","receivedAt":"2008-12-07T17:41:01Z","isPatch":false,"sender":{"key":"thomas.jarosch@intra2net.com","avatar":"https://avatars.githubusercontent.com/u/1146758?v=4"},"body":"Hello together,\n\nI've successfully imported a large subversion repository into git.\nThe tree contains source code and binary data (\"releases\"),\nthe resulting .git directory is about 11GB.\n\nAfter the import I recreated the tags/branches by converting the refs\nto the subversion tags using a small shell script from the web:\n\nfor branch in `git branch -r`; do\n     ...\n     version=`basename $branch`\n     git tag -s -f -m \"$subject\" \"$version\" \"$branch^\"\n     git branch -d -r $branch\ndone\n\nOk, so far everything went really smooth. I wanted to split this repository\ninto two repositories, one for the source code and one for the binary data.\nThe current tree layout is like this:\n\nsources/c++_xyz\nreleases/large_binary_data\n...\n\nThe original tree was imported from CVS to subversion and the layout\nof the trunk was once reorganized/moved later. Here's the command\nI used to split out the \"source\" tree:\n\ngit filter-branch --index-filter 'git rm --cached --ignore-unmatch -r -f\nCVSROOT Attic source/Attic develpkg/Attic\nsource/packages/Attic releases update_pkg' -- --all\n\nAfter that I ran these commands to reclaim the space:\n- git clone --no-hardlinks filtered_tree final_output\n- cd final_output\n- git gc\n- git prune\n- git repack -a -d --depth=250 --window=250\n\nUnfortunately the .git directory of the \"source\" tree is still 7.5GB big.\n\nWhen I just imported the \"trunk\" from subversion without any tags\nand then ran \"git filter-branch --subdirectory-filter source\" + git gc,\nthe .git directory was about 1.5GB afterwards.\n\nHow can I find out where those other 6GB go to?\nI already looked at the tags with gitk,\nthere's no sign of the releases/* stuff left.\n\nThe \"--all\" switch for \"git filter-branch\"\ndoesn't seem documented in git 1.6.0.4?\nI just learned about it from the example usage.\n\n\"git filter-branch\" also had trouble converting the tags\nand suggested I should add \"--tag-name-filter cat\", which I did.\nMaybe that's something for the examples, too?\n\nI also tried running \"git filter-branch --tag-name-filter cat \n--subdirectory-filter source -- --all\", but that commands aborts\nwith these messages:\n\nWARNING: 'refs/tags/v5-0-8' was rewritten into multiple commits:\nee180f6117597b60ee237e9da92047946dfdeec5\nfd7824d1926ce9e4c89b685583eb9a9c2f2537af\nWARNING: Ref 'refs/tags/v5-0-8' points to the first one now.\nerror: Ref refs/tags/v5-0-8 is at 4ea78238cfd6ee259c4e8bde7be4a90bc86295b0 \nbut expected 06c60261502acfb7b2bbe44c2e2ec371bea65827\nfatal: Cannot lock the ref 'refs/tags/v5-0-8'.\nCould not rewrite refs/tags/v5-0-8\n\n\nBesides that git really rocks :-)\n\nThanks in advance,\nThomas\n"},{"id":"97345","messageId":"493D2174.80500@drmicha.warpmail.net","threadId":"16625","inReplyTo":"493C0AAD.1040208@intra2net.com","subject":"Re: help needed: Splitting a git repository after subversion migration","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2008-12-08T13:30:28Z","receivedAt":"2008-12-08T13:30:28Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Thomas Jarosch venit, vidit, dixit 07.12.2008 18:41:\n> Hello together,\n> \n> I've successfully imported a large subversion repository into git.\n> The tree contains source code and binary data (\"releases\"),\n> the resulting .git directory is about 11GB.\n> \n> After the import I recreated the tags/branches by converting the refs\n> to the subversion tags using a small shell script from the web:\n> \n> for branch in `git branch -r`; do\n>      ...\n>      version=`basename $branch`\n>      git tag -s -f -m \"$subject\" \"$version\" \"$branch^\"\n>      git branch -d -r $branch\n> done\n> \n> Ok, so far everything went really smooth. I wanted to split this repository\n> into two repositories, one for the source code and one for the binary data.\n> The current tree layout is like this:\n> \n> sources/c++_xyz\n> releases/large_binary_data\n> ...\n> \n> The original tree was imported from CVS to subversion and the layout\n> of the trunk was once reorganized/moved later. Here's the command\n> I used to split out the \"source\" tree:\n> \n> git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r -f\n> CVSROOT Attic source/Attic develpkg/Attic\n> source/packages/Attic releases update_pkg' -- --all\n> \n> After that I ran these commands to reclaim the space:\n> - git clone --no-hardlinks filtered_tree final_output\n> - cd final_output\n> - git gc\n> - git prune\n> - git repack -a -d --depth=250 --window=250\n> \n> Unfortunately the .git directory of the \"source\" tree is still 7.5GB big.\n> \n> When I just imported the \"trunk\" from subversion without any tags\n> and then ran \"git filter-branch --subdirectory-filter source\" + git gc,\n> the .git directory was about 1.5GB afterwards.\n> \n> How can I find out where those other 6GB go to?\n> I already looked at the tags with gitk,\n> there's no sign of the releases/* stuff left.\n\nI strongly suspect the reorganization/move to be the cause. Most\nprobably some releases were put in places where you don't expect them,\nand therefore they are not filtered out by removing the releases subdir.\nIf they have distinguished file names (say you know a name from before\nthe move) you can find them using \"git log\". Or use gitk --all, switch\nto \"tree display\" and look for unexpected files in the earliest revisions.\n\nAlso, it may be better to do the tag creation (from tags/... branches)\nafter the filter-branch. If you don't rewrite the tags (have you?) then\nthe tags will still point to the original commits (before the rewrite)\nand therefore include all the \"fat blobs\". You avoid this best by\ncreating them after the rewrite.\n\nMichael\n"},{"id":"97349","messageId":"20081208142447.GA20186@atjola.homenet","threadId":"16625","inReplyTo":"493D2174.80500@drmicha.warpmail.net","subject":"Re: help needed: Splitting a git repository after subversion migration","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-12-08T14:24:47Z","receivedAt":"2008-12-08T14:24:47Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.12.08 14:30:28 +0100, Michael J Gruber wrote:\n> Thomas Jarosch venit, vidit, dixit 07.12.2008 18:41:\n> > Hello together,\n> > \n> > I've successfully imported a large subversion repository into git.\n> > The tree contains source code and binary data (\"releases\"),\n> > the resulting .git directory is about 11GB.\n> > \n> > After the import I recreated the tags/branches by converting the refs\n> > to the subversion tags using a small shell script from the web:\n> > \n> > for branch in `git branch -r`; do\n> >      ...\n> >      version=`basename $branch`\n> >      git tag -s -f -m \"$subject\" \"$version\" \"$branch^\"\n> >      git branch -d -r $branch\n> > done\n> > \n> > Ok, so far everything went really smooth. I wanted to split this repository\n> > into two repositories, one for the source code and one for the binary data.\n> > The current tree layout is like this:\n> > \n> > sources/c++_xyz\n> > releases/large_binary_data\n> > ...\n> > \n> > The original tree was imported from CVS to subversion and the layout\n> > of the trunk was once reorganized/moved later. Here's the command\n> > I used to split out the \"source\" tree:\n> > \n> > git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r -f\n> > CVSROOT Attic source/Attic develpkg/Attic\n> > source/packages/Attic releases update_pkg' -- --all\n> > \n> > After that I ran these commands to reclaim the space:\n> > - git clone --no-hardlinks filtered_tree final_output\n> > - cd final_output\n> > - git gc\n> > - git prune\n> > - git repack -a -d --depth=250 --window=250\n> > \n> > Unfortunately the .git directory of the \"source\" tree is still 7.5GB big.\n> > \n> > When I just imported the \"trunk\" from subversion without any tags\n> > and then ran \"git filter-branch --subdirectory-filter source\" + git gc,\n> > the .git directory was about 1.5GB afterwards.\n> > \n> > How can I find out where those other 6GB go to?\n> > I already looked at the tags with gitk,\n> > there's no sign of the releases/* stuff left.\n> \n> I strongly suspect the reorganization/move to be the cause. Most\n> probably some releases were put in places where you don't expect them,\n> and therefore they are not filtered out by removing the releases subdir.\n> If they have distinguished file names (say you know a name from before\n> the move) you can find them using \"git log\". Or use gitk --all, switch\n> to \"tree display\" and look for unexpected files in the earliest revisions.\n\nIf it's about huge objects, and not just lots of small objects, you can\nuse this:\n\n# Find large objects\ngit rev-list --objects --all | cut -f1 -d' ' | \\\n\tgit cat-file --batch-check | grep blob | sort -n -k 3\n\nThis outputs lines in the format:\n<object_hash> blob <object_size>\n\nsorted by object size, large objects come last. To make use of that\ninformation, you'll likely need to also find the filename(s) that are\nused for these blobs:\n\n# Find filenames for objects\ngit rev-list --all --objects | grep <object_hash>\n\nAnd then you can use the filenames to do some more filtering.\n\nBjörn\n"},{"id":"97370","messageId":"200812081834.26688.thomas.jarosch@intra2net.com","threadId":"16625","inReplyTo":"20081208142447.GA20186@atjola.homenet","subject":"Re: help needed: Splitting a git repository after subversion migration","fromName":"Thomas Jarosch","fromEmail":"thomas.jarosch@intra2net.com","sentAt":"2008-12-08T17:34:20Z","receivedAt":"2008-12-08T17:34:20Z","isPatch":false,"sender":{"key":"thomas.jarosch@intra2net.com","avatar":"https://avatars.githubusercontent.com/u/1146758?v=4"},"body":"On Monday, 8. December 2008 15:24:47 you wrote:\n> If it's about huge objects, and not just lots of small objects, you can\n> use this:\n\nThanks, those two commands have been really helpful. I've found some objects\nthat shouldn't be there and now I have two more questions:\n\n1. When I run \"git rev-list --all --objects\", I can see file names that look \nlike \"SVN-branchname/directory/filename\". Is it normal that \"git svn\"\ncreates a directory with the name of the branch and puts files below it?\n\n\"git rev-list --all --objects |grep 5-0-3-hotfix\":\n5fe3265b6941c2fa74c12da799ea23e2801efa8a 5-0-3-hotfix/source\n...\n\nThe branch in question existed for a limited time in branches/xyz\non the SVN tree and was deleted later on. Guessing the version number\nfrom the filename, it looks like a copy of the files when I started the branch\nas it's an old version number before I committed changes to it.\n(f.e. upgraded libpng). When I just grep for \"libpng\" on the whole index,\nI see all the various updates I made over the years.\n\n2. Something goes wrong after the filter branch:\n\nOutput from the full 11GB tree:\ngit rev-list --all --objects |grep 5-0-3-hotfix |grep xyz\n-> No match\n\nOutput from the filtered tree:\ngit rev-list --all --objects |grep 5-0-3-hotfix |grep xyz\n\n3a13f87bc116aee96e031441eaafc416652ba4bd 5-0-3-hotfix/update_pkg/xyz\nebebb84ccff26c949fb1f803c60034074e6603fe 5-0-3-hotfix/update_pkg/xyz\n5529ef51de887cc905fe460e4c4f6cd34b93b5a6 5-0-3-hotfix/update_pkg/xyz\nc264a9d5db30ebb131c96c4f93192bfe9a5c0a7b 5-0-3-hotfix/update_pkg/xyz\n\nI have no idea how those objects suddenly appeared there.\nIt feels like something was stitched together wrongly.\n\nWhen I converted the SVN tag to a git tag, I tagged the branches\nwith a \"branch-\" prefix. Might that be a problem, is \"branch-\" reserved?\n\nCheers,\nThomas\n"},{"id":"97498","messageId":"200812101733.36221.thomas.jarosch@intra2net.com","threadId":"16625","inReplyTo":"200812081834.26688.thomas.jarosch@intra2net.com","subject":"Re: help needed: Splitting a git repository after subversion migration","fromName":"Thomas Jarosch","fromEmail":"thomas.jarosch@intra2net.com","sentAt":"2008-12-10T16:33:28Z","receivedAt":"2008-12-10T16:33:28Z","isPatch":false,"sender":{"key":"thomas.jarosch@intra2net.com","avatar":"https://avatars.githubusercontent.com/u/1146758?v=4"},"body":"On Monday, 8. December 2008 18:34:20 Thomas Jarosch wrote:\n> 1. When I run \"git rev-list --all --objects\", I can see file names that\n> look like \"SVN-branchname/directory/filename\". Is it normal that \"git svn\"\n> creates a directory with the name of the branch and puts files below it?\n\nOk, this seems to be a PEBKAC: In the history of the subversion repository, \nf.e. I once copied the \"branches\" root folder to tags/xyz. One revision later \nI noticed this and retagged the correct branch. git-svn imports all branches\nfrom the first tag, which is the correct thing to do :o)\n\nNow I'll manually check the history of the tags/ and branches/ folder\nfor more funny tags and write down the revision. If I understood\nthe git-svn man page correctly, I should be able to specifiy\nrevision ranges it's going to import. I'll try to skip the broken tags.\n\nCheers,\nThomas\n"},{"id":"97587","messageId":"20081211081009.GA14639@atjola.homenet","threadId":"16625","inReplyTo":"200812101733.36221.thomas.jarosch@intra2net.com","subject":"Re: help needed: Splitting a git repository after subversion migration","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-12-11T08:10:09Z","receivedAt":"2008-12-11T08:10:09Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.12.10 17:33:28 +0100, Thomas Jarosch wrote:\n> On Monday, 8. December 2008 18:34:20 Thomas Jarosch wrote:\n> > 1. When I run \"git rev-list --all --objects\", I can see file names that\n> > look like \"SVN-branchname/directory/filename\". Is it normal that \"git svn\"\n> > creates a directory with the name of the branch and puts files below it?\n> \n> Ok, this seems to be a PEBKAC: In the history of the subversion repository, \n> f.e. I once copied the \"branches\" root folder to tags/xyz. One revision later \n> I noticed this and retagged the correct branch. git-svn imports all branches\n> from the first tag, which is the correct thing to do :o)\n> \n> Now I'll manually check the history of the tags/ and branches/ folder\n> for more funny tags and write down the revision. If I understood\n> the git-svn man page correctly, I should be able to specifiy\n> revision ranges it's going to import. I'll try to skip the broken tags.\n\nAs long as the breakage only involves branches/tags that are completely\nuseless, it's probably a lot easier to just delete them afterwards.\n\nAnd if you accidently added changes to a tag, after it was created, it's\nalso easier to manually tag to right version in git, and just forgetting\nabout the additional commit.\n\nAnd for a bunch of other cases, rebase -i/filter-branch are probably\nalso better options ;-)\n\nSkipping revisions in a git-svn import sounds rather annoying and\nerror-prone.\n\nBjörn\n"},{"id":"97694","messageId":"200812121522.38791.thomas.jarosch@intra2net.com","threadId":"16625","inReplyTo":"20081211081009.GA14639@atjola.homenet","subject":"Re: help needed: Splitting a git repository after subversion migration","fromName":"Thomas Jarosch","fromEmail":"thomas.jarosch@intra2net.com","sentAt":"2008-12-12T14:22:15Z","receivedAt":"2008-12-12T14:22:15Z","isPatch":false,"sender":{"key":"thomas.jarosch@intra2net.com","avatar":"https://avatars.githubusercontent.com/u/1146758?v=4"},"body":"On Thursday, 11. December 2008 09:10:09 you wrote:\n> > Now I'll manually check the history of the tags/ and branches/ folder\n> > for more funny tags and write down the revision. If I understood\n> > the git-svn man page correctly, I should be able to specifiy\n> > revision ranges it's going to import. I'll try to skip the broken tags.\n>\n> As long as the breakage only involves branches/tags that are completely\n> useless, it's probably a lot easier to just delete them afterwards.\n>\n> And if you accidently added changes to a tag, after it was created, it's\n> also easier to manually tag to right version in git, and just forgetting\n> about the additional commit.\n>\n> And for a bunch of other cases, rebase -i/filter-branch are probably\n> also better options ;-)\n>\n> Skipping revisions in a git-svn import sounds rather annoying and\n> error-prone.\n\nSounds very reasonable. When I'm done filtering with filter-branch,\nthe original commits are still stored in \"refs/originals\" and the reflogs.\nWhat's the best way to get rid of those to free up the space?\n\nA nice way to find the corresponding commit for a file can be found here: \nhttp://stackoverflow.com/questions/223678/git-which-commit-has-this-blob\n\nThanks for your help so far!\n\nThomas\n\nPS: Yes, I have a backup copy of the repository ;-)\n"},{"id":"97696","messageId":"20081212144929.GA27445@atjola.homenet","threadId":"16625","inReplyTo":"200812121522.38791.thomas.jarosch@intra2net.com","subject":"Re: help needed: Splitting a git repository after subversion migration","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-12-12T14:49:29Z","receivedAt":"2008-12-12T14:49:29Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.12.12 15:22:15 +0100, Thomas Jarosch wrote:\n> On Thursday, 11. December 2008 09:10:09 you wrote:\n> > > Now I'll manually check the history of the tags/ and branches/ folder\n> > > for more funny tags and write down the revision. If I understood\n> > > the git-svn man page correctly, I should be able to specifiy\n> > > revision ranges it's going to import. I'll try to skip the broken tags.\n> >\n> > As long as the breakage only involves branches/tags that are completely\n> > useless, it's probably a lot easier to just delete them afterwards.\n> >\n> > And if you accidently added changes to a tag, after it was created, it's\n> > also easier to manually tag to right version in git, and just forgetting\n> > about the additional commit.\n> >\n> > And for a bunch of other cases, rebase -i/filter-branch are probably\n> > also better options ;-)\n> >\n> > Skipping revisions in a git-svn import sounds rather annoying and\n> > error-prone.\n> \n> Sounds very reasonable. When I'm done filtering with filter-branch,\n> the original commits are still stored in \"refs/originals\" and the reflogs.\n> What's the best way to get rid of those to free up the space?\n\nSee the \"purging unwanted history\" thread:\n\nhttp://n2.nabble.com/purging-unwanted-history-td1507638.html\n\nThe commands there (starting with the \"git for-each-ref\") should clean\nout all the pre-filter-branch stuff.\n\n> A nice way to find the corresponding commit for a file can be found here: \n> http://stackoverflow.com/questions/223678/git-which-commit-has-this-blob\n\nYeah, I think something similar (or even the same?) is in the git wiki\nsomewhere. I never had any use for it though ;-)\n\nBjörn\n"}]}