{"thread":{"id":"32040","subject":"git-svn with ignore-paths misses/skips some revisions during fetch","startedAt":"2012-11-08T13:50:56Z","lastAt":"2012-11-12T15:00:52Z","messageCount":3,"participants":["McHenry, Matt","Enrico Weigelt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"202638","messageId":"D377A9280DB18546A2471214D5CBB0E9054C25E071@exchdb01","threadId":"32040","inReplyTo":null,"subject":"git-svn with ignore-paths misses/skips some revisions during fetch","fromName":"McHenry, Matt","fromEmail":"mmchenry@carnegielearning.com","sentAt":"2012-11-08T13:50:56Z","receivedAt":"2012-11-08T13:50:56Z","isPatch":false,"sender":{"key":"mmchenry@carnegielearning.com","avatar":"https://gravatar.com/avatar/2187a10f4f413a9b901f08107f275b70aa826c8f9bbe7c14fff4dae3ae58384b?d=mp&s=160"},"body":"\n        My company has a fairly large SVN repository, and I'm running into a bug with git-svn where some revisions aren't being fetched.\n\n        The repository has a standard trunk/tags/branches layout, but there are some top-level directories under trunk/ that clearly don't belong in Git, and some that do.  So, I've been experimenting with two approaches: a single git-svn clone that uses 'ignore-paths' to exclude the stuff that I don't want; and a series of per-subdir clones that use \"fetch = trunk/<subdir>:refs/remotes/trunk\" to pull in only that subdir.\n\n        The problem is that the 'ignore-paths' approach sometimes misses commits during a fetch, and then at some later time will \"realize\" it and squash those changes onto some other, unrelated commit.  (I've never seen this happen with the per-subdir 'fetch' approach.)  Here are three commits in SVN:\n\n$ svn log -v -r 172602 -r 172605 -r 172626 svn://dev\n------------------------------------------------------------------------\nr172602 | matt | 2012-10-31 16:03:08 -0400 (Wed, 31 Oct 2012) | 1 line\nChanged paths:\n   M /branches/localization-merge/buildprocess/antfiles/dmg.xml\n   D /branches/localization-merge/buildprocess/resources/CDs/JavaApplicationStub\n   M /branches/localization-merge/buildprocess/resources/launchanywhere/Cognitive Tutor.app.zip\n\nupdate to use newer java application stub\n------------------------------------------------------------------------\nr172605 | matt | 2012-10-31 16:29:25 -0400 (Wed, 31 Oct 2012) | 1 line\nChanged paths:\n   M /branches/localization-merge/buildprocess/antfiles/dmg.xml\n   M /branches/localization-merge/buildprocess/antfiles/lmstree.xml\n\nensure that sdk-fe code is installed; fix to get correct app-dir value for antcalls; add problem authoring tool\n------------------------------------------------------------------------\nr172626 | leslie | 2012-11-01 08:49:36 -0400 (Thu, 01 Nov 2012) | 1 line\nChanged paths:\n   M /branches/localization-merge/authoring/sdk/src/sdk-cc/src/main/java/cl/sdk/tdde/problemspace/problemtypes/geo/area/AreaPerimeterTutorInstance_PPV.java\n\ncheck the right shape-values for the gn\n------------------------------------------------------------------------\n\n\n        The first two are not to be found in git: 'git log --all --grep 17260[25]' returns nothing.  However, the third commit is there, and has had the changes from the earlier two (and some others) squashed into it somehow:\n\n$ git log --grep 172626 --numstat --summary\ncommit 6d9a10abc17c74396e07bb4bc7692059ac0e8b99\nAuthor: leslie <leslie@752fcc94-cd22-0410-baa8-ef54ac2c6973>\nDate:   Thu Nov 1 12:49:36 2012 +0000\n\n    check the right shape-values for the gn\n\n    git-svn-id: svn://dev.carnegielearning.com/branches/localization-merge@172626 752fcc94-cd22-0410-baa8-ef54ac2c6973\n\n13      8       authoring/sdk/src/sdk-cc/src/main/java/cl/sdk/tdde/problemspace/problemtypes/geo/area/AreaPerimeterTutorInstance_PPV.java\n11      2       buildprocess/antfiles/dmg.xml\n2       0       buildprocess/antfiles/lmstree.xml\n-       -       buildprocess/resources/CDs/JavaApplicationStub\n-       -       buildprocess/resources/launchanywhere/Cognitive Tutor.app.zip\n0       25      runtime/sdk-be/src/main/resources/cl/sdk/common/localized_words_es.properties\n9815    5143    runtime/sdk-be/src/main/resources/cl/sdk/tdde/problemspace/problemtypes/scratchpads/scratchpad-tutor.ctrules\n773     5       runtime/sdk-be/src/main/resources/cl/sdk/tdde/problemspace/problemtypes/scratchpads/scratchpad-tutor.ctrules_strings\n773     5       runtime/sdk-be/src/main/resources/cl/sdk/tdde/problemspace/problemtypes/scratchpads/scratchpad-tutor.ctrules_strings_es\n1562    79      runtime/sdk-be/src/main/resources/cl/sdk/tdde/problemspace/problemtypes/scratchpads/scratchpad-tutor.cttypes\n261     3       runtime/sdk-be/src/main/resources/cl/sdk/tdde/problemspace/problemtypes/scratchpads/scratchpad-tutor.cttypes_strings\n261     3       runtime/sdk-be/src/main/resources/cl/sdk/tdde/problemspace/problemtypes/scratchpads/scratchpad-tutor.cttypes_strings_es\n217     17      runtime/sdk-be/src/main/resources/cl/sdk/tdde/problemspace/problemtypes/scratchpads/scratchpad-tutor.ttspec\n25      0       runtime/sdk-be/src/main/resources_es/cl/sdk/common/localized_words_es.properties\n246     0       runtime/ui/src/main/resources/tp-tx-formula-sheet_es.html\n delete mode 100755 buildprocess/resources/CDs/JavaApplicationStub\n delete mode 100644 runtime/sdk-be/src/main/resources/cl/sdk/common/localized_words_es.properties\n create mode 100644 runtime/sdk-be/src/main/resources_es/cl/sdk/common/localized_words_es.properties\n create mode 100644 runtime/ui/src/main/resources/tp-tx-formula-sheet_es.html\n\n\n        The directories on the server are:\n\n$ svn ls svn://dev/trunk\nIDEs/\nQA-automation/\nauthoring/\nbugtracking/\nbuildprocess/\ncurriculum/\ndoc/\nimages/\ninstaller/\nlib/\nmisc-tools/\nresearch/\nruntime/\nscripts/\nserversearch/\nuser-assistance/\nweb/\n\n\n        My config file is:\n\n$ cat .git/config\n[core]\n        repositoryformatversion = 0\n        filemode = true\n        bare = false\n        logallrefupdates = true\n        ignorecase = true\n[svn-remote \"svn\"]\n        url = svn://dev.carnegielearning.com\n        fetch = trunk:refs/remotes/trunk\n        ignore-paths=\"^(trunk|tags/[^/]*|branches/[^/]*)/(doc|doc-old|IDEs|images|marathon-testing|QA-automation|research|Sikuli-Automation|bugtracking)/\"\n        branches = branches/{sdk-bug-258,2007-Q2-problemcounter,2008-Q2-bugfix,WEBEQ-REDUCTION,proof-gng,2009-Q2-marathon,2008-Q4,2009-Q4-bugfix,2009-Q2-bugfix-client,modules,2009-Q2-bugfix,2007-Q2-webstart,2011-Q3-math-lounge,2011-Q3-nyc,2011-Q2,2011-Q3-bugfix,cstoy,2010-Q2-bugfix,localization,localization-merge,2012-Q2-bugfix,hs_web_20120524,2012-research-casl,2012-Q3-math-208-9,2012-Q3-aug-localization}:refs/remotes/branches/*\n        tags = tags/{webeq-3.7,2009-Q2-release,2010-Q2-release,ui-old-impl-code,2011-Q3-release,2011-Q2-release,2012-Q2-release}:refs/remotes/tags/*\n\n\n        My git and OS versions are:\n\n$ uname -a\nLinux alpha 3.3.8-gentoo #1 SMP PREEMPT Thu Jul 26 21:25:22 EDT 2012 i686 Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz GenuineIntel GNU/Linux\n\n$ git svn --version\ngit-svn version 1.7.8.6 (svn 1.6.17)\n\n        On the svn server, I have:\n\nmmchenry@dev:~$ svnserve --version\nsvnserve, version 1.6.16 (r1073529)\n   compiled Mar  4 2011, 09:05:11\n\n\nMatt McHenry\nSoftware Developer\nCarnegie Learning, Inc.\n(888) 851-7094 x150 toll free\n(412) 690-2444 fax\nmmchenry@carnegielearning.com\nwww.carnegielearning.com\n\nDecision 2012: Election Math | Engaging Video Content | FREE Interactive Math Problems\nhttp://www.nbclearn.com/portal/site/learn/decision2012\n"},{"id":"202738","messageId":"be90fa6b-3927-47cb-9306-6dbb7bac2c04@zcs","threadId":"32040","inReplyTo":"D377A9280DB18546A2471214D5CBB0E9054C25E071@exchdb01","subject":"Re: git-svn with ignore-paths misses/skips some revisions during fetch","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-11-10T07:57:59Z","receivedAt":"2012-11-10T07:57:59Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"\n>         The problem is that the 'ignore-paths' approach sometimes\n>         misses commits during a fetch, and then at some later time\n>         will \"realize\" it and squash those changes onto some other,\n>         unrelated commit.  (I've never seen this happen with the\n>         per-subdir 'fetch' approach.)  Here are three commits in\n>         SVN:\n\nCould it be that certain files spent parts of their historical lifetime\ninside the ignored paths ?\n\n\ncu\n-- \nMit freundlichen Grüßen / Kind regards \n\nEnrico Weigelt \nVNC - Virtual Network Consult GmbH \nHead Of Development \n\nPariser Platz 4a, D-10117 Berlin\nTel.: +49 (30) 3464615-20\nFax: +49 (30) 3464615-59\n\nenrico.weigelt@vnc.biz; www.vnc.de \n"},{"id":"202954","messageId":"D377A9280DB18546A2471214D5CBB0E9054C3B4CF6@exchdb01","threadId":"32040","inReplyTo":"be90fa6b-3927-47cb-9306-6dbb7bac2c04@zcs","subject":"RE: git-svn with ignore-paths misses/skips some revisions during fetch","fromName":"McHenry, Matt","fromEmail":"mmchenry@carnegielearning.com","sentAt":"2012-11-12T15:00:52Z","receivedAt":"2012-11-12T15:00:52Z","isPatch":false,"sender":{"key":"mmchenry@carnegielearning.com","avatar":"https://gravatar.com/avatar/2187a10f4f413a9b901f08107f275b70aa826c8f9bbe7c14fff4dae3ae58384b?d=mp&s=160"},"body":"\nEnrico asked:\n> Could it be that certain files spent parts of their historical lifetime\n> inside the ignored paths ?\n\n\n        I left out one possibly important piece of information: My initial 'git svn fetch' used '-r' to \"cauterize\" the history, both because there is a lot of it (almost 12 years) and because the repository was reorganized significantly after a cvs -> svn migration.  The first revision I have is r83875:\n\n$ git log --max-parents=0 --all --date-order | tail -n 1\n    git-svn-id: svn://dev.carnegielearning.com/trunk@83875 752fcc94-cd22-0410-baa8-ef54ac2c6973\n\n\n        So to answer Enrico's question: Prior to the initial revision that was fetched into git, these files did live in a different top-level directory.  However it's not one that's matched by the 'ignore-paths' regex.  Here's one example:\n\n$ svn log -v svn://dev/branches/localization-merge/buildprocess/antfiles/dmg.xml | grep /dmg.xml | uniq -c\n      9    M /branches/localization-merge/buildprocess/antfiles/dmg.xml\n     35    M /trunk/buildprocess/antfiles/dmg.xml\n      1    A /trunk/buildprocess/antfiles/dmg.xml (from /trunk/buildprocess/assemble-support/dmg.xml:48305)\n      1    D /trunk/buildprocess/assemble-support/dmg.xml\n     57    M /trunk/assemble-support/dmg.xml\n      1    A /trunk/assemble-support/dmg.xml\n\n\nHere are the svn revisions that explain the transition from 'assemble-support' to 'buildprocess/assemble-support', just after the cvs -> svn migration.\n\n------------------------------------------------------------------------\nr48303 | matt | 2006-11-27 14:56:10 -0500 (Mon, 27 Nov 2006) | 1 line\nChanged paths:\n   D /old-trunk/assemble-support\n   A /trunk/buildprocess/assemble-support (from /old-trunk/assemble-support:48302)\n\nmoving old assemble-support into new buildprocess\n------------------------------------------------------------------------\nr48248 | matt | 2006-11-22 13:41:42 -0500 (Wed, 22 Nov 2006) | 1 line\nChanged paths:\n   A /old-trunk (from /trunk:48247)\n   D /trunk\n\nmoving old trunk out of the way\n------------------------------------------------------------------------\n\n\n\nMatt McHenry\nSoftware Developer\nCarnegie Learning, Inc.\n(888) 851-7094 x150 toll free\n(412) 690-2444 fax\nmmchenry@carnegielearning.com\nwww.carnegielearning.com\n\nDecision 2012: Election Math | Engaging Video Content | FREE Interactive Math Problems\nhttp://www.nbclearn.com/portal/site/learn/decision2012\n\n"}]}