{"thread":{"id":"18107","subject":"import files w/ history","startedAt":"2009-03-03T12:54:54Z","lastAt":"2009-03-11T00:11:34Z","messageCount":8,"participants":["Csaba Henk","Jeff King","Miklos Vajna"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"106831","messageId":"slrngqqa4l.1t4t.csaba-ml@beastie.creo.hu","threadId":"18107","inReplyTo":null,"subject":"import files w/ history","fromName":"Csaba Henk","fromEmail":"csaba-ml@creo.hu","sentAt":"2009-03-03T12:54:54Z","receivedAt":"2009-03-03T12:54:54Z","isPatch":false,"sender":{"key":"csaba-ml@creo.hu","avatar":null},"body":"Hi,\n\nHow could I import some files from an unrelated git repo with history?\nAnd if I'd like to use different paths? Eg:\n\nSay the other repo has these files:\n\nlib/trees/rb_tree.{c,h}\n\nand I want to import them into my repo as\n\ninclude/rb_tree.h\nsrc/rb_tree.c\n\n(In fact I just have a single file to import and I don't want to\nvary paths, yet I'm curious about this extended case, too.)\n\nThanks,\nCsaba\n"},{"id":"106832","messageId":"20090303130046.GA7867@coredump.intra.peff.net","threadId":"18107","inReplyTo":"slrngqqa4l.1t4t.csaba-ml@beastie.creo.hu","subject":"Re: import files w/ history","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-03-03T13:00:46Z","receivedAt":"2009-03-03T13:00:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 03, 2009 at 12:54:54PM +0000, Csaba Henk wrote:\n\n> How could I import some files from an unrelated git repo with history?\n\nJust \"git pull\" from the other repo, which will include all of its\nhistory. If you want to pretend that the other history contains just a\nsubset of the true history, use \"git filter-branch\" to rewrite it first.\n\n> And if I'd like to use different paths? Eg:\n> \n> Say the other repo has these files:\n> \n> lib/trees/rb_tree.{c,h}\n> \n> and I want to import them into my repo as\n> \n> include/rb_tree.h\n> src/rb_tree.c\n\nIf you are rewriting the history, you can rename the files as you see\nfit. There is even an example of this in \"git help filter-branch\".\n\n-Peff\n"},{"id":"107197","messageId":"slrngr299k.1t4t.csaba-ml@beastie.creo.hu","threadId":"18107","inReplyTo":"20090303130046.GA7867@coredump.intra.peff.net","subject":"Re: import files w/ history","fromName":"Csaba Henk","fromEmail":"csaba-ml@creo.hu","sentAt":"2009-03-06T13:29:38Z","receivedAt":"2009-03-06T13:29:38Z","isPatch":false,"sender":{"key":"csaba-ml@creo.hu","avatar":null},"body":"On 2009-03-03, Jeff King <peff@peff.net> wrote:\n> On Tue, Mar 03, 2009 at 12:54:54PM +0000, Csaba Henk wrote:\n>\n>> How could I import some files from an unrelated git repo with history?\n>\n> Just \"git pull\" from the other repo, which will include all of its\n> history. If you want to pretend that the other history contains just a\n> subset of the true history, use \"git filter-branch\" to rewrite it first.\n\nThanks Jeff, but it didn't work well for a large repo. At least not what\nI could carve out myself.\n\nThe repo in question is the DragonFlyBSD repository, and I wanted to\ncut out the history of sys/dev/disk/vn/vn.c. After reading\ngit-filter-branch(1) I come up with the following: first I wanted to\nselect those commits where the file in question was modified. I tried\nto use the following filtration:\n\n$ git filter-branch --commit-filter '\n   if [ $# -lt 3 ] || git diff --stat $3 $1 | grep -q 'sys/dev/disk/vn/vn\\.c'\n   then\n     git commit-tree \"$@\"\n   else\n     skip_commit \"$@\"\n   fi' HEAD\n\nIt should select those commits where vn.c differs from the vn.c in the _first_\nparent, so probably it's not exactly what I want, but anyway, I went on\nto give it a try.\n\nI have even tested this filter script on a small repo and it worked\nwell. Then I ran it against the Dfly repo, and after 23 hours of\nprocessing I ended up with:\n\n...\n23575b3e0b087120b0475ae93c505c72a9779fdb\n35ac2f0aa5ac0ca78109781817c524fa354e8691\n23575b3e0b087120b0475ae93c505c72a9779fdb\n35ac2f0aa5ac0ca78109781817c524fa354e8691\n23575b3e0b087120b0475ae93c505c72a9779fdb\n35ac2f0aa5ac0ca78109781817c524fa354e8691\n23575b3e0b087120b0475ae93c505c72a9779fdb\n35ac2f0aa5ac0ca78109781817c524fa354e8691\n23575b3e0b087120b0475ae93c505c72a9779fdb\n35ac2f0aa5ac0ca78109781817c524fa354e8691\nWARNING: Ref 'refs/heads/__rewrite' points to the first one now.\n\nAnd the result is completely f*cked up.\nNeither those two commits which occur repeatedly at the end of the\noutput, nor the commit at the actual position of the __rewrite brach\nhas a parent, and the upstream commits from which these were derived\ndidn't affect vn.c.\n\n  *  *  *\n\nOK, I then tried to do more RTFM and be more clever and efficient, and\nfind a way to specify directly those commits which affect vn.c. As \"git\nrev-list\" can be invoked like \"git rev-list <commit> <path>\", and the\nsynopsis of \"git filter-branch\" is like\n\n git filter-branch [options] [--] [<rev-list options>...]\n\nI then gave a try to:\n\n$ git filter-branch --  master sys/dev/disk/vn/vn.c\n\nbut no dice -- I got:\n\n  fatal: ambiguous argument 'sys/dev/disk/vn/vn.c': unknown revision or\n  path not in the working tree.\n  Use '--' to separate paths from revisions\n  Could not get the commits\n\nAny idea?\n\nThanks,\nCsaba\n"},{"id":"107211","messageId":"20090306153313.GL4371@genesis.frugalware.org","threadId":"18107","inReplyTo":"slrngr299k.1t4t.csaba-ml@beastie.creo.hu","subject":"Re: import files w/ history","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2009-03-06T15:33:13Z","receivedAt":"2009-03-06T15:33:13Z","isPatch":false,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"On Fri, Mar 06, 2009 at 01:29:38PM +0000, Csaba Henk <csaba-ml@creo.hu> wrote:\n> $ git filter-branch --commit-filter '\n>    if [ $# -lt 3 ] || git diff --stat $3 $1 | grep -q 'sys/dev/disk/vn/vn\\.c'\n>    then\n>      git commit-tree \"$@\"\n>    else\n>      skip_commit \"$@\"\n>    fi' HEAD\n\nDid you notice --subdirectory-filter? Maybe it would be more efficient\nto run --subdirectory-filter sys/dev/disk/vn first, then you can play\nwith the resulting small repo to suit your needs. :)\n"},{"id":"107358","messageId":"20090308001021.GA26167@coredump.intra.peff.net","threadId":"18107","inReplyTo":"slrngr299k.1t4t.csaba-ml@beastie.creo.hu","subject":"Re: import files w/ history","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-03-08T00:10:21Z","receivedAt":"2009-03-08T00:10:21Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 06, 2009 at 01:29:38PM +0000, Csaba Henk wrote:\n\n> $ git filter-branch --commit-filter '\n>    if [ $# -lt 3 ] || git diff --stat $3 $1 | grep -q 'sys/dev/disk/vn/vn\\.c'\n>    then\n>      git commit-tree \"$@\"\n>    else\n>      skip_commit \"$@\"\n>    fi' HEAD\n\nWow, I'll bet that was slow to run. And it's not really what you want.\n\nYou are picking commits that changed a particular file, and then\nincluding the _whole_ tree. Remember that commits really record a tree\nstate; we only think of them as \"changes\" because they point to a prior\ncommit with its own tree state. So you are just selecting some subset of\nthe states, but not cutting down the tree in each state.\n\nWhat you really want to do is say:\n\n  - for every commit, narrow the tree to _just_ the one file\n\n  - if there were no changes in the narrowed tree, just throw out the\n    commit\n\nYou can use an --index-filter to do the former, and a --commit-filter to\ndo the latter (or just use --prune-empty, which is a shorthand).\n\nAnother poster had a similar problem, and you can see the right\nfilter-branch recipe there:\n\n  http://article.gmane.org/gmane.comp.version-control.git/111991\n\n>   *  *  *\n> \n> OK, I then tried to do more RTFM and be more clever and efficient, and\n> find a way to specify directly those commits which affect vn.c. As \"git\n> rev-list\" can be invoked like \"git rev-list <commit> <path>\", and the\n> synopsis of \"git filter-branch\" is like\n> \n>  git filter-branch [options] [--] [<rev-list options>...]\n> \n> I then gave a try to:\n> \n> $ git filter-branch --  master sys/dev/disk/vn/vn.c\n> \n> but no dice -- I got:\n> \n>   fatal: ambiguous argument 'sys/dev/disk/vn/vn.c': unknown revision or\n>   path not in the working tree.\n>   Use '--' to separate paths from revisions\n>   Could not get the commits\n> \n> Any idea?\n\nI think you need an extra '--' to separate the paths from the revisions\nin the rev-list arguments:\n\n  git filter-branch -- master -- sys/dev/disk/vn/vn.c\n\nbut even that doesn't quite do what you want. It limits the commits that\nare shown, similar to your first attempt above, but it doesn't cut down\nthe tree itself (OTOH, limiting by path rather than using --prune-empty\nis likely to run faster, since you won't even look at commits that are\nuninteresting. However, it may change the shape of your history with\nrespect to branching and merging).\n\n-Peff\n"},{"id":"107438","messageId":"slrngr99ei.1t4t.csaba-ml@beastie.creo.hu","threadId":"18107","inReplyTo":"20090308001021.GA26167@coredump.intra.peff.net","subject":"Re: import files w/ history","fromName":"Csaba Henk","fromEmail":"csaba-ml@creo.hu","sentAt":"2009-03-09T05:15:16Z","receivedAt":"2009-03-09T05:15:16Z","isPatch":false,"sender":{"key":"csaba-ml@creo.hu","avatar":null},"body":"On 2009-03-08, Jeff King <peff@peff.net> wrote:\n> What you really want to do is say:\n>\n>   - for every commit, narrow the tree to _just_ the one file\n>\n>   - if there were no changes in the narrowed tree, just throw out the\n>     commit\n>\n> You can use an --index-filter to do the former, and a --commit-filter to\n> do the latter (or just use --prune-empty, which is a shorthand).\n>\n> Another poster had a similar problem, and you can see the right\n> filter-branch recipe there:\n>\n>   http://article.gmane.org/gmane.comp.version-control.git/111991\n\nThanks, this did the job.\n\n>\n> I think you need an extra '--' to separate the paths from the revisions\n> in the rev-list arguments:\n>\n>   git filter-branch -- master -- sys/dev/disk/vn/vn.c\n>\n> but even that doesn't quite do what you want. It limits the commits that\n> are shown, similar to your first attempt above, but it doesn't cut down\n> the tree itself (OTOH, limiting by path rather than using --prune-empty\n> is likely to run faster, since you won't even look at commits that are\n> uninteresting. However, it may change the shape of your history with\n> respect to branching and merging).\n\nFinally I choose to add the path to the rev-list args -- 80 vs\n15000 commits does make a difference. (I can still check if there was\nany histroy [I just coined this from \"history\" and \"destroy\" :)] and\ngo back to the full-scan way if yes.)\n\nBut I still had a hard time with it... Finally I realized that if I do\nfiltering this way, I have to start filtering from the topmost commit\nwhich affects the given file.\n\nIf I just start from origin/HEAD (assuming that it's on a commit which\ndoes not affect the file), then it won't be found as a key of the mapping\ncreated by git-filter-branch (as it's ignored because rev-listing was\nnarrowed down to the file), and therefore filter-branch finally punts\nwith \"WARNING: Ref '<sha1>' is unchanged\". I don't know if it's an\nintended behaviour, or something which could/should be improved, or at\nleast documented... seems to be some sort of POLS violation to me (at\nleast I was surprised :) ).\n\nRegards,\nCsaba\n"},{"id":"107599","messageId":"20090310180353.GC26351@sigill.intra.peff.net","threadId":"18107","inReplyTo":"slrngr99ei.1t4t.csaba-ml@beastie.creo.hu","subject":"Re: import files w/ history","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-03-10T18:03:53Z","receivedAt":"2009-03-10T18:03:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 09, 2009 at 05:15:16AM +0000, Csaba Henk wrote:\n\n> But I still had a hard time with it... Finally I realized that if I do\n> filtering this way, I have to start filtering from the topmost commit\n> which affects the given file.\n> \n> If I just start from origin/HEAD (assuming that it's on a commit which\n> does not affect the file), then it won't be found as a key of the mapping\n> created by git-filter-branch (as it's ignored because rev-listing was\n> narrowed down to the file), and therefore filter-branch finally punts\n> with \"WARNING: Ref '<sha1>' is unchanged\". I don't know if it's an\n> intended behaviour, or something which could/should be improved, or at\n> least documented... seems to be some sort of POLS violation to me (at\n> least I was surprised :) ).\n\nI think passing path limiters to filter-branch is just something that\nnobody ever really tried before. I think the solutions are, in order of\ndecreasing easiness and increasing difficulty:\n\n  1. document the problem in Documentation/git-filter-branch.txt\n\n  2. create a failing test for it in the test suite\n\n  3. fix the failing test. ;)\n\nDo you want to try a patch for one (or more!) of those?\n\n-Peff\n"},{"id":"107621","messageId":"slrngre0d2.1t4t.csaba-ml@beastie.creo.hu","threadId":"18107","inReplyTo":"20090310180353.GC26351@sigill.intra.peff.net","subject":"Re: import files w/ history","fromName":"Csaba Henk","fromEmail":"csaba-ml@creo.hu","sentAt":"2009-03-11T00:11:34Z","receivedAt":"2009-03-11T00:11:34Z","isPatch":false,"sender":{"key":"csaba-ml@creo.hu","avatar":null},"body":"On 2009-03-10, Jeff King <peff@peff.net> wrote:\n> I think passing path limiters to filter-branch is just something that\n> nobody ever really tried before. I think the solutions are, in order of\n> decreasing easiness and increasing difficulty:\n>\n>   1. document the problem in Documentation/git-filter-branch.txt\n>\n>   2. create a failing test for it in the test suite\n>\n>   3. fix the failing test. ;)\n>\n> Do you want to try a patch for one (or more!) of those?\n\nI'm eager to contribute... next month. I'm pretty much overwhelmed\nnow. (Accidentally I almost wrote \"overclocked\" instead of\n\"overwhelmed\"... only if that were true!)\n\nRegards,\nCsaba\n"}]}