{"thread":{"id":"26070","subject":"subdirectory-filter does not delete files before the directory came into existence?","startedAt":"2010-12-14T22:21:35Z","lastAt":"2010-12-19T22:51:48Z","messageCount":8,"participants":["Jan Wielemaker","Thomas Rast"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"158105","messageId":"1292365295.6666.22.camel@ct","threadId":"26070","inReplyTo":null,"subject":"subdirectory-filter does not delete files before the directory came into existence?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@cs.vu.nl","sentAt":"2010-12-14T22:21:35Z","receivedAt":"2010-12-14T22:21:35Z","isPatch":false,"sender":{"key":"j.wielemaker@cs.vu.nl","avatar":null},"body":"Hi,\n\nThere is a lot of information about extracting a directory from a git\nproject.  One thing I failed to find though is the following:\n\nI try to extract a directory.  The result is fine, but there is a lot\nof history in the result from *before* the directory was added to the\nproject.  Why?  How can I get rid of this?\n\nIf you want to see yourself, I did:\n\n\tgit clone git://www.swi-prolog.org/home/pl/git/pl-devel.git\n\tgit clone pl-devel odbc\n\tcd odbc\n\tgit filter-branch --subdirectory-filter packages/odbc --prune-empty\n--tag-name-filter cat -- --all\n\t\nNow use e.g. qgit to look at the history.  As from 03/07/2002, when\nthe packages/odbc directory was created, all looks just fine.  Before\nthough ...\n\n\tThanks for any hints\n\n\t\t--- Jan\n"},{"id":"158118","messageId":"201012150003.30127.trast@student.ethz.ch","threadId":"26070","inReplyTo":"1292365295.6666.22.camel@ct","subject":"Re: subdirectory-filter does not delete files before the directory came into existence?","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-12-14T23:03:29Z","receivedAt":"2010-12-14T23:03:29Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Jan Wielemaker wrote:\n> I try to extract a directory.  The result is fine, but there is a lot\n> of history in the result from *before* the directory was added to the\n> project.  Why?  How can I get rid of this?\n[...]\n> Now use e.g. qgit to look at the history.  As from 03/07/2002, when\n> the packages/odbc directory was created, all looks just fine.  Before\n> though ...\n\nThat history is not connected to the filtered one.  git-filter-branch\nalerts you to it with messages like\n\n  WARNING: Ref 'refs/tags/V5.0.4' is unchanged\n  WARNING: Ref 'refs/tags/V5.0.5' is unchanged\n  WARNING: Ref 'refs/tags/V5.0.6' is unchanged\n  WARNING: Ref 'refs/tags/V5.0.7' is unchanged\n\nI haven't made up my mind if this is a bug report or a feature\nrequest, but in any case you can delete all of them and the problem\ngoes away.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"158154","messageId":"1292406642.11607.37.camel@hppc323.few.vu.nl","threadId":"26070","inReplyTo":"201012150003.30127.trast@student.ethz.ch","subject":"Re: subdirectory-filter does not delete files before the directory came into existence?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@cs.vu.nl","sentAt":"2010-12-15T09:50:42Z","receivedAt":"2010-12-15T09:50:42Z","isPatch":false,"sender":{"key":"j.wielemaker@cs.vu.nl","avatar":null},"body":"Dear Thomas,\n\nOn Wed, 2010-12-15 at 00:03 +0100, Thomas Rast wrote:\n> Jan Wielemaker wrote:\n> > I try to extract a directory.  The result is fine, but there is a lot\n> > of history in the result from *before* the directory was added to the\n> > project.  Why?  How can I get rid of this?\n> [...]\n> > Now use e.g. qgit to look at the history.  As from 03/07/2002, when\n> > the packages/odbc directory was created, all looks just fine.  Before\n> > though ...\n> \n> That history is not connected to the filtered one.  git-filter-branch\n> alerts you to it with messages like\n> \n>   WARNING: Ref 'refs/tags/V5.0.4' is unchanged\n>   WARNING: Ref 'refs/tags/V5.0.5' is unchanged\n>   WARNING: Ref 'refs/tags/V5.0.6' is unchanged\n>   WARNING: Ref 'refs/tags/V5.0.7' is unchanged\n\nThanks for the insight.  Catching these errors and running git tag -d on\nthem gets me a nice and clean history.  Only ...  It starts in\n12/08/2008 instead of 03/07/2002.  This is (almost) compatible with the\nfiltering feedback that says it rewrote 174 commits.  The filtered and\ncleaned history contains 171.\n\nThis is a bit odd.  If I open qgit on the original (before filtering)\nand show the history of odbc.c, it looks like a nice and continuous\none going back to 2002.  Also\n\n   git log --oneline packages/odbc/odbc.c\n\nshows a history that starts with \"First public version of ODBC\ninterface\"\n\nOf course, this is a project with a long history that was converted\nfrom CVS, but the history looks unbroken, so why does filtering a\ndirectory breaks it?\n\n> I haven't made up my mind if this is a bug report or a feature\n> request, but in any case you can delete all of them and the problem\n> goes away.\n\nIsn't it true that you will have info from before introducing a\ndirectory whenever there are tags that are older than the directory?\nIf that is the case, it looks wrong to me.  I want to filter the \ndirectory, so the repository from before the existence of the \ndirectory is not interesting.  Of course, things change if the\ndirectory was created by renaming files that where already in\nthe repository.  I don't know what one should `expect' in that\ncase.  Here, the directory was added from new files, so it is\nquite clear what one should expect.\n\n\tRegards --- Jan\n"},{"id":"158155","messageId":"1292409621.11607.41.camel@hppc323.few.vu.nl","threadId":"26070","inReplyTo":"201012150003.30127.trast@student.ethz.ch","subject":"Re: subdirectory-filter does not delete files before the directory came into existence?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@cs.vu.nl","sentAt":"2010-12-15T10:40:21Z","receivedAt":"2010-12-15T10:40:21Z","isPatch":false,"sender":{"key":"j.wielemaker@cs.vu.nl","avatar":null},"body":"In addition to my previous reply: Looking at the result of the\ninitial filter, if remove all unchanged refs I loose the history\nbefore 2008.  Qgit however shows a broken history at the start\nof the directory in 2002.  If I keep deleting the tag that is\nthe head of older stuff I end up with what I hoped in the first\nplace.  This is of course a bit tedious :-(\n\nYou can view the result at\n\n   git://www.swi-prolog.org/home/pl/git/packages/odbc.git\n\nI'll split some more packages.  Curious to what is going to happen ...\n\n\tRegards --- Jan\n\nOn Wed, 2010-12-15 at 00:03 +0100, Thomas Rast wrote:\n> Jan Wielemaker wrote:\n> > I try to extract a directory.  The result is fine, but there is a lot\n> > of history in the result from *before* the directory was added to the\n> > project.  Why?  How can I get rid of this?\n> [...]\n> > Now use e.g. qgit to look at the history.  As from 03/07/2002, when\n> > the packages/odbc directory was created, all looks just fine.  Before\n> > though ...\n> \n> That history is not connected to the filtered one.  git-filter-branch\n> alerts you to it with messages like\n> \n>   WARNING: Ref 'refs/tags/V5.0.4' is unchanged\n>   WARNING: Ref 'refs/tags/V5.0.5' is unchanged\n>   WARNING: Ref 'refs/tags/V5.0.6' is unchanged\n>   WARNING: Ref 'refs/tags/V5.0.7' is unchanged\n> \n> I haven't made up my mind if this is a bug report or a feature\n> request, but in any case you can delete all of them and the problem\n> goes away.\n> \n"},{"id":"158156","messageId":"1292415733.11607.51.camel@hppc323.few.vu.nl","threadId":"26070","inReplyTo":"201012150003.30127.trast@student.ethz.ch","subject":"Re: subdirectory-filter does not delete files before the directory came into existence?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@cs.vu.nl","sentAt":"2010-12-15T12:22:13Z","receivedAt":"2010-12-15T12:22:13Z","isPatch":false,"sender":{"key":"j.wielemaker@cs.vu.nl","avatar":null},"body":"The reported problems also apply to the next module.  What appears to\nwork is this:\n\n  * Walk through the history, finding the commit where the directory\n  is created.\n  * use git tag -l --contains <commit that created dir> to get the \n  tags we want to keep.\n  * get all tags, use comm and delete the tags not in the `contained'\n  set above.\n\nNot very friendly and I'm (with Thomas) about the status of these\nfindings.  I like to thank Thomas for giving me the right clue.\n\n\tRegards --- Jan\n\nOn Wed, 2010-12-15 at 00:03 +0100, Thomas Rast wrote:\n> Jan Wielemaker wrote:\n> > I try to extract a directory.  The result is fine, but there is a lot\n> > of history in the result from *before* the directory was added to the\n> > project.  Why?  How can I get rid of this?\n> [...]\n> > Now use e.g. qgit to look at the history.  As from 03/07/2002, when\n> > the packages/odbc directory was created, all looks just fine.  Before\n> > though ...\n> \n> That history is not connected to the filtered one.  git-filter-branch\n> alerts you to it with messages like\n> \n>   WARNING: Ref 'refs/tags/V5.0.4' is unchanged\n>   WARNING: Ref 'refs/tags/V5.0.5' is unchanged\n>   WARNING: Ref 'refs/tags/V5.0.6' is unchanged\n>   WARNING: Ref 'refs/tags/V5.0.7' is unchanged\n> \n> I haven't made up my mind if this is a bug report or a feature\n> request, but in any case you can delete all of them and the problem\n> goes away.\n> \n"},{"id":"158350","messageId":"201012190323.59829.trast@student.ethz.ch","threadId":"26070","inReplyTo":"1292415733.11607.51.camel@hppc323.few.vu.nl","subject":"Re: subdirectory-filter does not delete files before the directory came into existence?","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-12-19T02:23:59Z","receivedAt":"2010-12-19T02:23:59Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Jan Wielemaker wrote:\n> The reported problems also apply to the next module.  What appears to\n> work is this:\n> \n>   * Walk through the history, finding the commit where the directory\n>   is created.\n>   * use git tag -l --contains <commit that created dir> to get the \n>   tags we want to keep.\n>   * get all tags, use comm and delete the tags not in the `contained'\n>   set above.\n> \n> Not very friendly and I'm (with Thomas) about the status of these\n> findings.  I like to thank Thomas for giving me the right clue.\n\nNow I finally remember where I knew this problem from:\n\n  http://article.gmane.org/gmane.comp.version-control.git/91708\n\n(My memory really sucks.)\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"158356","messageId":"1292751289.21229.26.camel@ct","threadId":"26070","inReplyTo":"201012190323.59829.trast@student.ethz.ch","subject":"Re: subdirectory-filter does not delete files before the directory came into existence?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@cs.vu.nl","sentAt":"2010-12-19T09:34:49Z","receivedAt":"2010-12-19T09:34:49Z","isPatch":false,"sender":{"key":"j.wielemaker@cs.vu.nl","avatar":null},"body":"On Sun, 2010-12-19 at 03:23 +0100, Thomas Rast wrote:\n> Jan Wielemaker wrote:\n> > The reported problems also apply to the next module.  What appears to\n> > work is this:\n> > \n> >   * Walk through the history, finding the commit where the directory\n> >   is created.\n> >   * use git tag -l --contains <commit that created dir> to get the \n> >   tags we want to keep.\n> >   * get all tags, use comm and delete the tags not in the `contained'\n> >   set above.\n> > \n> > Not very friendly and I'm (with Thomas) about the status of these\n> > findings.  I like to thank Thomas for giving me the right clue.\n> \n> Now I finally remember where I knew this problem from:\n> \n>   http://article.gmane.org/gmane.comp.version-control.git/91708\n> \n> (My memory really sucks.)\n\nFunny.  That was me having problems with filtering out directories\nas well :-)  I thought your patch was added using the --prune-empty\nflag.  I guess you can comment on that.  I can confirm that I've got\nnice and clean filtering using\n\n  * git filter-branch --subdirectory-filter <dir> --prune-empty\n--tag-name-filter cat -- --all\n  \nfollowed by the steps above.  I use qgit with the tree-view enabled\nto find the place where the hierarchy changes from the complete one\nto the only-this-dir one.  You can do a binary search for that and\nyou spot the exact commit easily by the gap in the history-line.  Then\nI run this little bit of code:\n\n#!/bin/bash\n\ncontains=\"$1\"\n\ngit tag | sort > tags.all\ngit tag -l --contains $contains | sort > tags.keep\n\nfor t in `comm -23 tags.all tags.keep`; do\n  git tag -d $t\ndone\n\nNot ideal, but doable.\n\n\tCheers --- Jan\n"},{"id":"158374","messageId":"201012192351.49279.trast@student.ethz.ch","threadId":"26070","inReplyTo":"1292751289.21229.26.camel@ct","subject":"Re: subdirectory-filter does not delete files before the directory came into existence?","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-12-19T22:51:48Z","receivedAt":"2010-12-19T22:51:48Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Jan Wielemaker wrote:\n> On Sun, 2010-12-19 at 03:23 +0100, Thomas Rast wrote:\n> > Jan Wielemaker wrote:\n> > >   * get all tags, use comm and delete the tags not in the `contained'\n> > >   set above.\n[...]\n> >   http://article.gmane.org/gmane.comp.version-control.git/91708\n[...]\n> Funny.  That was me having problems with filtering out directories\n> as well :-)  I thought your patch was added using the --prune-empty\n> flag.  I guess you can comment on that.  I can confirm that I've got\n> nice and clean filtering using\n\nNo, those two are rather different.  --prune-empty drops commits that\nbecame \"no-ops\" in the sense that their tree is the same as their\n(only) parent's.  In the case of --subdirectory-filter, --prune-empty\nis most likely[*] redundant since the former already enables history\nsimplification limited to that directory.\n\nAs you can see from \"TOY PATCH\", my patch wasn't really meant for\napplication anyway.  I'm now wondering what the ramifications would\nbe.  filter-branch only attempts to change refs that you told it to\n(listed positively on the command line), so maybe deleting anything\nthat was not rewritten is a sensible option (not default, mind you).\n\n\n[*] Read: I think it is redundant, I'm just too lazy to double-check.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"}]}