{"thread":{"id":"30239","subject":"Filter-branch's \"move tree to subdirectory\" example fails with BSD sed?","startedAt":"2012-04-14T16:00:28Z","lastAt":"2012-04-17T18:36:49Z","messageCount":9,"participants":["Christopher Tiwald","Johannes Sixt","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"189262","messageId":"20120414160028.GD1012@gmail.com","threadId":"30239","inReplyTo":null,"subject":"Filter-branch's \"move tree to subdirectory\" example fails with BSD sed?","fromName":"Christopher Tiwald","fromEmail":"christiwald@gmail.com","sentAt":"2012-04-14T16:00:28Z","receivedAt":"2012-04-14T16:00:28Z","isPatch":false,"sender":{"key":"christiwald@gmail.com","avatar":"https://avatars.githubusercontent.com/u/667276?v=4"},"body":"The \"Move tree to subdirectory\" example in the 'git filter-branch'\nmanpage fails on Mac OSX 10.7.3, but succeeds on Ubuntu 10.04. I'm\nusing git version 1.7.0.4, which happened to be the version installed\non the Ubuntu VM I had laying around. I think it's a difference between\n'sed' on the two systems.\n\nThe example:\ngit filter-branch --index-filter \\\n        'git ls-files -s | sed \"s-\\t\\\"*-&newsubdir/-\" |\n                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \\\n                        git update-index --index-info &&\n         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD\n\nDemonstrating the problem is with sed:\ngit init \"test\"\ncd \"test\"\nmkdir -p subdirA/subdirB\necho content > subdirA/subdirB/file\ngit add .\ngit commit -m \"initial commit\"\ngit ls-files -s | sed \"s-\\t\\\"*-&newsubdir/-\"\n\nOn Mac 10.7.3 the final command outputs:\n100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0\tsubdirA/subdirB/file\n\nOn Ubuntu 10.04:\n100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0\tnewsubdir/subdirA/subdirB/file\n\nI can solve my immediate problem using Ubuntu, but is there a way we\ncould rewrite the example to work on both systems? I'm afraid differences\nbetween BSD and GNU 'sed' are a bit beyond me.\n\n--\nChristopher Tiwald\n"},{"id":"189278","messageId":"4F89CF11.7060309@kdbg.org","threadId":"30239","inReplyTo":"20120414160028.GD1012@gmail.com","subject":"Re: Filter-branch's \"move tree to subdirectory\" example fails with BSD sed?","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2012-04-14T19:25:05Z","receivedAt":"2012-04-14T19:25:05Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 14.04.2012 18:00, schrieb Christopher Tiwald:\n> The \"Move tree to subdirectory\" example in the 'git filter-branch'\n> manpage fails on Mac OSX 10.7.3, but succeeds on Ubuntu 10.04.\n\n> git init \"test\"\n> cd \"test\"\n> mkdir -p subdirA/subdirB\n> echo content > subdirA/subdirB/file\n> git add .\n> git commit -m \"initial commit\"\n> git ls-files -s | sed \"s-\\t\\\"*-&newsubdir/-\"\n> \n> On Mac 10.7.3 the final command outputs:\n> 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0\tsubdirA/subdirB/file\n> \n> On Ubuntu 10.04:\n> 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0\tnewsubdir/subdirA/subdirB/file\n\nPerhaps a literal TAB instead of \\t makes the example work?\n\nIt would be difficult, though, to write this down in the manual in an\nunambiguous way.\n\n-- Hannes\n"},{"id":"189415","messageId":"20120416152737.GB14724@sigill.intra.peff.net","threadId":"30239","inReplyTo":"4F89CF11.7060309@kdbg.org","subject":"Re: Filter-branch's \"move tree to subdirectory\" example fails with BSD sed?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-04-16T15:27:38Z","receivedAt":"2012-04-16T15:27:38Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Apr 14, 2012 at 09:25:05PM +0200, Johannes Sixt wrote:\n\n> Am 14.04.2012 18:00, schrieb Christopher Tiwald:\n> > The \"Move tree to subdirectory\" example in the 'git filter-branch'\n> > manpage fails on Mac OSX 10.7.3, but succeeds on Ubuntu 10.04.\n> \n> > git init \"test\"\n> > cd \"test\"\n> > mkdir -p subdirA/subdirB\n> > echo content > subdirA/subdirB/file\n> > git add .\n> > git commit -m \"initial commit\"\n> > git ls-files -s | sed \"s-\\t\\\"*-&newsubdir/-\"\n> > \n> > On Mac 10.7.3 the final command outputs:\n> > 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0\tsubdirA/subdirB/file\n> > \n> > On Ubuntu 10.04:\n> > 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0\tnewsubdir/subdirA/subdirB/file\n> \n> Perhaps a literal TAB instead of \\t makes the example work?\n\nYes, I suspect that is the problem, too.\n\n> It would be difficult, though, to write this down in the manual in an\n> unambiguous way.\n\nMaybe it would be simpler to just use perl:\n\n git filter-branch --index-filter '\n   git ls-files -s |\n     perl -pe \"s{\\t\\\"?}{$&newsubdir/}\" |\n     GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&\n   mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE\n ' HEAD\n\nI also think tweaking the indentation and line breaks as I did above\nmakes it a little more readable. If we are using perl, we could\nalso just use \"-z\" to get rid of the funny quote handling.\n\nI wish we could also get rid of the temporary index. It exists solely\nfor the side effect of removing the existing entries (otherwise, you get\nboth \"foo\" and \"newsubdir/foo\" in the resulting history). It would be\nnice if there was some flag to update-index to say \"before you do\nanything, clear the existing index\" (or I guess, \"don't bother reading\nthe existing index\"). And then result could look like:\n\n  git filter-branch --index-filter '\n    git ls-files -sz |\n    perl -0pe \"s{\\t}{\\tnewsubdir/}\" |\n    git update-index --from-scratch -z --index-info\n  ' HEAD\n\nwhich is IMHO much easier to read and understand.\n\n-Peff\n"},{"id":"189423","messageId":"20120416160232.GA1350@sigill.intra.peff.net","threadId":"30239","inReplyTo":"20120416152737.GB14724@sigill.intra.peff.net","subject":"[RFC/PATCH 1/2] update-index: add --clear option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-04-16T16:02:32Z","receivedAt":"2012-04-16T16:02:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This just discards existing entries from the index, which\ncan be useful if you are rewriting all entries with\n\"--index-info\" or similar.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nI tried to make something like:\n\n  git update-index --from-scratch --index-info\n\nwork by avoiding reading all entries in the first place. However,\nupdate-index actually processes its arguments sequentially, so we _must_\nread the index before we start processing arguments. But because it's\nsequential, a \"clear\" operation makes sense, since you clear, then add\nnew entries.\n\n Documentation/git-update-index.txt |    3 +++\n builtin/update-index.c             |   10 ++++++++++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex a3081f4..47f0ae6 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -71,6 +71,9 @@ OPTIONS\n --cacheinfo <mode> <object> <path>::\n \tDirectly insert the specified info into the index.\n \n+--clear::\n+\tDiscard all existing entries from the index.\n+\n --index-info::\n         Read index information from stdin.\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex a6a23fa..559dfae 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -645,6 +645,13 @@ static int cacheinfo_callback(struct parse_opt_ctx_t *ctx,\n \treturn 0;\n }\n \n+static int clear_callback(const struct option *opt,\n+\t\t\t  const char *arg, int unset)\n+{\n+\tdiscard_cache();\n+\treturn 0;\n+}\n+\n static int stdin_cacheinfo_callback(struct parse_opt_ctx_t *ctx,\n \t\t\t      const struct option *opt, int unset)\n {\n@@ -774,6 +781,9 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t\"add entries from standard input to the index\",\n \t\t\tPARSE_OPT_NONEG | PARSE_OPT_NOARG,\n \t\t\t(parse_opt_cb *) stdin_cacheinfo_callback},\n+\t\t{OPTION_CALLBACK, 0, \"clear\", NULL, NULL,\n+\t\t\t\"drop all index entries\", PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n+\t\t\tclear_callback},\n \t\t{OPTION_LOWLEVEL_CALLBACK, 0, \"unresolve\", &has_errors, NULL,\n \t\t\t\"repopulate stages #2 and #3 for the listed paths\",\n \t\t\tPARSE_OPT_NONEG | PARSE_OPT_NOARG,\n-- \n1.7.9.6.8.g992e5\n"},{"id":"189424","messageId":"20120416160300.GB1350@sigill.intra.peff.net","threadId":"30239","inReplyTo":"20120416152737.GB14724@sigill.intra.peff.net","subject":"[RFC/PATCH 2/2] docs/filter-branch: clean up newsubdir example","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-04-16T16:03:00Z","receivedAt":"2012-04-16T16:03:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Over the years, this simple example has ended up quite hard\nto read because of the number of special cases that must be\nhandled. Let's simplify it a bit:\n\n  1. Use the new \"index-info --clear\" to avoid the need for\n     a temporary index.\n\n  2. Use \"-z\" and \"perl -0\" to avoid dealing with quoting\n     issues. As a bonus, using perl means that \"\\t\" will\n     work consistently in regexps (the previous example\n     using sed was reported to fail on OS X).\n\n  3. Change the indentation to keep one logical unit per\n     line and avoid extra backslash-escaping.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n Documentation/git-filter-branch.txt |   10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex 0f2f117..3df138f 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -358,11 +358,11 @@ git filter-branch ... D..H --not C\n To move the whole tree into a subdirectory, or remove it from there:\n \n ---------------------------------------------------------------\n-git filter-branch --index-filter \\\n-\t'git ls-files -s | sed \"s-\\t\\\"*-&newsubdir/-\" |\n-\t\tGIT_INDEX_FILE=$GIT_INDEX_FILE.new \\\n-\t\t\tgit update-index --index-info &&\n-\t mv \"$GIT_INDEX_FILE.new\" \"$GIT_INDEX_FILE\"' HEAD\n+git filter-branch --index-filter '\n+\tgit ls-files -sz |\n+\tperl -0pe \"s{\\t}{\\tnewsubdir/}\" |\n+\tgit update-index -z --clear --index-info\n+' HEAD\n ---------------------------------------------------------------\n \n \n-- \n1.7.9.6.8.g992e5\n"},{"id":"189434","messageId":"7vr4vnwqwo.fsf@alter.siamese.dyndns.org","threadId":"30239","inReplyTo":"20120416152737.GB14724@sigill.intra.peff.net","subject":"Re: Filter-branch's \"move tree to subdirectory\" example fails with BSD sed?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-04-16T17:03:03Z","receivedAt":"2012-04-16T17:03:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> I wish we could also get rid of the temporary index. It exists solely\n> for the side effect of removing the existing entries (otherwise, you get\n> both \"foo\" and \"newsubdir/foo\" in the resulting history). It would be\n> nice if there was some flag to update-index to say \"before you do\n> anything, clear the existing index\" (or I guess, \"don't bother reading\n> the existing index\"). And then result could look like:\n>\n>   git filter-branch --index-filter '\n>     git ls-files -sz |\n>     perl -0pe \"s{\\t}{\\tnewsubdir/}\" |\n>     git update-index --from-scratch -z --index-info\n>   ' HEAD\n>\n> which is IMHO much easier to read and understand.\n\nWouldn't \"git read-tree --prefix=newsubdir/\" suffice without the pipeline?\nI.e.\n\n        git filter-branch --index-filter '\n\t\trm -f \"$GIT_INDEX_FILE\"\n                git read-tree --prefix=newsubdir/ \"$GIT_COMMIT\"\n\t' HEAD\n"},{"id":"189436","messageId":"20120416171307.GA11221@sigill.intra.peff.net","threadId":"30239","inReplyTo":"7vr4vnwqwo.fsf@alter.siamese.dyndns.org","subject":"Re: Filter-branch's \"move tree to subdirectory\" example fails with BSD sed?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-04-16T17:13:07Z","receivedAt":"2012-04-16T17:13:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 16, 2012 at 10:03:03AM -0700, Junio C Hamano wrote:\n\n> >   git filter-branch --index-filter '\n> >     git ls-files -sz |\n> >     perl -0pe \"s{\\t}{\\tnewsubdir/}\" |\n> >     git update-index --from-scratch -z --index-info\n> >   ' HEAD\n> \n> Wouldn't \"git read-tree --prefix=newsubdir/\" suffice without the pipeline?\n> I.e.\n> \n>         git filter-branch --index-filter '\n> \t\trm -f \"$GIT_INDEX_FILE\"\n>                 git read-tree --prefix=newsubdir/ \"$GIT_COMMIT\"\n> \t' HEAD\n\nYeah, I think it does (at least it makes sense to me, and worked on a\nsimple test case). I think I might use \"git read-tree --empty\" instead\nof \"rm\", as it is a little more obvious what is going on. But other than\nthat, it is much more readable.\n\nIt is a slight shame not to show an \"ls-files | update-index\" example,\nbecause that is the most general form. But since this form is quite\noften cut-and-paste by people, I think simple is better here. If we want\nto show a more complex example, then we should add it separately.\n\n-Peff\n"},{"id":"189470","messageId":"20120416214855.GM12860@gmail.com","threadId":"30239","inReplyTo":"20120416160232.GA1350@sigill.intra.peff.net","subject":"Re: [RFC/PATCH 1/2] update-index: add --clear option","fromName":"Christopher Tiwald","fromEmail":"christiwald@gmail.com","sentAt":"2012-04-16T21:48:55Z","receivedAt":"2012-04-16T21:48:55Z","isPatch":true,"sender":{"key":"christiwald@gmail.com","avatar":"https://avatars.githubusercontent.com/u/667276?v=4"},"body":"On Mon, Apr 16, 2012 at 12:02:32PM -0400, Jeff King wrote:\n> I tried to make something like:\n> \n>   git update-index --from-scratch --index-info\n> \n> work by avoiding reading all entries in the first place. However,\n> update-index actually processes its arguments sequentially, so we _must_\n> read the index before we start processing arguments. But because it's\n> sequential, a \"clear\" operation makes sense, since you clear, then add\n> new entries.\n\nAt the very least I can give this patch series a go this evening. I'm in\nthe process of combining 40 highly-intertwined, nested repositories into\none, and should be able to test it under the conditions outlined in the\nupdated example.\n\n--\nChristopher Tiwald\n"},{"id":"189551","messageId":"CAHiigudcsEqaZwv7MpRVEw-5V60VJ7zemuh4u902vvhrAsRJGQ@mail.gmail.com","threadId":"30239","inReplyTo":"20120416214855.GM12860@gmail.com","subject":"Re: [RFC/PATCH 1/2] update-index: add --clear option","fromName":"Christopher Tiwald","fromEmail":"christiwald@gmail.com","sentAt":"2012-04-17T18:36:49Z","receivedAt":"2012-04-17T18:36:49Z","isPatch":true,"sender":{"key":"christiwald@gmail.com","avatar":"https://avatars.githubusercontent.com/u/667276?v=4"},"body":"On Mon, Apr 16, 2012 at 5:48 PM, Christopher Tiwald\n<christiwald@gmail.com> wrote:\n> At the very least I can give this patch series a go this evening. I'm in\n> the process of combining 40 highly-intertwined, nested repositories into\n> one, and should be able to test it under the conditions outlined in the\n> updated example.\n\nThis worked great for me, at least in the case of adding trees to\nsubdirectories.\nFwiw, I found the newer, patched example a little easier to parse, but\nI also had\nspent a while figuring out what the original version did.\n\n--\nChristopher Tiwald\n"}]}