{"thread":{"id":"27980","subject":"git-fast-export is returning streams with source code inside","startedAt":"2011-08-01T11:57:58Z","lastAt":"2011-08-05T10:30:16Z","messageCount":8,"participants":["James Gregory","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"172531","messageId":"CAFC9htxwRg=+RD68hnnRy0hfptq23x3bL+xxHieK1evfFCTPjw@mail.gmail.com","threadId":"27980","inReplyTo":null,"subject":"git-fast-export is returning streams with source code inside","fromName":"James Gregory","fromEmail":"j.gregory@epigenesys.co.uk","sentAt":"2011-08-01T11:57:58Z","receivedAt":"2011-08-01T11:57:58Z","isPatch":false,"sender":{"key":"j.gregory@epigenesys.co.uk","avatar":"https://gravatar.com/avatar/3f482c78bfc90928994cfbf8edb04e4c23722ac07647b580b9bf529a68039ccf?d=mp&s=160"},"body":"Hello,\n\nI'm not sure if this is the right place to post bug reports - if not,\nplease can you let me know.\n\nI've found (what appears to be) a problem in git-fast-export. I've\nfiled a bug in Launchpad, but I'm not sure if anyone from the Git\ncommunity actively monitors it so I'm reporting it here to see if it\ncan be fixed.\n\nThe bug can be viewed online at:\nhttps://bugs.launchpad.net/ubuntu/+source/git/+bug/815828\nand I've done my best to describe the bug in some detail. If you need\nme to elaborate on the problem further, don't hesitate to let me know.\n\nThanks,\nJames\n\n-- \n\nJames Gregory\nSenior Developer\nepiGenesys - a University of Sheffield company\nEnterprise Zone, Portobello, Sheffield, S1 4DP, UK\nT: +44 (0)114 22 21 884 ~ F: +44 (0)114 22 24 042\nwww.epigenesys.co.uk ~ @epigenesys\n--\nThis email and its attachments should be considered confidential and\nare intended solely for the individual to whom the email is addressed.\nIf you are not the intended recipient you should take no action based\nupon them, nor should you copy or show them to anyone. Any views or\nopinions expressed are solely those of the author and might not\nrepresent those of Epi Genesys Limited.\nEpi Genesys Limited is registered in England and Wales with company\nnumber 06388971 and has its registered office at 40 Leavygreave Road,\nSheffield, S3 7RD.\n"},{"id":"172888","messageId":"20110804070528.GA11805@sigill.intra.peff.net","threadId":"27980","inReplyTo":"CAFC9htxwRg=+RD68hnnRy0hfptq23x3bL+xxHieK1evfFCTPjw@mail.gmail.com","subject":"Re: git-fast-export is returning streams with source code inside","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-04T07:05:28Z","receivedAt":"2011-08-04T07:05:28Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 01, 2011 at 12:57:58PM +0100, James Gregory wrote:\n\n> I'm not sure if this is the right place to post bug reports - if not,\n> please can you let me know.\n\nThis is the right place.\n\n> I've found (what appears to be) a problem in git-fast-export. I've\n> filed a bug in Launchpad, but I'm not sure if anyone from the Git\n> community actively monitors it so I'm reporting it here to see if it\n> can be fixed.\n\nI don't think anybody monitors Launchpad explicitly; thanks for the\npointer.\n\n> The bug can be viewed online at:\n> https://bugs.launchpad.net/ubuntu/+source/git/+bug/815828\n> and I've done my best to describe the bug in some detail. If you need\n> me to elaborate on the problem further, don't hesitate to let me know.\n\nDefinitely looks like a bug. Is it possible for you to make the\noffending repo available either publicly or to me in private? There's\nnot enough information in your report for me to reproduce the problem\nhere.\n\n-Peff\n"},{"id":"172892","messageId":"CAFC9htxRD0z3O_k_OLp08KZFUOL_T3AryCXD+OqqXx9ZYeRutg@mail.gmail.com","threadId":"27980","inReplyTo":"20110804070528.GA11805@sigill.intra.peff.net","subject":"Re: git-fast-export is returning streams with source code inside","fromName":"James Gregory","fromEmail":"j.gregory@epigenesys.co.uk","sentAt":"2011-08-04T09:08:43Z","receivedAt":"2011-08-04T09:08:43Z","isPatch":false,"sender":{"key":"j.gregory@epigenesys.co.uk","avatar":"https://gravatar.com/avatar/3f482c78bfc90928994cfbf8edb04e4c23722ac07647b580b9bf529a68039ccf?d=mp&s=160"},"body":"Hi Peff,\n\nThanks for your reply. I'm afraid I won't be able to allow you access\nto repo as it is for a business project and covered by NDAs, etc. If I\ncan do anything to help with diagnosing the problem (debug logs, more\nexcerpts from the stream, etc.) then that shouldn't be a problem.\n\nJames\n\nOn 4 August 2011 08:05, Jeff King <peff@peff.net> wrote:\n> On Mon, Aug 01, 2011 at 12:57:58PM +0100, James Gregory wrote:\n>\n>> I'm not sure if this is the right place to post bug reports - if not,\n>> please can you let me know.\n>\n> This is the right place.\n>\n>> I've found (what appears to be) a problem in git-fast-export. I've\n>> filed a bug in Launchpad, but I'm not sure if anyone from the Git\n>> community actively monitors it so I'm reporting it here to see if it\n>> can be fixed.\n>\n> I don't think anybody monitors Launchpad explicitly; thanks for the\n> pointer.\n>\n>> The bug can be viewed online at:\n>> https://bugs.launchpad.net/ubuntu/+source/git/+bug/815828\n>> and I've done my best to describe the bug in some detail. If you need\n>> me to elaborate on the problem further, don't hesitate to let me know.\n>\n> Definitely looks like a bug. Is it possible for you to make the\n> offending repo available either publicly or to me in private? There's\n> not enough information in your report for me to reproduce the problem\n> here.\n>\n> -Peff\n>\n\n\n\n-- \n\nJames Gregory\nSenior Developer\nepiGenesys - a University of Sheffield company\nEnterprise Zone, Portobello, Sheffield, S1 4DP, UK\nT: +44 (0)114 22 21 884 ~ F: +44 (0)114 22 24 042\nwww.epigenesys.co.uk ~ @epigenesys\n--\nThis email and its attachments should be considered confidential and\nare intended solely for the individual to whom the email is addressed.\nIf you are not the intended recipient you should take no action based\nupon them, nor should you copy or show them to anyone. Any views or\nopinions expressed are solely those of the author and might not\nrepresent those of Epi Genesys Limited.\nEpi Genesys Limited is registered in England and Wales with company\nnumber 06388971 and has its registered office at 40 Leavygreave Road,\nSheffield, S3 7RD.\n"},{"id":"172941","messageId":"20110804183218.GA15943@sigill.intra.peff.net","threadId":"27980","inReplyTo":"CAFC9htxRD0z3O_k_OLp08KZFUOL_T3AryCXD+OqqXx9ZYeRutg@mail.gmail.com","subject":"Re: git-fast-export is returning streams with source code inside","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-04T18:32:18Z","receivedAt":"2011-08-04T18:32:18Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 04, 2011 at 10:08:43AM +0100, James Gregory wrote:\n\n> Thanks for your reply. I'm afraid I won't be able to allow you access\n> to repo as it is for a business project and covered by NDAs, etc. If I\n> can do anything to help with diagnosing the problem (debug logs, more\n> excerpts from the stream, etc.) then that shouldn't be a problem.\n\nSince it seems like blob data is making it into the stream in an odd\nplace, my first guess would be a memory problem. Can you try installing\nvalgrind (since you are on Ubuntu, there should be a package), and then\nrunning:\n\n  valgrind git fast-export --all >/dev/null 2>valgrind.out\n\nand then sharing the results of valgrind.out?\n\n-Peff\n"},{"id":"172963","messageId":"CAFC9htyFKm7NCNFvrUkxXpmj1jwatWkxrnRSEdztY4Syo+EQ-g@mail.gmail.com","threadId":"27980","inReplyTo":"20110804183218.GA15943@sigill.intra.peff.net","subject":"Re: git-fast-export is returning streams with source code inside","fromName":"James Gregory","fromEmail":"j.gregory@epigenesys.co.uk","sentAt":"2011-08-05T09:24:07Z","receivedAt":"2011-08-05T09:24:07Z","isPatch":false,"sender":{"key":"j.gregory@epigenesys.co.uk","avatar":"https://gravatar.com/avatar/3f482c78bfc90928994cfbf8edb04e4c23722ac07647b580b9bf529a68039ccf?d=mp&s=160"},"body":"Hi Peff,\n\nI've attached the valgrind.out file. If I'm reading the output\nproperly, it does look like it is suffering from a memory leak.\n\nJames\n\nOn 4 August 2011 19:32, Jeff King <peff@peff.net> wrote:\n> On Thu, Aug 04, 2011 at 10:08:43AM +0100, James Gregory wrote:\n>\n>> Thanks for your reply. I'm afraid I won't be able to allow you access\n>> to repo as it is for a business project and covered by NDAs, etc. If I\n>> can do anything to help with diagnosing the problem (debug logs, more\n>> excerpts from the stream, etc.) then that shouldn't be a problem.\n>\n> Since it seems like blob data is making it into the stream in an odd\n> place, my first guess would be a memory problem. Can you try installing\n> valgrind (since you are on Ubuntu, there should be a package), and then\n> running:\n>\n>  valgrind git fast-export --all >/dev/null 2>valgrind.out\n>\n> and then sharing the results of valgrind.out?\n>\n> -Peff\n>\n\n\n\n-- \n\nJames Gregory\nSenior Developer\nepiGenesys - a University of Sheffield company\nEnterprise Zone, Portobello, Sheffield, S1 4DP, UK\nT: +44 (0)114 22 21884 ~ F: +44 (0)114 22 24042\nwww.epigenesys.co.uk ~ @epigenesys\n--\nThis email and its attachments should be considered confidential and\nare intended solely for the individual to whom the email is addressed.\nIf you are not the intended recipient you should take no action based\nupon them, nor should you copy or show them to anyone. Any views or\nopinions expressed are solely those of the author and might not\nrepresent those of Epi Genesys Limited.\nEpi Genesys Limited is registered in England and Wales with company\nnumber 06388971 and has its registered office at 40 Leavygreave Road,\nSheffield, S3 7RD.\n"},{"id":"172964","messageId":"20110805093618.GA19062@sigill.intra.peff.net","threadId":"27980","inReplyTo":"CAFC9htyFKm7NCNFvrUkxXpmj1jwatWkxrnRSEdztY4Syo+EQ-g@mail.gmail.com","subject":"Re: git-fast-export is returning streams with source code inside","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-05T09:36:19Z","receivedAt":"2011-08-05T09:36:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 05, 2011 at 10:24:07AM +0100, James Gregory wrote:\n\n> I've attached the valgrind.out file. If I'm reading the output\n> properly, it does look like it is suffering from a memory leak.\n\nIt looks pretty innocuous:\n\n> ==3267== Conditional jump or move depends on uninitialised value(s)\n> ==3267==    at 0x4E39510: inflateReset2 (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)\n> ==3267==    by 0x4E39605: inflateInit2_ (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)\n> ==3267==    by 0x4D5B02: ??? (in /usr/bin/git)\n> ==3267==    by 0x4BB059: ??? (in /usr/bin/git)\n> ==3267==    by 0x4BC464: ??? (in /usr/bin/git)\n> ==3267==    by 0x4BCA29: ??? (in /usr/bin/git)\n> ==3267==    by 0x4BCAFD: ??? (in /usr/bin/git)\n> ==3267==    by 0x4BD161: ??? (in /usr/bin/git)\n> ==3267==    by 0x49AEAC: ??? (in /usr/bin/git)\n> ==3267==    by 0x4B38AA: ??? (in /usr/bin/git)\n> ==3267==    by 0x4B422A: ??? (in /usr/bin/git)\n> ==3267==    by 0x4A9CD2: ??? (in /usr/bin/git)\n\nThis is a well-known false positive caused by zlib, and is nothing to\nworry about.\n\n> ==3267== LEAK SUMMARY:\n> ==3267==    definitely lost: 48,016 bytes in 3 blocks\n> ==3267==    indirectly lost: 30,226 bytes in 1,889 blocks\n\nThese are actual leaks, but minor.\n\n> ==3267==      possibly lost: 0 bytes in 0 blocks\n> ==3267==    still reachable: 2,743,134 bytes in 4,334 blocks\n\nThese are not really leaks, but rather things we don't bother cleaning\nup since we're about to exit and let the OS reclaim memory (e.g., all of\nthe commit objects).\n\nSo yes, there's leaking, but it's not much. And more importantly, I was\nlooking not for leaks, but for memory access errors (of which there are\nnone, except for the zlib false positive).\n\nSo I'm not sure where to go from here. I can't reproduce the problem\nlocally. Is there anything else you can tell us about the problem? Does\nit always happen on the same commit? If you export just that commit,\ndoes the problem happen? Is there anything noteworthy in the contents of\nthat commit?\n\n-Peff\n"},{"id":"172965","messageId":"CAFC9htzzPQWFLGCpAP2WHhDQajfknwz1KUu6K0gvyhCVX5gyaQ@mail.gmail.com","threadId":"27980","inReplyTo":"20110805093618.GA19062@sigill.intra.peff.net","subject":"Re: git-fast-export is returning streams with source code inside","fromName":"James Gregory","fromEmail":"j.gregory@epigenesys.co.uk","sentAt":"2011-08-05T09:54:29Z","receivedAt":"2011-08-05T09:54:29Z","isPatch":false,"sender":{"key":"j.gregory@epigenesys.co.uk","avatar":"https://gravatar.com/avatar/3f482c78bfc90928994cfbf8edb04e4c23722ac07647b580b9bf529a68039ccf?d=mp&s=160"},"body":"Hi Peff,\n\nThanks for the feedback. I've just looked at the commit on gitweb, and\nthis could be the problem!\n\n---\nfixed dodgy filename\nspec/blueprints/sjt_blueprint.rb\t[moved from\nspec/blueprints/sjt_blueprint.rb\\n lead_in\\n scenario\\n answers {\nSham.answers_object }\\n justification_selected\\n\njustification_unselected\\n mark_scheme {\nSham.single_mark_scheme_object }\\nend\\nmcq_blueprint.rb with 100%\nsimilarity]\n---\n\nI'm guessing that is where the problem lies... somehow the git\ntransaction has got corrupt(?)\n\nJames\n\nOn 5 August 2011 10:36, Jeff King <peff@peff.net> wrote:\n> On Fri, Aug 05, 2011 at 10:24:07AM +0100, James Gregory wrote:\n>\n>> I've attached the valgrind.out file. If I'm reading the output\n>> properly, it does look like it is suffering from a memory leak.\n>\n> It looks pretty innocuous:\n>\n>> ==3267== Conditional jump or move depends on uninitialised value(s)\n>> ==3267==    at 0x4E39510: inflateReset2 (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)\n>> ==3267==    by 0x4E39605: inflateInit2_ (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)\n>> ==3267==    by 0x4D5B02: ??? (in /usr/bin/git)\n>> ==3267==    by 0x4BB059: ??? (in /usr/bin/git)\n>> ==3267==    by 0x4BC464: ??? (in /usr/bin/git)\n>> ==3267==    by 0x4BCA29: ??? (in /usr/bin/git)\n>> ==3267==    by 0x4BCAFD: ??? (in /usr/bin/git)\n>> ==3267==    by 0x4BD161: ??? (in /usr/bin/git)\n>> ==3267==    by 0x49AEAC: ??? (in /usr/bin/git)\n>> ==3267==    by 0x4B38AA: ??? (in /usr/bin/git)\n>> ==3267==    by 0x4B422A: ??? (in /usr/bin/git)\n>> ==3267==    by 0x4A9CD2: ??? (in /usr/bin/git)\n>\n> This is a well-known false positive caused by zlib, and is nothing to\n> worry about.\n>\n>> ==3267== LEAK SUMMARY:\n>> ==3267==    definitely lost: 48,016 bytes in 3 blocks\n>> ==3267==    indirectly lost: 30,226 bytes in 1,889 blocks\n>\n> These are actual leaks, but minor.\n>\n>> ==3267==      possibly lost: 0 bytes in 0 blocks\n>> ==3267==    still reachable: 2,743,134 bytes in 4,334 blocks\n>\n> These are not really leaks, but rather things we don't bother cleaning\n> up since we're about to exit and let the OS reclaim memory (e.g., all of\n> the commit objects).\n>\n> So yes, there's leaking, but it's not much. And more importantly, I was\n> looking not for leaks, but for memory access errors (of which there are\n> none, except for the zlib false positive).\n>\n> So I'm not sure where to go from here. I can't reproduce the problem\n> locally. Is there anything else you can tell us about the problem? Does\n> it always happen on the same commit? If you export just that commit,\n> does the problem happen? Is there anything noteworthy in the contents of\n> that commit?\n>\n> -Peff\n>\n\n\n\n-- \n\nJames Gregory\nSenior Developer\nepiGenesys - a University of Sheffield company\nEnterprise Zone, Portobello, Sheffield, S1 4DP, UK\nT: +44 (0)114 22 21884 ~ F: +44 (0)114 22 24042\nwww.epigenesys.co.uk ~ @epigenesys\n--\nThis email and its attachments should be considered confidential and\nare intended solely for the individual to whom the email is addressed.\nIf you are not the intended recipient you should take no action based\nupon them, nor should you copy or show them to anyone. Any views or\nopinions expressed are solely those of the author and might not\nrepresent those of Epi Genesys Limited.\nEpi Genesys Limited is registered in England and Wales with company\nnumber 06388971 and has its registered office at 40 Leavygreave Road,\nSheffield, S3 7RD.\n"},{"id":"172966","messageId":"20110805103016.GA19648@sigill.intra.peff.net","threadId":"27980","inReplyTo":"CAFC9htzzPQWFLGCpAP2WHhDQajfknwz1KUu6K0gvyhCVX5gyaQ@mail.gmail.com","subject":"Re: git-fast-export is returning streams with source code inside","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-05T10:30:16Z","receivedAt":"2011-08-05T10:30:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 05, 2011 at 10:54:29AM +0100, James Gregory wrote:\n\n> Thanks for the feedback. I've just looked at the commit on gitweb, and\n> this could be the problem!\n> \n> ---\n> fixed dodgy filename\n> spec/blueprints/sjt_blueprint.rb\t[moved from\n> spec/blueprints/sjt_blueprint.rb\\n lead_in\\n scenario\\n answers {\n> Sham.answers_object }\\n justification_selected\\n\n> justification_unselected\\n mark_scheme {\n> Sham.single_mark_scheme_object }\\nend\\nmcq_blueprint.rb with 100%\n> similarity]\n> ---\n> \n> I'm guessing that is where the problem lies... somehow the git\n> transaction has got corrupt(?)\n\nAh, OK. That makes sense. I can replicate your problem easily with:\n\n  $ touch 'file with\n    newline'\n  $ git init\n  $ git add .\n  $ git commit -m foo\n  $ git fast-export HEAD | git fast-import\n  fatal: Unsupported command: newline\n\nAccording to the fast-import manpage, fast-export should be quoting the\nembedded line-feed. It looks like it isn't doing any quoting at all of\npathnames right now, which is just wrong.\n\nDoes the patch below fix your issue?\n\n---\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 9247871..bd27f08 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -16,6 +16,7 @@\n #include \"string-list.h\"\n #include \"utf8.h\"\n #include \"parse-options.h\"\n+#include \"quote.h\"\n \n static const char *fast_export_usage[] = {\n \t\"git fast-export [rev-list-opts]\",\n@@ -167,6 +168,15 @@ static int depth_first(const void *a_, const void *b_)\n \treturn (a->status == 'R') - (b->status == 'R');\n }\n \n+static void print_path(const char *path)\n+{\n+\tint need_quote = quote_c_style(path, NULL, NULL, 0);\n+\tif (need_quote)\n+\t\tquote_c_style(path, NULL, stdout, 0);\n+\telse\n+\t\tprintf(\"%s\", path);\n+}\n+\n static void show_filemodify(struct diff_queue_struct *q,\n \t\t\t    struct diff_options *options, void *data)\n {\n@@ -184,13 +194,18 @@ static void show_filemodify(struct diff_queue_struct *q,\n \n \t\tswitch (q->queue[i]->status) {\n \t\tcase DIFF_STATUS_DELETED:\n-\t\t\tprintf(\"D %s\\n\", spec->path);\n+\t\t\tprintf(\"D \");\n+\t\t\tprint_path(spec->path);\n+\t\t\tputchar('\\n');\n \t\t\tbreak;\n \n \t\tcase DIFF_STATUS_COPIED:\n \t\tcase DIFF_STATUS_RENAMED:\n-\t\t\tprintf(\"%c \\\"%s\\\" \\\"%s\\\"\\n\", q->queue[i]->status,\n-\t\t\t       ospec->path, spec->path);\n+\t\t\tprintf(\"%c \", q->queue[i]->status);\n+\t\t\tprint_path(ospec->path);\n+\t\t\tputchar(' ');\n+\t\t\tprint_path(spec->path);\n+\t\t\tputchar('\\n');\n \n \t\t\tif (!hashcmp(ospec->sha1, spec->sha1) &&\n \t\t\t    ospec->mode == spec->mode)\n@@ -205,13 +220,15 @@ static void show_filemodify(struct diff_queue_struct *q,\n \t\t\t * output the SHA-1 verbatim.\n \t\t\t */\n \t\t\tif (no_data || S_ISGITLINK(spec->mode))\n-\t\t\t\tprintf(\"M %06o %s %s\\n\", spec->mode,\n-\t\t\t\t       sha1_to_hex(spec->sha1), spec->path);\n+\t\t\t\tprintf(\"M %06o %s \", spec->mode,\n+\t\t\t\t       sha1_to_hex(spec->sha1));\n \t\t\telse {\n \t\t\t\tstruct object *object = lookup_object(spec->sha1);\n-\t\t\t\tprintf(\"M %06o :%d %s\\n\", spec->mode,\n-\t\t\t\t       get_object_mark(object), spec->path);\n+\t\t\t\tprintf(\"M %06o :%d \", spec->mode,\n+\t\t\t\t       get_object_mark(object));\n \t\t\t}\n+\t\t\tprint_path(spec->path);\n+\t\t\tputchar('\\n');\n \t\t\tbreak;\n \n \t\tdefault:\n"}]}