{"thread":{"id":"20227","subject":"[PATCH] git fast-export: add --no-data option","startedAt":"2009-07-25T13:45:19Z","lastAt":"2009-07-28T08:01:08Z","messageCount":9,"participants":["Geoffrey Irving","Johannes Schindelin","Junio C Hamano","Stephen Boyd"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"118716","messageId":"7f9d599f0907250645s6e6f9b81w3cf20f07eff088eb@mail.gmail.com","threadId":"20227","inReplyTo":null,"subject":"[PATCH] git fast-export: add --no-data option","fromName":"Geoffrey Irving","fromEmail":"irving@naml.us","sentAt":"2009-07-25T13:45:19Z","receivedAt":"2009-07-25T13:45:19Z","isPatch":true,"sender":{"key":"irving@naml.us","avatar":"https://gravatar.com/avatar/52d7452fcd134aac0fa12f57a3bb7ef5f3f7e73ca0ab36736d06c6a6132de718?d=mp&s=160"},"body":"When using git fast-export and git fast-import to rewrite the history\nof a repository with large binary files, almost all of the time is\nspent dealing with blobs. This is extremely inefficient if all we want\nto do is rewrite the commits and tree structure. --no-data skips the\noutput of blobs and writes SHA-1s instead of marks, which provides a\nmassive speedup.\n\nSigned-off-by: Geoffrey Irving \n---\n\nI've already done all I need with this change (for now, at least), but\nhere it is in case it proves useful to others. Amusingly, rewriting\nhistory with\n\n git fast-export --no-data | | git fast-import\n\nis now much, much faster than the equivalent\n\n git filter-branch --prune-empty --msg-filter ...\n\nI haven't investigated why.\n\n Documentation/git-fast-export.txt | 7 +++++++\n builtin-fast-export.c | 8 +++++++-\n 2 files changed, 14 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt\nb/Documentation/git-fast-export.txt\nindex 0c9eb56..47a96dd 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -71,6 +71,13 @@ marks the same across runs.\n \tallow that. So fake a tagger to be able to fast-import the\n \toutput.\n\n+--no-data::\n+\tSkip output of blob objects and instead refer to blobs via\n+\ttheir original SHA-1 hash. This is useful when rewriting the\n+\tdirectory structure or history of a repository without\n+\ttouching the contents of individual files. Note that the\n+\tresulting stream can only be used by a repository which\n+\talready contains the necessary objects.\n\n EXAMPLES\n --------\ndiff --git a/builtin-fast-export.c b/builtin-fast-export.c\nindex 9a8a6fc..ac72791 100644\n--- a/builtin-fast-export.c\n+++ b/builtin-fast-export.c\n@@ -25,6 +25,7 @@ static const char *fast_export_usage[] = {\n static int progress;\n static enum { VERBATIM, WARN, STRIP, ABORT } signed_tag_mode = ABORT;\n static int fake_missing_tagger;\n+static int no_data;\n\n static int parse_opt_signed_tag_mode(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -101,6 +102,9 @@ static void handle_object(const unsigned char *sha1)\n \tchar *buf;\n \tstruct object *object;\n\n+\tif (no_data)\n+\t\treturn;\n+\n \tif (is_null_sha1(sha1))\n \t\treturn;\n\n@@ -158,7 +162,7 @@ static void show_filemodify(struct diff_queue_struct *q,\n \t\t\t * Links refer to objects in another repositories;\n \t\t\t * output the SHA-1 verbatim.\n \t\t\t */\n-\t\t\tif (S_ISGITLINK(spec->mode))\n+\t\t\tif (no_data || S_ISGITLINK(spec->mode))\n \t\t\t\tprintf(\"M %06o %s %s\\n\", spec->mode,\n \t\t\t\t sha1_to_hex(spec->sha1), spec->path);\n \t\t\telse {\n@@ -504,6 +508,8 @@ int cmd_fast_export(int argc, const char **argv,\nconst char *prefix)\n \t\t\t \"Import marks from this file\"),\n \t\tOPT_BOOLEAN(0, \"fake-missing-tagger\", &fake_missing_tagger,\n \t\t\t \"Fake a tagger when tags lack one\"),\n+\t\tOPT_BOOLEAN(0, \"no-data\", &no_data,\n+\t\t\t \"Skip output of blob data\"),\n \t\tOPT_END()\n \t};\n\n-- \n1.6.3.1\n"},{"id":"118719","messageId":"alpine.DEB.1.00.0907251628220.8306@pacific.mpi-cbg.de","threadId":"20227","inReplyTo":"7f9d599f0907250645s6e6f9b81w3cf20f07eff088eb@mail.gmail.com","subject":"Re: [PATCH] git fast-export: add --no-data option","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-25T14:28:48Z","receivedAt":"2009-07-25T14:28:48Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 25 Jul 2009, Geoffrey Irving wrote:\n\n> When using git fast-export and git fast-import to rewrite the history\n> of a repository with large binary files, almost all of the time is\n> spent dealing with blobs. This is extremely inefficient if all we want\n> to do is rewrite the commits and tree structure. --no-data skips the\n> output of blobs and writes SHA-1s instead of marks, which provides a\n> massive speedup.\n\nACK. I was looking for such an option recently, and was disappointed that \nit was not there yet.\n\nCiao,\nDscho\n"},{"id":"118733","messageId":"7vfxcku13i.fsf@alter.siamese.dyndns.org","threadId":"20227","inReplyTo":"7f9d599f0907250645s6e6f9b81w3cf20f07eff088eb@mail.gmail.com","subject":"Re: [PATCH] git fast-export: add --no-data option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-25T17:25:05Z","receivedAt":"2009-07-25T17:25:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Geoffrey Irving writes:\n\n> @@ -504,6 +508,8 @@ int cmd_fast_export(int argc, const char **argv,\n> const char *prefix)\n> \t\t\t \"Import marks from this file\"),\n> \t\tOPT_BOOLEAN(0, \"fake-missing-tagger\", &fake_missing_tagger,\n> \t\t\t \"Fake a tagger when tags lack one\"),\n> +\t\tOPT_BOOLEAN(0, \"no-data\", &no_data,\n> +\t\t\t \"Skip output of blob data\"),\n\nShouldn't this be --[no-]data option that defaults to true? Otherwise you\nwould accept --no-no-data that looks silly.\n\n> \t\tOPT_END()\n> \t};\n>\n> -- \n> 1.6.3.1\n"},{"id":"118737","messageId":"alpine.DEB.1.00.0907251942390.8306@pacific.mpi-cbg.de","threadId":"20227","inReplyTo":"7vfxcku13i.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] git fast-export: add --no-data option","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-25T17:44:05Z","receivedAt":"2009-07-25T17:44:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 25 Jul 2009, Junio C Hamano wrote:\n\n> Geoffrey Irving writes:\n> \n> > @@ -504,6 +508,8 @@ int cmd_fast_export(int argc, const char **argv,\n> > const char *prefix)\n> > \t\t\t \"Import marks from this file\"),\n> > \t\tOPT_BOOLEAN(0, \"fake-missing-tagger\", &fake_missing_tagger,\n> > \t\t\t \"Fake a tagger when tags lack one\"),\n> > +\t\tOPT_BOOLEAN(0, \"no-data\", &no_data,\n> > +\t\t\t \"Skip output of blob data\"),\n> \n> Shouldn't this be --[no-]data option that defaults to true? Otherwise you\n> would accept --no-no-data that looks silly.\n\nMaybe\n\n\t\tOPT_NEGBIT(0, \"data\", &no_data,\n\t\t\t \"Skip output of blob data\", 1),\n\nHmm?\n\nCiao,\nDscho\n\t\t\n"},{"id":"118875","messageId":"7f9d599f0907270548k15c51a01j1fedba20c66b66f6@mail.gmail.com","threadId":"20227","inReplyTo":"alpine.DEB.1.00.0907251942390.8306@pacific.mpi-cbg.de","subject":"Re: [PATCH] git fast-export: add --no-data option","fromName":"Geoffrey Irving","fromEmail":"irving@naml.us","sentAt":"2009-07-27T12:48:34Z","receivedAt":"2009-07-27T12:48:34Z","isPatch":true,"sender":{"key":"irving@naml.us","avatar":"https://gravatar.com/avatar/52d7452fcd134aac0fa12f57a3bb7ef5f3f7e73ca0ab36736d06c6a6132de718?d=mp&s=160"},"body":"On Sat, Jul 25, 2009 at 1:44 PM, Johannes\nSchindelin wrote:\n> Hi,\n>\n> On Sat, 25 Jul 2009, Junio C Hamano wrote:\n>\n>> Geoffrey Irving writes:\n>>\n>> > @@ -504,6 +508,8 @@ int cmd_fast_export(int argc, const char **argv,\n>> > const char *prefix)\n>> >                          \"Import marks from this file\"),\n>> >             OPT_BOOLEAN(0, \"fake-missing-tagger\", &fake_missing_tagger,\n>> >                          \"Fake a tagger when tags lack one\"),\n>> > +           OPT_BOOLEAN(0, \"no-data\", &no_data,\n>> > +                        \"Skip output of blob data\"),\n>>\n>> Shouldn't this be --[no-]data option that defaults to true?  Otherwise you\n>> would accept --no-no-data that looks silly.\n>\n> Maybe\n>\n>                OPT_NEGBIT(0, \"data\", &no_data,\n>                             \"Skip output of blob data\", 1),\n>\n> Hmm?\n\nNot quite. That produces\n\nusage: git fast-export [rev-list-opts]\n\n --progress show progress after objects\n --signed-tags select handling of signed tags\n --export-marks \n Dump marks to this file\n --import-marks \n Import marks from this file\n --fake-missing-tagger\n Fake a tagger when tags lack one\n --data Skip output of blob data\n\nI don't see similar uses of OPT_NEGBIT, so maybe the necessary option\ncase hasn't been written yet (or I'm missing something obvious)?\n\nGeoffrey\n"},{"id":"118893","messageId":"alpine.DEB.1.00.0907272042450.6883@intel-tinevez-2-302","threadId":"20227","inReplyTo":"7f9d599f0907270548k15c51a01j1fedba20c66b66f6@mail.gmail.com","subject":"Re: [PATCH] git fast-export: add --no-data option","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-27T18:49:56Z","receivedAt":"2009-07-27T18:49:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 27 Jul 2009, Geoffrey Irving wrote:\n\n> On Sat, Jul 25, 2009 at 1:44 PM, Johannes\n> Schindelin wrote:\n>\n> > On Sat, 25 Jul 2009, Junio C Hamano wrote:\n> >\n> >> Geoffrey Irving writes:\n> >>\n> >> > @@ -504,6 +508,8 @@ int cmd_fast_export(int argc, const char **argv,\n> >> > const char *prefix)\n> >> >                          \"Import marks from this file\"),\n> >> >             OPT_BOOLEAN(0, \"fake-missing-tagger\", &fake_missing_tagger,\n> >> >                          \"Fake a tagger when tags lack one\"),\n> >> > +           OPT_BOOLEAN(0, \"no-data\", &no_data,\n> >> > +                        \"Skip output of blob data\"),\n> >>\n> >> Shouldn't this be --[no-]data option that defaults to true? \n> >>  Otherwise you would accept --no-no-data that looks silly.\n> >\n> > Maybe\n> >\n> >                OPT_NEGBIT(0, \"data\", &no_data,\n> >                             \"Skip output of blob data\", 1),\n> >\n> > Hmm?\n> \n> Not quite. That produces\n> \n> usage: git fast-export [rev-list-opts]\n> \n> --progress show progress after objects\n> --signed-tags select handling of signed tags\n> --export-marks \n> Dump marks to this file\n> --import-marks \n> Import marks from this file\n> --fake-missing-tagger\n> Fake a tagger when tags lack one\n> --data Skip output of blob data\n> \n> I don't see similar uses of OPT_NEGBIT, so maybe the necessary option\n> case hasn't been written yet (or I'm missing something obvious)?\n\nThere is an ugly solution:\n\n\t\t{ OPTION_NEGBIT, 0, \"no-data\", &no_data, NULL, NULL,\n\t\t\tPARSE_OPT_NOARG | PARSE_OPT_HIDDEN, NULL, 0 },\n\t\t{ OPTION_BIT, 0, \"no-data\", NULL, NULL,\n\t\t\t\"Skip output of blob data\",\n\t\t\tPARSE_OPT_NOARG, NULL, 1 },\n\nand there is a more elegant solution:\n\n[PATCH] parse-opt: optionally show \"--no-\" option string\n\nIt is usually better to have positive options, to avoid confusing double\nnegations. However, sometimes it is desirable to show the negative option\nin the help.\n\nIntroduce the flag PARSE_OPT_NEGHELP to do that.\n\nSigned-off-by: Johannes Schindelin \n\n---\n\n parse-options.c | 6 ++++--\n parse-options.h | 1 +\n 2 files changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/parse-options.c b/parse-options.c\nindex 68accef..a64a4d6 100644\n--- a/parse-options.c\n+++ b/parse-options.c\n@@ -511,7 +511,7 @@ static int usage_with_options_internal(const char * const *usagestr,\n \t\t\tcontinue;\n \n \t\tpos = fprintf(stderr, \" \");\n-\t\tif (opts->short_name) {\n+\t\tif (opts->short_name && !(opts->flags & PARSE_OPT_NEGHELP)) {\n \t\t\tif (opts->flags & PARSE_OPT_NODASH)\n \t\t\t\tpos += fprintf(stderr, \"%c\", opts->short_name);\n \t\t\telse\n@@ -520,7 +520,9 @@ static int usage_with_options_internal(const char * const *usagestr,\n \t\tif (opts->long_name && opts->short_name)\n \t\t\tpos += fprintf(stderr, \", \");\n \t\tif (opts->long_name)\n-\t\t\tpos += fprintf(stderr, \"--%s\", opts->long_name);\n+\t\t\tpos += fprintf(stderr, \"--%s%s\",\n+\t\t\t\t(opts->flags & PARSE_OPT_NEGHELP) ? \"no-\" : \"\",\n+\t\t\t\topts->long_name);\n \t\tif (opts->type == OPTION_NUMBER)\n \t\t\tpos += fprintf(stderr, \"-NUM\");\n \ndiff --git a/parse-options.h b/parse-options.h\nindex 4b62361..8f0035a 100644\n--- a/parse-options.h\n+++ b/parse-options.h\n@@ -36,6 +36,7 @@ enum parse_opt_option_flags {\n \tPARSE_OPT_LASTARG_DEFAULT = 16,\n \tPARSE_OPT_NODASH = 32,\n \tPARSE_OPT_LITERAL_ARGHELP = 64,\n+\tPARSE_OPT_NEGHELP = 128,\n };\n \n struct option;"},{"id":"118917","messageId":"7f9d599f0907271920y58709842v1316758aa2a23e20@mail.gmail.com","threadId":"20227","inReplyTo":"alpine.DEB.1.00.0907272042450.6883@intel-tinevez-2-302","subject":"[PATCH] git fast-export: add --no-data option","fromName":"Geoffrey Irving","fromEmail":"irving@naml.us","sentAt":"2009-07-28T02:20:22Z","receivedAt":"2009-07-28T02:20:22Z","isPatch":true,"sender":{"key":"irving@naml.us","avatar":"https://gravatar.com/avatar/52d7452fcd134aac0fa12f57a3bb7ef5f3f7e73ca0ab36736d06c6a6132de718?d=mp&s=160"},"body":"When using git fast-export and git fast-import to rewrite the history\nof a repository with large binary files, almost all of the time is\nspent dealing with blobs. This is extremely inefficient if all we want\nto do is rewrite the commits and tree structure. --no-data skips the\noutput of blobs and writes SHA-1s instead of marks, which provides a\nmassive speedup.\n\nSigned-off-by: Geoffrey Irving \n---\n\nHere's my modified patch on top of Johannes' fix to parse-options. On github:\n\n http://github.com/girving/git/commit/98549f6809a4dc22d088f3c2ee1f798e858cce3e\n http://github.com/girving/git/commit/00a7c591b9a1fc6880ad5f88d118bb1d6ea86878\n\n Documentation/git-fast-export.txt | 7 +++++++\n builtin-fast-export.c | 9 ++++++++-\n 2 files changed, 15 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt\nb/Documentation/git-fast-export.txt\nindex 0c9eb56..47a96dd 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -71,6 +71,13 @@ marks the same across runs.\n \tallow that. So fake a tagger to be able to fast-import the\n \toutput.\n\n+--no-data::\n+\tSkip output of blob objects and instead refer to blobs via\n+\ttheir original SHA-1 hash. This is useful when rewriting the\n+\tdirectory structure or history of a repository without\n+\ttouching the contents of individual files. Note that the\n+\tresulting stream can only be used by a repository which\n+\talready contains the necessary objects.\n\n EXAMPLES\n --------\ndiff --git a/builtin-fast-export.c b/builtin-fast-export.c\nindex 9a8a6fc..a0f0284 100644\n--- a/builtin-fast-export.c\n+++ b/builtin-fast-export.c\n@@ -25,6 +25,7 @@ static const char *fast_export_usage[] = {\n static int progress;\n static enum { VERBATIM, WARN, STRIP, ABORT } signed_tag_mode = ABORT;\n static int fake_missing_tagger;\n+static int no_data;\n\n static int parse_opt_signed_tag_mode(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -101,6 +102,9 @@ static void handle_object(const unsigned char *sha1)\n \tchar *buf;\n \tstruct object *object;\n\n+\tif (no_data)\n+\t\treturn;\n+\n \tif (is_null_sha1(sha1))\n \t\treturn;\n\n@@ -158,7 +162,7 @@ static void show_filemodify(struct diff_queue_struct *q,\n \t\t\t * Links refer to objects in another repositories;\n \t\t\t * output the SHA-1 verbatim.\n \t\t\t */\n-\t\t\tif (S_ISGITLINK(spec->mode))\n+\t\t\tif (no_data || S_ISGITLINK(spec->mode))\n \t\t\t\tprintf(\"M %06o %s %s\\n\", spec->mode,\n \t\t\t\t sha1_to_hex(spec->sha1), spec->path);\n \t\t\telse {\n@@ -504,6 +508,9 @@ int cmd_fast_export(int argc, const char **argv,\nconst char *prefix)\n \t\t\t \"Import marks from this file\"),\n \t\tOPT_BOOLEAN(0, \"fake-missing-tagger\", &fake_missing_tagger,\n \t\t\t \"Fake a tagger when tags lack one\"),\n+\t\t{ OPTION_NEGBIT, 0, \"data\", &no_data, NULL,\n+\t\t\t\"Skip output of blob data\",\n+\t\t\tPARSE_OPT_NOARG | PARSE_OPT_NEGHELP, NULL, 1 },\n \t\tOPT_END()\n \t};\n\n-- \n1.6.3.1\n"},{"id":"118920","messageId":"4A6E7A80.9040809@gmail.com","threadId":"20227","inReplyTo":"alpine.DEB.1.00.0907272042450.6883@intel-tinevez-2-302","subject":"Re: [PATCH] git fast-export: add --no-data option","fromName":"Stephen Boyd","fromEmail":"bebarino@gmail.com","sentAt":"2009-07-28T04:11:44Z","receivedAt":"2009-07-28T04:11:44Z","isPatch":true,"sender":{"key":"bebarino@gmail.com","avatar":"https://avatars.githubusercontent.com/u/38832?v=4"},"body":"Johannes Schindelin wrote:\n> There is an ugly solution:\n>\n> \t\t{ OPTION_NEGBIT, 0, \"no-data\", &no_data, NULL, NULL,\n> \t\t\tPARSE_OPT_NOARG | PARSE_OPT_HIDDEN, NULL, 0 },\n> \t\t{ OPTION_BIT, 0, \"no-data\", NULL, NULL,\n> \t\t\t\"Skip output of blob data\",\n> \t\t\tPARSE_OPT_NOARG, NULL, 1 },\n>\n> and there is a more elegant solution:\n>\n> [PATCH] parse-opt: optionally show \"--no-\" option string\n>\n> It is usually better to have positive options, to avoid confusing double\n> negations. However, sometimes it is desirable to show the negative option\n> in the help.\n>\n> Introduce the flag PARSE_OPT_NEGHELP to do that.\n\nPerhaps with this documentation throw in?\n\ndiff --git a/parse-options.h b/parse-options.h\nindex 90e577d..14162e9 100644\n--- a/parse-options.h\n+++ b/parse-options.h\n@@ -81,6 +81,9 @@ typedef int parse_opt_cb(const struct option *, const char *arg, int unset);\n * PARSE_OPT_LITERAL_ARGHELP: says that argh shouldn't be enclosed in brackets\n * (i.e. '') in the help message.\n * Useful for options with multiple parameters.\n+ * PARSE_OPT_NEGHELP: says that the long option should always be shown with\n+ * the --no prefix in the usage message. Sometimes\n+ * useful for users of OPTION_NEGBIT.\n *\n * `callback`::\n * pointer to the callback to use for OPTION_CALLBACK.\n"},{"id":"118928","messageId":"alpine.DEB.1.00.0907281000360.8306@pacific.mpi-cbg.de","threadId":"20227","inReplyTo":"4A6E7A80.9040809@gmail.com","subject":"Re: [PATCH] git fast-export: add --no-data option","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-28T08:01:08Z","receivedAt":"2009-07-28T08:01:08Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 27 Jul 2009, Stephen Boyd wrote:\n\n> Johannes Schindelin wrote:\n> >\n> > [PATCH] parse-opt: optionally show \"--no-\" option string\n> >\n> > It is usually better to have positive options, to avoid confusing \n> > double negations. However, sometimes it is desirable to show the \n> > negative option in the help.\n> >\n> > Introduce the flag PARSE_OPT_NEGHELP to do that.\n> \n> Perhaps with this documentation throw in?\n> \n> diff --git a/parse-options.h b/parse-options.h\n> index 90e577d..14162e9 100644\n> --- a/parse-options.h\n> +++ b/parse-options.h\n> @@ -81,6 +81,9 @@ typedef int parse_opt_cb(const struct option *, const char *arg, int unset);\n> * PARSE_OPT_LITERAL_ARGHELP: says that argh shouldn't be enclosed in brackets\n> * (i.e. '') in the help message.\n> * Useful for options with multiple parameters.\n> + * PARSE_OPT_NEGHELP: says that the long option should always be shown with\n> + * the --no prefix in the usage message. Sometimes\n> + * useful for users of OPTION_NEGBIT.\n> *\n> * `callback`::\n> * pointer to the callback to use for OPTION_CALLBACK.\n> \n\nThanks.\n\nCiao,\nDscho\n"}]}