{"thread":{"id":"25805","subject":"[PATCH/RFC] fast-import: treat SIGUSR1 as a request to access objects early","startedAt":"2010-11-22T08:16:02Z","lastAt":"2010-11-24T20:53:19Z","messageCount":3,"participants":["Jonathan Nieder","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"156305","messageId":"20101122081601.GA9722@burratino","threadId":"25805","inReplyTo":null,"subject":"[PATCH/RFC] fast-import: treat SIGUSR1 as a request to access objects early","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-22T08:16:02Z","receivedAt":"2010-11-22T08:16:02Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"It can be tedious to wait for a multi-million-revision import.\nUnfortunately it is hard to spy on the import because fast-import\nworks by continuously streaming out objects, without updating the pack\nindex or refs until a checkpoint command or the end of the stream.\n\nSo allow the impatient operator to request checkpoints by sending a\nsignal, like so:\n\n\tkillall -USR1 git-fast-import\n\nWhen receiving such a signal, fast-import would schedule a checkpoint\nto take place after the current top-level command (usually a \"commit\"\nor \"blob\" request) finishes.\n\nCaveats: just like ordinary checkpoint commands, such requests slow\ndown the import.  Switching to a new pack at a suboptimal moment is\nalso likely to result in a less dense initial collection of packs.\nThat's the price.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nThoughts?\n\n Documentation/git-fast-import.txt |    8 +++++++\n fast-import.c                     |   42 ++++++++++++++++++++++++++++++++++++-\n 2 files changed, 49 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 5d0c245..526670a 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -1233,6 +1233,14 @@ and lazy loading of subtrees, allows fast-import to efficiently import\n projects with 2,000+ branches and 45,114+ files in a very limited\n memory footprint (less than 2.7 MiB per active branch).\n \n+Signals\n+-------\n+Sending *SIGUSR1* to the 'git fast-import' process ends the current\n+packfile early, simulating a `checkpoint` command.  The impatient\n+operator can use this facility to peek at the objects and refs from an\n+import in progress, at the cost of some added running time and worse\n+compression.\n+\n \n Author\n ------\ndiff --git a/fast-import.c b/fast-import.c\nindex 77549eb..4779e3a 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -361,6 +361,9 @@ static uintmax_t next_mark;\n static struct strbuf new_data = STRBUF_INIT;\n static int seen_data_command;\n \n+/* Signal handling */\n+static volatile sig_atomic_t checkpoint_requested;\n+\n static void parse_argv(void);\n \n static void write_branch_report(FILE *rpt, struct branch *b)\n@@ -500,6 +503,33 @@ static NORETURN void die_nicely(const char *err, va_list params)\n \texit(128);\n }\n \n+#ifndef SIGUSR1\t/* Windows, for example */\n+\n+static void set_checkpoint_signal(void)\n+{\n+\t/* nothing */\n+}\n+\n+#else\n+\n+static void checkpoint_signal(int signo)\n+{\n+\tcheckpoint_requested = 1;\n+}\n+\n+static void set_checkpoint_signal(void)\n+{\n+\tstruct sigaction sa;\n+\n+\tmemset(&sa, 0, sizeof(sa));\n+\tsa.sa_handler = checkpoint_signal;\n+\tsigemptyset(&sa.sa_mask);\n+\tsa.sa_flags = SA_RESTART;\n+\tsigaction(SIGUSR1, &sa, NULL);\n+}\n+\n+#endif\n+\n static void alloc_objects(unsigned int cnt)\n {\n \tstruct object_entry_pool *b;\n@@ -2689,14 +2719,20 @@ static void parse_reset_branch(void)\n \t\tunread_command_buf = 1;\n }\n \n-static void parse_checkpoint(void)\n+static void do_checkpoint(void)\n {\n+\tcheckpoint_requested = 0;\n \tif (object_count) {\n \t\tcycle_packfile();\n \t\tdump_branches();\n \t\tdump_tags();\n \t\tdump_marks();\n \t}\n+}\n+\n+static void parse_checkpoint(void)\n+{\n+\tdo_checkpoint();\n \tskip_optional_lf();\n }\n \n@@ -2953,6 +2989,7 @@ int main(int argc, const char **argv)\n \tprepare_packed_git();\n \tstart_packfile();\n \tset_die_routine(die_nicely);\n+\tset_checkpoint_signal();\n \twhile (read_next_command() != EOF) {\n \t\tif (!strcmp(\"blob\", command_buf.buf))\n \t\t\tparse_new_blob();\n@@ -2974,6 +3011,9 @@ int main(int argc, const char **argv)\n \t\t\t/* ignore non-git options*/;\n \t\telse\n \t\t\tdie(\"Unsupported command: %s\", command_buf.buf);\n+\n+\t\tif (checkpoint_requested)\n+\t\t\tdo_checkpoint();\n \t}\n \n \t/* argv hasn't been parsed yet, do so */\n-- \n1.7.2.3\n"},{"id":"156500","messageId":"7vpqtusbyx.fsf@alter.siamese.dyndns.org","threadId":"25805","inReplyTo":"20101122081601.GA9722@burratino","subject":"Re: [PATCH/RFC] fast-import: treat SIGUSR1 as a request to access objects early","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-11-24T20:28:22Z","receivedAt":"2010-11-24T20:28:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n> +static void parse_checkpoint(void)\n> +{\n> +\tdo_checkpoint();\n>  \tskip_optional_lf();\n>  }\n\nI would have expected the new implementation of parse_checkpoint() to just\nset the checkpoint_requested variable to true, so that the checkpoint\nalways happens in the main loop, which somehow feels cleaner.  But that is\na minor point.\n"},{"id":"156502","messageId":"20101124205319.GA12245@burratino","threadId":"25805","inReplyTo":"7vpqtusbyx.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC] fast-import: treat SIGUSR1 as a request to access objects early","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-24T20:53:19Z","receivedAt":"2010-11-24T20:53:19Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Junio C Hamano wrote:\n\n> I would have expected the new implementation of parse_checkpoint() to just\n> set the checkpoint_requested variable to true, so that the checkpoint\n> always happens in the main loop, which somehow feels cleaner.\n\nNice.  Here's that plus some minor cleanups I had in my tree.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 6be3938..5937cc0 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -1119,7 +1119,6 @@ operator can use this facility to peek at the objects and refs from an\n import in progress, at the cost of some added running time and worse\n compression.\n \n-\n Author\n ------\n Written by Shawn O. Pearce <spearce@spearce.org>.\ndiff --git a/fast-import.c b/fast-import.c\nindex e8f08fb..a348994 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -490,7 +490,6 @@ static NORETURN void die_nicely(const char *err, va_list params)\n \n static void set_checkpoint_signal(void)\n {\n-\t/* nothing */\n }\n \n #else\n@@ -2351,7 +2350,7 @@ static void parse_reset_branch(void)\n \t\tunread_command_buf = 1;\n }\n \n-static void do_checkpoint(void)\n+static void checkpoint(void)\n {\n \tcheckpoint_requested = 0;\n \tif (object_count) {\n@@ -2364,7 +2363,7 @@ static void do_checkpoint(void)\n \n static void parse_checkpoint(void)\n {\n-\tdo_checkpoint();\n+\tcheckpoint_requested = 1;\n \tskip_optional_lf();\n }\n \n@@ -2524,7 +2523,7 @@ int main(int argc, const char **argv)\n \t\t\tdie(\"Unsupported command: %s\", command_buf.buf);\n \n \t\tif (checkpoint_requested)\n-\t\t\tdo_checkpoint();\n+\t\t\tcheckpoint();\n \t}\n \tend_packfile();\n \n"}]}