{"thread":{"id":"11929","subject":"[PATCH v2.5] Make git prune remove temporary packs that look like write failures","startedAt":"2008-02-07T02:55:14Z","lastAt":"2008-02-11T16:10:36Z","messageCount":2,"participants":["David Steven Tweed","Nicolas Pitre"],"isPatch":true,"patchVersion":2,"patchTotal":null},"messages":[{"id":"67743","messageId":"Pine.GSO.4.63.0802070248190.15699@suma3","threadId":"11929","inReplyTo":null,"subject":"[PATCH v2.5] Make git prune remove temporary packs that look like write failures","fromName":"David Steven Tweed","fromEmail":"d.s.tweed@reading.ac.uk","sentAt":"2008-02-07T02:55:14Z","receivedAt":"2008-02-07T02:55:14Z","isPatch":true,"sender":{"key":"d.s.tweed@reading.ac.uk","avatar":null},"body":"Write errors when repacking (eg, due to out-of-space conditions)\ncan leave temporary packs (and possibly other files beginning\nwith \"tmp_\") lying around which no existing\ncodepath removes and which aren't obvious to the casual user.\nThese can also be multi-megabyte files wasting noticeable space.\nUnfortunately there's no way to definitely tell in builtin-prune\nthat a tmp_ file is not being used by a concurrent process,\nsuch as a fetch. However, it is documented that pruning should\nonly be done on a quiet repository and --expire is honoured\n(using code from Johannes Schindelin, along with a test case\nhe wrote) so that its safety is the same as that of loose\nobject pruning.\n\nSince they might be signs of a problem (unlike orphaned loose\nobjects) the names of any removed files are printed.\n\nSigned-off-by: David Tweed (david.tweed@gmail.com)\n---\n\nThis version incorporates all the agreed with changes from the\nprevious version, and includes Johannes Schindelin's\nmodifications and also his test. If you want to change the attribution\nlines to be him that's fine.\n\n(BTW, I won't be reading mail until after the weekend now.)\n\n builtin-prune.c  |   39 +++++++++++++++++++++++++++++++++++++++\n t/t5304-prune.sh |   32 ++++++++++++++++++++++++++++++++\n 2 files changed, 71 insertions(+), 0 deletions(-)\n create mode 100644 t/t5304-prune.sh\n\ndiff --git a/builtin-prune.c b/builtin-prune.c\nindex b5e7684..f25ddb6 100644\n--- a/builtin-prune.c\n+++ b/builtin-prune.c\n@@ -83,6 +83,44 @@ static void prune_object_dir(const char *path)\n \t}\n }\n\n+/*\n+ * Write errors (particularly out of space) can result in\n+ * failed temporary packs (and more rarely indexes and other\n+ * files begining with \"tmp_\") accumulating in the\n+ * object directory.\n+ */\n+static void remove_temporary_files(void)\n+{\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tchar* dirname=get_object_directory();\n+\n+\tdir = opendir(dirname);\n+\tif (!dir) {\n+\t\tfprintf(stderr, \"Unable to open object directory %s\\n\",\n+\t\t\tdirname);\n+\t\treturn;\n+\t}\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tif (!prefixcmp(de->d_name, \"tmp_\")) {\n+\t\t\tchar name[PATH_MAX];\n+\t\t\tint c = snprintf(name, PATH_MAX, \"%s/%s\",\n+\t\t\t\t\t dirname, de->d_name);\n+\t\t\tif (c < 0 || c >= PATH_MAX)\n+\t\t\t\tcontinue;\n+\t\t\tif (expire) {\n+\t\t\t\tstruct stat st;\n+\t\t\t\tif (stat(name, &st) != 0 || st.st_mtime >= expire)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tprintf(\"Removing stale temporary file %s\\n\", name);\n+\t\t\tif (!show_only)\n+\t\t\t\tunlink(name);\n+\t\t}\n+\t}\n+\tclosedir(dir);\n+}\n+\n int cmd_prune(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n@@ -115,5 +153,6 @@ int cmd_prune(int argc, const char **argv, const char *prefix)\n\n \tsync();\n \tprune_packed_objects(show_only);\n+\tremove_temporary_files();\n \treturn 0;\n }\ndiff --git a/t/t5304-prune.sh b/t/t5304-prune.sh\nnew file mode 100644\nindex 0000000..6560af7\n--- /dev/null\n+++ b/t/t5304-prune.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2008 Johannes E. Schindelin\n+#\n+\n+test_description='prune'\n+. ./test-lib.sh\n+\n+test_expect_success setup '\n+\n+\t: > file &&\n+\tgit add file &&\n+\ttest_tick &&\n+\tgit commit -m initial &&\n+\tgit gc\n+\n+'\n+\n+test_expect_success 'prune stale packs' '\n+\n+\torig_pack=$(echo .git/objects/pack/*.pack) &&\n+\t: > .git/objects/tmp_1.pack &&\n+\t: > .git/objects/tmp_2.pack &&\n+\ttest-chmtime -86501 .git/objects/tmp_1.pack &&\n+\tgit prune --expire 1.day &&\n+\ttest -f $orig_pack &&\n+\ttest -f .git/objects/tmp_2.pack &&\n+\t! test -f .git/objects/tmp_1.pack\n+\n+'\n+\n+test_done\n"},{"id":"68406","messageId":"alpine.LFD.1.00.0802111109500.2732@xanadu.home","threadId":"11929","inReplyTo":"Pine.GSO.4.63.0802070248190.15699@suma3","subject":"Re: [PATCH v2.5] Make git prune remove temporary packs that look like write failures","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-02-11T16:10:36Z","receivedAt":"2008-02-11T16:10:36Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 7 Feb 2008, David Steven Tweed wrote:\n\n> Write errors when repacking (eg, due to out-of-space conditions)\n> can leave temporary packs (and possibly other files beginning\n> with \"tmp_\") lying around which no existing\n> codepath removes and which aren't obvious to the casual user.\n> These can also be multi-megabyte files wasting noticeable space.\n> Unfortunately there's no way to definitely tell in builtin-prune\n> that a tmp_ file is not being used by a concurrent process,\n> such as a fetch. However, it is documented that pruning should\n> only be done on a quiet repository and --expire is honoured\n> (using code from Johannes Schindelin, along with a test case\n> he wrote) so that its safety is the same as that of loose\n> object pruning.\n> \n> Since they might be signs of a problem (unlike orphaned loose\n> objects) the names of any removed files are printed.\n> \n> Signed-off-by: David Tweed (david.tweed@gmail.com)\n\nAcked-by: Nicolas Pitre <nico@cam.org>\n\n> ---\n> \n> This version incorporates all the agreed with changes from the\n> previous version, and includes Johannes Schindelin's\n> modifications and also his test. If you want to change the attribution\n> lines to be him that's fine.\n> \n> (BTW, I won't be reading mail until after the weekend now.)\n> \n>  builtin-prune.c  |   39 +++++++++++++++++++++++++++++++++++++++\n>  t/t5304-prune.sh |   32 ++++++++++++++++++++++++++++++++\n>  2 files changed, 71 insertions(+), 0 deletions(-)\n>  create mode 100644 t/t5304-prune.sh\n> \n> diff --git a/builtin-prune.c b/builtin-prune.c\n> index b5e7684..f25ddb6 100644\n> --- a/builtin-prune.c\n> +++ b/builtin-prune.c\n> @@ -83,6 +83,44 @@ static void prune_object_dir(const char *path)\n>  \t}\n>  }\n> \n> +/*\n> + * Write errors (particularly out of space) can result in\n> + * failed temporary packs (and more rarely indexes and other\n> + * files begining with \"tmp_\") accumulating in the\n> + * object directory.\n> + */\n> +static void remove_temporary_files(void)\n> +{\n> +\tDIR *dir;\n> +\tstruct dirent *de;\n> +\tchar* dirname=get_object_directory();\n> +\n> +\tdir = opendir(dirname);\n> +\tif (!dir) {\n> +\t\tfprintf(stderr, \"Unable to open object directory %s\\n\",\n> +\t\t\tdirname);\n> +\t\treturn;\n> +\t}\n> +\twhile ((de = readdir(dir)) != NULL) {\n> +\t\tif (!prefixcmp(de->d_name, \"tmp_\")) {\n> +\t\t\tchar name[PATH_MAX];\n> +\t\t\tint c = snprintf(name, PATH_MAX, \"%s/%s\",\n> +\t\t\t\t\t dirname, de->d_name);\n> +\t\t\tif (c < 0 || c >= PATH_MAX)\n> +\t\t\t\tcontinue;\n> +\t\t\tif (expire) {\n> +\t\t\t\tstruct stat st;\n> +\t\t\t\tif (stat(name, &st) != 0 || st.st_mtime >= expire)\n> +\t\t\t\t\tcontinue;\n> +\t\t\t}\n> +\t\t\tprintf(\"Removing stale temporary file %s\\n\", name);\n> +\t\t\tif (!show_only)\n> +\t\t\t\tunlink(name);\n> +\t\t}\n> +\t}\n> +\tclosedir(dir);\n> +}\n> +\n>  int cmd_prune(int argc, const char **argv, const char *prefix)\n>  {\n>  \tint i;\n> @@ -115,5 +153,6 @@ int cmd_prune(int argc, const char **argv, const char *prefix)\n> \n>  \tsync();\n>  \tprune_packed_objects(show_only);\n> +\tremove_temporary_files();\n>  \treturn 0;\n>  }\n> diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh\n> new file mode 100644\n> index 0000000..6560af7\n> --- /dev/null\n> +++ b/t/t5304-prune.sh\n> @@ -0,0 +1,32 @@\n> +#!/bin/sh\n> +#\n> +# Copyright (c) 2008 Johannes E. Schindelin\n> +#\n> +\n> +test_description='prune'\n> +. ./test-lib.sh\n> +\n> +test_expect_success setup '\n> +\n> +\t: > file &&\n> +\tgit add file &&\n> +\ttest_tick &&\n> +\tgit commit -m initial &&\n> +\tgit gc\n> +\n> +'\n> +\n> +test_expect_success 'prune stale packs' '\n> +\n> +\torig_pack=$(echo .git/objects/pack/*.pack) &&\n> +\t: > .git/objects/tmp_1.pack &&\n> +\t: > .git/objects/tmp_2.pack &&\n> +\ttest-chmtime -86501 .git/objects/tmp_1.pack &&\n> +\tgit prune --expire 1.day &&\n> +\ttest -f $orig_pack &&\n> +\ttest -f .git/objects/tmp_2.pack &&\n> +\t! test -f .git/objects/tmp_1.pack\n> +\n> +'\n> +\n> +test_done\n> \n\n\nNicolas\n"}]}