# [PATCH] Teach fsck and prune about the new location of temporary objects 5 messages from 2008-07-24 to 2008-07-27. Participants: Brandon Casey, Johannes Schindelin, Junio C Hamano, Shawn O. Pearce. Thread: https://gitlist.dev/t/14664 ## Brandon Casey, 2008-07-24 22:41 Subject: [PATCH] Teach fsck and prune about the new location of temporary objects Message-ID: <6ruv3Y98_kSSVnJFTkV0PNdiNcQ3g-a3M4BhGoT7S1PorElp5tJAkw@cipher.nrlssc.navy.mil> URL: https://gitlist.dev/e/6ruv3Y98_kSSVnJFTkV0PNdiNcQ3g-a3M4BhGoT7S1PorElp5tJAkw%40cipher.nrlssc.navy.mil ``` Since 5723fe7e, temporary objects are now created in their final destination directories, rather than in .git/objects/. Teach fsck to recognize and ignore the temporary objects it encounters, and teach prune to remove them. Signed-off-by: Brandon Casey --- builtin-fsck.c | 4 ++++ builtin-prune.c | 41 ++++++++++++++++++++++++----------------- 2 files changed, 28 insertions(+), 17 deletions(-) diff --git a/builtin-fsck.c b/builtin-fsck.c index 7326dc3..7a4a4f1 100644 --- a/builtin-fsck.c +++ b/builtin-fsck.c @@ -377,6 +377,10 @@ static void fsck_dir(int i, char *path) if (de->d_name[0] != '.') break; continue; + case 14: + if (prefixcmp(de->d_name, "tmp_obj_")) + break; + continue; case 38: sprintf(name, "%02x", i); memcpy(name+2, de->d_name, len+1); diff --git a/builtin-prune.c b/builtin-prune.c index 7de4cab..947de8c 100644 --- a/builtin-prune.c +++ b/builtin-prune.c @@ -13,6 +13,22 @@ static const char * const prune_usage[] = { static int show_only; static unsigned long expire; +static int prune_tmp_object(char *path, const char *filename) +{ + const char *fullpath = mkpath("%s/%s", path, filename); + if (expire) { + struct stat st; + if (lstat(fullpath, &st)) + return error("Could not stat '%s'", fullpath); + if (st.st_mtime > expire) + return 0; + } + printf("Removing stale temporary file %s\n", fullpath); + if (!show_only) + unlink(fullpath); + return 0; +} + static int prune_object(char *path, const char *filename, const unsigned char *sha1) { const char *fullpath = mkpath("%s/%s", path, filename); @@ -53,6 +69,11 @@ static int prune_dir(int i, char *path) if (de->d_name[0] != '.') break; continue; + case 14: + if (prefixcmp(de->d_name, "tmp_obj_")) + break; + prune_tmp_object(path, de->d_name); + continue; case 38: sprintf(name, "%02x", i); memcpy(name+2, de->d_name, len+1); @@ -105,23 +126,9 @@ static void remove_temporary_files(void) dirname); return; } - while ((de = readdir(dir)) != NULL) { - if (!prefixcmp(de->d_name, "tmp_")) { - char name[PATH_MAX]; - int c = snprintf(name, PATH_MAX, "%s/%s", - dirname, de->d_name); - if (c < 0 || c >= PATH_MAX) - continue; - if (expire) { - struct stat st; - if (stat(name, &st) != 0 || st.st_mtime >= expire) - continue; - } - printf("Removing stale temporary file %s\n", name); - if (!show_only) - unlink(name); - } - } + while ((de = readdir(dir)) != NULL) + if (!prefixcmp(de->d_name, "tmp_")) + prune_tmp_object(dirname, de->d_name); closedir(dir); } -- 1.6.0.rc0.55.g160ae.dirty ``` ## Johannes Schindelin, 2008-07-25 00:33 Subject: Re: [PATCH] Teach fsck and prune about the new location of temporary objects Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0807250233031.4140%40eeepc-johanness In-Reply-To: <6ruv3Y98_kSSVnJFTkV0PNdiNcQ3g-a3M4BhGoT7S1PorElp5tJAkw@cipher.nrlssc.navy.mil> ``` Hi, On Thu, 24 Jul 2008, Brandon Casey wrote: > Since 5723fe7e, temporary objects are now created in their final destination > directories, rather than in .git/objects/. Teach fsck to recognize and > ignore the temporary objects it encounters, It somehow feels wrong for fsck to ignore anything that could be used. Ciao, Dscho ``` ## Junio C Hamano, 2008-07-25 06:07 Subject: Re: [PATCH] Teach fsck and prune about the new location of temporary objects Message-ID: <7vd4l2xzrd.fsf@gitster.siamese.dyndns.org> URL: https://gitlist.dev/e/7vd4l2xzrd.fsf%40gitster.siamese.dyndns.org In-Reply-To: ``` Johannes Schindelin writes: > On Thu, 24 Jul 2008, Brandon Casey wrote: > >> Since 5723fe7e, temporary objects are now created in their final destination >> directories, rather than in .git/objects/. Teach fsck to recognize and >> ignore the temporary objects it encounters, > > It somehow feels wrong for fsck to ignore anything that could be used. Back when we created temporaries in .git/objects/, these tmp_obj files were not even checked nor looked at, because fsck_object_dir() checked only those 256 fan-out directories, so this is nothing new. And these tmp_obj files are something we _expect_ to be in the object directory. The user can ^C out object creation codepath anytime, and we do not reference the final destination objects from refs nor the index to preserve the integrity of the repository when that happens. In other words, they are crufts, but they are not something unusual nor get alarmed about. And it also is the right thing to do to remove them in prune. ``` ## Shawn O. Pearce, 2008-07-27 02:33 Subject: [PATCH] fsck: Don't require tmp_obj_ file names are 14 bytes in length Message-ID: <20080727023300.GB17425@spearce.org> URL: https://gitlist.dev/e/20080727023300.GB17425%40spearce.org In-Reply-To: <6ruv3Y98_kSSVnJFTkV0PNdiNcQ3g-a3M4BhGoT7S1PorElp5tJAkw@cipher.nrlssc.navy.mil> ``` Not all temporary file creation routines will ensure 14 bytes are used to generate the temporary file name. In C Git this may be true, but alternate implementations such as jgit are not always able to generate a temporary file name with a specific prefix and also ensure the file name length is 14 bytes long. Since temporary files in a directory we are fsck'ing should be uncommon (as they are short lived only long enough for an active writer to finish writing the file and rename it) we shouldn't see these show up very often. Always using a prefixcmp() call and ignoring the length opens up room for other implementations to use different name generation schemes. Signed-off-by: Shawn O. Pearce --- Brandon Casey wrote: > Since 5723fe7e, temporary objects are now created in their final destination > directories, rather than in .git/objects/. Teach fsck to recognize and > ignore the temporary objects it encounters, and teach prune to remove them. jgit can't exactly follow the 14 character naming rule. This should be a safe way around it. builtin-fsck.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/builtin-fsck.c b/builtin-fsck.c index 7a4a4f1..6eb7da8 100644 --- a/builtin-fsck.c +++ b/builtin-fsck.c @@ -377,10 +377,6 @@ static void fsck_dir(int i, char *path) if (de->d_name[0] != '.') break; continue; - case 14: - if (prefixcmp(de->d_name, "tmp_obj_")) - break; - continue; case 38: sprintf(name, "%02x", i); memcpy(name+2, de->d_name, len+1); @@ -389,6 +385,8 @@ static void fsck_dir(int i, char *path) add_sha1_list(sha1, DIRENT_SORT_HINT(de)); continue; } + if (prefixcmp(de->d_name, "tmp_obj_")) + continue; fprintf(stderr, "bad sha1 file: %s/%s\n", path, de->d_name); } closedir(dir); -- 1.6.0.rc0.182.gb96c7 -- Shawn. ``` ## Junio C Hamano, 2008-07-27 04:15 Subject: Re: [PATCH] fsck: Don't require tmp_obj_ file names are 14 bytes in length Message-ID: <7vwsj83quc.fsf@gitster.siamese.dyndns.org> URL: https://gitlist.dev/e/7vwsj83quc.fsf%40gitster.siamese.dyndns.org In-Reply-To: <20080727023300.GB17425@spearce.org> ``` "Shawn O. Pearce" writes: > Brandon Casey wrote: > > Since 5723fe7e, temporary objects are now created in their final destination > > directories, rather than in .git/objects/. Teach fsck to recognize and > > ignore the temporary objects it encounters, and teach prune to remove them. > > jgit can't exactly follow the 14 character naming rule. > This should be a safe way around it. More importantly, we might want to change the temporary filename to ensure that it is 38 bytes in length in the future (cf. $gmane/85106). ```