threads / patch / 14664

patchTeach fsck and prune about the new location of temporary objects

Subject: [PATCH] Teach fsck and prune about the new location of temporary objects

## tl;dr

5 messages between Jul 24, 2008 and Jul 27, 2008. Diffs are folded; open one to read it.

replies: 4people: 4as markdown or json

Brandon Casey· Jul 24, 2008, 22:41 UTC · lore

Since 5723fe7e, temporary objects are now created in their final destination directories, rather than in .git/objects/. Teach fsck to recognize and ignore the temporary objects it encounters, and teach prune to remove them.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 builtin-fsck.c  |    4 ++++
 builtin-prune.c |   41 ++++++++++++++++++++++++-----------------
 2 files changed, 28 insertions(+), 17 deletions(-)
Show changes to 2 files +28 −17

builtin-fsck.c, builtin-prune.c

diff --git a/builtin-fsck.c b/builtin-fsck.c
index 7326dc3..7a4a4f1 100644
--- a/builtin-fsck.c
+++ b/builtin-fsck.c
@@ -377,6 +377,10 @@ static void fsck_dir(int i, char *path)
 			if (de->d_name[0] != '.')
 				break;
 			continue;
+		case 14:
+			if (prefixcmp(de->d_name, "tmp_obj_"))
+				break;
+			continue;
 		case 38:
 			sprintf(name, "%02x", i);
 			memcpy(name+2, de->d_name, len+1);
diff --git a/builtin-prune.c b/builtin-prune.c
index 7de4cab..947de8c 100644
--- a/builtin-prune.c
+++ b/builtin-prune.c
@@ -13,6 +13,22 @@ static const char * const prune_usage[] = {
 static int show_only;
 static unsigned long expire;
 
+static int prune_tmp_object(char *path, const char *filename)
+{
+	const char *fullpath = mkpath("%s/%s", path, filename);
+	if (expire) {
+		struct stat st;
+		if (lstat(fullpath, &st))
+			return error("Could not stat '%s'", fullpath);
+		if (st.st_mtime > expire)
+			return 0;
+	}
+	printf("Removing stale temporary file %s\n", fullpath);
+	if (!show_only)
+		unlink(fullpath);
+	return 0;
+}
+
 static int prune_object(char *path, const char *filename, const unsigned char *sha1)
 {
 	const char *fullpath = mkpath("%s/%s", path, filename);
@@ -53,6 +69,11 @@ static int prune_dir(int i, char *path)
 			if (de->d_name[0] != '.')
 				break;
 			continue;
+		case 14:
+			if (prefixcmp(de->d_name, "tmp_obj_"))
+				break;
+			prune_tmp_object(path, de->d_name);
+			continue;
 		case 38:
 			sprintf(name, "%02x", i);
 			memcpy(name+2, de->d_name, len+1);
@@ -105,23 +126,9 @@ static void remove_temporary_files(void)
 			dirname);
 		return;
 	}
-	while ((de = readdir(dir)) != NULL) {
-		if (!prefixcmp(de->d_name, "tmp_")) {
-			char name[PATH_MAX];
-			int c = snprintf(name, PATH_MAX, "%s/%s",
-					 dirname, de->d_name);
-			if (c < 0 || c >= PATH_MAX)
-				continue;
-			if (expire) {
-				struct stat st;
-				if (stat(name, &st) != 0 || st.st_mtime >= expire)
-					continue;
-			}
-			printf("Removing stale temporary file %s\n", name);
-			if (!show_only)
-				unlink(name);
-		}
-	}
+	while ((de = readdir(dir)) != NULL)
+		if (!prefixcmp(de->d_name, "tmp_"))
+			prune_tmp_object(dirname, de->d_name);
 	closedir(dir);
 }
 

-- 
1.6.0.rc0.55.g160ae.dirty
Johannes Schindelin· Jul 25, 2008, 00:33 UTC · re: Brandon Casey · lore

Re: [PATCH] Teach fsck and prune about the new location of temporary objects

Hi,
On Thu, 24 Jul 2008, Brandon Casey wrote:
> Since 5723fe7e, temporary objects are now created in their final destination
> directories, rather than in .git/objects/. Teach fsck to recognize and
> ignore the temporary objects it encounters,
It somehow feels wrong for fsck to ignore anything that could be used.

Ciao, Dscho

Junio C Hamano· Jul 25, 2008, 06:07 UTC · re: Johannes Schindelin · lore

Re: [PATCH] Teach fsck and prune about the new location of temporary objects

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
Show 7 quoted lines
> On Thu, 24 Jul 2008, Brandon Casey wrote:
>
>> Since 5723fe7e, temporary objects are now created in their final destination
>> directories, rather than in .git/objects/. Teach fsck to recognize and
>> ignore the temporary objects it encounters,
>
> It somehow feels wrong for fsck to ignore anything that could be used.

Back when we created temporaries in .git/objects/, these tmp_obj files were not even checked nor looked at, because fsck_object_dir() checked only those 256 fan-out directories, so this is nothing new.

And these tmp_obj files are something we _expect_ to be in the object directory. The user can ^C out object creation codepath anytime, and we do not reference the final destination objects from refs nor the index to preserve the integrity of the repository when that happens. In other words, they are crufts, but they are not something unusual nor get alarmed about.

And it also is the right thing to do to remove them in prune.
Shawn O. Pearce· Jul 27, 2008, 02:33 UTC · re: Brandon Casey · lore

[PATCH] fsck: Don't require tmp_obj_ file names are 14 bytes in length

Not all temporary file creation routines will ensure 14 bytes are used to generate the temporary file name. In C Git this may be true, but alternate implementations such as jgit are not always able to generate a temporary file name with a specific prefix and also ensure the file name length is 14 bytes long.

Since temporary files in a directory we are fsck'ing should be uncommon (as they are short lived only long enough for an active writer to finish writing the file and rename it) we shouldn't see these show up very often. Always using a prefixcmp() call and ignoring the length opens up room for other implementations to use different name generation schemes.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
  Brandon Casey <casey@nrlssc.navy.mil> wrote:
  > Since 5723fe7e, temporary objects are now created in their final destination
  > directories, rather than in .git/objects/. Teach fsck to recognize and
  > ignore the temporary objects it encounters, and teach prune to remove them.
  jgit can't exactly follow the 14 character naming rule.
  This should be a safe way around it.
 builtin-fsck.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)
Show changes to builtin-fsck.c +2 −4
diff --git a/builtin-fsck.c b/builtin-fsck.c
index 7a4a4f1..6eb7da8 100644
--- a/builtin-fsck.c
+++ b/builtin-fsck.c
@@ -377,10 +377,6 @@ static void fsck_dir(int i, char *path)
 			if (de->d_name[0] != '.')
 				break;
 			continue;
-		case 14:
-			if (prefixcmp(de->d_name, "tmp_obj_"))
-				break;
-			continue;
 		case 38:
 			sprintf(name, "%02x", i);
 			memcpy(name+2, de->d_name, len+1);
@@ -389,6 +385,8 @@ static void fsck_dir(int i, char *path)
 			add_sha1_list(sha1, DIRENT_SORT_HINT(de));
 			continue;
 		}
+		if (prefixcmp(de->d_name, "tmp_obj_"))
+			continue;
 		fprintf(stderr, "bad sha1 file: %s/%s\n", path, de->d_name);
 	}
 	closedir(dir);

-- 
1.6.0.rc0.182.gb96c7

-- 
Shawn.
Junio C Hamano· Jul 27, 2008, 04:15 UTC · re: Shawn O. Pearce · lore

Re: [PATCH] fsck: Don't require tmp_obj_ file names are 14 bytes in length

"Shawn O. Pearce" <spearce@spearce.org> writes:
Show 7 quoted lines
>   Brandon Casey <casey@nrlssc.navy.mil> wrote:
>   > Since 5723fe7e, temporary objects are now created in their final destination
>   > directories, rather than in .git/objects/. Teach fsck to recognize and
>   > ignore the temporary objects it encounters, and teach prune to remove them.
>
>   jgit can't exactly follow the 14 character naming rule.
>   This should be a safe way around it.

More importantly, we might want to change the temporary filename to ensure that it is 38 bytes in length in the future (cf. $gmane/85106).

← back to recent threads