threads / patch / 38543

patchfast-import: avoid running end_packfile recursively

Subject: [PATCH] fast-import: avoid running end_packfile recursively

## tl;dr

3 messages between Feb 10, 2015 and Feb 10, 2015. Diffs are folded; open one to read it.

replies: 2people: 2as markdown or json

Jeff King· Feb 10, 2015, 01:07 UTC · lore

When an import has finished, we run end_packfile() to finalize the data and move the packfile into place. If this process fails, we call die() and end up in our die_nicely() handler. Which unfortunately includes running end_packfile to save any progress we made. We enter the function again, and start operating on the pack_data struct while it is in an inconsistent state, leading to a segfault.

One way to trigger this is to simply start two identical fast-imports at the same time. They will both create the same packfiles, which will then try to create identically named ".keep" files. One will win the race, and the other will die(), and end up with the segfault.

Since 3c078b9, we already reset the pack_data pointer to NULL at the end of end_packfile. That covers the case of us calling die() right after end_packfile, before we have reinitialized the pack_data pointer. This new problem is quite similar, except that we are worried about calling die() _during_ end_packfile, not right after. Ideally we would simply set pack_data to NULL as soon as we enter the function, and operate on a copy of the pointer.

Unfortunately, it is not so easy. pack_data is a global, and end_packfile calls into other functions which operate on the global directly. We would have to teach each of these to take an argument, and there is no guarantee that we would catch all of the spots.

Instead, we can simply use a static flag to avoid recursively entering the function. This is a little less elegant, but it's short and fool-proof.

Signed-off-by: Jeff King <peff@peff.net>
---
 fast-import.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
Show changes to fast-import.c +5 −1
diff --git a/fast-import.c b/fast-import.c
index d0bd285..aac2c24 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -947,9 +947,12 @@ static void unkeep_all_packs(void)
 
 static void end_packfile(void)
 {
-	if (!pack_data)
+	static int running;
+
+	if (running || !pack_data)
 		return;
 
+	running = 1;
 	clear_delta_base_cache();
 	if (object_count) {
 		struct packed_git *new_p;
@@ -999,6 +1002,7 @@ static void end_packfile(void)
 	}
 	free(pack_data);
 	pack_data = NULL;
+	running = 0;
 
 	/* We can't carry a delta across packfiles. */
 	strbuf_release(&last_blob.data);

-- 
2.3.0.rc1.287.g761fd19
Junio C Hamano· Feb 10, 2015, 18:45 UTC · re: Jeff King · lore

Re: [PATCH] fast-import: avoid running end_packfile recursively

Jeff King <peff@peff.net> writes:
Show 12 quoted lines
> When an import has finished, we run end_packfile() to
> finalize the data and move the packfile into place. If this
> process fails, we call die() and end up in our die_nicely()
> handler.  Which unfortunately includes running end_packfile
> to save any progress we made. We enter the function again,
> and start operating on the pack_data struct while it is in
> an inconsistent state, leading to a segfault.
> ... This new problem is
> quite similar, except that we are worried about calling
> die() _during_ end_packfile, not right after. Ideally we
> would simply set pack_data to NULL as soon as we enter the
> function, and operate on a copy of the pointer.
Nicely analyzed and well done.
Show 5 quoted lines
> Unfortunately, it is not so easy. pack_data is a global, and
> end_packfile calls into other functions which operate on the
> global directly. We would have to teach each of these to
> take an argument, and there is no guarantee that we would
> catch all of the spots.

Well, you can rename the global to something else to make sure ;-) But I think that the approach with a simple flag is better.

If we were planning to do the global-to-parameter surgery for other reasons (perhaps need to make things reentrant?) then the equation might become different, but I do not think we are doing that right now, so...

Thanks.
Jeff King· Feb 10, 2015, 18:58 UTC · re: Junio C Hamano · lore

Re: [PATCH] fast-import: avoid running end_packfile recursively

On Tue, Feb 10, 2015 at 10:45:20AM -0800, Junio C Hamano wrote:
Show 8 quoted lines
> > Unfortunately, it is not so easy. pack_data is a global, and
> > end_packfile calls into other functions which operate on the
> > global directly. We would have to teach each of these to
> > take an argument, and there is no guarantee that we would
> > catch all of the spots.
> 
> Well, you can rename the global to something else to make sure ;-)
> But I think that the approach with a simple flag is better.
:) True. The problems I had in mind were more:
  1. One of the problems with that is that there are a whole bunch of
     helper functions that use the variable. But only the ones that are
     called as part of end_packfile need this treatment. So either we
     need to touch all of them, or we need to figure out reliably which
     ones are part of this code path.
  2. Unless we get rid of the global completely, we open ourselves up to
     end_packfile calling new functions, bringing the problem back. This
     is probably not a huge concern, though, as this code has basically
     not changed much since its inception.

I did work through it, though, and the result is not _too_ bad. Here it is for reference, in case you want to change your mind. The remaining references are only in start_packfile, and in gfi_load_object, both of which are hopefully unlikely to be called from end_packfile.

---
Show changes to fast-import.c +18 −16
diff --git a/fast-import.c b/fast-import.c
index d0bd285..e842386 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -883,7 +883,7 @@ static void start_packfile(void)
 	all_packs[pack_id] = p;
 }
 
-static const char *create_index(void)
+static const char *create_index(struct packed_git *pack)
 {
 	const char *tmpfile;
 	struct pack_idx_entry **idx, **c, **last;
@@ -901,18 +901,18 @@ static const char *create_index(void)
 	if (c != last)
 		die("internal consistency error creating the index");
 
-	tmpfile = write_idx_file(NULL, idx, object_count, &pack_idx_opts, pack_data->sha1);
+	tmpfile = write_idx_file(NULL, idx, object_count, &pack_idx_opts, pack->sha1);
 	free(idx);
 	return tmpfile;
 }
 
-static char *keep_pack(const char *curr_index_name)
+static char *keep_pack(struct packed_git *pack, const char *curr_index_name)
 {
 	static char name[PATH_MAX];
 	static const char *keep_msg = "fast-import";
 	int keep_fd;
 
-	keep_fd = odb_pack_keep(name, sizeof(name), pack_data->sha1);
+	keep_fd = odb_pack_keep(name, sizeof(name), pack->sha1);
 	if (keep_fd < 0)
 		die_errno("cannot create keep file");
 	write_or_die(keep_fd, keep_msg, strlen(keep_msg));
@@ -920,12 +920,12 @@ static char *keep_pack(const char *curr_index_name)
 		die_errno("failed to write keep file");
 
 	snprintf(name, sizeof(name), "%s/pack/pack-%s.pack",
-		 get_object_directory(), sha1_to_hex(pack_data->sha1));
-	if (move_temp_to_file(pack_data->pack_name, name))
+		 get_object_directory(), sha1_to_hex(pack->sha1));
+	if (move_temp_to_file(pack->pack_name, name))
 		die("cannot store pack file");
 
 	snprintf(name, sizeof(name), "%s/pack/pack-%s.idx",
-		 get_object_directory(), sha1_to_hex(pack_data->sha1));
+		 get_object_directory(), sha1_to_hex(pack->sha1));
 	if (move_temp_to_file(curr_index_name, name))
 		die("cannot store index file");
 	free((void *)curr_index_name);
@@ -947,8 +947,11 @@ static void unkeep_all_packs(void)
 
 static void end_packfile(void)
 {
+	struct packed_git *old_p = pack_data;
+
 	if (!pack_data)
 		return;
+	pack_data = NULL;
 
 	clear_delta_base_cache();
 	if (object_count) {
@@ -959,13 +962,13 @@ static void end_packfile(void)
 		struct branch *b;
 		struct tag *t;
 
-		close_pack_windows(pack_data);
+		close_pack_windows(old_p);
 		sha1close(pack_file, cur_pack_sha1, 0);
-		fixup_pack_header_footer(pack_data->pack_fd, pack_data->sha1,
-				    pack_data->pack_name, object_count,
+		fixup_pack_header_footer(old_p->pack_fd, old_p->sha1,
+				    old_p->pack_name, object_count,
 				    cur_pack_sha1, pack_size);
-		close(pack_data->pack_fd);
-		idx_name = keep_pack(create_index());
+		close(old_p->pack_fd);
+		idx_name = keep_pack(old_p, create_index(old_p));
 
 		/* Register the packfile with core git's machinery. */
 		new_p = add_packed_git(idx_name, strlen(idx_name), 1);
@@ -994,11 +997,10 @@ static void end_packfile(void)
 		pack_id++;
 	}
 	else {
-		close(pack_data->pack_fd);
-		unlink_or_warn(pack_data->pack_name);
+		close(old_p->pack_fd);
+		unlink_or_warn(old_p->pack_name);
 	}
-	free(pack_data);
-	pack_data = NULL;
+	free(old_p);
 
 	/* We can't carry a delta across packfiles. */
 	strbuf_release(&last_blob.data);

← back to recent threads