threads / patch / 7636

patch, 2 partsTeach git list-objects logic not to follow gitlinks

Subject: [PATCH 1/2] Teach git list-objects logic not to follow gitlinks

## tl;dr

5 messages between Apr 13, 2007 and Apr 13, 2007. Diffs are folded; open one to read it.

replies: 4people: 2as markdown or json

Linus Torvalds· Apr 13, 2007, 04:03 UTC · lore

This allows us to pack superprojects and thus clone them (but not yet check them out on the receiving side - that's the next patch)

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---
Most of this diff is a comment saying why there is nothing to do ;)
 list-objects.c |   34 ++++++++++++++++++++++++++++++++++
 1 files changed, 34 insertions(+), 0 deletions(-)
Show changes to list-objects.c +34 −0
diff --git a/list-objects.c b/list-objects.c
index 2ba2c95..310f8d3 100644
--- a/list-objects.c
+++ b/list-objects.c
@@ -25,6 +25,37 @@ static void process_blob(struct rev_info *revs,
 	add_object(obj, p, path, name);
 }
 
+/*
+ * Processing a gitlink entry currently does nothing, since
+ * we do not recurse into the subproject.
+ *
+ * We *could* eventually add a flag that actually does that,
+ * which would involve:
+ *  - is the subproject actually checked out?
+ *  - if so, see if the subproject has already been added
+ *    to the alternates list, and add it if not.
+ *  - process the commit (or tag) the gitlink points to
+ *    recursively.
+ *
+ * However, it's unclear whether there is really ever any
+ * reason to see superprojects and subprojects as such a
+ * "unified" object pool (potentially resulting in a totally
+ * humongous pack - avoiding which was the whole point of
+ * having gitlinks in the first place!).
+ *
+ * So for now, there is just a note that we *could* follow
+ * the link, and how to do it. Whether it necessarily makes
+ * any sense what-so-ever to ever do that is another issue.
+ */
+static void process_gitlink(struct rev_info *revs,
+			    const unsigned char *sha1,
+			    struct object_array *p,
+			    struct name_path *path,
+			    const char *name)
+{
+	/* Nothing to do */
+}
+
 static void process_tree(struct rev_info *revs,
 			 struct tree *tree,
 			 struct object_array *p,
@@ -56,6 +87,9 @@ static void process_tree(struct rev_info *revs,
 			process_tree(revs,
 				     lookup_tree(entry.sha1),
 				     p, &me, entry.path);
+		else if (S_ISDIRLNK(entry.mode))
+			process_gitlink(revs, entry.sha1,
+					p, &me, entry.path);
 		else
 			process_blob(revs,
 				     lookup_blob(entry.sha1),
Linus Torvalds· Apr 13, 2007, 04:08 UTC · re: Linus Torvalds · lore

[PATCH 2/2] Teach "git-read-tree -u" to check out submodules as a directory

This actually allows us to check out a supermodule after cloning, although the submodules will obviously not be checked out, and will just be an empty subdirectory.

[ Side note: this also shows that we currently don't correctly handle
  such subprojects that aren't checked out correctly yet.  They should
  always show up as not being modified, but failing to resolve the
  gitlink HEAD does not properly trigger the "not modified" logic in all 
  places it needs to..
  So more work to be done, but that's a separate issue, unrelated to
  the action of checking out the superproject. ]

The bulk of this patch is simply because we need to check the type of the index entry *before* we try to read the object it points to, and that meant that the code needed some re-organization. So I moved some of the code in common to both symlinks and files to be a trivial helper function.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---
 entry.c |   42 +++++++++++++++++++++++++++++-------------
 1 files changed, 29 insertions(+), 13 deletions(-)
Show changes to entry.c +29 −13
diff --git a/entry.c b/entry.c
index d72f811..9545e89 100644
--- a/entry.c
+++ b/entry.c
@@ -62,26 +62,33 @@ static int create_file(const char *path, unsigned int mode)
 	return open(path, O_WRONLY | O_CREAT | O_EXCL, mode);
 }
 
+static void *read_blob_entry(struct cache_entry *ce, const char *path, unsigned long *size)
+{
+	enum object_type type;
+	void *new = read_sha1_file(ce->sha1, &type, size);
+
+	if (new) {
+		if (type == OBJ_BLOB)
+			return new;
+		free(new);
+	}
+	return NULL;
+}
+
 static int write_entry(struct cache_entry *ce, char *path, struct checkout *state, int to_tempfile)
 {
 	int fd;
-	void *new;
-	unsigned long size;
 	long wrote;
-	enum object_type type;
 
-	new = read_sha1_file(ce->sha1, &type, &size);
-	if (!new || type != OBJ_BLOB) {
-		if (new)
-			free(new);
-		return error("git-checkout-index: unable to read sha1 file of %s (%s)",
-			path, sha1_to_hex(ce->sha1));
-	}
 	switch (ntohl(ce->ce_mode) & S_IFMT) {
-		char *buf;
-		unsigned long nsize;
+		char *buf, *new;
+		unsigned long size, nsize;
 
 	case S_IFREG:
+		new = read_blob_entry(ce, path, &size);
+		if (!new)
+			return error("git-checkout-index: unable to read sha1 file of %s (%s)",
+				path, sha1_to_hex(ce->sha1));
 		if (to_tempfile) {
 			strcpy(path, ".merge_file_XXXXXX");
 			fd = mkstemp(path);
@@ -111,6 +118,10 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
 			return error("git-checkout-index: unable to write file %s", path);
 		break;
 	case S_IFLNK:
+		new = read_blob_entry(ce, path, &size);
+		if (!new)
+			return error("git-checkout-index: unable to read sha1 file of %s (%s)",
+				path, sha1_to_hex(ce->sha1));
 		if (to_tempfile || !has_symlinks) {
 			if (to_tempfile) {
 				strcpy(path, ".merge_link_XXXXXX");
@@ -136,8 +147,13 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
 						 "symlink %s (%s)", path, strerror(errno));
 		}
 		break;
+	case S_IFDIRLNK:
+		if (to_tempfile)
+			return error("git-checkout-index: cannot create temporary subproject %s", path);
+		if (mkdir(path, 0777) < 0)
+			return error("git-checkout-index: cannot create subproject directory %s", path);
+		break;
 	default:
-		free(new);
 		return error("git-checkout-index: unknown file mode for %s", path);
 	}
 
Linus Torvalds· Apr 13, 2007, 04:15 UTC · re: Linus Torvalds · lore

Re: [PATCH 2/2] Teach "git-read-tree -u" to check out submodules as a directory

On Thu, 12 Apr 2007, Linus Torvalds wrote:
Show 9 quoted lines
> 
> [ Side note: this also shows that we currently don't correctly handle
>   such subprojects that aren't checked out correctly yet.  They should
>   always show up as not being modified, but failing to resolve the
>   gitlink HEAD does not properly trigger the "not modified" logic in all 
>   places it needs to..
> 
>   So more work to be done, but that's a separate issue, unrelated to
>   the action of checking out the superproject. ]

Apart from this issue, and the fact that we don't actually diff subprojects at all right now (even to the point of saying "subproject XyZ has changed from commit X to commit Y" - the raw format knows to say that, but the patch format does not), I think the "really core plumbing" is actually mostly done with this series.

So there's certainly some loose ends to tie up and some missing functionality even at the core level, but I suspect this is getting to the point where a big chunk of the remaining part is actually to teach the upper layers (like "fetch", "checkout", "merge" etc) to actually iterate over subprojects using the theoretical higher-level ".subprojects" file, and do the plumbing-level operations on those.

Of course, actually trying to start to *use* these things is bound to find tons and tons of issues, but I still feel like a lot of the core is at least "fleshed out", if not necessarily actually usable yet.

I may take a few days off on this, and hope others will step in and fix up some of the remaining problems ..

			Linus
Junio C Hamano· Apr 13, 2007, 04:32 UTC · re: Linus Torvalds · lore

Re: [PATCH 2/2] Teach "git-read-tree -u" to check out submodules as a directory

Linus Torvalds <torvalds@linux-foundation.org> writes:
Show 14 quoted lines
> @@ -136,8 +147,13 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
>  						 "symlink %s (%s)", path, strerror(errno));
>  		}
>  		break;
> +	case S_IFDIRLNK:
> +		if (to_tempfile)
> +			return error("git-checkout-index: cannot create temporary subproject %s", path);
> +		if (mkdir(path, 0777) < 0)
> +			return error("git-checkout-index: cannot create subproject directory %s", path);
> +		break;
>  	default:
>  		return error("git-checkout-index: unknown file mode for %s", path);
>  	}
>  
Hmm.  Perhaps something like this on top?
Show changes to entry.c +22 −10
diff --git a/entry.c b/entry.c
index 9545e89..0874d61 100644
--- a/entry.c
+++ b/entry.c
@@ -1,6 +1,23 @@
 #include "cache.h"
 #include "blob.h"
 
+static int make_directory(const char *path, int unlink_as_needed)
+{
+	if (mkdir(path, 0777) < 0) {
+		if (errno == EEXIST) {
+			struct stat st;
+			if (unlink_as_needed &&
+			    !unlink(path) &&
+			    !mkdir(path, 0777))
+				return 0; /* ok */
+			if (!stat(path, &st) && S_ISDIR(st.st_mode))
+				return 0; /* ok */
+		}
+		return -1;
+	}
+	return 0;
+}
+
 static void create_directories(const char *path, struct checkout *state)
 {
 	int len = strlen(path);
@@ -8,19 +25,14 @@ static void create_directories(const char *path, struct checkout *state)
 	const char *slash = path;
 
 	while ((slash = strchr(slash+1, '/')) != NULL) {
+		int unlink_as_needed;
+
 		len = slash - path;
 		memcpy(buf, path, len);
 		buf[len] = 0;
-		if (mkdir(buf, 0777)) {
-			if (errno == EEXIST) {
-				struct stat st;
-				if (len > state->base_dir_len && state->force && !unlink(buf) && !mkdir(buf, 0777))
-					continue;
-				if (!stat(buf, &st) && S_ISDIR(st.st_mode))
-					continue; /* ok */
-			}
+		unlink_as_needed = (state->base_dir_len < len && state->force);
+		if (make_directory(buf, unlink_as_needed) < 0)
 			die("cannot create directory at %s", buf);
-		}
 	}
 	free(buf);
 }
@@ -150,7 +162,7 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
 	case S_IFDIRLNK:
 		if (to_tempfile)
 			return error("git-checkout-index: cannot create temporary subproject %s", path);
-		if (mkdir(path, 0777) < 0)
+		if (make_directory(path, 0))
 			return error("git-checkout-index: cannot create subproject directory %s", path);
 		break;
 	default:
Linus Torvalds· Apr 13, 2007, 04:54 UTC · re: Junio C Hamano · lore

Re: [PATCH 2/2] Teach "git-read-tree -u" to check out submodules as a directory

On Thu, 12 Apr 2007, Junio C Hamano wrote:
Show 12 quoted lines
> 
> Hmm.  Perhaps something like this on top?
> 
> diff --git a/entry.c b/entry.c
> index 9545e89..0874d61 100644
> --- a/entry.c
> +++ b/entry.c
> @@ -1,6 +1,23 @@
>  #include "cache.h"
>  #include "blob.h"
>  
> +static int make_directory(const char *path, int unlink_as_needed)

I actually tried to think this through, and I don't *think* we should need to do this. It should literally be sufficient to just do the "mkdir()" call.

Why? Because we've already done the "create_directories()" calls earlier, and if you actually look at "create_file()", that one also just does a simple "open(path, O_WRONLY | O_CREAT | O_EXCL, mode)" - so doing just the mkdir() for creating a subdirectory should be the logically equivalent operation.

Similarly, the S_IFLNK case just does a "symlink()" system call. No "try to unlink if there was an old entry there before" code.

Basically, once we are inside "write_entry()", we expect to get a success or a failure, not a "needs cleanup". If it needed resolving of other state before, we shouldn't even have gotten to that stage in the first place.

That said, looking closer, you're right - the cleanup that is done earlier is actually wrong for the gitlink case. So I think the *real* problem is the cleanup in "checkout_entry()". In particular, note the

	if (!lstat(path, &st)) {
		...
		if (S_ISDIR(st.st_mode)) {
			...
		}
	}

code. If there was a directory there, we should actually leave it alone for the gitlink case, because it is up to the *subproject* checkout (not the superproject checkout) to handle any issues within that subdirectory!

So I think the *right* patch (on top of the one I send out) is actually just this:

		Linus
---
Show changes to entry.c +3 −0
diff --git a/entry.c b/entry.c
index 9545e89..50ffae4 100644
--- a/entry.c
+++ b/entry.c
@@ -195,6 +195,9 @@ int checkout_entry(struct cache_entry *ce, struct checkout *state, char *topath)
 		 */
 		unlink(path);
 		if (S_ISDIR(st.st_mode)) {
+			/* If it is a gitlink, leave it alone! */
+			if (S_ISDIRLNK(ntohl(ce->ce_mode)))
+				return 0;
 			if (!state->force)
 				return error("%s is a directory", path);
 			remove_subtree(path);

← back to recent threads