threads / patch / 44488

v3, 4 partsSpeedup finding of unpushed submodules

Subject: [PATCH v3 0/4] Speedup finding of unpushed submodules

## tl;dr

14 messages between Nov 15, 2016 and Nov 16, 2016. Diffs are folded; open one to read it.

replies: 13people: 4as markdown or json

Heiko Voigt· Nov 15, 2016, 14:56 UTC · lore
You can find the second iteration of this series here:
http://public-inbox.org/git/cover.1475851621.git.hvoigt@hvoigt.net/

All mentioned issues should be fixed. I put the NEEDSWORK comment in a seperate patch since it seemed to me as if we did not fully agree on that. So in case we decide against it we can just drop that patch.

Cheers Heiko
Heiko Voigt (4):
  serialize collection of changed submodules
  serialize collection of refs that contain submodule changes
  batch check whether submodule needs pushing into one call
  submodule_needs_pushing() NEEDSWORK when we can not answer this
    question
 submodule.c | 120 +++++++++++++++++++++++++++++++++++++++++++++++-------------
 submodule.h |   5 ++-
 transport.c |  29 +++++++++++----
 3 files changed, 118 insertions(+), 36 deletions(-)
-- 
2.10.1.386.gc503e45
Heiko Voigt· Nov 15, 2016, 14:56 UTC · re: Heiko Voigt · lore

[PATCH v3 3/4] batch check whether submodule needs pushing into one call

We run a command for each sha1 change in a submodule. This is unnecessary since we can simply batch all sha1's we want to check into one command. Lets do it so we can speedup the check when many submodule changes are in need of checking.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
---
 submodule.c | 63 ++++++++++++++++++++++++++++++++-----------------------------
 1 file changed, 33 insertions(+), 30 deletions(-)
Show changes to submodule.c +33 −30
diff --git a/submodule.c b/submodule.c
index 769d666..e1196fd 100644
--- a/submodule.c
+++ b/submodule.c
@@ -507,27 +507,49 @@ static int append_sha1_to_argv(const unsigned char sha1[20], void *data)
 	return 0;
 }
 
-static int submodule_needs_pushing(const char *path, const unsigned char sha1[20])
+static int check_has_commit(const unsigned char sha1[20], void *data)
 {
-	if (add_submodule_odb(path) || !lookup_commit_reference(sha1))
+	int *has_commit = (int *) data;
+
+	if (!lookup_commit_reference(sha1))
+		*has_commit = 0;
+
+	return 0;
+}
+
+static int submodule_has_commits(const char *path, struct sha1_array *commits)
+{
+	int has_commit = 1;
+
+	if (add_submodule_odb(path))
+		return 0;
+
+	sha1_array_for_each_unique(commits, check_has_commit, &has_commit);
+	return has_commit;
+}
+
+static int submodule_needs_pushing(const char *path, struct sha1_array *commits)
+{
+	if (!submodule_has_commits(path, commits))
 		return 0;
 
 	if (for_each_remote_ref_submodule(path, has_remote, NULL) > 0) {
 		struct child_process cp = CHILD_PROCESS_INIT;
-		const char *argv[] = {"rev-list", NULL, "--not", "--remotes", "-n", "1" , NULL};
 		struct strbuf buf = STRBUF_INIT;
 		int needs_pushing = 0;
 
-		argv[1] = sha1_to_hex(sha1);
-		cp.argv = argv;
+		argv_array_push(&cp.args, "rev-list");
+		sha1_array_for_each_unique(commits, append_sha1_to_argv, &cp.args);
+		argv_array_pushl(&cp.args, "--not", "--remotes", "-n", "1" , NULL);
+
 		prepare_submodule_repo_env(&cp.env_array);
 		cp.git_cmd = 1;
 		cp.no_stdin = 1;
 		cp.out = -1;
 		cp.dir = path;
 		if (start_command(&cp))
-			die("Could not run 'git rev-list %s --not --remotes -n 1' command in submodule %s",
-				sha1_to_hex(sha1), path);
+			die("Could not run 'git rev-list <commits> --not --remotes -n 1' command in submodule %s",
+					path);
 		if (strbuf_read(&buf, cp.out, 41))
 			needs_pushing = 1;
 		finish_command(&cp);
@@ -582,23 +604,6 @@ static void find_unpushed_submodule_commits(struct commit *commit,
 	diff_tree_combined_merge(commit, 1, &rev);
 }
 
-struct collect_submodule_from_sha1s_data {
-	char *submodule_path;
-	struct string_list *needs_pushing;
-};
-
-static int collect_submodules_from_sha1s(const unsigned char sha1[20],
-		void *data)
-{
-	struct collect_submodule_from_sha1s_data *me =
-		(struct collect_submodule_from_sha1s_data *) data;
-
-	if (submodule_needs_pushing(me->submodule_path, sha1))
-		string_list_insert(me->needs_pushing, me->submodule_path);
-
-	return 0;
-}
-
 static void free_submodules_sha1s(struct string_list *submodules)
 {
 	struct string_list_item *item;
@@ -635,12 +640,10 @@ int find_unpushed_submodules(struct sha1_array *commits,
 	argv_array_clear(&argv);
 
 	for_each_string_list_item(submodule, &submodules) {
-		struct collect_submodule_from_sha1s_data data;
-		data.submodule_path = submodule->string;
-		data.needs_pushing = needs_pushing;
-		sha1_array_for_each_unique((struct sha1_array *) submodule->util,
-				collect_submodules_from_sha1s,
-				&data);
+		struct sha1_array *commits = (struct sha1_array *) submodule->util;
+
+		if (submodule_needs_pushing(submodule->string, commits))
+			string_list_insert(needs_pushing, submodule->string);
 	}
 	free_submodules_sha1s(&submodules);
 
-- 
2.10.1.386.gc503e45
Stefan Beller· Nov 15, 2016, 22:28 UTC · re: Heiko Voigt · lore

Re: [PATCH v3 3/4] batch check whether submodule needs pushing into one call

On Tue, Nov 15, 2016 at 6:56 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
Show 5 quoted lines
> -static int submodule_needs_pushing(const char *path, const unsigned char sha1[20])
> +static int check_has_commit(const unsigned char sha1[20], void *data)
>  {
> -       if (add_submodule_odb(path) || !lookup_commit_reference(sha1))
> +       int *has_commit = (int *) data;
nit: just as prior patches ;) void* can be cast implicitly.
Heiko Voigt· Nov 16, 2016, 14:29 UTC · re: Stefan Beller · lore

Re: [PATCH v3 3/4] batch check whether submodule needs pushing into one call

On Tue, Nov 15, 2016 at 02:28:31PM -0800, Stefan Beller wrote:
Show 9 quoted lines
> On Tue, Nov 15, 2016 at 6:56 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> 
> > -static int submodule_needs_pushing(const char *path, const unsigned char sha1[20])
> > +static int check_has_commit(const unsigned char sha1[20], void *data)
> >  {
> > -       if (add_submodule_odb(path) || !lookup_commit_reference(sha1))
> > +       int *has_commit = (int *) data;
> 
> nit: just as prior patches ;) void* can be cast implicitly.
Even though its just a nit: Will remove all the void casts. :)
Cheers Heiko
Heiko Voigt· Nov 15, 2016, 14:56 UTC · re: Heiko Voigt · lore

[PATCH v3 2/4] serialize collection of refs that contain submodule changes

We are iterating over each pushed ref and want to check whether it contains changes to submodules. Instead of immediately checking each ref lets first collect them and then do the check for all of them in one revision walk.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
---
 submodule.c | 35 ++++++++++++++++++++---------------
 submodule.h |  5 +++--
 transport.c | 29 +++++++++++++++++++++--------
 3 files changed, 44 insertions(+), 25 deletions(-)
Show changes to 3 files +44 −25

submodule.c, submodule.h, transport.c

diff --git a/submodule.c b/submodule.c
index b91585e..769d666 100644
--- a/submodule.c
+++ b/submodule.c
@@ -500,6 +500,13 @@ static int has_remote(const char *refname, const struct object_id *oid,
 	return 1;
 }
 
+static int append_sha1_to_argv(const unsigned char sha1[20], void *data)
+{
+	struct argv_array *argv = (struct argv_array *) data;
+	argv_array_push(argv, sha1_to_hex(sha1));
+	return 0;
+}
+
 static int submodule_needs_pushing(const char *path, const unsigned char sha1[20])
 {
 	if (add_submodule_odb(path) || !lookup_commit_reference(sha1))
@@ -600,25 +607,24 @@ static void free_submodules_sha1s(struct string_list *submodules)
 	string_list_clear(submodules, 1);
 }
 
-int find_unpushed_submodules(unsigned char new_sha1[20],
+int find_unpushed_submodules(struct sha1_array *commits,
 		const char *remotes_name, struct string_list *needs_pushing)
 {
 	struct rev_info rev;
 	struct commit *commit;
-	const char *argv[] = {NULL, NULL, "--not", "NULL", NULL};
-	int argc = ARRAY_SIZE(argv) - 1;
-	char *sha1_copy;
 	struct string_list submodules = STRING_LIST_INIT_DUP;
 	struct string_list_item *submodule;
+	struct argv_array argv = ARGV_ARRAY_INIT;
 
-	struct strbuf remotes_arg = STRBUF_INIT;
-
-	strbuf_addf(&remotes_arg, "--remotes=%s", remotes_name);
 	init_revisions(&rev, NULL);
-	sha1_copy = xstrdup(sha1_to_hex(new_sha1));
-	argv[1] = sha1_copy;
-	argv[3] = remotes_arg.buf;
-	setup_revisions(argc, argv, &rev, NULL);
+
+	/* argv.argv[0] will be ignored by setup_revisions */
+	argv_array_push(&argv, "find_unpushed_submodules");
+	sha1_array_for_each_unique(commits, append_sha1_to_argv, &argv);
+	argv_array_push(&argv, "--not");
+	argv_array_pushf(&argv, "--remotes=%s", remotes_name);
+
+	setup_revisions(argv.argc, argv.argv, &rev, NULL);
 	if (prepare_revision_walk(&rev))
 		die("revision walk setup failed");
 
@@ -626,8 +632,7 @@ int find_unpushed_submodules(unsigned char new_sha1[20],
 		find_unpushed_submodule_commits(commit, &submodules);
 
 	reset_revision_walk();
-	free(sha1_copy);
-	strbuf_release(&remotes_arg);
+	argv_array_clear(&argv);
 
 	for_each_string_list_item(submodule, &submodules) {
 		struct collect_submodule_from_sha1s_data data;
@@ -664,12 +669,12 @@ static int push_submodule(const char *path)
 	return 1;
 }
 
-int push_unpushed_submodules(unsigned char new_sha1[20], const char *remotes_name)
+int push_unpushed_submodules(struct sha1_array *commits, const char *remotes_name)
 {
 	int i, ret = 1;
 	struct string_list needs_pushing = STRING_LIST_INIT_DUP;
 
-	if (!find_unpushed_submodules(new_sha1, remotes_name, &needs_pushing))
+	if (!find_unpushed_submodules(commits, remotes_name, &needs_pushing))
 		return 1;
 
 	for (i = 0; i < needs_pushing.nr; i++) {
diff --git a/submodule.h b/submodule.h
index d9e197a..9454806 100644
--- a/submodule.h
+++ b/submodule.h
@@ -3,6 +3,7 @@
 
 struct diff_options;
 struct argv_array;
+struct sha1_array;
 
 enum {
 	RECURSE_SUBMODULES_CHECK = -4,
@@ -62,9 +63,9 @@ int submodule_uses_gitfile(const char *path);
 int ok_to_remove_submodule(const char *path);
 int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],
 		    const unsigned char a[20], const unsigned char b[20], int search);
-int find_unpushed_submodules(unsigned char new_sha1[20], const char *remotes_name,
+int find_unpushed_submodules(struct sha1_array *commits, const char *remotes_name,
 		struct string_list *needs_pushing);
-int push_unpushed_submodules(unsigned char new_sha1[20], const char *remotes_name);
+int push_unpushed_submodules(struct sha1_array *commits, const char *remotes_name);
 void connect_work_tree_and_git_dir(const char *work_tree, const char *git_dir);
 int parallel_submodules(void);
 
diff --git a/transport.c b/transport.c
index d57e8de..f482869 100644
--- a/transport.c
+++ b/transport.c
@@ -949,23 +949,36 @@ int transport_push(struct transport *transport,
 
 		if ((flags & TRANSPORT_RECURSE_SUBMODULES_ON_DEMAND) && !is_bare_repository()) {
 			struct ref *ref = remote_refs;
+			struct sha1_array commits = SHA1_ARRAY_INIT;
+
 			for (; ref; ref = ref->next)
-				if (!is_null_oid(&ref->new_oid) &&
-				    !push_unpushed_submodules(ref->new_oid.hash,
-					    transport->remote->name))
-				    die ("Failed to push all needed submodules!");
+				if (!is_null_oid(&ref->new_oid))
+					sha1_array_append(&commits, ref->new_oid.hash);
+
+			if (!push_unpushed_submodules(&commits, transport->remote->name)) {
+				sha1_array_clear(&commits);
+				die("Failed to push all needed submodules!");
+			}
+			sha1_array_clear(&commits);
 		}
 
 		if ((flags & (TRANSPORT_RECURSE_SUBMODULES_ON_DEMAND |
 			      TRANSPORT_RECURSE_SUBMODULES_CHECK)) && !is_bare_repository()) {
 			struct ref *ref = remote_refs;
 			struct string_list needs_pushing = STRING_LIST_INIT_DUP;
+			struct sha1_array commits = SHA1_ARRAY_INIT;
 
 			for (; ref; ref = ref->next)
-				if (!is_null_oid(&ref->new_oid) &&
-				    find_unpushed_submodules(ref->new_oid.hash,
-					    transport->remote->name, &needs_pushing))
-					die_with_unpushed_submodules(&needs_pushing);
+				if (!is_null_oid(&ref->new_oid))
+					sha1_array_append(&commits, ref->new_oid.hash);
+
+			if (find_unpushed_submodules(&commits, transport->remote->name,
+						&needs_pushing)) {
+				sha1_array_clear(&commits);
+				die_with_unpushed_submodules(&needs_pushing);
+			}
+			string_list_clear(&needs_pushing, 0);
+			sha1_array_clear(&commits);
 		}
 
 		push_ret = transport->push_refs(transport, remote_refs, flags);
-- 
2.10.1.386.gc503e45
Stefan Beller· Nov 15, 2016, 22:20 UTC · re: Heiko Voigt · lore

Re: [PATCH v3 2/4] serialize collection of refs that contain submodule changes

On Tue, Nov 15, 2016 at 6:56 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
Show 8 quoted lines
> +++ b/submodule.c
> @@ -500,6 +500,13 @@ static int has_remote(const char *refname, const struct object_id *oid,
>         return 1;
>  }
>
> +static int append_sha1_to_argv(const unsigned char sha1[20], void *data)
> +{
> +       struct argv_array *argv = (struct argv_array *) data;
nit: no explicit cast needed when coming from a void pointer.
Heiko Voigt· Nov 15, 2016, 14:56 UTC · re: Heiko Voigt · lore

[PATCH v3 1/4] serialize collection of changed submodules

To check whether a submodule needs to be pushed we need to collect all changed submodules. Lets collect them first and then execute the possibly expensive test whether certain revisions are already pushed only once per submodule.

There is further potential for optimization since we can assemble one command and only issued that instead of one call for each remote ref in the submodule.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
---
 submodule.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 56 insertions(+), 4 deletions(-)
Show changes to submodule.c +56 −4
diff --git a/submodule.c b/submodule.c
index 6f7d883..b91585e 100644
--- a/submodule.c
+++ b/submodule.c
@@ -532,19 +532,34 @@ static int submodule_needs_pushing(const char *path, const unsigned char sha1[20
 	return 0;
 }
 
+static struct sha1_array *submodule_commits(struct string_list *submodules,
+					    const char *path)
+{
+	struct string_list_item *item;
+
+	item = string_list_insert(submodules, path);
+	if (item->util)
+		return (struct sha1_array *) item->util;
+
+	/* NEEDSWORK: should we have sha1_array_init()? */
+	item->util = xcalloc(1, sizeof(struct sha1_array));
+	return (struct sha1_array *) item->util;
+}
+
 static void collect_submodules_from_diff(struct diff_queue_struct *q,
 					 struct diff_options *options,
 					 void *data)
 {
 	int i;
-	struct string_list *needs_pushing = data;
+	struct string_list *submodules = data;
 
 	for (i = 0; i < q->nr; i++) {
 		struct diff_filepair *p = q->queue[i];
+		struct sha1_array *commits;
 		if (!S_ISGITLINK(p->two->mode))
 			continue;
-		if (submodule_needs_pushing(p->two->path, p->two->oid.hash))
-			string_list_insert(needs_pushing, p->two->path);
+		commits = submodule_commits(submodules, p->two->path);
+		sha1_array_append(commits, p->two->oid.hash);
 	}
 }
 
@@ -560,6 +575,31 @@ static void find_unpushed_submodule_commits(struct commit *commit,
 	diff_tree_combined_merge(commit, 1, &rev);
 }
 
+struct collect_submodule_from_sha1s_data {
+	char *submodule_path;
+	struct string_list *needs_pushing;
+};
+
+static int collect_submodules_from_sha1s(const unsigned char sha1[20],
+		void *data)
+{
+	struct collect_submodule_from_sha1s_data *me =
+		(struct collect_submodule_from_sha1s_data *) data;
+
+	if (submodule_needs_pushing(me->submodule_path, sha1))
+		string_list_insert(me->needs_pushing, me->submodule_path);
+
+	return 0;
+}
+
+static void free_submodules_sha1s(struct string_list *submodules)
+{
+	struct string_list_item *item;
+	for_each_string_list_item(item, submodules)
+		sha1_array_clear((struct sha1_array *) item->util);
+	string_list_clear(submodules, 1);
+}
+
 int find_unpushed_submodules(unsigned char new_sha1[20],
 		const char *remotes_name, struct string_list *needs_pushing)
 {
@@ -568,6 +608,8 @@ int find_unpushed_submodules(unsigned char new_sha1[20],
 	const char *argv[] = {NULL, NULL, "--not", "NULL", NULL};
 	int argc = ARRAY_SIZE(argv) - 1;
 	char *sha1_copy;
+	struct string_list submodules = STRING_LIST_INIT_DUP;
+	struct string_list_item *submodule;
 
 	struct strbuf remotes_arg = STRBUF_INIT;
 
@@ -581,12 +623,22 @@ int find_unpushed_submodules(unsigned char new_sha1[20],
 		die("revision walk setup failed");
 
 	while ((commit = get_revision(&rev)) != NULL)
-		find_unpushed_submodule_commits(commit, needs_pushing);
+		find_unpushed_submodule_commits(commit, &submodules);
 
 	reset_revision_walk();
 	free(sha1_copy);
 	strbuf_release(&remotes_arg);
 
+	for_each_string_list_item(submodule, &submodules) {
+		struct collect_submodule_from_sha1s_data data;
+		data.submodule_path = submodule->string;
+		data.needs_pushing = needs_pushing;
+		sha1_array_for_each_unique((struct sha1_array *) submodule->util,
+				collect_submodules_from_sha1s,
+				&data);
+	}
+	free_submodules_sha1s(&submodules);
+
 	return needs_pushing->nr;
 }
 
-- 
2.10.1.386.gc503e45
Stefan Beller· Nov 15, 2016, 22:15 UTC · re: Heiko Voigt · lore

Re: [PATCH v3 1/4] serialize collection of changed submodules

On Tue, Nov 15, 2016 at 6:56 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
Show 14 quoted lines
> @@ -560,6 +575,31 @@ static void find_unpushed_submodule_commits(struct commit *commit,
>         diff_tree_combined_merge(commit, 1, &rev);
>  }
>
> +struct collect_submodule_from_sha1s_data {
> +       char *submodule_path;
> +       struct string_list *needs_pushing;
> +};
> +
> +static int collect_submodules_from_sha1s(const unsigned char sha1[20],
> +               void *data)
> +{
> +       struct collect_submodule_from_sha1s_data *me =
> +               (struct collect_submodule_from_sha1s_data *) data;
nit: no explicit cast needed when coming from void* ?
Heiko Voigt· Nov 15, 2016, 14:56 UTC · re: Heiko Voigt · lore

[PATCH v3 4/4] submodule_needs_pushing() NEEDSWORK when we can not answer this question

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
---
 submodule.c | 8 ++++++++
 1 file changed, 8 insertions(+)
Show changes to submodule.c +8 −0
diff --git a/submodule.c b/submodule.c
index e1196fd..29efee9 100644
--- a/submodule.c
+++ b/submodule.c
@@ -531,6 +531,14 @@ static int submodule_has_commits(const char *path, struct sha1_array *commits)
 static int submodule_needs_pushing(const char *path, struct sha1_array *commits)
 {
 	if (!submodule_has_commits(path, commits))
+		/* NEEDSWORK: The correct answer here is "We do not
+		 * know" instead of "No push needed". We currently
+		 * proceed pushing here as if the submodules commits are
+		 * available on a remote. Since we can not check the
+		 * remote availability for this submodule we should
+		 * consider changing this behavior to: Stop here and
+		 * tell the user how to skip this check if wanted.
+		 */
 		return 0;
 
 	if (for_each_remote_ref_submodule(path, has_remote, NULL) > 0) {
-- 
2.10.1.386.gc503e45
Stefan Beller· Nov 15, 2016, 22:39 UTC · re: Heiko Voigt · lore

Re: [PATCH v3 4/4] submodule_needs_pushing() NEEDSWORK when we can not answer this question

On Tue, Nov 15, 2016 at 6:56 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
Show 14 quoted lines
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
>  submodule.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/submodule.c b/submodule.c
> index e1196fd..29efee9 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -531,6 +531,14 @@ static int submodule_has_commits(const char *path, struct sha1_array *commits)
>  static int submodule_needs_pushing(const char *path, struct sha1_array *commits)
>  {
>         if (!submodule_has_commits(path, commits))
> +               /* NEEDSWORK: The correct answer here is "We do not
style nit:
/*
 * Usually we prefer comments with both the first and last line of the
comment "empty".
 */
/* or just a one liner */

AFAICT these are the only two modes that we prefer in Git. For a discussion of all the other style, enjoy Linus' guidance. ;) http://lkml.iu.edu/hypermail/linux/kernel/1607.1/00627.html

> "We do not know" ...

... because there is no way to check for us as we don't have the submodule commits.

    " We do consider it safe as no one in their sane mind would
    have changed the submodule pointers without having the
    submodule around. If a user did however change the submodules
    without having the submodule commits around, this indicates an
    expert who knows what they were doing."
Show 8 quoted lines
>   We currently
> +                * proceed pushing here as if the submodules commits are
> +                * available on a remote. Since we can not check the
> +                * remote availability for this submodule we should
> +                * consider changing this behavior to: Stop here and
> +                * tell the user how to skip this check if wanted.
> +                */
>                 return 0;

Thanks for adding the NEEDSWORK, I just wrote the above lines to clarify my thought process, not as a suggestion for change.

Overall the series looks good to me; the nits are minor IMHO.

Thanks, Stefan

Junio C Hamano· Nov 16, 2016, 00:13 UTC · re: Stefan Beller · lore

Re: [PATCH v3 4/4] submodule_needs_pushing() NEEDSWORK when we can not answer this question

Stefan Beller <sbeller@google.com> writes:
Show 10 quoted lines
>> "We do not know" ...
>
> ... because there is no way to check for us as we don't have the
> submodule commits.
>
>     " We do consider it safe as no one in their sane mind would
>     have changed the submodule pointers without having the
>     submodule around. If a user did however change the submodules
>     without having the submodule commits around, this indicates an
>     expert who knows what they were doing."

I didn't think it through myself to arrive at such a conclusion, but to me the above sounds like a sensible reasoning [*1*].

Show 11 quoted lines
>>   We currently
>> +                * proceed pushing here as if the submodules commits are
>> +                * available on a remote. Since we can not check the
>> +                * remote availability for this submodule we should
>> +                * consider changing this behavior to: Stop here and
>> +                * tell the user how to skip this check if wanted.
>> +                */
>>                 return 0;
>
> Thanks for adding the NEEDSWORK, I just wrote the above lines
> to clarify my thought process, not as a suggestion for change.

One thing I would suggest would be "Stop here, explain the situation and then tell the user how to skip". I am not convinced that it would _help_ users and make it _safer_ if we stopped here, though.

> Overall the series looks good to me; the nits are minor IMHO.
Ditto.
[Footnote]
*1* My version was more like "we do not know if they would get into
    a situation where they do not have enough submodule commits if
    we pushed our superproject, but more importantly, we DO KNOW
    that it would not help an iota if we pushed our submodule to
    them, so there is no point stopping the push of superproject
    saying 'no, no, no, you must push the submodule first'".
Heiko Voigt· Nov 16, 2016, 14:26 UTC · re: Junio C Hamano · lore

Re: [PATCH v3 4/4] submodule_needs_pushing() NEEDSWORK when we can not answer this question

On Tue, Nov 15, 2016 at 04:13:51PM -0800, Junio C Hamano wrote:
Show 15 quoted lines
> Stefan Beller <sbeller@google.com> writes:
> 
> >> "We do not know" ...
> >
> > ... because there is no way to check for us as we don't have the
> > submodule commits.
> >
> >     " We do consider it safe as no one in their sane mind would
> >     have changed the submodule pointers without having the
> >     submodule around. If a user did however change the submodules
> >     without having the submodule commits around, this indicates an
> >     expert who knows what they were doing."
> 
> I didn't think it through myself to arrive at such a conclusion, but
> to me the above sounds like a sensible reasoning [*1*].

I think you have a point here. If I rephrase it like this: "We do consider it safe as no one in their sane mind *could* have changed the submodule pointers without having the submodule around..."

Since its actually hard to create such a situation without the submodule commit around I agree here.

Show 6 quoted lines
> *1* My version was more like "we do not know if they would get into
>     a situation where they do not have enough submodule commits if
>     we pushed our superproject, but more importantly, we DO KNOW
>     that it would not help an iota if we pushed our submodule to
>     them, so there is no point stopping the push of superproject
>     saying 'no, no, no, you must push the submodule first'".

Yes saying that would be wrong. I was rather suggesting that we tell the user that we could not find the submodule commits to and that if he wants to proceed he should either pass --recurse-submodules=no or initialize the submodule.

But I think the above reasoning obsoletes my suggestion. I would adjust the comment accordingly but still keep the patch so we have documentation that this behavior is on purpose.

Cheers Heiko
Stefan Beller· Nov 15, 2016, 17:43 UTC · re: Heiko Voigt · lore

Re: [PATCH v3 0/4] Speedup finding of unpushed submodules

On Tue, Nov 15, 2016 at 6:56 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
Show 10 quoted lines
> You can find the second iteration of this series here:
>
> http://public-inbox.org/git/cover.1475851621.git.hvoigt@hvoigt.net/
>
> All mentioned issues should be fixed. I put the NEEDSWORK comment in a
> seperate patch since it seemed to me as if we did not fully agree on
> that. So in case we decide against it we can just drop that patch.
>
> Cheers Heiko
>
+cc Brandon who started building a series on top of yours.
Brandon Williams· Nov 15, 2016, 17:52 UTC · re: Stefan Beller · lore

Re: [PATCH v3 0/4] Speedup finding of unpushed submodules

On 11/15, Stefan Beller wrote:
Show 13 quoted lines
> On Tue, Nov 15, 2016 at 6:56 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > You can find the second iteration of this series here:
> >
> > http://public-inbox.org/git/cover.1475851621.git.hvoigt@hvoigt.net/
> >
> > All mentioned issues should be fixed. I put the NEEDSWORK comment in a
> > seperate patch since it seemed to me as if we did not fully agree on
> > that. So in case we decide against it we can just drop that patch.
> >
> > Cheers Heiko
> >
> 
> +cc Brandon who started building a series on top of yours.

I don't think there should be too much I'll have to change with my series, I'll just rebase against these changes once Junio updates his branch.

If you want to take a look at my series its here: https://public-inbox.org/git/1479172735-698-1-git-send-email-bmwill@google.com/

Thanks for the heads up Stefan.
-- 
Brandon Williams

← back to recent threads