threads / patch / 34452

patch, 7 partsFirst class shallow clone

Subject: [PATCH 0/7] First class shallow clone

## tl;dr

44 messages between Jul 17, 2013 and Jul 24, 2013. Diffs are folded; open one to read it.

replies: 43people: 4as markdown or json

Nguyễn Thái Ngọc Duy· Jul 17, 2013, 12:47 UTC · lore

This is probably the first attempt to treat shallow clones just like ordinary ones. Which means you can push or fetch/clone between any two repos, regardless of their shallow status. There are two purposes behind this:

 - to make local/shallow clone <-> (complete) upstream repo workflow
   smoother, the note about shallow clone limitation in git-clone.txt
   can be removed
 - to make it possible for upstream to provide a lightweight repo that
   others can use. For example, big repos with lots of activities can
   be split into new base repo that only contains the work of maximum one
   year and a complete repo mostly for archive.

This is a naive approach. I might overlook something again, which is why I publish it early to get more eyes on it.

The idea is simple: in shallow case, we provide the pack _and_ .git/shallow file to the other end. The other end will setup extra grafting to make sure the updated repo is "complete". More in individual patch messages.

There might be issues with generating optimum pack for transfer when both ends are shallow.. There's also an interesting issue, whether we can take advantage of commit bitmaps in shallow clones if they are more widely used.. We also might need a config key to protect a repo from becoming shallow by a fetch or push, if the repo is to be a backup one..

Nguyễn Thái Ngọc Duy (7):
  transport.h: remove send_pack prototype, already defined in send-pack.h
  {receive,upload}-pack: advertise shallow graft information
  connect.c: teach get_remote_heads to parse "shallow" lines
  Move setup_alternate_shallow and write_shallow_commits to shallow.c
  fetch-pack: support fetching from a shallow repository
  {send,receive}-pack: support pushing from a shallow clone
  send-pack: support pushing to a shallow clone
 Documentation/technical/pack-protocol.txt |   7 +-
 builtin/fetch-pack.c                      |   6 +-
 builtin/receive-pack.c                    |  54 ++++++++++++----
 builtin/send-pack.c                       |   7 +-
 cache.h                                   |   1 +
 commit.h                                  |   8 +++
 connect.c                                 |  12 +++-
 fetch-pack.c                              |  78 +++++++---------------
 fetch-pack.h                              |   1 +
 remote-curl.c                             |   2 +-
 send-pack.c                               |  41 ++++++++++--
 send-pack.h                               |   4 +-
 shallow.c                                 | 103 ++++++++++++++++++++++++++++++
 t/t5536-fetch-shallow.sh (new +x)         |  75 ++++++++++++++++++++++
 t/t5537-push-shallow.sh (new +x)          |  85 ++++++++++++++++++++++++
 transport.c                               |  14 ++--
 transport.h                               |   6 --
 upload-pack.c                             |   3 +-
 18 files changed, 414 insertions(+), 93 deletions(-)
 create mode 100755 t/t5536-fetch-shallow.sh
 create mode 100755 t/t5537-push-shallow.sh
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 17, 2013, 12:47 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH 1/7] transport.h: remove send_pack prototype, already defined in send-pack.h

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 transport.h | 6 ------
 1 file changed, 6 deletions(-)
Show changes to transport.h +0 −6
diff --git a/transport.h b/transport.h
index ea70ea7..3178dc9 100644
--- a/transport.h
+++ b/transport.h
@@ -182,10 +182,4 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 
 typedef void alternate_ref_fn(const struct ref *, void *);
 extern void for_each_alternate_ref(alternate_ref_fn, void *);
-
-struct send_pack_args;
-extern int send_pack(struct send_pack_args *args,
-		     int fd[], struct child_process *conn,
-		     struct ref *remote_refs,
-		     struct extra_have_objects *extra_have);
 #endif
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 17, 2013, 12:47 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH 2/7] {receive,upload}-pack: advertise shallow graft information

If either receive-pack or upload-pack is called on a shallow repository, shallow graft points will be sent after the ref advertisement (but before the packet flush).

This breaks the protocol for all clients trying to push to a shallow repo, or fetch from one. Which is basically the same end result as today's "is_repository_shallow() && die()" in receive-pack and upload-pack. New clients will be made aware of shallow upstream and can make use of this information.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/technical/pack-protocol.txt |  3 +++
 builtin/receive-pack.c                    |  5 ++---
 commit.h                                  |  1 +
 shallow.c                                 | 16 ++++++++++++++++
 upload-pack.c                             |  3 +--
 5 files changed, 23 insertions(+), 5 deletions(-)
Show changes to 5 files +23 −5

Documentation/technical/pack-protocol.txt, builtin/receive-pack.c, commit.h, shallow.c, upload-pack.c

diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index b898e97..eb8edd1 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -161,6 +161,7 @@ MUST peel the ref if it's an annotated tag.
 
 ----
   advertised-refs  =  (no-refs / list-of-refs)
+		      *shallow
 		      flush-pkt
 
   no-refs          =  PKT-LINE(zero-id SP "capabilities^{}"
@@ -174,6 +175,8 @@ MUST peel the ref if it's an annotated tag.
   other-tip        =  obj-id SP refname LF
   other-peeled     =  obj-id SP refname "^{}" LF
 
+  shallow          =  PKT-LINE("shallow" SP obj-id)
+
   capability-list  =  capability *(SP capability)
   capability       =  1*(LC_ALPHA / DIGIT / "-" / "_")
   LC_ALPHA         =  %x61-7A
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index e3eb5fc..2d8e19b 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -176,6 +176,8 @@ static void write_head_info(void)
 	if (!sent_capabilities)
 		show_ref("capabilities^{}", null_sha1);
 
+	advertise_shallow_grafts(1);
+
 	/* EOF */
 	packet_flush(1);
 }
@@ -990,9 +992,6 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 	if (!enter_repo(dir, 0))
 		die("'%s' does not appear to be a git repository", dir);
 
-	if (is_repository_shallow())
-		die("attempt to push into a shallow repository");
-
 	git_config(receive_pack_config, NULL);
 
 	if (0 <= transfer_unpack_limit)
diff --git a/commit.h b/commit.h
index 4d452dc..e0688c3 100644
--- a/commit.h
+++ b/commit.h
@@ -187,6 +187,7 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
 		int depth, int shallow_flag, int not_shallow_flag);
 extern void check_shallow_file_for_update(void);
 extern void set_alternate_shallow_file(const char *path);
+extern void advertise_shallow_grafts(int);
 
 int is_descendant_of(struct commit *, struct commit_list *);
 int in_merge_bases(struct commit *, struct commit *);
diff --git a/shallow.c b/shallow.c
index cbe2526..ccdfefc 100644
--- a/shallow.c
+++ b/shallow.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "commit.h"
 #include "tag.h"
+#include "pkt-line.h"
 
 static int is_shallow = -1;
 static struct stat shallow_stat;
@@ -146,3 +147,18 @@ void check_shallow_file_for_update(void)
 		   )
 		die("shallow file was changed during fetch");
 }
+
+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)
+{
+	int fd = *(int*)cb;
+	if (graft->nr_parent == -1)
+		packet_write(fd, "shallow %s\n", sha1_to_hex(graft->sha1));
+	return 0;
+}
+
+void advertise_shallow_grafts(int fd)
+{
+	if (!is_repository_shallow())
+		return;
+	for_each_commit_graft(advertise_shallow_grafts_cb, &fd);
+}
diff --git a/upload-pack.c b/upload-pack.c
index 127e59a..6cefe43 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -766,6 +766,7 @@ static void upload_pack(void)
 		reset_timeout();
 		head_ref_namespaced(send_ref, NULL);
 		for_each_namespaced_ref(send_ref, NULL);
+		advertise_shallow_grafts(1);
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(mark_our_ref, NULL);
@@ -837,8 +838,6 @@ int main(int argc, char **argv)
 
 	if (!enter_repo(dir, strict))
 		die("'%s' does not appear to be a git repository", dir);
-	if (is_repository_shallow())
-		die("attempt to fetch/clone from a shallow repository");
 	git_config(upload_pack_config, NULL);
 	upload_pack();
 	return 0;
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 17, 2013, 12:47 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH 3/7] connect.c: teach get_remote_heads to parse "shallow" lines

No callers pass a non-empty pointer as shallow_points at this stage. As a result, all clients still refuse to talk to shallow repository on the other end.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/fetch-pack.c |  2 +-
 builtin/send-pack.c  |  2 +-
 cache.h              |  1 +
 connect.c            | 12 +++++++++++-
 remote-curl.c        |  2 +-
 transport.c          |  7 ++++---
 6 files changed, 19 insertions(+), 7 deletions(-)
Show changes to 6 files +19 −7

builtin/fetch-pack.c, builtin/send-pack.c, cache.h, connect.c, remote-curl.c, transport.c

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index aba4465..080e599 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -144,7 +144,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 				   args.verbose ? CONNECT_VERBOSE : 0);
 	}
 
-	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL);
+	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
 
 	ref = fetch_pack(&args, fd, conn, ref, dest,
 			 sought, nr_sought, pack_lockfile_ptr);
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 152c4ea..f0ad0ce 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -207,7 +207,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 
 	memset(&extra_have, 0, sizeof(extra_have));
 
-	get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);
+	get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);
 
 	transport_verify_remote_names(nr_refspecs, refspecs);
 
diff --git a/cache.h b/cache.h
index dd0fb33..7665e03 100644
--- a/cache.h
+++ b/cache.h
@@ -1091,6 +1091,7 @@ struct extra_have_objects {
 };
 extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 				     struct ref **list, unsigned int flags,
+				     struct extra_have_objects *,
 				     struct extra_have_objects *);
 extern int server_supports(const char *feature);
 extern int parse_feature_request(const char *features, const char *feature);
diff --git a/connect.c b/connect.c
index a0783d4..1cba424 100644
--- a/connect.c
+++ b/connect.c
@@ -64,7 +64,8 @@ static void die_initial_contact(int got_at_least_one_head)
  */
 struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 			      struct ref **list, unsigned int flags,
-			      struct extra_have_objects *extra_have)
+			      struct extra_have_objects *extra_have,
+			      struct extra_have_objects *shallow_points)
 {
 	int got_at_least_one_head = 0;
 
@@ -89,6 +90,15 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 		if (len > 4 && !prefixcmp(buffer, "ERR "))
 			die("remote error: %s", buffer + 4);
 
+		if (len == 48 && !prefixcmp(buffer, "shallow ")) {
+			if (get_sha1_hex(buffer + 8, old_sha1))
+				die("protocol error: expected shallow sha, got '%s'", buffer + 8);
+			if (!shallow_points)
+				die("repository on the other end cannot be shallow");
+			add_extra_have(shallow_points, old_sha1);
+			continue;
+		}
+
 		if (len < 42 || get_sha1_hex(buffer, old_sha1) || buffer[40] != ' ')
 			die("protocol error: expected sha/ref, got '%s'", buffer);
 		name = buffer + 41;
diff --git a/remote-curl.c b/remote-curl.c
index 5b3ce9e..c329bd3 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -86,7 +86,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
 {
 	struct ref *list = NULL;
 	get_remote_heads(-1, heads->buf, heads->len, &list,
-			 for_push ? REF_NORMAL : 0, NULL);
+			 for_push ? REF_NORMAL : 0, NULL, NULL);
 	return list;
 }
 
diff --git a/transport.c b/transport.c
index e15db98..10a8cb8 100644
--- a/transport.c
+++ b/transport.c
@@ -509,7 +509,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 
 	connect_setup(transport, for_push, 0);
 	get_remote_heads(data->fd[0], NULL, 0, &refs,
-			 for_push ? REF_NORMAL : 0, &data->extra_have);
+			 for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
 	data->got_remote_heads = 1;
 
 	return refs;
@@ -539,7 +539,8 @@ static int fetch_refs_via_pack(struct transport *transport,
 
 	if (!data->got_remote_heads) {
 		connect_setup(transport, 0, 0);
-		get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0, NULL);
+		get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
+				 NULL, NULL);
 		data->got_remote_heads = 1;
 	}
 
@@ -799,7 +800,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 		struct ref *tmp_refs;
 		connect_setup(transport, 1, 0);
 
-		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL);
+		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
 		data->got_remote_heads = 1;
 	}
 
-- 
1.8.2.83.gc99314b
Junio C Hamano· Jul 20, 2013, 03:27 UTC · re: Nguyễn Thái Ngọc Duy · lore

Re: [PATCH 3/7] connect.c: teach get_remote_heads to parse "shallow" lines

Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
Show 10 quoted lines
> diff --git a/cache.h b/cache.h
> index dd0fb33..7665e03 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1091,6 +1091,7 @@ struct extra_have_objects {
>  };
>  extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
>  				     struct ref **list, unsigned int flags,
> +				     struct extra_have_objects *,
>  				     struct extra_have_objects *);

When it is the sole parameter of a specific type (i.e. not "char *", int, size_t, etc. but an application specific structure like "struct extra_have_objects *"), it is perfectly fine (and even preferrable) to omit the parameter name from the declaration, as it is clear what the parameter is and means.

But when you add another of the same type, you need to give both of them a descriptive name (e.g. the ones you use in the definition).

Otherwise, somebody who wants to write a caller cannot tell which "struct extra_have_objects *" parameter is the extra and which one is the shallow graft points list.

Nguyễn Thái Ngọc Duy· Jul 17, 2013, 12:47 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH 4/7] Move setup_alternate_shallow and write_shallow_commits to shallow.c

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 commit.h     |  3 +++
 fetch-pack.c | 53 +----------------------------------------------------
 shallow.c    | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+), 52 deletions(-)
Show changes to 3 files +57 −52

commit.h, fetch-pack.c, shallow.c

diff --git a/commit.h b/commit.h
index e0688c3..678fa20 100644
--- a/commit.h
+++ b/commit.h
@@ -188,6 +188,9 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
 extern void check_shallow_file_for_update(void);
 extern void set_alternate_shallow_file(const char *path);
 extern void advertise_shallow_grafts(int);
+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);
+extern void setup_alternate_shallow(struct lock_file *shallow_lock,
+				    const char **alternate_shallow_file);
 
 int is_descendant_of(struct commit *, struct commit_list *);
 int in_merge_bases(struct commit *, struct commit *);
diff --git a/fetch-pack.c b/fetch-pack.c
index abe5ffb..dc71a2b 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -185,36 +185,6 @@ static void consume_shallow_list(struct fetch_pack_args *args, int fd)
 	}
 }
 
-struct write_shallow_data {
-	struct strbuf *out;
-	int use_pack_protocol;
-	int count;
-};
-
-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
-{
-	struct write_shallow_data *data = cb_data;
-	const char *hex = sha1_to_hex(graft->sha1);
-	data->count++;
-	if (data->use_pack_protocol)
-		packet_buf_write(data->out, "shallow %s", hex);
-	else {
-		strbuf_addstr(data->out, hex);
-		strbuf_addch(data->out, '\n');
-	}
-	return 0;
-}
-
-static int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
-{
-	struct write_shallow_data data;
-	data.out = out;
-	data.use_pack_protocol = use_pack_protocol;
-	data.count = 0;
-	for_each_commit_graft(write_one_shallow, &data);
-	return data.count;
-}
-
 static enum ack_type get_ack(int fd, unsigned char *result_sha1)
 {
 	int len;
@@ -795,27 +765,6 @@ static int cmp_ref_by_name(const void *a_, const void *b_)
 	return strcmp(a->name, b->name);
 }
 
-static void setup_alternate_shallow(void)
-{
-	struct strbuf sb = STRBUF_INIT;
-	int fd;
-
-	check_shallow_file_for_update();
-	fd = hold_lock_file_for_update(&shallow_lock, git_path("shallow"),
-				       LOCK_DIE_ON_ERROR);
-	if (write_shallow_commits(&sb, 0)) {
-		if (write_in_full(fd, sb.buf, sb.len) != sb.len)
-			die_errno("failed to write to %s", shallow_lock.filename);
-		alternate_shallow_file = shallow_lock.filename;
-	} else
-		/*
-		 * is_repository_shallow() sees empty string as "no
-		 * shallow file".
-		 */
-		alternate_shallow_file = "";
-	strbuf_release(&sb);
-}
-
 static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 				 int fd[2],
 				 const struct ref *orig_ref,
@@ -896,7 +845,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 	if (args->stateless_rpc)
 		packet_flush(fd[1]);
 	if (args->depth > 0)
-		setup_alternate_shallow();
+		setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
 	if (get_pack(args, fd, pack_lockfile))
 		die("git fetch-pack: fetch failed.");
 
diff --git a/shallow.c b/shallow.c
index ccdfefc..ee9edd4 100644
--- a/shallow.c
+++ b/shallow.c
@@ -162,3 +162,56 @@ void advertise_shallow_grafts(int fd)
 		return;
 	for_each_commit_graft(advertise_shallow_grafts_cb, &fd);
 }
+
+struct write_shallow_data {
+	struct strbuf *out;
+	int use_pack_protocol;
+	int count;
+};
+
+static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
+{
+	struct write_shallow_data *data = cb_data;
+	const char *hex = sha1_to_hex(graft->sha1);
+	data->count++;
+	if (data->use_pack_protocol)
+		packet_buf_write(data->out, "shallow %s", hex);
+	else {
+		strbuf_addstr(data->out, hex);
+		strbuf_addch(data->out, '\n');
+	}
+	return 0;
+}
+
+int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
+{
+	struct write_shallow_data data;
+	data.out = out;
+	data.use_pack_protocol = use_pack_protocol;
+	data.count = 0;
+	for_each_commit_graft(write_one_shallow, &data);
+	return data.count;
+}
+
+void setup_alternate_shallow(struct lock_file *shallow_lock,
+			     const char **alternate_shallow_file)
+{
+	struct strbuf sb = STRBUF_INIT;
+	int fd;
+
+	check_shallow_file_for_update();
+	fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
+				       LOCK_DIE_ON_ERROR);
+	if (write_shallow_commits(&sb, 0)) {
+		if (write_in_full(fd, sb.buf, sb.len) != sb.len)
+			die_errno("failed to write to %s",
+				  shallow_lock->filename);
+		*alternate_shallow_file = shallow_lock->filename;
+	} else
+		/*
+		 * is_repository_shallow() sees empty string as "no
+		 * shallow file".
+		 */
+		*alternate_shallow_file = "";
+	strbuf_release(&sb);
+}
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 17, 2013, 12:47 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH 5/7] fetch-pack: support fetching from a shallow repository

upload-pack already advertises all shallow grafts if server repository is shallow. This information can be used to add more grafts to the client if the server sends commit chains down to its graft points.

If the server is shallow, before we receive the pack, we setup a temporary shallow file that contains both local graft points and the server's. This stops index-pack from going beyond server's graft points.

Only server graft points that do not have corresponding SHA-1s in local repo are added to the temp shallow file because we don't want to accidentally cut the client history because the server's is shorter. The client cutting can only happen when --depth is requested.

After index-pack finishes successfully, we write the temporary shallow down with one exception: unused graft points provided by the server are removed. We don't want those lying around and suddenly become active.

Note that in the "shallow -> shallow" case, the server might not have enough information to find common roots to create an optimum pack. It might send complete commit chains down to the graft points as a result. I don't think we can improve this, unless upload-pack somehow has access to a full repository.

"shallow -> shallow" case only makes sense when the upstream provides a stable shallow repo (e.g. make a cut every year or so and ask devs to all move to the new base). If the cloned repos are all based on a stable (shallow) upstream, the above problem is unlikely to happen.

A side effect of this change is we can now clone from a shallow repository. And a full repository may automatically become shallow if you fetch from a shallow repository.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/fetch-pack.c              |  6 ++--
 commit.h                          |  8 +++--
 fetch-pack.c                      | 27 +++++++++++---
 fetch-pack.h                      |  1 +
 shallow.c                         | 46 ++++++++++++++++++++----
 t/t5536-fetch-shallow.sh (new +x) | 75 +++++++++++++++++++++++++++++++++++++++
 transport.c                       |  8 +++--
 7 files changed, 153 insertions(+), 18 deletions(-)
 create mode 100755 t/t5536-fetch-shallow.sh
Show changes to 7 files +153 −18

builtin/fetch-pack.c, commit.h, fetch-pack.c, fetch-pack.h, shallow.c, t/t5536-fetch-shallow.sh, transport.c

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 080e599..b89d753 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -37,6 +37,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 	char **pack_lockfile_ptr = NULL;
 	struct child_process *conn;
 	struct fetch_pack_args args;
+	struct extra_have_objects shallow;
 
 	packet_trace_identity("fetch-pack");
 
@@ -144,10 +145,11 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 				   args.verbose ? CONNECT_VERBOSE : 0);
 	}
 
-	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
+	memset(&shallow, 0, sizeof(shallow));
+	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
 
 	ref = fetch_pack(&args, fd, conn, ref, dest,
-			 sought, nr_sought, pack_lockfile_ptr);
+			 sought, nr_sought, &shallow, pack_lockfile_ptr);
 	if (pack_lockfile) {
 		printf("lock %s\n", pack_lockfile);
 		fflush(stdout);
diff --git a/commit.h b/commit.h
index 678fa20..7faf0e4 100644
--- a/commit.h
+++ b/commit.h
@@ -188,9 +188,13 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
 extern void check_shallow_file_for_update(void);
 extern void set_alternate_shallow_file(const char *path);
 extern void advertise_shallow_grafts(int);
-extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);
+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
+				 struct extra_have_objects *extra,
+				 int remove_unused_grafts);
 extern void setup_alternate_shallow(struct lock_file *shallow_lock,
-				    const char **alternate_shallow_file);
+				    const char **alternate_shallow_file,
+				    struct extra_have_objects *extra,
+				    int rewrite);
 
 int is_descendant_of(struct commit *, struct commit_list *);
 int in_merge_bases(struct commit *, struct commit *);
diff --git a/fetch-pack.c b/fetch-pack.c
index dc71a2b..68b95a5 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args *args,
 	}
 
 	if (is_repository_shallow())
-		write_shallow_commits(&req_buf, 1);
+		write_shallow_commits(&req_buf, 1, NULL, 0);
 	if (args->depth > 0)
 		packet_buf_write(&req_buf, "deepen %d", args->depth);
 	packet_buf_flush(&req_buf);
@@ -769,6 +769,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 				 int fd[2],
 				 const struct ref *orig_ref,
 				 struct ref **sought, int nr_sought,
+				 struct extra_have_objects *shallow,
 				 char **pack_lockfile)
 {
 	struct ref *ref = copy_ref_list(orig_ref);
@@ -844,8 +845,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 
 	if (args->stateless_rpc)
 		packet_flush(fd[1]);
-	if (args->depth > 0)
-		setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
+	if (args->depth > 0 || shallow->nr)
+		setup_alternate_shallow(&shallow_lock, &alternate_shallow_file,
+					shallow, 0);
 	if (get_pack(args, fd, pack_lockfile))
 		die("git fetch-pack: fetch failed.");
 
@@ -922,6 +924,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 		       const struct ref *ref,
 		       const char *dest,
 		       struct ref **sought, int nr_sought,
+		       struct extra_have_objects *shallow,
 		       char **pack_lockfile)
 {
 	struct ref *ref_cpy;
@@ -934,14 +937,28 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 		packet_flush(fd[1]);
 		die("no matching remote head");
 	}
-	ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);
+	ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
+				shallow, pack_lockfile);
 
 	if (alternate_shallow_file) {
 		if (*alternate_shallow_file == '\0') { /* --unshallow */
 			unlink_or_warn(git_path("shallow"));
 			rollback_lock_file(&shallow_lock);
-		} else
+		} else {
+			/*
+			 *  The server is a shallow clone and it sends
+			 *  us all of its shallow grafts. Some may be
+			 *  needed if the server sends objects down to
+			 *  the bottom. Remove all unused grafts.
+			 */
+			if (shallow->nr) {
+				reprepare_packed_git();
+				setup_alternate_shallow(&shallow_lock,
+							&alternate_shallow_file,
+							shallow, 1);
+			}
 			commit_lock_file(&shallow_lock);
+		}
 	}
 
 	reprepare_packed_git();
diff --git a/fetch-pack.h b/fetch-pack.h
index 40f08ba..c55bf1c 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -32,6 +32,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 		       const char *dest,
 		       struct ref **sought,
 		       int nr_sought,
+		       struct extra_have_objects *shallow,
 		       char **pack_lockfile);
 
 #endif
diff --git a/shallow.c b/shallow.c
index ee9edd4..d909d95 100644
--- a/shallow.c
+++ b/shallow.c
@@ -183,26 +183,60 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
 	return 0;
 }
 
-int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
+			  struct extra_have_objects *extra,
+			  int remove_unused_grafts)
 {
 	struct write_shallow_data data;
+	int i;
 	data.out = out;
 	data.use_pack_protocol = use_pack_protocol;
 	data.count = 0;
 	for_each_commit_graft(write_one_shallow, &data);
+	if (!extra)
+		return data.count;
+
+	for (i = 0; i < extra->nr; i++) {
+		if (!remove_unused_grafts && has_sha1_file(extra->array[i]))
+			/*
+			 * The server may have even shorter history
+			 * than the client (e.g. the client has
+			 * A-B-C-D but the server only has C-D). We do
+			 * NOT want to cut client's history down to
+			 * C-D simply because the server is set up
+			 * so. If we don't have "C" and the server
+			 * sends C-D to us, then we set up a graft at
+			 * "C".
+			 */
+			continue;
+		if (remove_unused_grafts && !has_sha1_file(extra->array[i]))
+			continue;
+		strbuf_addstr(out, sha1_to_hex(extra->array[i]));
+		strbuf_addch(out, '\n');
+		data.count++;
+	}
 	return data.count;
 }
 
 void setup_alternate_shallow(struct lock_file *shallow_lock,
-			     const char **alternate_shallow_file)
+			     const char **alternate_shallow_file,
+			     struct extra_have_objects *extra,
+			     int rewrite)
 {
 	struct strbuf sb = STRBUF_INIT;
 	int fd;
 
-	check_shallow_file_for_update();
-	fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
-				       LOCK_DIE_ON_ERROR);
-	if (write_shallow_commits(&sb, 0)) {
+	if (!rewrite)
+		fd = hold_lock_file_for_update(shallow_lock,
+					       git_path("shallow"),
+					       LOCK_DIE_ON_ERROR);
+	else
+		fd = shallow_lock->fd;
+	if (write_shallow_commits(&sb, 0, extra, rewrite)) {
+		if (rewrite && (lseek(fd, 0, SEEK_SET) == -1 ||
+				ftruncate(fd, 0) == -1))
+			die_errno("unable to truncate the new shallow file %s",
+				  shallow_lock->filename);
 		if (write_in_full(fd, sb.buf, sb.len) != sb.len)
 			die_errno("failed to write to %s",
 				  shallow_lock->filename);
diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
new file mode 100755
index 0000000..599928f
--- /dev/null
+++ b/t/t5536-fetch-shallow.sh
@@ -0,0 +1,75 @@
+#!/bin/sh
+
+test_description='fetch/clone from a shallow clone'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	test_commit 1 &&
+	test_commit 2 &&
+	test_commit 3 &&
+	test_commit 4
+'
+
+test_expect_success 'setup shallow clone' '
+	git clone --no-local --depth=2 .git shallow &&
+	git --git-dir=shallow/.git log --format=%s >actual &&
+	cat <<EOF >expect &&
+4
+3
+EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'clone from shallow clone' '
+	git clone --no-local shallow shallow2 &&
+	(
+	cd shallow2 &&
+	git fsck &&
+	git log --format=%s >actual &&
+	cat <<EOF >expect &&
+4
+3
+EOF
+	test_cmp expect actual
+	)
+'
+
+test_expect_success 'fetch from shallow clone' '
+	(
+	cd shallow &&
+	test_commit 5
+	) &&
+	(
+	cd shallow2 &&
+	git fetch &&
+	git fsck &&
+	git log --format=%s origin/master >actual &&
+	cat <<EOF >expect &&
+5
+4
+3
+EOF
+	test_cmp expect actual
+	)
+'
+
+test_expect_success 'fetch --depth from shallow clone' '
+	(
+	cd shallow &&
+	test_commit 6
+	) &&
+	(
+	cd shallow2 &&
+	git fetch --depth=2 &&
+	git fsck &&
+	git log --format=%s origin/master >actual &&
+	cat <<EOF >expect &&
+6
+5
+EOF
+	test_cmp expect actual
+	)
+'
+
+test_done
diff --git a/transport.c b/transport.c
index 10a8cb8..17fef16 100644
--- a/transport.c
+++ b/transport.c
@@ -453,6 +453,7 @@ struct git_transport_data {
 	int fd[2];
 	unsigned got_remote_heads : 1;
 	struct extra_have_objects extra_have;
+	struct extra_have_objects shallow;
 };
 
 static int set_git_option(struct git_transport_options *opts,
@@ -509,7 +510,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 
 	connect_setup(transport, for_push, 0);
 	get_remote_heads(data->fd[0], NULL, 0, &refs,
-			 for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
+			 for_push ? REF_NORMAL : 0,
+			 &data->extra_have, &data->shallow);
 	data->got_remote_heads = 1;
 
 	return refs;
@@ -540,13 +542,13 @@ static int fetch_refs_via_pack(struct transport *transport,
 	if (!data->got_remote_heads) {
 		connect_setup(transport, 0, 0);
 		get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
-				 NULL, NULL);
+				 NULL, &data->shallow);
 		data->got_remote_heads = 1;
 	}
 
 	refs = fetch_pack(&args, data->fd, data->conn,
 			  refs_tmp ? refs_tmp : transport->remote_refs,
-			  dest, to_fetch, nr_heads,
+			  dest, to_fetch, nr_heads, &data->shallow,
 			  &transport->pack_lockfile);
 	close(data->fd[0]);
 	close(data->fd[1]);
-- 
1.8.2.83.gc99314b
Junio C Hamano· Jul 20, 2013, 03:17 UTC · re: Nguyễn Thái Ngọc Duy · lore

Re: [PATCH 5/7] fetch-pack: support fetching from a shallow repository

Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
> Only server graft points that do not have corresponding SHA-1s in
> local repo are added to the temp shallow file because we don't want to
> accidentally cut the client history because the server's is
> shorter. The client cutting can only happen when --depth is requested.

This "have corresponding SHA-1" is iffy from connectivity point of view, isn't it? That is,...

> +	for (i = 0; i < extra->nr; i++) {
> +		if (!remove_unused_grafts && has_sha1_file(extra->array[i]))

...shouldn't this require much more than "has_sha1_file()", like "has it, and we have everything behind it with respect to the shallow graft points we started with" or something?

Otherwise an isolated island that was left behind by an earlier aborted commit walker fetch may mislead us to believe that we have the complete history behind this commit object.

Nguyễn Thái Ngọc Duy· Jul 17, 2013, 12:47 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH 6/7] {send,receive}-pack: support pushing from a shallow clone

Pushing from a shallow clone using today's send-pack and receive-pack may work, if the transferred pack does not ends up at any graft points. If it does, recent receive-pack that does connectivity check will reject the push. If receive-pack is old, the upstream repo becomes corrupt.

The pack protocol is updated and send-pack now sends all shallow grafts before it sends the commands, if the repo is shallow. This protocol extension will break current receive-pack, which is intended, mostly to stop corrupting the upstream repo.

The receiver end, the newreceive-pack, does something similar to fetch-pack: it creates a temporary shallow file with grafts from send-pack, then receives the pack, and finally writes the refined shallow file down.

shadow file is not cleaned up after deleting (or force updating) a ref if that ref is the only way to reach the graft points. The reason is once we delete graft points, we can't recover. That may make reflog entries on server useless. Leave that for the administrators to decide when to clean up shadow file (maybe at repack/gc time).

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/technical/pack-protocol.txt |  4 +-
 builtin/receive-pack.c                    | 49 ++++++++++++++++++----
 send-pack.c                               |  3 ++
 t/t5537-push-shallow.sh (new +x)          | 67 +++++++++++++++++++++++++++++++
 4 files changed, 114 insertions(+), 9 deletions(-)
 create mode 100755 t/t5537-push-shallow.sh
Show changes to 4 files +114 −9

Documentation/technical/pack-protocol.txt, builtin/receive-pack.c, send-pack.c, t/t5537-push-shallow.sh

diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index eb8edd1..c73b62f 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -464,7 +464,9 @@ contain all the objects that the server will need to complete the new
 references.
 
 ----
-  update-request    =  command-list [pack-file]
+  update-request    =  *shallow command-list [pack-file]
+
+  shallow           =  PKT-LINE("shallow" SP obj-id)
 
   command-list      =  PKT-LINE(command NUL capability-list LF)
 		       *PKT-LINE(command LF)
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 2d8e19b..0537e26 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -41,6 +41,10 @@ static int auto_gc = 1;
 static const char *head_name;
 static void *head_name_to_free;
 static int sent_capabilities;
+static int shallow_changed;
+static const char* alternate_shallow_file;
+static struct lock_file shallow_lock;
+static struct extra_have_objects shallow;
 
 static enum deny_action parse_deny_action(const char *var, const char *value)
 {
@@ -751,6 +755,13 @@ static void execute_commands(struct command *commands, const char *unpacker_erro
 	}
 }
 
+static void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)
+{
+	ALLOC_GROW(extra->array, extra->nr + 1, extra->alloc);
+	hashcpy(&(extra->array[extra->nr][0]), sha1);
+	extra->nr++;
+}
+
 static struct command *read_head_info(void)
 {
 	struct command *commands = NULL;
@@ -765,6 +776,17 @@ static struct command *read_head_info(void)
 		line = packet_read_line(0, &len);
 		if (!line)
 			break;
+
+		if (len == 48 && !prefixcmp(line, "shallow ")) {
+			if (get_sha1_hex(line + 8, old_sha1))
+				die("protocol error: expected shallow sha, got '%s'", line + 8);
+			if (!has_sha1_file(old_sha1)) {
+				add_extra_have(&shallow, old_sha1);
+				shallow_changed = 1;
+			}
+			continue;
+		}
+
 		if (len < 83 ||
 		    line[40] != ' ' ||
 		    line[81] != ' ' ||
@@ -827,6 +849,12 @@ static const char *unpack(int err_fd)
 			    ? transfer_fsck_objects
 			    : 0);
 
+	if (shallow_changed)
+		setup_alternate_shallow(&shallow_lock,
+					&alternate_shallow_file,
+					&shallow, 0);
+
+
 	hdr_err = parse_pack_header(&hdr);
 	if (hdr_err) {
 		if (err_fd > 0)
@@ -854,9 +882,8 @@ static const char *unpack(int err_fd)
 		child.err = err_fd;
 		child.git_cmd = 1;
 		code = run_command(&child);
-		if (!code)
-			return NULL;
-		return "unpack-objects abnormal exit";
+		if (code)
+			return "unpack-objects abnormal exit";
 	} else {
 		const char *keeper[7];
 		int s, status, i = 0;
@@ -887,12 +914,18 @@ static const char *unpack(int err_fd)
 		pack_lockfile = index_pack_lockfile(ip.out);
 		close(ip.out);
 		status = finish_command(&ip);
-		if (!status) {
-			reprepare_packed_git();
-			return NULL;
-		}
-		return "index-pack abnormal exit";
+		if (status)
+			return "index-pack abnormal exit";
+		reprepare_packed_git();
+	}
+
+	if (shallow_changed) {
+		setup_alternate_shallow(&shallow_lock,
+					&alternate_shallow_file,
+					&shallow, 1);
+		commit_lock_file(&shallow_lock);
 	}
+	return NULL;
 }
 
 static const char *unpack_with_sideband(void)
diff --git a/send-pack.c b/send-pack.c
index 7d172ef..81d4b1c 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -212,6 +212,9 @@ int send_pack(struct send_pack_args *args,
 		return 0;
 	}
 
+	if (!args->dry_run && !args->stateless_rpc)
+		advertise_shallow_grafts(out);
+
 	/*
 	 * Finally, tell the other end!
 	 */
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
new file mode 100755
index 0000000..30fdbc8
--- /dev/null
+++ b/t/t5537-push-shallow.sh
@@ -0,0 +1,67 @@
+#!/bin/sh
+
+test_description='push from/to a shallow clone'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	test_commit 1 &&
+	test_commit 2 &&
+	test_commit 3 &&
+	test_commit 4
+'
+
+test_expect_success 'setup shallow clone' '
+	git clone --no-local --depth=2 .git shallow &&
+	git --git-dir=shallow/.git log --format=%s >actual &&
+	cat <<EOF >expect &&
+4
+3
+EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'push from shallow clone' '
+	(
+	cd shallow &&
+	test_commit 5 &&
+	git push ../.git +master:refs/remotes/shallow/master
+	) &&
+	git log --format=%s shallow/master >actual &&
+	git fsck &&
+	cat <<EOF >expect &&
+5
+4
+3
+2
+1
+EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'push from shallow clone, with grafted roots' '
+	git init shallow2 &&
+	(
+	cd shallow2 &&
+	test_commit a &&
+	test_commit b &&
+	test_commit c &&
+	git rev-parse b > .git/shallow &&
+	git log --format=%s >actual &&
+	cat <<EOF >expect &&
+c
+b
+EOF
+	test_cmp expect actual &&
+	git push ../.git +master:refs/remotes/shallow2/master
+	) &&
+	git log --format=%s shallow2/master >actual &&
+	git fsck &&
+	cat <<EOF >expect &&
+c
+b
+EOF
+	test_cmp expect actual
+'
+
+test_done
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 17, 2013, 12:47 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH 7/7] send-pack: support pushing to a shallow clone

When send-pack receives "shallow" lines from receive-pack, it knows the other end does not have a complete commit chains. It restrict itself to the commits that are not cut out by either end to make sure the result pack is usuable by receive-pack.

The same technique here, using setup_alternate_shallow() and --shallow-file, might simplify similar code in upload-pack.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/send-pack.c     |  7 +++++--
 send-pack.c             | 38 +++++++++++++++++++++++++++++++++-----
 send-pack.h             |  4 +++-
 t/t5537-push-shallow.sh | 18 ++++++++++++++++++
 transport.c             |  5 +++--
 5 files changed, 62 insertions(+), 10 deletions(-)
Show changes to 5 files +62 −10

builtin/send-pack.c, send-pack.c, send-pack.h, t/t5537-push-shallow.sh, transport.c

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index f0ad0ce..b177120 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -94,6 +94,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int fd[2];
 	struct child_process *conn;
 	struct extra_have_objects extra_have;
+	struct extra_have_objects shallow;
 	struct ref *remote_refs, *local_refs;
 	int ret;
 	int helper_status = 0;
@@ -206,8 +207,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	}
 
 	memset(&extra_have, 0, sizeof(extra_have));
+	memset(&shallow, 0, sizeof(shallow));
 
-	get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);
+	get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,
+			 &extra_have, &shallow);
 
 	transport_verify_remote_names(nr_refspecs, refspecs);
 
@@ -227,7 +230,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	set_ref_status_for_push(remote_refs, args.send_mirror,
 		args.force_update);
 
-	ret = send_pack(&args, fd, conn, remote_refs, &extra_have);
+	ret = send_pack(&args, fd, conn, remote_refs, &extra_have, &shallow);
 
 	if (helper_status)
 		print_helper_status(remote_refs);
diff --git a/send-pack.c b/send-pack.c
index 81d4b1c..08de681 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -27,14 +27,19 @@ static int feed_object(const unsigned char *sha1, int fd, int negative)
 /*
  * Make a pack stream and spit it out into file descriptor fd
  */
-static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *extra, struct send_pack_args *args)
+static int pack_objects(int fd, struct ref *refs,
+			struct extra_have_objects *extra,
+			struct extra_have_objects *extra_shallow,
+			struct send_pack_args *args)
 {
 	/*
 	 * The child becomes pack-objects --revs; we feed
 	 * the revision parameters to it via its stdin and
 	 * let its stdout go back to the other end.
 	 */
-	const char *argv[] = {
+	const char *av[] = {
+		"--shallow-file",
+		NULL,
 		"pack-objects",
 		"--all-progress-implied",
 		"--revs",
@@ -45,10 +50,27 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
 		NULL,
 		NULL,
 	};
+	const char **argv;
 	struct child_process po;
+	static struct lock_file shallow_lock;
+	const char *alternate_shallow_file = NULL;
 	int i;
 
-	i = 4;
+	if (extra_shallow->nr) {
+		memset(&shallow_lock, 0, sizeof(shallow_lock));
+		/* just to load up .git/shallow if exists */
+		is_repository_shallow();
+		setup_alternate_shallow(&shallow_lock,
+					&alternate_shallow_file,
+					extra_shallow,
+					0);
+		av[1] = alternate_shallow_file;
+		argv = av;
+		i = 6;
+	} else {
+		argv = &av[2];
+		i = 4;
+	}
 	if (args->use_thin_pack)
 		argv[i++] = "--thin";
 	if (args->use_ofs_delta)
@@ -100,6 +122,10 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
 
 	if (finish_command(&po))
 		return -1;
+
+	if (extra_shallow->nr)
+		rollback_lock_file(&shallow_lock);
+
 	return 0;
 }
 
@@ -176,7 +202,8 @@ static int sideband_demux(int in, int out, void *data)
 int send_pack(struct send_pack_args *args,
 	      int fd[], struct child_process *conn,
 	      struct ref *remote_refs,
-	      struct extra_have_objects *extra_have)
+	      struct extra_have_objects *extra_have,
+	      struct extra_have_objects *extra_shallow)
 {
 	int in = fd[0];
 	int out = fd[1];
@@ -294,7 +321,8 @@ int send_pack(struct send_pack_args *args,
 	}
 
 	if (new_refs && cmds_sent) {
-		if (pack_objects(out, remote_refs, extra_have, args) < 0) {
+		if (pack_objects(out, remote_refs, extra_have, extra_shallow,
+				 args) < 0) {
 			for (ref = remote_refs; ref; ref = ref->next)
 				ref->status = REF_STATUS_NONE;
 			if (args->stateless_rpc)
diff --git a/send-pack.h b/send-pack.h
index 05d7ab1..1c8a343 100644
--- a/send-pack.h
+++ b/send-pack.h
@@ -16,6 +16,8 @@ struct send_pack_args {
 
 int send_pack(struct send_pack_args *args,
 	      int fd[], struct child_process *conn,
-	      struct ref *remote_refs, struct extra_have_objects *extra_have);
+	      struct ref *remote_refs,
+	      struct extra_have_objects *extra_have,
+	      struct extra_have_objects *extra_shallow);
 
 #endif
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index 30fdbc8..76d3d51 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -64,4 +64,22 @@ EOF
 	test_cmp expect actual
 '
 
+test_expect_success 'push from shallow to shallow' '
+	(
+	cd shallow &&
+	git push ../shallow2/.git +master:refs/remotes/shallow/master
+	) &&
+	(
+	cd shallow2 &&
+	git log --format=%s shallow/master >actual &&
+	git fsck &&
+	cat <<EOF >expect &&
+5
+4
+3
+EOF
+	test_cmp expect actual
+	)
+'
+
 test_done
diff --git a/transport.c b/transport.c
index 17fef16..d592e8d 100644
--- a/transport.c
+++ b/transport.c
@@ -802,7 +802,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 		struct ref *tmp_refs;
 		connect_setup(transport, 1, 0);
 
-		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
+		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,
+				 NULL, &data->shallow);
 		data->got_remote_heads = 1;
 	}
 
@@ -817,7 +818,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 	args.porcelain = !!(flags & TRANSPORT_PUSH_PORCELAIN);
 
 	ret = send_pack(&args, data->fd, data->conn, remote_refs,
-			&data->extra_have);
+			&data->extra_have, &data->shallow);
 
 	close(data->fd[1]);
 	close(data->fd[0]);
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:57 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 00/16] First class shallow clone

v2 includes:
 - fix Junio comments, especially the one that may lead to incomplete
   commit islands.
 - fix send-pack setting up temporary shallow file, but never passes
   it to index-pack/unpack-objects (also fix the tests to catch this)
 - support smart http
 - add core.noshallow for repos that wish to be always complete
 - fix locally cloning a shallow repository
 - make upload-pack pass --shallow-file to pack-objects in order to
   remove duplicate object counting code just for shallow case.
Nguyễn Thái Ngọc Duy (16):
  send-pack: forbid pushing from a shallow repository
  {receive,upload}-pack: advertise shallow graft information
  connect.c: teach get_remote_heads to parse "shallow" lines
  Move setup_alternate_shallow and write_shallow_commits to shallow.c
  fetch-pack: support fetching from a shallow repository
  {send,receive}-pack: support pushing from a shallow clone
  send-pack: support pushing to a shallow clone
  upload-pack: let pack-objects do the object counting in shallow case
  pack-protocol.txt: a bit about smart http
  Add document for command arguments for supporting smart http
  {fetch,upload}-pack: support fetching from a shallow clone via smart http
  receive-pack: support pushing to a shallow clone via http
  send-pack: support pushing from a shallow clone via http
  git-clone.txt: remove shallow clone limitations
  config: add core.noshallow to prevent turning a repo into a shallow one
  clone: use git protocol for cloning shallow repo locally
 Documentation/config.txt                  |   5 +
 Documentation/git-clone.txt               |   7 +-
 Documentation/git-fetch-pack.txt          |  11 +-
 Documentation/git-receive-pack.txt        |  16 ++-
 Documentation/git-send-pack.txt           |   9 +-
 Documentation/git-upload-pack.txt         |  13 ++-
 Documentation/technical/pack-protocol.txt |  76 ++++++++++++-
 builtin/clone.c                           |  14 ++-
 builtin/fetch-pack.c                      |   6 +-
 builtin/receive-pack.c                    |  76 +++++++++++--
 builtin/send-pack.c                       |   7 +-
 cache.h                                   |   4 +-
 commit.h                                  |  27 +++++
 config.c                                  |   5 +
 connect.c                                 |  12 +-
 environment.c                             |   1 +
 fetch-pack.c                              |  90 ++++++---------
 fetch-pack.h                              |   1 +
 remote-curl.c                             |   4 +-
 send-pack.c                               |  57 +++++++++-
 send-pack.h                               |   4 +-
 shallow.c                                 | 147 +++++++++++++++++++++++++
 t/t5530-upload-pack-error.sh              |   3 -
 t/t5536-fetch-shallow.sh (new +x)         | 141 ++++++++++++++++++++++++
 t/t5537-push-shallow.sh (new +x)          | 176 ++++++++++++++++++++++++++++++
 t/t5601-clone.sh                          |   7 ++
 transport.c                               |  14 ++-
 upload-pack.c                             | 132 ++++++----------------
 28 files changed, 858 insertions(+), 207 deletions(-)
 create mode 100755 t/t5536-fetch-shallow.sh
 create mode 100755 t/t5537-push-shallow.sh
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:57 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 01/16] send-pack: forbid pushing from a shallow repository

send-pack can send a pack with loose ends to the server. receive-pack before 6d4bb38 (fetch: verify we have everything we need before updating our ref - 2011-09-01) does not detect this and keeps the pack anyway, which corrupts the repository, at least from fsck point of view.

send-pack will learn to safely push from a shallow repository later.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/send-pack.c | 3 +++
 1 file changed, 3 insertions(+)
Show changes to builtin/send-pack.c +3 −0
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 152c4ea..d4cbf64 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -184,6 +184,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	    (send_all && args.send_mirror))
 		usage(send_pack_usage);
 
+	if (is_repository_shallow())
+		die("attempt to push from a shallow repository");
+
 	if (remote_name) {
 		remote = remote_get(remote_name);
 		if (!remote_has_url(remote, dest)) {
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:57 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 02/16] {receive,upload}-pack: advertise shallow graft information

If either receive-pack or upload-pack is called on a shallow repository, shallow graft points will be sent after the ref advertisement (but before the packet flush), so that the client has the full "shape" of the server's commit graph.

This breaks the protocol for all clients trying to push to a shallow repo, or fetch from one. Which is basically the same end result as today's "is_repository_shallow() && die()" in receive-pack and upload-pack. New clients will be made aware of shallow upstream and can make use of this information.

Smart HTTP is not affected by this patch. Shallow support on smart-http comes later separately.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/technical/pack-protocol.txt |  3 +++
 builtin/receive-pack.c                    |  4 +++-
 commit.h                                  |  1 +
 shallow.c                                 | 16 ++++++++++++++++
 upload-pack.c                             |  6 ++++--
 5 files changed, 27 insertions(+), 3 deletions(-)
Show changes to 5 files +27 −3

Documentation/technical/pack-protocol.txt, builtin/receive-pack.c, commit.h, shallow.c, upload-pack.c

diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index b898e97..eb8edd1 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -161,6 +161,7 @@ MUST peel the ref if it's an annotated tag.
 
 ----
   advertised-refs  =  (no-refs / list-of-refs)
+		      *shallow
 		      flush-pkt
 
   no-refs          =  PKT-LINE(zero-id SP "capabilities^{}"
@@ -174,6 +175,8 @@ MUST peel the ref if it's an annotated tag.
   other-tip        =  obj-id SP refname LF
   other-peeled     =  obj-id SP refname "^{}" LF
 
+  shallow          =  PKT-LINE("shallow" SP obj-id)
+
   capability-list  =  capability *(SP capability)
   capability       =  1*(LC_ALPHA / DIGIT / "-" / "_")
   LC_ALPHA         =  %x61-7A
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index e3eb5fc..6ffe526 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -176,6 +176,8 @@ static void write_head_info(void)
 	if (!sent_capabilities)
 		show_ref("capabilities^{}", null_sha1);
 
+	advertise_shallow_grafts(1);
+
 	/* EOF */
 	packet_flush(1);
 }
@@ -990,7 +992,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 	if (!enter_repo(dir, 0))
 		die("'%s' does not appear to be a git repository", dir);
 
-	if (is_repository_shallow())
+	if (is_repository_shallow() && stateless_rpc)
 		die("attempt to push into a shallow repository");
 
 	git_config(receive_pack_config, NULL);
diff --git a/commit.h b/commit.h
index 4d452dc..e0688c3 100644
--- a/commit.h
+++ b/commit.h
@@ -187,6 +187,7 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
 		int depth, int shallow_flag, int not_shallow_flag);
 extern void check_shallow_file_for_update(void);
 extern void set_alternate_shallow_file(const char *path);
+extern void advertise_shallow_grafts(int);
 
 int is_descendant_of(struct commit *, struct commit_list *);
 int in_merge_bases(struct commit *, struct commit *);
diff --git a/shallow.c b/shallow.c
index cbe2526..ccdfefc 100644
--- a/shallow.c
+++ b/shallow.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "commit.h"
 #include "tag.h"
+#include "pkt-line.h"
 
 static int is_shallow = -1;
 static struct stat shallow_stat;
@@ -146,3 +147,18 @@ void check_shallow_file_for_update(void)
 		   )
 		die("shallow file was changed during fetch");
 }
+
+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)
+{
+	int fd = *(int*)cb;
+	if (graft->nr_parent == -1)
+		packet_write(fd, "shallow %s\n", sha1_to_hex(graft->sha1));
+	return 0;
+}
+
+void advertise_shallow_grafts(int fd)
+{
+	if (!is_repository_shallow())
+		return;
+	for_each_commit_graft(advertise_shallow_grafts_cb, &fd);
+}
diff --git a/upload-pack.c b/upload-pack.c
index 127e59a..c377a3e 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -766,6 +766,7 @@ static void upload_pack(void)
 		reset_timeout();
 		head_ref_namespaced(send_ref, NULL);
 		for_each_namespaced_ref(send_ref, NULL);
+		advertise_shallow_grafts(1);
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(mark_our_ref, NULL);
@@ -837,8 +838,9 @@ int main(int argc, char **argv)
 
 	if (!enter_repo(dir, strict))
 		die("'%s' does not appear to be a git repository", dir);
-	if (is_repository_shallow())
-		die("attempt to fetch/clone from a shallow repository");
+	if (is_repository_shallow() && stateless_rpc)
+		die("attempt to push into a shallow repository");
+
 	git_config(upload_pack_config, NULL);
 	upload_pack();
 	return 0;
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:57 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 03/16] connect.c: teach get_remote_heads to parse "shallow" lines

No callers pass a non-empty pointer as shallow_points at this stage. As a result, all clients still refuse to talk to shallow repository on the other end.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/fetch-pack.c |  2 +-
 builtin/send-pack.c  |  2 +-
 cache.h              |  3 ++-
 connect.c            | 12 +++++++++++-
 remote-curl.c        |  2 +-
 transport.c          |  7 ++++---
 6 files changed, 20 insertions(+), 8 deletions(-)
Show changes to 6 files +20 −8

builtin/fetch-pack.c, builtin/send-pack.c, cache.h, connect.c, remote-curl.c, transport.c

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index aba4465..080e599 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -144,7 +144,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 				   args.verbose ? CONNECT_VERBOSE : 0);
 	}
 
-	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL);
+	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
 
 	ref = fetch_pack(&args, fd, conn, ref, dest,
 			 sought, nr_sought, pack_lockfile_ptr);
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index d4cbf64..4be5931 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -210,7 +210,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 
 	memset(&extra_have, 0, sizeof(extra_have));
 
-	get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);
+	get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);
 
 	transport_verify_remote_names(nr_refspecs, refspecs);
 
diff --git a/cache.h b/cache.h
index dd0fb33..7f17228 100644
--- a/cache.h
+++ b/cache.h
@@ -1091,7 +1091,8 @@ struct extra_have_objects {
 };
 extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 				     struct ref **list, unsigned int flags,
-				     struct extra_have_objects *);
+				     struct extra_have_objects *have,
+				     struct extra_have_objects *shallow);
 extern int server_supports(const char *feature);
 extern int parse_feature_request(const char *features, const char *feature);
 extern const char *server_feature_value(const char *feature, int *len_ret);
diff --git a/connect.c b/connect.c
index a0783d4..1cba424 100644
--- a/connect.c
+++ b/connect.c
@@ -64,7 +64,8 @@ static void die_initial_contact(int got_at_least_one_head)
  */
 struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 			      struct ref **list, unsigned int flags,
-			      struct extra_have_objects *extra_have)
+			      struct extra_have_objects *extra_have,
+			      struct extra_have_objects *shallow_points)
 {
 	int got_at_least_one_head = 0;
 
@@ -89,6 +90,15 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 		if (len > 4 && !prefixcmp(buffer, "ERR "))
 			die("remote error: %s", buffer + 4);
 
+		if (len == 48 && !prefixcmp(buffer, "shallow ")) {
+			if (get_sha1_hex(buffer + 8, old_sha1))
+				die("protocol error: expected shallow sha, got '%s'", buffer + 8);
+			if (!shallow_points)
+				die("repository on the other end cannot be shallow");
+			add_extra_have(shallow_points, old_sha1);
+			continue;
+		}
+
 		if (len < 42 || get_sha1_hex(buffer, old_sha1) || buffer[40] != ' ')
 			die("protocol error: expected sha/ref, got '%s'", buffer);
 		name = buffer + 41;
diff --git a/remote-curl.c b/remote-curl.c
index 5b3ce9e..c329bd3 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -86,7 +86,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
 {
 	struct ref *list = NULL;
 	get_remote_heads(-1, heads->buf, heads->len, &list,
-			 for_push ? REF_NORMAL : 0, NULL);
+			 for_push ? REF_NORMAL : 0, NULL, NULL);
 	return list;
 }
 
diff --git a/transport.c b/transport.c
index e15db98..10a8cb8 100644
--- a/transport.c
+++ b/transport.c
@@ -509,7 +509,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 
 	connect_setup(transport, for_push, 0);
 	get_remote_heads(data->fd[0], NULL, 0, &refs,
-			 for_push ? REF_NORMAL : 0, &data->extra_have);
+			 for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
 	data->got_remote_heads = 1;
 
 	return refs;
@@ -539,7 +539,8 @@ static int fetch_refs_via_pack(struct transport *transport,
 
 	if (!data->got_remote_heads) {
 		connect_setup(transport, 0, 0);
-		get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0, NULL);
+		get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
+				 NULL, NULL);
 		data->got_remote_heads = 1;
 	}
 
@@ -799,7 +800,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 		struct ref *tmp_refs;
 		connect_setup(transport, 1, 0);
 
-		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL);
+		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
 		data->got_remote_heads = 1;
 	}
 
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:57 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 04/16] Move setup_alternate_shallow and write_shallow_commits to shallow.c

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 commit.h     |  3 +++
 fetch-pack.c | 53 +----------------------------------------------------
 shallow.c    | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+), 52 deletions(-)
Show changes to 3 files +57 −52

commit.h, fetch-pack.c, shallow.c

diff --git a/commit.h b/commit.h
index e0688c3..678fa20 100644
--- a/commit.h
+++ b/commit.h
@@ -188,6 +188,9 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
 extern void check_shallow_file_for_update(void);
 extern void set_alternate_shallow_file(const char *path);
 extern void advertise_shallow_grafts(int);
+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);
+extern void setup_alternate_shallow(struct lock_file *shallow_lock,
+				    const char **alternate_shallow_file);
 
 int is_descendant_of(struct commit *, struct commit_list *);
 int in_merge_bases(struct commit *, struct commit *);
diff --git a/fetch-pack.c b/fetch-pack.c
index abe5ffb..dc71a2b 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -185,36 +185,6 @@ static void consume_shallow_list(struct fetch_pack_args *args, int fd)
 	}
 }
 
-struct write_shallow_data {
-	struct strbuf *out;
-	int use_pack_protocol;
-	int count;
-};
-
-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
-{
-	struct write_shallow_data *data = cb_data;
-	const char *hex = sha1_to_hex(graft->sha1);
-	data->count++;
-	if (data->use_pack_protocol)
-		packet_buf_write(data->out, "shallow %s", hex);
-	else {
-		strbuf_addstr(data->out, hex);
-		strbuf_addch(data->out, '\n');
-	}
-	return 0;
-}
-
-static int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
-{
-	struct write_shallow_data data;
-	data.out = out;
-	data.use_pack_protocol = use_pack_protocol;
-	data.count = 0;
-	for_each_commit_graft(write_one_shallow, &data);
-	return data.count;
-}
-
 static enum ack_type get_ack(int fd, unsigned char *result_sha1)
 {
 	int len;
@@ -795,27 +765,6 @@ static int cmp_ref_by_name(const void *a_, const void *b_)
 	return strcmp(a->name, b->name);
 }
 
-static void setup_alternate_shallow(void)
-{
-	struct strbuf sb = STRBUF_INIT;
-	int fd;
-
-	check_shallow_file_for_update();
-	fd = hold_lock_file_for_update(&shallow_lock, git_path("shallow"),
-				       LOCK_DIE_ON_ERROR);
-	if (write_shallow_commits(&sb, 0)) {
-		if (write_in_full(fd, sb.buf, sb.len) != sb.len)
-			die_errno("failed to write to %s", shallow_lock.filename);
-		alternate_shallow_file = shallow_lock.filename;
-	} else
-		/*
-		 * is_repository_shallow() sees empty string as "no
-		 * shallow file".
-		 */
-		alternate_shallow_file = "";
-	strbuf_release(&sb);
-}
-
 static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 				 int fd[2],
 				 const struct ref *orig_ref,
@@ -896,7 +845,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 	if (args->stateless_rpc)
 		packet_flush(fd[1]);
 	if (args->depth > 0)
-		setup_alternate_shallow();
+		setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
 	if (get_pack(args, fd, pack_lockfile))
 		die("git fetch-pack: fetch failed.");
 
diff --git a/shallow.c b/shallow.c
index ccdfefc..ee9edd4 100644
--- a/shallow.c
+++ b/shallow.c
@@ -162,3 +162,56 @@ void advertise_shallow_grafts(int fd)
 		return;
 	for_each_commit_graft(advertise_shallow_grafts_cb, &fd);
 }
+
+struct write_shallow_data {
+	struct strbuf *out;
+	int use_pack_protocol;
+	int count;
+};
+
+static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
+{
+	struct write_shallow_data *data = cb_data;
+	const char *hex = sha1_to_hex(graft->sha1);
+	data->count++;
+	if (data->use_pack_protocol)
+		packet_buf_write(data->out, "shallow %s", hex);
+	else {
+		strbuf_addstr(data->out, hex);
+		strbuf_addch(data->out, '\n');
+	}
+	return 0;
+}
+
+int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
+{
+	struct write_shallow_data data;
+	data.out = out;
+	data.use_pack_protocol = use_pack_protocol;
+	data.count = 0;
+	for_each_commit_graft(write_one_shallow, &data);
+	return data.count;
+}
+
+void setup_alternate_shallow(struct lock_file *shallow_lock,
+			     const char **alternate_shallow_file)
+{
+	struct strbuf sb = STRBUF_INIT;
+	int fd;
+
+	check_shallow_file_for_update();
+	fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
+				       LOCK_DIE_ON_ERROR);
+	if (write_shallow_commits(&sb, 0)) {
+		if (write_in_full(fd, sb.buf, sb.len) != sb.len)
+			die_errno("failed to write to %s",
+				  shallow_lock->filename);
+		*alternate_shallow_file = shallow_lock->filename;
+	} else
+		/*
+		 * is_repository_shallow() sees empty string as "no
+		 * shallow file".
+		 */
+		*alternate_shallow_file = "";
+	strbuf_release(&sb);
+}
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:57 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 05/16] fetch-pack: support fetching from a shallow repository

upload-pack already advertises all shallow grafts if server repository is shallow. This information can be used to add more grafts to the client if the server sends commit chains down to its graft points.

If the server is shallow, before we receive the pack, we setup a temporary shallow file that contains both local graft points and the server's. This stops index-pack from going beyond server's graft points.

Only server graft points that do not have corresponding SHA-1s in local repo are added to the temp shallow file because we don't want to accidentally cut the client history because the server's is shorter. The client cutting can only happen when --depth is requested.

After index-pack finishes successfully, we write the temporary shallow down with one exception: unused graft points provided by the server are removed. We don't want those lying around and suddenly become active.

Note that in the "shallow -> shallow" case, the server might not have enough information to find common roots to create an optimum pack. It might send complete commit chains down to the graft points as a result. I don't think we can improve this, unless upload-pack somehow has access to a full repository.

"shallow -> shallow" case only makes sense when the upstream provides a stable shallow repo (e.g. make a cut every year or so and ask devs to all move to the new base). If the cloned repos are all based on a stable (shallow) upstream, the above problem is unlikely to happen.

A side effect of this change is we can now clone from a shallow repository. And a full repository may automatically become shallow if you fetch from a shallow repository.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/fetch-pack.c              |   7 ++-
 commit.h                          |  27 +++++++++-
 fetch-pack.c                      |  30 +++++++++--
 fetch-pack.h                      |   1 +
 shallow.c                         |  90 +++++++++++++++++++++++++++++---
 t/t5536-fetch-shallow.sh (new +x) | 105 ++++++++++++++++++++++++++++++++++++++
 transport.c                       |  10 ++--
 7 files changed, 252 insertions(+), 18 deletions(-)
 create mode 100755 t/t5536-fetch-shallow.sh
Show changes to 7 files +252 −18

builtin/fetch-pack.c, commit.h, fetch-pack.c, fetch-pack.h, shallow.c, t/t5536-fetch-shallow.sh, transport.c

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 080e599..f6a6d76 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -37,6 +37,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 	char **pack_lockfile_ptr = NULL;
 	struct child_process *conn;
 	struct fetch_pack_args args;
+	struct extra_have_objects shallow;
 
 	packet_trace_identity("fetch-pack");
 
@@ -144,10 +145,12 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 				   args.verbose ? CONNECT_VERBOSE : 0);
 	}
 
-	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
+	memset(&shallow, 0, sizeof(shallow));
+	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL,
+			 args.stateless_rpc ? NULL : &shallow);
 
 	ref = fetch_pack(&args, fd, conn, ref, dest,
-			 sought, nr_sought, pack_lockfile_ptr);
+			 sought, nr_sought, &shallow, pack_lockfile_ptr);
 	if (pack_lockfile) {
 		printf("lock %s\n", pack_lockfile);
 		fflush(stdout);
diff --git a/commit.h b/commit.h
index 678fa20..97fd54e 100644
--- a/commit.h
+++ b/commit.h
@@ -188,9 +188,32 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
 extern void check_shallow_file_for_update(void);
 extern void set_alternate_shallow_file(const char *path);
 extern void advertise_shallow_grafts(int);
-extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);
+
+/*
+ * Ignore graft points that cut out existing "good" parts of our
+ * history. Graft points pointing to incomplete commit islands are
+ * _not_ ignore.
+ */
+#define WRITE_SHALLOW_NO_CUT         1
+/*
+ * Remove graft points that are not attached to any SHA-1 in current
+ * history.
+ */
+#define WRITE_SHALLOW_NO_UNUSED      2
+/*
+ * Not really write_shallow_commits() flag, this lets
+ * setup_alternate_shallow know this is the "final" setup before
+ * committing to .git/shallow
+ */
+#define WRITE_SHALLOW_REWRITE        4
+
+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
+				 struct extra_have_objects *extra,
+				 unsigned int flags);
 extern void setup_alternate_shallow(struct lock_file *shallow_lock,
-				    const char **alternate_shallow_file);
+				    const char **alternate_shallow_file,
+				    struct extra_have_objects *extra,
+				    unsigned int flags);
 
 int is_descendant_of(struct commit *, struct commit_list *);
 int in_merge_bases(struct commit *, struct commit *);
diff --git a/fetch-pack.c b/fetch-pack.c
index dc71a2b..f337526 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args *args,
 	}
 
 	if (is_repository_shallow())
-		write_shallow_commits(&req_buf, 1);
+		write_shallow_commits(&req_buf, 1, NULL, WRITE_SHALLOW_NO_CUT);
 	if (args->depth > 0)
 		packet_buf_write(&req_buf, "deepen %d", args->depth);
 	packet_buf_flush(&req_buf);
@@ -769,6 +769,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 				 int fd[2],
 				 const struct ref *orig_ref,
 				 struct ref **sought, int nr_sought,
+				 struct extra_have_objects *shallow,
 				 char **pack_lockfile)
 {
 	struct ref *ref = copy_ref_list(orig_ref);
@@ -844,8 +845,10 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 
 	if (args->stateless_rpc)
 		packet_flush(fd[1]);
-	if (args->depth > 0)
-		setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
+	if (args->depth > 0 || shallow->nr)
+		setup_alternate_shallow(&shallow_lock, &alternate_shallow_file,
+					shallow,
+					WRITE_SHALLOW_NO_CUT);
 	if (get_pack(args, fd, pack_lockfile))
 		die("git fetch-pack: fetch failed.");
 
@@ -922,6 +925,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 		       const struct ref *ref,
 		       const char *dest,
 		       struct ref **sought, int nr_sought,
+		       struct extra_have_objects *shallow,
 		       char **pack_lockfile)
 {
 	struct ref *ref_cpy;
@@ -934,14 +938,30 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 		packet_flush(fd[1]);
 		die("no matching remote head");
 	}
-	ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);
+	ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
+				shallow, pack_lockfile);
 
 	if (alternate_shallow_file) {
 		if (*alternate_shallow_file == '\0') { /* --unshallow */
 			unlink_or_warn(git_path("shallow"));
 			rollback_lock_file(&shallow_lock);
-		} else
+		} else {
+			/*
+			 *  The server is a shallow clone and it sends
+			 *  us all of its shallow grafts. Some may be
+			 *  needed if the server sends objects down to
+			 *  the bottom. Remove all unused grafts.
+			 */
+			if (shallow->nr) {
+				reprepare_packed_git();
+				setup_alternate_shallow(&shallow_lock,
+							&alternate_shallow_file,
+							shallow,
+							WRITE_SHALLOW_NO_CUT |
+							WRITE_SHALLOW_REWRITE);
+			}
 			commit_lock_file(&shallow_lock);
+		}
 	}
 
 	reprepare_packed_git();
diff --git a/fetch-pack.h b/fetch-pack.h
index 40f08ba..c55bf1c 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -32,6 +32,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 		       const char *dest,
 		       struct ref **sought,
 		       int nr_sought,
+		       struct extra_have_objects *shallow,
 		       char **pack_lockfile);
 
 #endif
diff --git a/shallow.c b/shallow.c
index ee9edd4..337f676 100644
--- a/shallow.c
+++ b/shallow.c
@@ -2,6 +2,7 @@
 #include "commit.h"
 #include "tag.h"
 #include "pkt-line.h"
+#include "connected.h"
 
 static int is_shallow = -1;
 static struct stat shallow_stat;
@@ -183,26 +184,103 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
 	return 0;
 }
 
-int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
+static int sha1_iterate(void *cb, unsigned char *sha1)
+{
+	if (is_null_sha1(cb))
+		return 1;
+	hashcpy(sha1, cb);
+	hashclr(cb);
+	return 0;
+}
+
+struct connectivity_cache_item {
+	unsigned char sha1[20];
+	int good;
+};
+
+static int good_connectivity_from(const unsigned char *sha1)
+{
+	static struct connectivity_cache_item *cache;
+	static int nr, alloc;
+	int i;
+	unsigned char sha1_cb[20];
+
+	for (i = 0; i < nr; i++)
+		if (!hashcmp(cache[i].sha1, sha1))
+			return cache[i].good;
+
+	ALLOC_GROW(cache, nr + 1, alloc);
+	hashcpy(sha1_cb, sha1);
+	hashcpy(cache[nr].sha1, sha1);
+	cache[nr].good = !check_everything_connected(sha1_iterate, 1, sha1_cb);
+	nr++;
+	return cache[nr - 1].good;
+}
+
+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
+			  struct extra_have_objects *extra,
+			  unsigned int flags)
 {
 	struct write_shallow_data data;
+	int i;
 	data.out = out;
 	data.use_pack_protocol = use_pack_protocol;
 	data.count = 0;
 	for_each_commit_graft(write_one_shallow, &data);
+	if (!extra)
+		return data.count;
+
+	for (i = 0; i < extra->nr; i++) {
+		if ((flags & WRITE_SHALLOW_NO_CUT) &&
+		    good_connectivity_from(extra->array[i]))
+			/*
+			 * The server may have even shorter history
+			 * than the client (e.g. the client has
+			 * A-B-C-D but the server only has C-D). We do
+			 * NOT want to cut client's history down to
+			 * C-D simply because the server is set up
+			 * so. If we don't have "C" and the server
+			 * sends C-D to us, then we set up a graft at
+			 * "C".
+			 *
+			 * If the client has (broken)-B-C-D, then "C"
+			 * _is_ accepted. The -B-C-D is an incomplete
+			 * island that normally has no connectivity to
+			 * the rest of the repo.
+			 */
+			continue;
+		if ((flags & WRITE_SHALLOW_NO_UNUSED) &&
+		    !has_sha1_file(extra->array[i]))
+			continue;
+		strbuf_addstr(out, sha1_to_hex(extra->array[i]));
+		strbuf_addch(out, '\n');
+		data.count++;
+	}
 	return data.count;
 }
 
 void setup_alternate_shallow(struct lock_file *shallow_lock,
-			     const char **alternate_shallow_file)
+			     const char **alternate_shallow_file,
+			     struct extra_have_objects *extra,
+			     unsigned int flags)
 {
 	struct strbuf sb = STRBUF_INIT;
+	int rewrite = flags & WRITE_SHALLOW_REWRITE;
 	int fd;
 
-	check_shallow_file_for_update();
-	fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
-				       LOCK_DIE_ON_ERROR);
-	if (write_shallow_commits(&sb, 0)) {
+	if (!rewrite)
+		fd = hold_lock_file_for_update(shallow_lock,
+					       git_path("shallow"),
+					       LOCK_DIE_ON_ERROR);
+	else {
+		flags |= WRITE_SHALLOW_NO_UNUSED;
+		fd = shallow_lock->fd;
+	}
+	if (write_shallow_commits(&sb, 0, extra, flags)) {
+		if (rewrite && (lseek(fd, 0, SEEK_SET) == -1 ||
+				ftruncate(fd, 0) == -1))
+			die_errno("unable to truncate the new shallow file %s",
+				  shallow_lock->filename);
 		if (write_in_full(fd, sb.buf, sb.len) != sb.len)
 			die_errno("failed to write to %s",
 				  shallow_lock->filename);
diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
new file mode 100755
index 0000000..15a8208
--- /dev/null
+++ b/t/t5536-fetch-shallow.sh
@@ -0,0 +1,105 @@
+#!/bin/sh
+
+test_description='fetch/clone from a shallow clone'
+
+. ./test-lib.sh
+
+commit() {
+	echo "$1" >tracked &&
+	git add tracked &&
+	git commit -m "$1"
+}
+
+test_expect_success 'setup' '
+	commit 1 &&
+	commit 2 &&
+	commit 3 &&
+	commit 4 &&
+	git config --global transfer.fsckObjects true
+'
+
+test_expect_success 'setup shallow clone' '
+	git clone --no-local --depth=2 .git shallow &&
+	git --git-dir=shallow/.git log --format=%s >actual &&
+	cat <<EOF >expect &&
+4
+3
+EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'clone from shallow clone' '
+	git clone --no-local shallow shallow2 &&
+	(
+	cd shallow2 &&
+	git fsck &&
+	git log --format=%s >actual &&
+	cat <<EOF >expect &&
+4
+3
+EOF
+	test_cmp expect actual
+	)
+'
+
+test_expect_success 'fetch from shallow clone' '
+	(
+	cd shallow &&
+	commit 5
+	) &&
+	(
+	cd shallow2 &&
+	git fetch &&
+	git fsck &&
+	git log --format=%s origin/master >actual &&
+	cat <<EOF >expect &&
+5
+4
+3
+EOF
+	test_cmp expect actual
+	)
+'
+
+test_expect_success 'fetch --depth from shallow clone' '
+	(
+	cd shallow &&
+	commit 6
+	) &&
+	(
+	cd shallow2 &&
+	git fetch --depth=2 &&
+	git fsck &&
+	git log --format=%s origin/master >actual &&
+	cat <<EOF >expect &&
+6
+5
+EOF
+	test_cmp expect actual
+	)
+'
+
+test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '
+	# the blob "1" is available in .git but hidden by the
+	# shallow2/.git/shallow and it should be resent
+	! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null &&
+	echo 1 > 1.t &&
+	git add 1.t &&
+	git commit -m add-1-back &&
+	(
+	cd shallow2 &&
+	git fetch ../.git +refs/heads/master:refs/remotes/top/master &&
+	git fsck &&
+	git log --format=%s top/master >actual &&
+	cat <<EOF >expect &&
+add-1-back
+4
+3
+EOF
+	test_cmp expect actual
+	) &&
+	git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null
+
+'
+
+test_done
diff --git a/transport.c b/transport.c
index 10a8cb8..f167ca5 100644
--- a/transport.c
+++ b/transport.c
@@ -453,6 +453,7 @@ struct git_transport_data {
 	int fd[2];
 	unsigned got_remote_heads : 1;
 	struct extra_have_objects extra_have;
+	struct extra_have_objects shallow;
 };
 
 static int set_git_option(struct git_transport_options *opts,
@@ -509,7 +510,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 
 	connect_setup(transport, for_push, 0);
 	get_remote_heads(data->fd[0], NULL, 0, &refs,
-			 for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
+			 for_push ? REF_NORMAL : 0,
+			 &data->extra_have, &data->shallow);
 	data->got_remote_heads = 1;
 
 	return refs;
@@ -540,13 +542,13 @@ static int fetch_refs_via_pack(struct transport *transport,
 	if (!data->got_remote_heads) {
 		connect_setup(transport, 0, 0);
 		get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
-				 NULL, NULL);
+				 NULL, &data->shallow);
 		data->got_remote_heads = 1;
 	}
 
 	refs = fetch_pack(&args, data->fd, data->conn,
 			  refs_tmp ? refs_tmp : transport->remote_refs,
-			  dest, to_fetch, nr_heads,
+			  dest, to_fetch, nr_heads, &data->shallow,
 			  &transport->pack_lockfile);
 	close(data->fd[0]);
 	close(data->fd[1]);
@@ -803,6 +805,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
 		data->got_remote_heads = 1;
 	}
+	if (data->shallow.nr)
+		die("pushing to a shallow repository is not supported");
 
 	memset(&args, 0, sizeof(args));
 	args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
-- 
1.8.2.83.gc99314b
Philip Oakley· Jul 22, 2013, 19:10 UTC · re: Nguyễn Thái Ngọc Duy · lore

Re: [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository

From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
Show 32 quoted lines
> upload-pack already advertises all shallow grafts if server repository
> is shallow. This information can be used to add more grafts to the
> client if the server sends commit chains down to its graft points.
>
> If the server is shallow, before we receive the pack, we setup a
> temporary shallow file that contains both local graft points and the
> server's. This stops index-pack from going beyond server's graft
> points.
>
> Only server graft points that do not have corresponding SHA-1s in
> local repo are added to the temp shallow file because we don't want to
> accidentally cut the client history because the server's is
> shorter. The client cutting can only happen when --depth is requested.
>
> After index-pack finishes successfully, we write the temporary shallow
> down with one exception: unused graft points provided by the server
> are removed. We don't want those lying around and suddenly become
> active.
>
> Note that in the "shallow -> shallow" case, the server might not have
> enough information to find common roots to create an optimum pack. It
> might send complete commit chains down to the graft points as a
> result. I don't think we can improve this, unless upload-pack somehow
> has access to a full repository.
>
> "shallow -> shallow" case only makes sense when the upstream provides
> a stable shallow repo (e.g. make a cut every year or so and ask devs
> to all move to the new base). If the cloned repos are all based on a
> stable (shallow) upstream, the above problem is unlikely to happen.
>
> A side effect of this change is we can now clone from a shallow
> repository.
>             And a full repository may automatically become shallow if
> you fetch from a shallow repository.

This "sounds" risky: Perhaps add a note about '[PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one' which will become available later in the series. The later core.noshallow may be a hostage to fortune.

Show 483 quoted lines
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
> builtin/fetch-pack.c              |   7 ++-
> commit.h                          |  27 +++++++++-
> fetch-pack.c                      |  30 +++++++++--
> fetch-pack.h                      |   1 +
> shallow.c                         |  90 
> +++++++++++++++++++++++++++++---
> t/t5536-fetch-shallow.sh (new +x) | 105 
> ++++++++++++++++++++++++++++++++++++++
> transport.c                       |  10 ++--
> 7 files changed, 252 insertions(+), 18 deletions(-)
> create mode 100755 t/t5536-fetch-shallow.sh
>
> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
> index 080e599..f6a6d76 100644
> --- a/builtin/fetch-pack.c
> +++ b/builtin/fetch-pack.c
> @@ -37,6 +37,7 @@ int cmd_fetch_pack(int argc, const char **argv, 
> const char *prefix)
>  char **pack_lockfile_ptr = NULL;
>  struct child_process *conn;
>  struct fetch_pack_args args;
> + struct extra_have_objects shallow;
>
>  packet_trace_identity("fetch-pack");
>
> @@ -144,10 +145,12 @@ int cmd_fetch_pack(int argc, const char **argv, 
> const char *prefix)
>     args.verbose ? CONNECT_VERBOSE : 0);
>  }
>
> - get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
> + memset(&shallow, 0, sizeof(shallow));
> + get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL,
> + args.stateless_rpc ? NULL : &shallow);
>
>  ref = fetch_pack(&args, fd, conn, ref, dest,
> - sought, nr_sought, pack_lockfile_ptr);
> + sought, nr_sought, &shallow, pack_lockfile_ptr);
>  if (pack_lockfile) {
>  printf("lock %s\n", pack_lockfile);
>  fflush(stdout);
> diff --git a/commit.h b/commit.h
> index 678fa20..97fd54e 100644
> --- a/commit.h
> +++ b/commit.h
> @@ -188,9 +188,32 @@ extern struct commit_list 
> *get_shallow_commits(struct object_array *heads,
> extern void check_shallow_file_for_update(void);
> extern void set_alternate_shallow_file(const char *path);
> extern void advertise_shallow_grafts(int);
> -extern int write_shallow_commits(struct strbuf *out, int 
> use_pack_protocol);
> +
> +/*
> + * Ignore graft points that cut out existing "good" parts of our
> + * history. Graft points pointing to incomplete commit islands are
> + * _not_ ignore.
> + */
> +#define WRITE_SHALLOW_NO_CUT         1
> +/*
> + * Remove graft points that are not attached to any SHA-1 in current
> + * history.
> + */
> +#define WRITE_SHALLOW_NO_UNUSED      2
> +/*
> + * Not really write_shallow_commits() flag, this lets
> + * setup_alternate_shallow know this is the "final" setup before
> + * committing to .git/shallow
> + */
> +#define WRITE_SHALLOW_REWRITE        4
> +
> +extern int write_shallow_commits(struct strbuf *out, int 
> use_pack_protocol,
> + struct extra_have_objects *extra,
> + unsigned int flags);
> extern void setup_alternate_shallow(struct lock_file *shallow_lock,
> -     const char **alternate_shallow_file);
> +     const char **alternate_shallow_file,
> +     struct extra_have_objects *extra,
> +     unsigned int flags);
>
> int is_descendant_of(struct commit *, struct commit_list *);
> int in_merge_bases(struct commit *, struct commit *);
> diff --git a/fetch-pack.c b/fetch-pack.c
> index dc71a2b..f337526 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args 
> *args,
>  }
>
>  if (is_repository_shallow())
> - write_shallow_commits(&req_buf, 1);
> + write_shallow_commits(&req_buf, 1, NULL, WRITE_SHALLOW_NO_CUT);
>  if (args->depth > 0)
>  packet_buf_write(&req_buf, "deepen %d", args->depth);
>  packet_buf_flush(&req_buf);
> @@ -769,6 +769,7 @@ static struct ref *do_fetch_pack(struct 
> fetch_pack_args *args,
>  int fd[2],
>  const struct ref *orig_ref,
>  struct ref **sought, int nr_sought,
> + struct extra_have_objects *shallow,
>  char **pack_lockfile)
> {
>  struct ref *ref = copy_ref_list(orig_ref);
> @@ -844,8 +845,10 @@ static struct ref *do_fetch_pack(struct 
> fetch_pack_args *args,
>
>  if (args->stateless_rpc)
>  packet_flush(fd[1]);
> - if (args->depth > 0)
> - setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
> + if (args->depth > 0 || shallow->nr)
> + setup_alternate_shallow(&shallow_lock, &alternate_shallow_file,
> + shallow,
> + WRITE_SHALLOW_NO_CUT);
>  if (get_pack(args, fd, pack_lockfile))
>  die("git fetch-pack: fetch failed.");
>
> @@ -922,6 +925,7 @@ struct ref *fetch_pack(struct fetch_pack_args 
> *args,
>         const struct ref *ref,
>         const char *dest,
>         struct ref **sought, int nr_sought,
> +        struct extra_have_objects *shallow,
>         char **pack_lockfile)
> {
>  struct ref *ref_cpy;
> @@ -934,14 +938,30 @@ struct ref *fetch_pack(struct fetch_pack_args 
> *args,
>  packet_flush(fd[1]);
>  die("no matching remote head");
>  }
> - ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, 
> pack_lockfile);
> + ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
> + shallow, pack_lockfile);
>
>  if (alternate_shallow_file) {
>  if (*alternate_shallow_file == '\0') { /* --unshallow */
>  unlink_or_warn(git_path("shallow"));
>  rollback_lock_file(&shallow_lock);
> - } else
> + } else {
> + /*
> + *  The server is a shallow clone and it sends
> + *  us all of its shallow grafts. Some may be
> + *  needed if the server sends objects down to
> + *  the bottom. Remove all unused grafts.
> + */
> + if (shallow->nr) {
> + reprepare_packed_git();
> + setup_alternate_shallow(&shallow_lock,
> + &alternate_shallow_file,
> + shallow,
> + WRITE_SHALLOW_NO_CUT |
> + WRITE_SHALLOW_REWRITE);
> + }
>  commit_lock_file(&shallow_lock);
> + }
>  }
>
>  reprepare_packed_git();
> diff --git a/fetch-pack.h b/fetch-pack.h
> index 40f08ba..c55bf1c 100644
> --- a/fetch-pack.h
> +++ b/fetch-pack.h
> @@ -32,6 +32,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
>         const char *dest,
>         struct ref **sought,
>         int nr_sought,
> +        struct extra_have_objects *shallow,
>         char **pack_lockfile);
>
> #endif
> diff --git a/shallow.c b/shallow.c
> index ee9edd4..337f676 100644
> --- a/shallow.c
> +++ b/shallow.c
> @@ -2,6 +2,7 @@
> #include "commit.h"
> #include "tag.h"
> #include "pkt-line.h"
> +#include "connected.h"
>
> static int is_shallow = -1;
> static struct stat shallow_stat;
> @@ -183,26 +184,103 @@ static int write_one_shallow(const struct 
> commit_graft *graft, void *cb_data)
>  return 0;
> }
>
> -int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
> +static int sha1_iterate(void *cb, unsigned char *sha1)
> +{
> + if (is_null_sha1(cb))
> + return 1;
> + hashcpy(sha1, cb);
> + hashclr(cb);
> + return 0;
> +}
> +
> +struct connectivity_cache_item {
> + unsigned char sha1[20];
> + int good;
> +};
> +
> +static int good_connectivity_from(const unsigned char *sha1)
> +{
> + static struct connectivity_cache_item *cache;
> + static int nr, alloc;
> + int i;
> + unsigned char sha1_cb[20];
> +
> + for (i = 0; i < nr; i++)
> + if (!hashcmp(cache[i].sha1, sha1))
> + return cache[i].good;
> +
> + ALLOC_GROW(cache, nr + 1, alloc);
> + hashcpy(sha1_cb, sha1);
> + hashcpy(cache[nr].sha1, sha1);
> + cache[nr].good = !check_everything_connected(sha1_iterate, 1, 
> sha1_cb);
> + nr++;
> + return cache[nr - 1].good;
> +}
> +
> +int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
> +   struct extra_have_objects *extra,
> +   unsigned int flags)
> {
>  struct write_shallow_data data;
> + int i;
>  data.out = out;
>  data.use_pack_protocol = use_pack_protocol;
>  data.count = 0;
>  for_each_commit_graft(write_one_shallow, &data);
> + if (!extra)
> + return data.count;
> +
> + for (i = 0; i < extra->nr; i++) {
> + if ((flags & WRITE_SHALLOW_NO_CUT) &&
> +     good_connectivity_from(extra->array[i]))
> + /*
> + * The server may have even shorter history
> + * than the client (e.g. the client has
> + * A-B-C-D but the server only has C-D). We do
> + * NOT want to cut client's history down to
> + * C-D simply because the server is set up
> + * so. If we don't have "C" and the server
> + * sends C-D to us, then we set up a graft at
> + * "C".
> + *
> + * If the client has (broken)-B-C-D, then "C"
> + * _is_ accepted. The -B-C-D is an incomplete
> + * island that normally has no connectivity to
> + * the rest of the repo.
> + */
> + continue;
> + if ((flags & WRITE_SHALLOW_NO_UNUSED) &&
> +     !has_sha1_file(extra->array[i]))
> + continue;
> + strbuf_addstr(out, sha1_to_hex(extra->array[i]));
> + strbuf_addch(out, '\n');
> + data.count++;
> + }
>  return data.count;
> }
>
> void setup_alternate_shallow(struct lock_file *shallow_lock,
> -      const char **alternate_shallow_file)
> +      const char **alternate_shallow_file,
> +      struct extra_have_objects *extra,
> +      unsigned int flags)
> {
>  struct strbuf sb = STRBUF_INIT;
> + int rewrite = flags & WRITE_SHALLOW_REWRITE;
>  int fd;
>
> - check_shallow_file_for_update();
> - fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
> -        LOCK_DIE_ON_ERROR);
> - if (write_shallow_commits(&sb, 0)) {
> + if (!rewrite)
> + fd = hold_lock_file_for_update(shallow_lock,
> +        git_path("shallow"),
> +        LOCK_DIE_ON_ERROR);
> + else {
> + flags |= WRITE_SHALLOW_NO_UNUSED;
> + fd = shallow_lock->fd;
> + }
> + if (write_shallow_commits(&sb, 0, extra, flags)) {
> + if (rewrite && (lseek(fd, 0, SEEK_SET) == -1 ||
> + ftruncate(fd, 0) == -1))
> + die_errno("unable to truncate the new shallow file %s",
> +   shallow_lock->filename);
>  if (write_in_full(fd, sb.buf, sb.len) != sb.len)
>  die_errno("failed to write to %s",
>    shallow_lock->filename);
> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
> new file mode 100755
> index 0000000..15a8208
> --- /dev/null
> +++ b/t/t5536-fetch-shallow.sh
> @@ -0,0 +1,105 @@
> +#!/bin/sh
> +
> +test_description='fetch/clone from a shallow clone'
> +
> +. ./test-lib.sh
> +
> +commit() {
> + echo "$1" >tracked &&
> + git add tracked &&
> + git commit -m "$1"
> +}
> +
> +test_expect_success 'setup' '
> + commit 1 &&
> + commit 2 &&
> + commit 3 &&
> + commit 4 &&
> + git config --global transfer.fsckObjects true
> +'
> +
> +test_expect_success 'setup shallow clone' '
> + git clone --no-local --depth=2 .git shallow &&
> + git --git-dir=shallow/.git log --format=%s >actual &&
> + cat <<EOF >expect &&
> +4
> +3
> +EOF
> + test_cmp expect actual
> +'
> +
> +test_expect_success 'clone from shallow clone' '
> + git clone --no-local shallow shallow2 &&
> + (
> + cd shallow2 &&
> + git fsck &&
> + git log --format=%s >actual &&
> + cat <<EOF >expect &&
> +4
> +3
> +EOF
> + test_cmp expect actual
> + )
> +'
> +
> +test_expect_success 'fetch from shallow clone' '
> + (
> + cd shallow &&
> + commit 5
> + ) &&
> + (
> + cd shallow2 &&
> + git fetch &&
> + git fsck &&
> + git log --format=%s origin/master >actual &&
> + cat <<EOF >expect &&
> +5
> +4
> +3
> +EOF
> + test_cmp expect actual
> + )
> +'
> +
> +test_expect_success 'fetch --depth from shallow clone' '
> + (
> + cd shallow &&
> + commit 6
> + ) &&
> + (
> + cd shallow2 &&
> + git fetch --depth=2 &&
> + git fsck &&
> + git log --format=%s origin/master >actual &&
> + cat <<EOF >expect &&
> +6
> +5
> +EOF
> + test_cmp expect actual
> + )
> +'
> +
> +test_expect_success 'fetch something upstream has but hidden by 
> clients shallow boundaries' '
> + # the blob "1" is available in .git but hidden by the
> + # shallow2/.git/shallow and it should be resent
> + ! git --git-dir=shallow2/.git cat-file blob `echo 1|git 
> hash-object --stdin` >/dev/null &&
> + echo 1 > 1.t &&
> + git add 1.t &&
> + git commit -m add-1-back &&
> + (
> + cd shallow2 &&
> + git fetch ../.git +refs/heads/master:refs/remotes/top/master &&
> + git fsck &&
> + git log --format=%s top/master >actual &&
> + cat <<EOF >expect &&
> +add-1-back
> +4
> +3
> +EOF
> + test_cmp expect actual
> + ) &&
> + git --git-dir=shallow2/.git cat-file blob `echo 1|git 
> hash-object --stdin` >/dev/null
> +
> +'
> +
> +test_done
> diff --git a/transport.c b/transport.c
> index 10a8cb8..f167ca5 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -453,6 +453,7 @@ struct git_transport_data {
>  int fd[2];
>  unsigned got_remote_heads : 1;
>  struct extra_have_objects extra_have;
> + struct extra_have_objects shallow;
> };
>
> static int set_git_option(struct git_transport_options *opts,
> @@ -509,7 +510,8 @@ static struct ref *get_refs_via_connect(struct 
> transport *transport, int for_pus
>
>  connect_setup(transport, for_push, 0);
>  get_remote_heads(data->fd[0], NULL, 0, &refs,
> - for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
> + for_push ? REF_NORMAL : 0,
> + &data->extra_have, &data->shallow);
>  data->got_remote_heads = 1;
>
>  return refs;
> @@ -540,13 +542,13 @@ static int fetch_refs_via_pack(struct transport 
> *transport,
>  if (!data->got_remote_heads) {
>  connect_setup(transport, 0, 0);
>  get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
> - NULL, NULL);
> + NULL, &data->shallow);
>  data->got_remote_heads = 1;
>  }
>
>  refs = fetch_pack(&args, data->fd, data->conn,
>    refs_tmp ? refs_tmp : transport->remote_refs,
> -   dest, to_fetch, nr_heads,
> +   dest, to_fetch, nr_heads, &data->shallow,
>    &transport->pack_lockfile);
>  close(data->fd[0]);
>  close(data->fd[1]);
> @@ -803,6 +805,8 @@ static int git_transport_push(struct transport 
> *transport, struct ref *remote_re
>  get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, 
> NULL);
>  data->got_remote_heads = 1;
>  }
> + if (data->shallow.nr)
> + die("pushing to a shallow repository is not supported");
>
>  memset(&args, 0, sizeof(args));
>  args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
> -- 
> 1.8.2.83.gc99314b
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.3349 / Virus Database: 3204/6504 - Release Date: 
> 07/19/13
> 
Duy Nguyen· Jul 23, 2013, 02:06 UTC · re: Philip Oakley · lore

Re: [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository

On Tue, Jul 23, 2013 at 2:10 AM, Philip Oakley <philipoakley@iee.org> wrote:
Show 8 quoted lines
>>             And a full repository may automatically become shallow if
>> you fetch from a shallow repository.
>
>
> This "sounds" risky: Perhaps add a note about '[PATCH v2 15/16] config: add
> core.noshallow to prevent turning a repo into a shallow one' which will
> become available later in the series. The later core.noshallow may be a
> hostage to fortune.

core.noshallow is one way. Another is reject new shallow history (i.e. no additions to .git/shallow) unless the user explicitly asks so either via --depth or a new option --shallow. This does not mean that fetching from a shallow clone always fails without either of those options. The shallow clone may provide updates to an existing ref and _not_ introduce new shallow grafts and that would be successful. core.noshallow (or an opposite alternative) is probably the only way for push. -- Duy

Duy Nguyen· Jul 23, 2013, 14:39 UTC · re: Duy Nguyen · lore

Re: [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository

On Tue, Jul 23, 2013 at 9:06 AM, Duy Nguyen <pclouds@gmail.com> wrote:
Show 5 quoted lines
> Another is reject new shallow history (i.e.
> no additions to .git/shallow) unless the user explicitly asks so
> either via --depth or a new option --shallow. This does not mean that
> fetching from a shallow clone always fails without either of those
> options.

The above was at pack level when I wrote that, although I think, from the user perspective, rejecting at ref level makes more sense. That is if a fetch request returns one ref update with incremental updates and one with new shallow history, instead of rejecting the whole request, we reject the second ref update and accept the first one. -- Duy

Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 06/16] {send,receive}-pack: support pushing from a shallow clone

Pushing from a shallow clone using today's send-pack and receive-pack may work, if the transferred pack does not ends up at any graft points. If it does, recent receive-pack that does connectivity check will reject the push. If receive-pack is old, the upstream repo becomes corrupt.

The pack protocol is updated and send-pack now sends all shallow grafts before it sends the commands, if the repo is shallow. This protocol extension will break current receive-pack, which is intended, mostly to stop corrupting the upstream repo.

The receiver end, the newreceive-pack, does something similar to fetch-pack: it creates a temporary shallow file with grafts from send-pack, then receives the pack, and finally writes the refined shallow file down.

shadow file is not cleaned up after deleting (or force updating) a ref if that ref is the only way to reach the graft points. The reason is once we delete graft points, we can't recover. That may make reflog entries on server useless. Leave that for the administrators to decide when to clean up shadow file (maybe at repack/gc time).

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/technical/pack-protocol.txt |  4 +-
 builtin/receive-pack.c                    | 66 ++++++++++++++++++++++-----
 builtin/send-pack.c                       |  2 +-
 send-pack.c                               |  3 ++
 t/t5537-push-shallow.sh (new +x)          | 74 +++++++++++++++++++++++++++++++
 5 files changed, 137 insertions(+), 12 deletions(-)
 create mode 100755 t/t5537-push-shallow.sh
Show changes to 5 files +137 −12

Documentation/technical/pack-protocol.txt, builtin/receive-pack.c, builtin/send-pack.c, send-pack.c, t/t5537-push-shallow.sh

diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index eb8edd1..c73b62f 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -464,7 +464,9 @@ contain all the objects that the server will need to complete the new
 references.
 
 ----
-  update-request    =  command-list [pack-file]
+  update-request    =  *shallow command-list [pack-file]
+
+  shallow           =  PKT-LINE("shallow" SP obj-id)
 
   command-list      =  PKT-LINE(command NUL capability-list LF)
 		       *PKT-LINE(command LF)
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 6ffe526..cbb2025 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -41,6 +41,10 @@ static int auto_gc = 1;
 static const char *head_name;
 static void *head_name_to_free;
 static int sent_capabilities;
+static int shallow_changed;
+static const char* alternate_shallow_file;
+static struct lock_file shallow_lock;
+static struct extra_have_objects shallow;
 
 static enum deny_action parse_deny_action(const char *var, const char *value)
 {
@@ -751,6 +755,13 @@ static void execute_commands(struct command *commands, const char *unpacker_erro
 	}
 }
 
+static void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)
+{
+	ALLOC_GROW(extra->array, extra->nr + 1, extra->alloc);
+	hashcpy(&(extra->array[extra->nr][0]), sha1);
+	extra->nr++;
+}
+
 static struct command *read_head_info(void)
 {
 	struct command *commands = NULL;
@@ -765,6 +776,17 @@ static struct command *read_head_info(void)
 		line = packet_read_line(0, &len);
 		if (!line)
 			break;
+
+		if (len == 48 && !prefixcmp(line, "shallow ")) {
+			if (get_sha1_hex(line + 8, old_sha1))
+				die("protocol error: expected shallow sha, got '%s'", line + 8);
+			if (!has_sha1_file(old_sha1)) {
+				add_extra_have(&shallow, old_sha1);
+				shallow_changed = 1;
+			}
+			continue;
+		}
+
 		if (len < 83 ||
 		    line[40] != ' ' ||
 		    line[81] != ' ' ||
@@ -827,6 +849,13 @@ static const char *unpack(int err_fd)
 			    ? transfer_fsck_objects
 			    : 0);
 
+	if (shallow_changed)
+		setup_alternate_shallow(&shallow_lock,
+					&alternate_shallow_file,
+					&shallow,
+					WRITE_SHALLOW_NO_CUT);
+
+
 	hdr_err = parse_pack_header(&hdr);
 	if (hdr_err) {
 		if (err_fd > 0)
@@ -840,7 +869,12 @@ static const char *unpack(int err_fd)
 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
 		int code, i = 0;
 		struct child_process child;
-		const char *unpacker[5];
+		const char *unpacker[7];
+		if (alternate_shallow_file &&
+		    *alternate_shallow_file) {
+			unpacker[i++] = "--shallow-file";
+			unpacker[i++] = alternate_shallow_file;
+		}
 		unpacker[i++] = "unpack-objects";
 		if (quiet)
 			unpacker[i++] = "-q";
@@ -854,15 +888,19 @@ static const char *unpack(int err_fd)
 		child.err = err_fd;
 		child.git_cmd = 1;
 		code = run_command(&child);
-		if (!code)
-			return NULL;
-		return "unpack-objects abnormal exit";
+		if (code)
+			return "unpack-objects abnormal exit";
 	} else {
-		const char *keeper[7];
+		const char *keeper[9];
 		int s, status, i = 0;
 		char keep_arg[256];
 		struct child_process ip;
 
+		if (alternate_shallow_file &&
+		    *alternate_shallow_file) {
+			keeper[i++] = "--shallow-file";
+			keeper[i++] = alternate_shallow_file;
+		}
 		s = sprintf(keep_arg, "--keep=receive-pack %"PRIuMAX" on ", (uintmax_t) getpid());
 		if (gethostname(keep_arg + s, sizeof(keep_arg) - s))
 			strcpy(keep_arg + s, "localhost");
@@ -887,12 +925,20 @@ static const char *unpack(int err_fd)
 		pack_lockfile = index_pack_lockfile(ip.out);
 		close(ip.out);
 		status = finish_command(&ip);
-		if (!status) {
-			reprepare_packed_git();
-			return NULL;
-		}
-		return "index-pack abnormal exit";
+		if (status)
+			return "index-pack abnormal exit";
+		reprepare_packed_git();
+	}
+
+	if (shallow_changed) {
+		setup_alternate_shallow(&shallow_lock,
+					&alternate_shallow_file,
+					&shallow,
+					WRITE_SHALLOW_NO_CUT |
+					WRITE_SHALLOW_REWRITE);
+		commit_lock_file(&shallow_lock);
 	}
+	return NULL;
 }
 
 static const char *unpack_with_sideband(void)
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 4be5931..7dd254f 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -184,7 +184,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	    (send_all && args.send_mirror))
 		usage(send_pack_usage);
 
-	if (is_repository_shallow())
+	if (is_repository_shallow() && args.stateless_rpc)
 		die("attempt to push from a shallow repository");
 
 	if (remote_name) {
diff --git a/send-pack.c b/send-pack.c
index 7d172ef..dcd3345 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -212,6 +212,9 @@ int send_pack(struct send_pack_args *args,
 		return 0;
 	}
 
+	if (!args->dry_run)
+		advertise_shallow_grafts(out);
+
 	/*
 	 * Finally, tell the other end!
 	 */
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
new file mode 100755
index 0000000..37ad69b
--- /dev/null
+++ b/t/t5537-push-shallow.sh
@@ -0,0 +1,74 @@
+#!/bin/sh
+
+test_description='push from/to a shallow clone'
+
+. ./test-lib.sh
+
+commit() {
+	echo "$1" >tracked &&
+	git add tracked &&
+	git commit -m "$1"
+}
+
+test_expect_success 'setup' '
+	git config --global transfer.fsckObjects true &&
+	commit 1 &&
+	commit 2 &&
+	commit 3 &&
+	commit 4 &&
+	(
+	git init full-abc &&
+	cd full-abc &&
+	commit a &&
+	commit b &&
+	commit c
+	) &&
+	git clone --no-local --depth=2 .git shallow &&
+	git --git-dir=shallow/.git log --format=%s >actual &&
+	cat <<EOF >expect &&
+4
+3
+EOF
+	test_cmp expect actual &&
+	git clone --no-local --depth=2 full-abc/.git shallow2 &&
+	git --git-dir=shallow2/.git log --format=%s >actual &&
+	cat <<EOF >expect &&
+c
+b
+EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'push from shallow clone' '
+	(
+	cd shallow &&
+	commit 5 &&
+	git push ../.git +master:refs/remotes/shallow/master
+	) &&
+	git log --format=%s shallow/master >actual &&
+	git fsck &&
+	cat <<EOF >expect &&
+5
+4
+3
+2
+1
+EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'push from shallow clone, with grafted roots' '
+	(
+	cd shallow2 &&
+	git push ../.git +master:refs/remotes/shallow2/master
+	) &&
+	git log --format=%s shallow2/master >actual &&
+	git fsck &&
+	cat <<EOF >expect &&
+c
+b
+EOF
+	test_cmp expect actual
+'
+
+test_done
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 07/16] send-pack: support pushing to a shallow clone

When send-pack receives "shallow" lines from receive-pack, it knows the other end does not have a complete commit chains. It restrict itself to the commits that are not cut out by either end to make sure the result pack is usuable by receive-pack.

The same technique here, using setup_alternate_shallow() and --shallow-file, might simplify similar code in upload-pack.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/send-pack.c     |  7 +++++--
 send-pack.c             | 37 ++++++++++++++++++++++++++++++++-----
 send-pack.h             |  4 +++-
 t/t5537-push-shallow.sh | 36 ++++++++++++++++++++++++++++++++++++
 transport.c             |  7 +++----
 5 files changed, 79 insertions(+), 12 deletions(-)
Show changes to 5 files +79 −12

builtin/send-pack.c, send-pack.c, send-pack.h, t/t5537-push-shallow.sh, transport.c

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 7dd254f..14a73ba 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -94,6 +94,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int fd[2];
 	struct child_process *conn;
 	struct extra_have_objects extra_have;
+	struct extra_have_objects shallow;
 	struct ref *remote_refs, *local_refs;
 	int ret;
 	int helper_status = 0;
@@ -209,8 +210,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	}
 
 	memset(&extra_have, 0, sizeof(extra_have));
+	memset(&shallow, 0, sizeof(shallow));
 
-	get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);
+	get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,
+			 &extra_have, &shallow);
 
 	transport_verify_remote_names(nr_refspecs, refspecs);
 
@@ -230,7 +233,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	set_ref_status_for_push(remote_refs, args.send_mirror,
 		args.force_update);
 
-	ret = send_pack(&args, fd, conn, remote_refs, &extra_have);
+	ret = send_pack(&args, fd, conn, remote_refs, &extra_have, &shallow);
 
 	if (helper_status)
 		print_helper_status(remote_refs);
diff --git a/send-pack.c b/send-pack.c
index dcd3345..1b828b0 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -27,14 +27,19 @@ static int feed_object(const unsigned char *sha1, int fd, int negative)
 /*
  * Make a pack stream and spit it out into file descriptor fd
  */
-static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *extra, struct send_pack_args *args)
+static int pack_objects(int fd, struct ref *refs,
+			struct extra_have_objects *extra,
+			struct extra_have_objects *extra_shallow,
+			struct send_pack_args *args)
 {
 	/*
 	 * The child becomes pack-objects --revs; we feed
 	 * the revision parameters to it via its stdin and
 	 * let its stdout go back to the other end.
 	 */
-	const char *argv[] = {
+	const char *av[] = {
+		"--shallow-file",
+		NULL,
 		"pack-objects",
 		"--all-progress-implied",
 		"--revs",
@@ -45,10 +50,26 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
 		NULL,
 		NULL,
 	};
+	const char **argv;
 	struct child_process po;
+	static struct lock_file shallow_lock;
+	const char *alternate_shallow_file = NULL;
 	int i;
 
-	i = 4;
+	if (extra_shallow->nr) {
+		memset(&shallow_lock, 0, sizeof(shallow_lock));
+		/* just to load up .git/shallow if exists */
+		is_repository_shallow();
+		setup_alternate_shallow(&shallow_lock,
+					&alternate_shallow_file,
+					extra_shallow, 0);
+		av[1] = alternate_shallow_file;
+		argv = av;
+		i = 6;
+	} else {
+		argv = &av[2];
+		i = 4;
+	}
 	if (args->use_thin_pack)
 		argv[i++] = "--thin";
 	if (args->use_ofs_delta)
@@ -100,6 +121,10 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
 
 	if (finish_command(&po))
 		return -1;
+
+	if (extra_shallow->nr)
+		rollback_lock_file(&shallow_lock);
+
 	return 0;
 }
 
@@ -176,7 +201,8 @@ static int sideband_demux(int in, int out, void *data)
 int send_pack(struct send_pack_args *args,
 	      int fd[], struct child_process *conn,
 	      struct ref *remote_refs,
-	      struct extra_have_objects *extra_have)
+	      struct extra_have_objects *extra_have,
+	      struct extra_have_objects *extra_shallow)
 {
 	int in = fd[0];
 	int out = fd[1];
@@ -294,7 +320,8 @@ int send_pack(struct send_pack_args *args,
 	}
 
 	if (new_refs && cmds_sent) {
-		if (pack_objects(out, remote_refs, extra_have, args) < 0) {
+		if (pack_objects(out, remote_refs, extra_have, extra_shallow,
+				 args) < 0) {
 			for (ref = remote_refs; ref; ref = ref->next)
 				ref->status = REF_STATUS_NONE;
 			if (args->stateless_rpc)
diff --git a/send-pack.h b/send-pack.h
index 05d7ab1..1c8a343 100644
--- a/send-pack.h
+++ b/send-pack.h
@@ -16,6 +16,8 @@ struct send_pack_args {
 
 int send_pack(struct send_pack_args *args,
 	      int fd[], struct child_process *conn,
-	      struct ref *remote_refs, struct extra_have_objects *extra_have);
+	      struct ref *remote_refs,
+	      struct extra_have_objects *extra_have,
+	      struct extra_have_objects *extra_shallow);
 
 #endif
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index 37ad69b..beb85ce 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -71,4 +71,40 @@ EOF
 	test_cmp expect actual
 '
 
+test_expect_success 'push from shallow to shallow' '
+	(
+	cd shallow &&
+	git push ../shallow2/.git +master:refs/remotes/shallow/master
+	) &&
+	(
+	cd shallow2 &&
+	git log --format=%s shallow/master >actual &&
+	git fsck &&
+	cat <<EOF >expect &&
+5
+4
+3
+EOF
+	test_cmp expect actual
+	)
+'
+
+test_expect_success 'push from full to shallow' '
+	! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` &&
+	commit 1 &&
+	git push shallow2/.git +master:refs/remotes/top/master &&
+	(
+	cd shallow2 &&
+	git log --format=%s top/master >actual &&
+	git fsck &&
+	cat <<EOF >expect &&
+1
+4
+3
+EOF
+	test_cmp expect actual &&
+	git cat-file blob `echo 1|git hash-object --stdin` >/dev/null
+	)
+'
+
 test_done
diff --git a/transport.c b/transport.c
index f167ca5..d592e8d 100644
--- a/transport.c
+++ b/transport.c
@@ -802,11 +802,10 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 		struct ref *tmp_refs;
 		connect_setup(transport, 1, 0);
 
-		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
+		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,
+				 NULL, &data->shallow);
 		data->got_remote_heads = 1;
 	}
-	if (data->shallow.nr)
-		die("pushing to a shallow repository is not supported");
 
 	memset(&args, 0, sizeof(args));
 	args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
@@ -819,7 +818,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 	args.porcelain = !!(flags & TRANSPORT_PUSH_PORCELAIN);
 
 	ret = send_pack(&args, data->fd, data->conn, remote_refs,
-			&data->extra_have);
+			&data->extra_have, &data->shallow);
 
 	close(data->fd[1]);
 	close(data->fd[0]);
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 08/16] upload-pack: let pack-objects do the object counting in shallow case

Remove the duplicate object counting code in upload-pack, dump out all register_shallow()'d SHA-1 into a temporary shallow file and feed it to pack-objects. The end result is the same, except with less code, and fewer bytes sending over pipe to pack-objects.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 t/t5530-upload-pack-error.sh |   3 -
 upload-pack.c                | 128 ++++++++++---------------------------------
 2 files changed, 30 insertions(+), 101 deletions(-)
Show changes to 2 files +30 −101

t/t5530-upload-pack-error.sh, upload-pack.c

diff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh
index c983d36..3932e79 100755
--- a/t/t5530-upload-pack-error.sh
+++ b/t/t5530-upload-pack-error.sh
@@ -54,9 +54,6 @@ test_expect_success 'upload-pack fails due to error in rev-list' '
 	printf "0032want %s\n0034shallow %s00000009done\n0000" \
 		$(git rev-parse HEAD) $(git rev-parse HEAD^) >input &&
 	test_must_fail git upload-pack . <input >/dev/null 2>output.err &&
-	# pack-objects survived
-	grep "Total.*, reused" output.err &&
-	# but there was an error, which must have been in rev-list
 	grep "bad tree object" output.err
 '
 
diff --git a/upload-pack.c b/upload-pack.c
index c377a3e..c3e68ae 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -35,7 +35,6 @@ static int no_done;
 static int use_thin_pack, use_ofs_delta, use_include_tag;
 static int no_progress, daemon_mode;
 static int allow_tip_sha1_in_want;
-static int shallow_nr;
 static struct object_array have_obj;
 static struct object_array want_obj;
 static struct object_array extra_edge_obj;
@@ -68,87 +67,32 @@ static ssize_t send_client_data(int fd, const char *data, ssize_t sz)
 	return sz;
 }
 
-static FILE *pack_pipe = NULL;
-static void show_commit(struct commit *commit, void *data)
-{
-	if (commit->object.flags & BOUNDARY)
-		fputc('-', pack_pipe);
-	if (fputs(sha1_to_hex(commit->object.sha1), pack_pipe) < 0)
-		die("broken output pipe");
-	fputc('\n', pack_pipe);
-	fflush(pack_pipe);
-	free(commit->buffer);
-	commit->buffer = NULL;
-}
-
-static void show_object(struct object *obj,
-			const struct name_path *path, const char *component,
-			void *cb_data)
-{
-	show_object_with_name(pack_pipe, obj, path, component);
-}
-
-static void show_edge(struct commit *commit)
-{
-	fprintf(pack_pipe, "-%s\n", sha1_to_hex(commit->object.sha1));
-}
-
-static int do_rev_list(int in, int out, void *user_data)
-{
-	int i;
-	struct rev_info revs;
-
-	pack_pipe = xfdopen(out, "w");
-	init_revisions(&revs, NULL);
-	revs.tag_objects = 1;
-	revs.tree_objects = 1;
-	revs.blob_objects = 1;
-	if (use_thin_pack)
-		revs.edge_hint = 1;
-
-	for (i = 0; i < want_obj.nr; i++) {
-		struct object *o = want_obj.objects[i].item;
-		/* why??? */
-		o->flags &= ~UNINTERESTING;
-		add_pending_object(&revs, o, NULL);
-	}
-	for (i = 0; i < have_obj.nr; i++) {
-		struct object *o = have_obj.objects[i].item;
-		o->flags |= UNINTERESTING;
-		add_pending_object(&revs, o, NULL);
-	}
-	setup_revisions(0, NULL, &revs, NULL);
-	if (prepare_revision_walk(&revs))
-		die("revision walk setup failed");
-	mark_edges_uninteresting(revs.commits, &revs, show_edge);
-	if (use_thin_pack)
-		for (i = 0; i < extra_edge_obj.nr; i++)
-			fprintf(pack_pipe, "-%s\n", sha1_to_hex(
-					extra_edge_obj.objects[i].item->sha1));
-	traverse_commit_list(&revs, show_commit, show_object, NULL);
-	fflush(pack_pipe);
-	fclose(pack_pipe);
-	return 0;
-}
-
 static void create_pack_file(void)
 {
-	struct async rev_list;
 	struct child_process pack_objects;
 	char data[8193], progress[128];
 	char abort_msg[] = "aborting due to possible repository "
 		"corruption on the remote side.";
 	int buffered = -1;
 	ssize_t sz;
-	const char *argv[10];
-	int arg = 0;
+	const char *argv[12];
+	int i, arg = 0;
+	FILE *pipe_fd;
+	static struct lock_file shallow_lock;
+	const char *alternate_shallow_file = NULL;
+
+	setup_alternate_shallow(&shallow_lock,
+				&alternate_shallow_file,
+				NULL, 0);
+	if (*alternate_shallow_file) {
+		argv[arg++] = "--shallow-file";
+		argv[arg++] = alternate_shallow_file;
+	}
 
 	argv[arg++] = "pack-objects";
-	if (!shallow_nr) {
-		argv[arg++] = "--revs";
-		if (use_thin_pack)
-			argv[arg++] = "--thin";
-	}
+	argv[arg++] = "--revs";
+	if (use_thin_pack)
+		argv[arg++] = "--thin";
 
 	argv[arg++] = "--stdout";
 	if (!no_progress)
@@ -169,28 +113,18 @@ static void create_pack_file(void)
 	if (start_command(&pack_objects))
 		die("git upload-pack: unable to fork git-pack-objects");
 
-	if (shallow_nr) {
-		memset(&rev_list, 0, sizeof(rev_list));
-		rev_list.proc = do_rev_list;
-		rev_list.out = pack_objects.in;
-		if (start_async(&rev_list))
-			die("git upload-pack: unable to fork git-rev-list");
-	}
-	else {
-		FILE *pipe_fd = xfdopen(pack_objects.in, "w");
-		int i;
+	pipe_fd = xfdopen(pack_objects.in, "w");
 
-		for (i = 0; i < want_obj.nr; i++)
-			fprintf(pipe_fd, "%s\n",
-				sha1_to_hex(want_obj.objects[i].item->sha1));
-		fprintf(pipe_fd, "--not\n");
-		for (i = 0; i < have_obj.nr; i++)
-			fprintf(pipe_fd, "%s\n",
-				sha1_to_hex(have_obj.objects[i].item->sha1));
-		fprintf(pipe_fd, "\n");
-		fflush(pipe_fd);
-		fclose(pipe_fd);
-	}
+	for (i = 0; i < want_obj.nr; i++)
+		fprintf(pipe_fd, "%s\n",
+			sha1_to_hex(want_obj.objects[i].item->sha1));
+	fprintf(pipe_fd, "--not\n");
+	for (i = 0; i < have_obj.nr; i++)
+		fprintf(pipe_fd, "%s\n",
+			sha1_to_hex(have_obj.objects[i].item->sha1));
+	fprintf(pipe_fd, "\n");
+	fflush(pipe_fd);
+	fclose(pipe_fd);
 
 
 	/* We read from pack_objects.err to capture stderr output for
@@ -290,8 +224,9 @@ static void create_pack_file(void)
 		error("git upload-pack: git-pack-objects died with error.");
 		goto fail;
 	}
-	if (shallow_nr && finish_async(&rev_list))
-		goto fail;	/* error was already reported */
+
+	if (*alternate_shallow_file)
+		rollback_lock_file(&shallow_lock);
 
 	/* flush the data */
 	if (0 <= buffered) {
@@ -575,7 +510,6 @@ static void receive_needs(void)
 	int depth = 0;
 	int has_non_tip = 0;
 
-	shallow_nr = 0;
 	for (;;) {
 		struct object *o;
 		const char *features;
@@ -679,7 +613,6 @@ static void receive_needs(void)
 				packet_write(1, "shallow %s",
 						sha1_to_hex(object->sha1));
 				register_shallow(object->sha1);
-				shallow_nr++;
 			}
 			result = result->next;
 		}
@@ -715,7 +648,6 @@ static void receive_needs(void)
 				register_shallow(shallows.objects[i].item->sha1);
 		}
 
-	shallow_nr += shallows.nr;
 	free(shallows.objects);
 }
 
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 09/16] pack-protocol.txt: a bit about smart http

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/technical/pack-protocol.txt | 66 +++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
Show changes to Documentation/technical/pack-protocol.txt +66 −0
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index c73b62f..a1672bc 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -551,3 +551,69 @@ An example client/server communication might look like this:
    S: 0018ok refs/heads/debug\n
    S: 002ang refs/heads/master non-fast-forward\n
 ----
+
+Smart HTTP Transport
+--------------------
+
+Smart HTTP protocol is basically "git protocol on top of http". The
+base protocol is modified slightly to fit HTTP processing model: no
+bidirectional full-duplex connections, the program may read the
+request, write a response and must exit.
+
+Reference Discovery
+-------------------
+
+The server end always sends the list of references in both push and
+fetch cases. This ref list is retrieved by the client's sending HTTP
+GET request to a smart http url ending with
+"/info/refs?service=<service>" where <service> could be either
+git-upload-pack or git-receive-pack for fetching or pushing
+respectively. The output is in pkt-line format.
+
+----
+  advertised-refs  =  service
+		      flush-pkt
+		      (no-refs / list-of-refs)
+		      flush-pkt
+
+  service          =  PKT-LINE("# service=" service-name)
+  service-name     =  ("git-upload-pack" / "git-receive-pack")
+
+  no-refs          =  PKT-LINE(zero-id SP "capabilities^{}"
+		      NUL capability-list LF)
+
+  list-of-refs     =  first-ref *other-ref
+  first-ref        =  PKT-LINE(obj-id SP refname
+		      NUL capability-list LF)
+
+  other-ref        =  PKT-LINE(other-tip / other-peeled)
+  other-tip        =  obj-id SP refname LF
+  other-peeled     =  obj-id SP refname "^{}" LF
+
+  capability-list  =  capability *(SP capability)
+  capability       =  1*(LC_ALPHA / DIGIT / "-" / "_")
+  LC_ALPHA         =  %x61-7A
+----
+
+Packfile Negotiation
+--------------------
+
+For fetching, packet negotiation is via a series of HTTP POST requests
+to an url ending with "/git-upload-pack" with the content in pkt-line
+format. git-upload-pack's response consists of a "service" line like
+in Reference Discovery followed by normal git-upload-pack packet
+lines. Capability multi_ack_detailed is required by Smart HTTP.
+
+Common objects that are discovered are appended onto the request as
+have lines and are sent again on the next request. This allows the
+remote side to reinitialize its in-memory list of common objects
+during the next request and the remote does not need to maintain the
+negotiation state.
+
+Reference Update Request
+------------------------
+
+For pushing, a HTTP POST request is sent to an url ending with
+"/git-receive-pack" with the content in pkt-line format.
+git-receive-pack's response consists of a "service" line like in
+Reference Discovery followed by normal git-receive-pack packet lines.
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 10/16] Add document for command arguments for supporting smart http

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-fetch-pack.txt   | 11 +++++++----
 Documentation/git-receive-pack.txt | 16 +++++++++++++++-
 Documentation/git-send-pack.txt    |  9 ++++++++-
 Documentation/git-upload-pack.txt  | 13 ++++++++++++-
 4 files changed, 42 insertions(+), 7 deletions(-)
Show changes to 4 files +42 −7

Documentation/git-fetch-pack.txt, Documentation/git-receive-pack.txt, Documentation/git-send-pack.txt, Documentation/git-upload-pack.txt

diff --git a/Documentation/git-fetch-pack.txt b/Documentation/git-fetch-pack.txt
index 1e71754..85a9437 100644
--- a/Documentation/git-fetch-pack.txt
+++ b/Documentation/git-fetch-pack.txt
@@ -9,10 +9,7 @@ git-fetch-pack - Receive missing objects from another repository
 SYNOPSIS
 --------
 [verse]
-'git fetch-pack' [--all] [--quiet|-q] [--keep|-k] [--thin] [--include-tag]
-	[--upload-pack=<git-upload-pack>]
-	[--depth=<n>] [--no-progress]
-	[-v] [<host>:]<directory> [<refs>...]
+'git fetch-pack' [options] [<host>:]<directory> [<refs>...]
 
 DESCRIPTION
 -----------
@@ -90,6 +87,12 @@ be in a separate packet, and the list must end with a flush packet.
 --no-progress::
 	Do not show the progress.
 
+--stateless-rpc::
+	Smart HTTP mode.
+
+--lock-pack::
+	Issue "lock" command to the remote helper via stdout.
+
 -v::
 	Run verbosely.
 
diff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt
index b1f7dc6..b56d2eb 100644
--- a/Documentation/git-receive-pack.txt
+++ b/Documentation/git-receive-pack.txt
@@ -9,7 +9,7 @@ git-receive-pack - Receive what is pushed into the repository
 SYNOPSIS
 --------
 [verse]
-'git-receive-pack' <directory>
+'git-receive-pack' [options] <directory>
 
 DESCRIPTION
 -----------
@@ -35,6 +35,20 @@ are not fast-forwards.
 
 OPTIONS
 -------
+--stateless-rpc::
+	git-receive-pack performs only a single read-write cycle with
+	stdin and stdout to fit with the HTTP POST request processing
+	model where a program may read the request, write a response,
+	and must exit.
+
+--advertise-refs::
+	Only the initial ref advertisement is output then exits
+	immediately to fit with the HTTP GET request model, where no
+	request content is received but a response must be produced.
+
+--quiet::
+	Make unpack-objects at the receive-pack end quiet.
+
 <directory>::
 	The repository to sync into.
 
diff --git a/Documentation/git-send-pack.txt b/Documentation/git-send-pack.txt
index dc3a568..a88e7e0 100644
--- a/Documentation/git-send-pack.txt
+++ b/Documentation/git-send-pack.txt
@@ -9,7 +9,7 @@ git-send-pack - Push objects over Git protocol to another repository
 SYNOPSIS
 --------
 [verse]
-'git send-pack' [--all] [--dry-run] [--force] [--receive-pack=<git-receive-pack>] [--verbose] [--thin] [<host>:]<directory> [<ref>...]
+'git send-pack' [options] [<host>:]<directory> [<ref>...]
 
 DESCRIPTION
 -----------
@@ -52,6 +52,13 @@ OPTIONS
 	Send a "thin" pack, which records objects in deltified form based
 	on objects not included in the pack to reduce network traffic.
 
+--stateless-rpc::
+	Smart HTTP mode.
+
+--helper-status:
+	Issue status commands (e.g. "ok" or "error") to the remote
+	help via stdout.
+
 <host>::
 	A remote host to house the repository.  When this
 	part is specified, 'git-receive-pack' is invoked via
diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt
index 0abc806..98d73cc 100644
--- a/Documentation/git-upload-pack.txt
+++ b/Documentation/git-upload-pack.txt
@@ -9,7 +9,7 @@ git-upload-pack - Send objects packed back to git-fetch-pack
 SYNOPSIS
 --------
 [verse]
-'git-upload-pack' [--strict] [--timeout=<n>] <directory>
+'git-upload-pack' [options] <directory>
 
 DESCRIPTION
 -----------
@@ -31,6 +31,17 @@ OPTIONS
 --timeout=<n>::
 	Interrupt transfer after <n> seconds of inactivity.
 
+--stateless-rpc::
+	git-upload-pack performs only a single read-write cycle with
+	stdin and stdout to fit with the HTTP POST request processing
+	model where a program may read the request, write a response,
+	and must exit.
+
+--advertise-refs::
+	Only the initial ref advertisement is output then exits
+	immediately to fit with the HTTP GET request model, where no
+	request content is received but a response must be produced.
+
 <directory>::
 	The repository to sync from.
 
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 11/16] {fetch,upload}-pack: support fetching from a shallow clone via smart http

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/technical/pack-protocol.txt |  3 +++
 builtin/fetch-pack.c                      |  3 +--
 remote-curl.c                             |  4 +++-
 t/t5536-fetch-shallow.sh                  | 27 +++++++++++++++++++++++++++
 upload-pack.c                             |  2 --
 5 files changed, 34 insertions(+), 5 deletions(-)
Show changes to 5 files +34 −5

Documentation/technical/pack-protocol.txt, builtin/fetch-pack.c, remote-curl.c, t/t5536-fetch-shallow.sh, upload-pack.c

diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index a1672bc..5013652 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -574,6 +574,7 @@ respectively. The output is in pkt-line format.
   advertised-refs  =  service
 		      flush-pkt
 		      (no-refs / list-of-refs)
+		      *shallow
 		      flush-pkt
 
   service          =  PKT-LINE("# service=" service-name)
@@ -590,6 +591,8 @@ respectively. The output is in pkt-line format.
   other-tip        =  obj-id SP refname LF
   other-peeled     =  obj-id SP refname "^{}" LF
 
+  shallow           =  PKT-LINE("shallow" SP obj-id)
+
   capability-list  =  capability *(SP capability)
   capability       =  1*(LC_ALPHA / DIGIT / "-" / "_")
   LC_ALPHA         =  %x61-7A
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index f6a6d76..b89d753 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -146,8 +146,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 	}
 
 	memset(&shallow, 0, sizeof(shallow));
-	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL,
-			 args.stateless_rpc ? NULL : &shallow);
+	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
 
 	ref = fetch_pack(&args, fd, conn, ref, dest,
 			 sought, nr_sought, &shallow, pack_lockfile_ptr);
diff --git a/remote-curl.c b/remote-curl.c
index c329bd3..de2cc8a 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -78,6 +78,7 @@ struct discovery {
 	char *buf;
 	size_t len;
 	struct ref *refs;
+	struct extra_have_objects shallow;
 	unsigned proto_git : 1;
 };
 static struct discovery *last_discovery;
@@ -86,7 +87,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
 {
 	struct ref *list = NULL;
 	get_remote_heads(-1, heads->buf, heads->len, &list,
-			 for_push ? REF_NORMAL : 0, NULL, NULL);
+			 for_push ? REF_NORMAL : 0, NULL, &heads->shallow);
 	return list;
 }
 
@@ -146,6 +147,7 @@ static void free_discovery(struct discovery *d)
 	if (d) {
 		if (d == last_discovery)
 			last_discovery = NULL;
+		free(d->shallow.array);
 		free(d->buf_alloc);
 		free_refs(d->refs);
 		free(d);
diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
index 15a8208..6ea6347 100755
--- a/t/t5536-fetch-shallow.sh
+++ b/t/t5536-fetch-shallow.sh
@@ -102,4 +102,31 @@ EOF
 
 '
 
+if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
+	say 'skipping remaining tests, git built without http support'
+	test_done
+fi
+
+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5536'}
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
+
+test_expect_success 'clone http repository' '
+	git clone --bare --no-local shallow "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git clone --quiet $HTTPD_URL/smart/repo.git clone &&
+	(
+	cd clone &&
+	git fsck &&
+	git log --format=%s origin/master >actual &&
+	cat <<EOF >expect &&
+6
+5
+4
+3
+EOF
+	test_cmp expect actual
+	)
+'
+
+stop_httpd
 test_done
diff --git a/upload-pack.c b/upload-pack.c
index c3e68ae..263ae08 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -770,8 +770,6 @@ int main(int argc, char **argv)
 
 	if (!enter_repo(dir, strict))
 		die("'%s' does not appear to be a git repository", dir);
-	if (is_repository_shallow() && stateless_rpc)
-		die("attempt to push into a shallow repository");
 
 	git_config(upload_pack_config, NULL);
 	upload_pack();
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 12/16] receive-pack: support pushing to a shallow clone via http

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/receive-pack.c  |  3 ---
 t/t5537-push-shallow.sh | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 3 deletions(-)
Show changes to 2 files +35 −3

builtin/receive-pack.c, t/t5537-push-shallow.sh

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index cbb2025..54bf6b2 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1038,9 +1038,6 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 	if (!enter_repo(dir, 0))
 		die("'%s' does not appear to be a git repository", dir);
 
-	if (is_repository_shallow() && stateless_rpc)
-		die("attempt to push into a shallow repository");
-
 	git_config(receive_pack_config, NULL);
 
 	if (0 <= transfer_unpack_limit)
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index beb85ce..02d9198 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -16,6 +16,7 @@ test_expect_success 'setup' '
 	commit 2 &&
 	commit 3 &&
 	commit 4 &&
+	git clone . full &&
 	(
 	git init full-abc &&
 	cd full-abc &&
@@ -107,4 +108,38 @@ EOF
 	)
 '
 
+if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
+	say 'skipping remaining tests, git built without http support'
+	test_done
+fi
+
+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5537'}
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
+
+test_expect_success 'push to shallow repo via http' '
+	git clone --bare --no-local shallow "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	(
+	cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git config http.receivepack true
+	) &&
+	(
+	cd full &&
+	commit 9 &&
+	git push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master
+	) &&
+	(
+	cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git fsck &&
+	git log --format=%s top/master >actual &&
+	cat <<EOF >expect &&
+9
+4
+3
+EOF
+	test_cmp expect actual
+	)
+'
+
+stop_httpd
 test_done
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 13/16] send-pack: support pushing from a shallow clone via http

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/send-pack.c     |  3 ---
 send-pack.c             | 19 +++++++++++++++++--
 t/t5537-push-shallow.sh | 25 +++++++++++++++++++++++++
 3 files changed, 42 insertions(+), 5 deletions(-)
Show changes to 3 files +42 −5

builtin/send-pack.c, send-pack.c, t/t5537-push-shallow.sh

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 14a73ba..b177120 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -185,9 +185,6 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	    (send_all && args.send_mirror))
 		usage(send_pack_usage);
 
-	if (is_repository_shallow() && args.stateless_rpc)
-		die("attempt to push from a shallow repository");
-
 	if (remote_name) {
 		remote = remote_get(remote_name);
 		if (!remote_has_url(remote, dest)) {
diff --git a/send-pack.c b/send-pack.c
index 1b828b0..520c838 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -198,6 +198,21 @@ static int sideband_demux(int in, int out, void *data)
 	return ret;
 }
 
+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)
+{
+	struct strbuf *sb = cb;
+	if (graft->nr_parent == -1)
+		packet_buf_write(sb, "shallow %s\n", sha1_to_hex(graft->sha1));
+	return 0;
+}
+
+void advertise_shallow_grafts_buf(struct strbuf *sb)
+{
+	if (!is_repository_shallow())
+		return;
+	for_each_commit_graft(advertise_shallow_grafts_cb, sb);
+}
+
 int send_pack(struct send_pack_args *args,
 	      int fd[], struct child_process *conn,
 	      struct ref *remote_refs,
@@ -239,7 +254,7 @@ int send_pack(struct send_pack_args *args,
 	}
 
 	if (!args->dry_run)
-		advertise_shallow_grafts(out);
+		advertise_shallow_grafts_buf(&req_buf);
 
 	/*
 	 * Finally, tell the other end!
@@ -299,7 +314,7 @@ int send_pack(struct send_pack_args *args,
 	}
 
 	if (args->stateless_rpc) {
-		if (!args->dry_run && cmds_sent) {
+		if (!args->dry_run && (cmds_sent || is_repository_shallow())) {
 			packet_buf_flush(&req_buf);
 			send_sideband(out, -1, req_buf.buf, req_buf.len, LARGE_PACKET_MAX);
 		}
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index 02d9198..8bea496 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -141,5 +141,30 @@ EOF
 	)
 '
 
+test_expect_success 'push from shallow repo via http' '
+	mv "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" shallow-upstream.git &&
+	git clone --bare --no-local full "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	(
+	cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git config http.receivepack true
+	) &&
+	commit 10 &&
+	git push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master &&
+	(
+	cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git fsck &&
+	git log --format=%s top/master >actual &&
+	cat <<EOF >expect &&
+10
+1
+4
+3
+2
+1
+EOF
+	test_cmp expect actual
+	)
+'
+
 stop_httpd
 test_done
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 14/16] git-clone.txt: remove shallow clone limitations

Now that git supports push/pull from/to a shallow clone, these limitations are not true anymore.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-clone.txt | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
Show changes to Documentation/git-clone.txt +1 −6
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 450f158..4987857 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -181,12 +181,7 @@ objects from the source repository into a pack in the cloned repository.
 
 --depth <depth>::
 	Create a 'shallow' clone with a history truncated to the
-	specified number of revisions.  A shallow repository has a
-	number of limitations (you cannot clone or fetch from
-	it, nor push from nor into it), but is adequate if you
-	are only interested in the recent history of a large project
-	with a long history, and would want to send in fixes
-	as patches.
+	specified number of revisions.
 
 --[no-]single-branch::
 	Clone only the history leading to the tip of a single branch,
-- 
1.8.2.83.gc99314b
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/config.txt | 5 +++++
 builtin/receive-pack.c   | 9 ++++++++-
 cache.h                  | 1 +
 config.c                 | 5 +++++
 environment.c            | 1 +
 fetch-pack.c             | 9 ++++++++-
 t/t5536-fetch-shallow.sh | 9 +++++++++
 t/t5537-push-shallow.sh  | 6 ++++++
 8 files changed, 43 insertions(+), 2 deletions(-)
Show changes to 8 files +43 −2

Documentation/config.txt, builtin/receive-pack.c, cache.h, config.c, environment.c, fetch-pack.c, t/t5536-fetch-shallow.sh, t/t5537-push-shallow.sh

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 81856dd..e811180 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -655,6 +655,11 @@ core.abbrev::
 	for abbreviated object names to stay unique for sufficiently long
 	time.
 
+core.noshallow::
+	If true, reject any pushes or fetches that may turn the
+	repository into a shallow one. This setting is ignored if the
+	repository is already shallow.
+
 add.ignore-errors::
 add.ignoreErrors::
 	Tells 'git add' to continue adding files when some files cannot be
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 54bf6b2..95ea481 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -936,7 +936,14 @@ static const char *unpack(int err_fd)
 					&shallow,
 					WRITE_SHALLOW_NO_CUT |
 					WRITE_SHALLOW_REWRITE);
-		commit_lock_file(&shallow_lock);
+		if (*alternate_shallow_file == '\0') {
+			unlink_or_warn(git_path("shallow"));
+			rollback_lock_file(&shallow_lock);
+		} else {
+			if (!is_repository_shallow() && cannot_be_shallow)
+				die("not allowed to turn this repository into a shallow one");
+			commit_lock_file(&shallow_lock);
+		}
 	}
 	return NULL;
 }
diff --git a/cache.h b/cache.h
index 7f17228..3a52b08 100644
--- a/cache.h
+++ b/cache.h
@@ -592,6 +592,7 @@ extern int fsync_object_files;
 extern int core_preload_index;
 extern int core_apply_sparse_checkout;
 extern int precomposed_unicode;
+extern int cannot_be_shallow;
 
 /*
  * The character that begins a commented line in user-editable file
diff --git a/config.c b/config.c
index d04e815..31f5a57 100644
--- a/config.c
+++ b/config.c
@@ -784,6 +784,11 @@ static int git_default_core_config(const char *var, const char *value)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.noshallow")) {
+		cannot_be_shallow = git_config_bool(var, value);
+		return 0;
+	}
+
 	/* Add other config variables here and to Documentation/config.txt. */
 	return 0;
 }
diff --git a/environment.c b/environment.c
index 0cb67b2..14c8005 100644
--- a/environment.c
+++ b/environment.c
@@ -61,6 +61,7 @@ int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 struct startup_info *startup_info;
 unsigned long pack_size_limit_cfg;
+int cannot_be_shallow;
 
 /*
  * The character that begins a commented line in user-editable file
diff --git a/fetch-pack.c b/fetch-pack.c
index f337526..40e7aa2 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -960,7 +960,14 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 							WRITE_SHALLOW_NO_CUT |
 							WRITE_SHALLOW_REWRITE);
 			}
-			commit_lock_file(&shallow_lock);
+			if (*alternate_shallow_file == '\0') {
+				unlink_or_warn(git_path("shallow"));
+				rollback_lock_file(&shallow_lock);
+			} else {
+				if (!is_repository_shallow() && cannot_be_shallow)
+					die("not allowed to turn this repository into a shallow one");
+				commit_lock_file(&shallow_lock);
+			}
 		}
 	}
 
diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
index 6ea6347..b7f89b1 100755
--- a/t/t5536-fetch-shallow.sh
+++ b/t/t5536-fetch-shallow.sh
@@ -102,6 +102,15 @@ EOF
 
 '
 
+test_expect_success 'core.noshallow' '
+	git init clean &&
+	(
+	cd clean &&
+	git config core.noshallow true &&
+	test_must_fail git fetch ../shallow/.git
+	)
+'
+
 if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
 	say 'skipping remaining tests, git built without http support'
 	test_done
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index 8bea496..0edd51f 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -108,6 +108,12 @@ EOF
 	)
 '
 
+test_expect_success 'core.noshallow' '
+	git init clean &&
+	git --git-dir=clean/.git config core.noshallow true &&
+	test_must_fail git --git-dir=shallow/.git push clean master:refs/remotes/shallow/master
+'
+
 if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
 	say 'skipping remaining tests, git built without http support'
 	test_done
-- 
1.8.2.83.gc99314b
Philip Oakley· Jul 22, 2013, 19:23 UTC · re: Nguyễn Thái Ngọc Duy · lore

Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one

From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
Subject: [PATCH v2 15/16] config: add core.noshallow to prevent turning 
a repo into a shallow one

Surely this should be the default now that it is possible to corrupt a golden repo by pushing/fetching a shallow repository to it and it then becomes shallow, and all the followers become shallow as well, with consequent problems (IIUC) [PATCH v2 05/16].

It would be just as easy to change the config to core.allowshallow which then must be enabled by the user, and can be mentioned in the shallow clone option's documentation.

Show 166 quoted lines
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
> Documentation/config.txt | 5 +++++
> builtin/receive-pack.c   | 9 ++++++++-
> cache.h                  | 1 +
> config.c                 | 5 +++++
> environment.c            | 1 +
> fetch-pack.c             | 9 ++++++++-
> t/t5536-fetch-shallow.sh | 9 +++++++++
> t/t5537-push-shallow.sh  | 6 ++++++
> 8 files changed, 43 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 81856dd..e811180 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -655,6 +655,11 @@ core.abbrev::
>  for abbreviated object names to stay unique for sufficiently long
>  time.
>
> +core.noshallow::
> + If true, reject any pushes or fetches that may turn the
> + repository into a shallow one. This setting is ignored if the
> + repository is already shallow.
> +
> add.ignore-errors::
> add.ignoreErrors::
>  Tells 'git add' to continue adding files when some files cannot be
> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index 54bf6b2..95ea481 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -936,7 +936,14 @@ static const char *unpack(int err_fd)
>  &shallow,
>  WRITE_SHALLOW_NO_CUT |
>  WRITE_SHALLOW_REWRITE);
> - commit_lock_file(&shallow_lock);
> + if (*alternate_shallow_file == '\0') {
> + unlink_or_warn(git_path("shallow"));
> + rollback_lock_file(&shallow_lock);
> + } else {
> + if (!is_repository_shallow() && cannot_be_shallow)
> + die("not allowed to turn this repository into a shallow one");
> + commit_lock_file(&shallow_lock);
> + }
>  }
>  return NULL;
> }
> diff --git a/cache.h b/cache.h
> index 7f17228..3a52b08 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -592,6 +592,7 @@ extern int fsync_object_files;
> extern int core_preload_index;
> extern int core_apply_sparse_checkout;
> extern int precomposed_unicode;
> +extern int cannot_be_shallow;
>
> /*
>  * The character that begins a commented line in user-editable file
> diff --git a/config.c b/config.c
> index d04e815..31f5a57 100644
> --- a/config.c
> +++ b/config.c
> @@ -784,6 +784,11 @@ static int git_default_core_config(const char 
> *var, const char *value)
>  return 0;
>  }
>
> + if (!strcmp(var, "core.noshallow")) {
> + cannot_be_shallow = git_config_bool(var, value);
> + return 0;
> + }
> +
>  /* Add other config variables here and to Documentation/config.txt. 
> */
>  return 0;
> }
> diff --git a/environment.c b/environment.c
> index 0cb67b2..14c8005 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -61,6 +61,7 @@ int merge_log_config = -1;
> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() 
> */
> struct startup_info *startup_info;
> unsigned long pack_size_limit_cfg;
> +int cannot_be_shallow;
>
> /*
>  * The character that begins a commented line in user-editable file
> diff --git a/fetch-pack.c b/fetch-pack.c
> index f337526..40e7aa2 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -960,7 +960,14 @@ struct ref *fetch_pack(struct fetch_pack_args 
> *args,
>  WRITE_SHALLOW_NO_CUT |
>  WRITE_SHALLOW_REWRITE);
>  }
> - commit_lock_file(&shallow_lock);
> + if (*alternate_shallow_file == '\0') {
> + unlink_or_warn(git_path("shallow"));
> + rollback_lock_file(&shallow_lock);
> + } else {
> + if (!is_repository_shallow() && cannot_be_shallow)
> + die("not allowed to turn this repository into a shallow one");
> + commit_lock_file(&shallow_lock);
> + }
>  }
>  }
>
> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
> index 6ea6347..b7f89b1 100755
> --- a/t/t5536-fetch-shallow.sh
> +++ b/t/t5536-fetch-shallow.sh
> @@ -102,6 +102,15 @@ EOF
>
> '
>
> +test_expect_success 'core.noshallow' '
> + git init clean &&
> + (
> + cd clean &&
> + git config core.noshallow true &&
> + test_must_fail git fetch ../shallow/.git
> + )
> +'
> +
> if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
>  say 'skipping remaining tests, git built without http support'
>  test_done
> diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
> index 8bea496..0edd51f 100755
> --- a/t/t5537-push-shallow.sh
> +++ b/t/t5537-push-shallow.sh
> @@ -108,6 +108,12 @@ EOF
>  )
> '
>
> +test_expect_success 'core.noshallow' '
> + git init clean &&
> + git --git-dir=clean/.git config core.noshallow true &&
> + test_must_fail git --git-dir=shallow/.git push clean 
> master:refs/remotes/shallow/master
> +'
> +
> if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
>  say 'skipping remaining tests, git built without http support'
>  test_done
> -- 
> 1.8.2.83.gc99314b
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.3349 / Virus Database: 3204/6504 - Release Date: 
> 07/19/13
> 
Duy Nguyen· Jul 23, 2013, 01:28 UTC · re: Philip Oakley · lore

Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one

On Tue, Jul 23, 2013 at 2:23 AM, Philip Oakley <philipoakley@iee.org> wrote:
Show 12 quoted lines
> From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
> Subject: [PATCH v2 15/16] config: add core.noshallow to prevent turning a
> repo into a shallow one
>
> Surely this should be the default now that it is possible to corrupt a
> golden repo by pushing/fetching a shallow repository to it and it then
> becomes shallow, and all the followers become shallow as well, with
> consequent problems (IIUC) [PATCH v2 05/16].
>
> It would be just as easy to change the config to core.allowshallow which
> then must be enabled by the user, and can be mentioned in the shallow clone
> option's documentation.

Clarification, it's not really "corrupt". If you have full history from a ref "A", fetching from another shallow clone does not touch the history of ref A at all (that is if you do _not_ specify --depth). It may add a a shallow ref B, which is the reason the whole repo becomes shallow. The same goes for push. This is not implemented, but I'm thinking of adding "clean .git/shallow" to git repack -ad. Then if you delete ref B and repack -ad, the repo could become full again.

But yeah, maybe defaulting to no shallow is better. Will do so in the reroll unless someone objects. -- Duy

Junio C Hamano· Jul 23, 2013, 04:06 UTC · re: Duy Nguyen · lore

Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one

Duy Nguyen <pclouds@gmail.com> writes:
> But yeah, maybe defaulting to no shallow is better. Will do so in the
> reroll unless someone objects.

No objections from me ;-). Thanks.

Philip Oakley· Jul 23, 2013, 19:44 UTC · re: Duy Nguyen · lore

Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one

From: "Duy Nguyen" <pclouds@gmail.com>
Sent: Tuesday, July 23, 2013 2:28 AM
Show 23 quoted lines
> On Tue, Jul 23, 2013 at 2:23 AM, Philip Oakley <philipoakley@iee.org> 
> wrote:
>> From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
>> Subject: [PATCH v2 15/16] config: add core.noshallow to prevent 
>> turning a
>> repo into a shallow one
>>
>> Surely this should be the default now that it is possible to corrupt 
>> a
>> golden repo by pushing/fetching a shallow repository to it and it 
>> then
>> becomes shallow, and all the followers become shallow as well, with
>> consequent problems (IIUC) [PATCH v2 05/16].
>>
>> It would be just as easy to change the config to core.allowshallow 
>> which
>> then must be enabled by the user, and can be mentioned in the shallow 
>> clone
>> option's documentation.
>
> Clarification, it's not really "corrupt". If you have full history
> from a ref "A", fetching from another shallow clone does not touch the
> history of ref A at all
>   (that is if you do _not_ specify --depth).
I hadn't appreciated this conditional.
> It
> may add a a shallow ref B, which is the reason the whole repo becomes
> shallow.

I had read the initial commit message (in 05/16) as implying that it was possible to fool someone into pulling a shallow repo and that would make their repo just as shallow (that's without a --depth argument to the command). Had that been the case then it would have been possible to loose some data from deep in the DAG. Glad to hear I was mistaken. Perhaps if your comment above is included in the commit message to ensure that clarification is there.

Show 9 quoted lines
>The same goes for push. This is not implemented, but I'm
> thinking of adding "clean .git/shallow" to git repack -ad. Then if you
> delete ref B and repack -ad, the repo could become full again.
>
> But yeah, maybe defaulting to no shallow is better. Will do so in the
> reroll unless someone objects.
> --
> Duy
> --
Philip 
Nguyễn Thái Ngọc Duy· Jul 20, 2013, 09:58 UTC · re: Nguyễn Thái Ngọc Duy · lore

[PATCH v2 16/16] clone: use git protocol for cloning shallow repo locally

clone_local() does not handle $SRC/shallow. It could be made so, but it's simpler to use fetch-pack/upload-pack instead.

This used by be caught by the check in upload-pack, which is triggered by transport_get_remote_refs(), even in local clone case. The check is now gone and check_everything_connected() should catch the result incomplete repo. But check_everything_connected() will soon be skipped in local clone case, opening a door to corrupt repo. This patch should close that door.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/clone.c  | 14 ++++++++++++--
 t/t5601-clone.sh |  7 +++++++
 2 files changed, 19 insertions(+), 2 deletions(-)
Show changes to 2 files +19 −2

builtin/clone.c, t/t5601-clone.sh

diff --git a/builtin/clone.c b/builtin/clone.c
index 17f57cd..09e383e 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -789,8 +789,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	else
 		repo = repo_name;
 	is_local = option_local != 0 && path && !is_bundle;
-	if (is_local && option_depth)
-		warning(_("--depth is ignored in local clones; use file:// instead."));
+	if (is_local) {
+		struct strbuf sb = STRBUF_INIT;
+		if (option_depth)
+			warning(_("--depth is ignored in local clones; use file:// instead."));
+		strbuf_addf(&sb, "%s/shallow", path);
+		if (!access(sb.buf, F_OK)) {
+			if (option_local > 0)
+				warning(_("source repository is shallow, ignoring --local"));
+			is_local = 0;
+		}
+		strbuf_release(&sb);
+	}
 	if (option_local > 0 && !is_local)
 		warning(_("--local is ignored"));
 
diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
index 0629149..35a2a3d 100755
--- a/t/t5601-clone.sh
+++ b/t/t5601-clone.sh
@@ -285,4 +285,11 @@ test_expect_success NOT_MINGW,NOT_CYGWIN 'clone local path foo:bar' '
 	git clone "./foo:bar" foobar
 '
 
+test_expect_success 'shallow clone locally' '
+	git clone --depth=1 --no-local src ssrrcc &&
+	git clone ssrrcc ddsstt &&
+	test_cmp ssrrcc/.git/shallow ddsstt/.git/shallow &&
+	( cd ddsstt && git fsck )
+'
+
 test_done
-- 
1.8.2.83.gc99314b
Philip Oakley· Jul 22, 2013, 23:41 UTC · re: Nguyễn Thái Ngọc Duy · lore

Re: [PATCH v2 00/16] First class shallow clone

From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
Subject: [PATCH v2 00/16] First class shallow clone
It's nice to see that shallow can be a first class clone.

Thinking outside the box, does this infrastructure offer the opportunity to maybe add a date based depth option that would establish the shallow watermark based on date rather than count. (e.g. the "deepen" SP depth could have an alternate with a leading 'T' to indicate a time limit ratherv than revision count - I'm expecting such a format would be an error for existing servers).

My other thought was this style of cut limit list may also allow a big file limit to do a similar process of listing objects (e.g. blobs) that are size-shallow in the repo, though it maybe a long list on some repos, or with a small size limit.

Philip
Show 82 quoted lines
> v2 includes:
>
> - fix Junio comments, especially the one that may lead to incomplete
>   commit islands.
> - fix send-pack setting up temporary shallow file, but never passes
>   it to index-pack/unpack-objects (also fix the tests to catch this)
> - support smart http
> - add core.noshallow for repos that wish to be always complete
> - fix locally cloning a shallow repository
> - make upload-pack pass --shallow-file to pack-objects in order to
>   remove duplicate object counting code just for shallow case.
>
> Nguyễn Thái Ngọc Duy (16):
>  send-pack: forbid pushing from a shallow repository
>  {receive,upload}-pack: advertise shallow graft information
>  connect.c: teach get_remote_heads to parse "shallow" lines
>  Move setup_alternate_shallow and write_shallow_commits to shallow.c
>  fetch-pack: support fetching from a shallow repository
>  {send,receive}-pack: support pushing from a shallow clone
>  send-pack: support pushing to a shallow clone
>  upload-pack: let pack-objects do the object counting in shallow case
>  pack-protocol.txt: a bit about smart http
>  Add document for command arguments for supporting smart http
>  {fetch,upload}-pack: support fetching from a shallow clone via smart 
> http
>  receive-pack: support pushing to a shallow clone via http
>  send-pack: support pushing from a shallow clone via http
>  git-clone.txt: remove shallow clone limitations
>  config: add core.noshallow to prevent turning a repo into a shallow 
> one
>  clone: use git protocol for cloning shallow repo locally
>
> Documentation/config.txt                  |   5 +
> Documentation/git-clone.txt               |   7 +-
> Documentation/git-fetch-pack.txt          |  11 +-
> Documentation/git-receive-pack.txt        |  16 ++-
> Documentation/git-send-pack.txt           |   9 +-
> Documentation/git-upload-pack.txt         |  13 ++-
> Documentation/technical/pack-protocol.txt |  76 ++++++++++++-
> builtin/clone.c                           |  14 ++-
> builtin/fetch-pack.c                      |   6 +-
> builtin/receive-pack.c                    |  76 +++++++++++--
> builtin/send-pack.c                       |   7 +-
> cache.h                                   |   4 +-
> commit.h                                  |  27 +++++
> config.c                                  |   5 +
> connect.c                                 |  12 +-
> environment.c                             |   1 +
> fetch-pack.c                              |  90 ++++++---------
> fetch-pack.h                              |   1 +
> remote-curl.c                             |   4 +-
> send-pack.c                               |  57 +++++++++-
> send-pack.h                               |   4 +-
> shallow.c                                 | 147 
> +++++++++++++++++++++++++
> t/t5530-upload-pack-error.sh              |   3 -
> t/t5536-fetch-shallow.sh (new +x)         | 141 
> ++++++++++++++++++++++++
> t/t5537-push-shallow.sh (new +x)          | 176 
> ++++++++++++++++++++++++++++++
> t/t5601-clone.sh                          |   7 ++
> transport.c                               |  14 ++-
> upload-pack.c                             | 132 ++++++----------------
> 28 files changed, 858 insertions(+), 207 deletions(-)
> create mode 100755 t/t5536-fetch-shallow.sh
> create mode 100755 t/t5537-push-shallow.sh
>
> -- 
> 1.8.2.83.gc99314b
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.3349 / Virus Database: 3204/6504 - Release Date: 
> 07/19/13
> 
Duy Nguyen· Jul 23, 2013, 01:20 UTC · re: Philip Oakley · lore

Re: [PATCH v2 00/16] First class shallow clone

On Tue, Jul 23, 2013 at 6:41 AM, Philip Oakley <philipoakley@iee.org> wrote:
Show 8 quoted lines
> From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
> Subject: [PATCH v2 00/16] First class shallow clone
>
> It's nice to see that shallow can be a first class clone.
>
> Thinking outside the box, does this infrastructure offer the opportunity to
> maybe add a date based depth option that would establish the shallow
> watermark based on date rather than count. (e.g. the "deepen" SP depth could

I've been carefully avoiding the deepen issues because, as you see, it's complicated. But no, this series does not enable or disable new deeepen mechanisms. They can always be added as protocol extensions. Still thinking if it's worth exposing a (restricted form of) rev-list to the protocol..

Show 8 quoted lines
> have an alternate with a leading 'T' to indicate a time limit ratherv than
> revision count - I'm expecting such a format would be an error for existing
> servers).
>
> My other thought was this style of cut limit list may also allow a big file
> limit to do a similar process of listing objects (e.g. blobs) that are
> size-shallow in the repo, though it maybe a long list on some repos, or with
> a small size limit.

This one, on the other hand, changes the "shape" of the repo (now with holes) and might need to go through the same process we do with this series. Maybe we should prepare for it now. Do you have a use case for size-based filtering? What can we do with a repo with some arbitrary blobs missing? Another form of this is narrow clone, where we cut by paths, not by blob size. Narrow clone sounds more useful to me because it's easier to control what we leave out. -- Duy

Junio C Hamano· Jul 23, 2013, 04:08 UTC · re: Duy Nguyen · lore

Re: [PATCH v2 00/16] First class shallow clone

Duy Nguyen <pclouds@gmail.com> writes:
Show 7 quoted lines
> This one, on the other hand, changes the "shape" of the repo (now with
> holes) and might need to go through the same process we do with this
> series. Maybe we should prepare for it now. Do you have a use case for
> size-based filtering? What can we do with a repo with some arbitrary
> blobs missing? Another form of this is narrow clone, where we cut by
> paths, not by blob size. Narrow clone sounds more useful to me because
> it's easier to control what we leave out.

I was about to say "Hear, hear", but then stopped with a question to myself: why are these "some people do not want them" paths in the same repository in the first place?

Duy Nguyen· Jul 23, 2013, 05:01 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 00/16] First class shallow clone

On Tue, Jul 23, 2013 at 11:08 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 13 quoted lines
> Duy Nguyen <pclouds@gmail.com> writes:
>
>> This one, on the other hand, changes the "shape" of the repo (now with
>> holes) and might need to go through the same process we do with this
>> series. Maybe we should prepare for it now. Do you have a use case for
>> size-based filtering? What can we do with a repo with some arbitrary
>> blobs missing? Another form of this is narrow clone, where we cut by
>> paths, not by blob size. Narrow clone sounds more useful to me because
>> it's easier to control what we leave out.
>
> I was about to say "Hear, hear", but then stopped with a question to
> myself: why are these "some people do not want them" paths in the
> same repository in the first place?

I think there are situations that splitting repos is not the best choice but I can't think of any. There's one case though that such "some people" exist: when they migrate from another version control to git and do not want to change the directory layout (because it used to work ok, because of the cost of updating toolchain...) -- Duy

Philip Oakley· Jul 23, 2013, 22:33 UTC · re: Duy Nguyen · lore

Re: [PATCH v2 00/16] First class shallow clone

From: "Duy Nguyen" <pclouds@gmail.com>
Sent: Tuesday, July 23, 2013 2:20 AM
Show 18 quoted lines
> On Tue, Jul 23, 2013 at 6:41 AM, Philip Oakley <philipoakley@iee.org> 
> wrote:
>> From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
>> Subject: [PATCH v2 00/16] First class shallow clone
>>
>> It's nice to see that shallow can be a first class clone.
>>
>> Thinking outside the box, does this infrastructure offer the 
>> opportunity to
>> maybe add a date based depth option that would establish the shallow
>> watermark based on date rather than count. (e.g. the "deepen" SP 
>> depth could
>
> I've been carefully avoiding the deepen issues because, as you see,
> it's complicated. But no, this series does not enable or disable new
> deeepen mechanisms. They can always be added as protocol extensions.
> Still thinking if it's worth exposing a (restricted form of) rev-list
> to the protocol..
Interesting idea.
Show 22 quoted lines
>
>> have an alternate with a leading 'T' to indicate a time limit ratherv 
>> than
>> revision count - I'm expecting such a format would be an error for 
>> existing
>> servers).
>>
>> My other thought was this style of cut limit list may also allow a 
>> big file
>> limit to do a similar process of listing objects (e.g. blobs) that 
>> are
>> size-shallow in the repo, though it maybe a long list on some repos, 
>> or with
>> a small size limit.
>
> This one, on the other hand, changes the "shape" of the repo (now with
> holes) and might need to go through the same process we do with this
> series. Maybe we should prepare for it now. Do you have a use case for
> size-based filtering? What can we do with a repo with some arbitrary
> blobs missing? Another form of this is narrow clone, where we cut by
> paths, not by blob size. Narrow clone sounds more useful to me because
> it's easier to control what we leave out.

In some sense a project with a sub-module is a narrow clone, split at a 'commit' object. There have been comments on the git-user list about the problem of accidental adding of large files which then make the repo's foot print pretty large as one use case [Git is consuming very much RAM]. The bigFileThreshold being one way of spotting such files as separate objects, and 'trimming' them.

It doesn't feel right to 'track files and directories` as paths for doing a narrow clone - it'd probably fall into the same trap as tracking file renames. However if one tracks trees and blobs (as a list of sha1 values, possibly with their source path) then it should it should be possible to allow work on the repo with those empty directories/files in the same manner as is used for sub-modules, possibly with some form of git-link file as an alternate marker.

The thought process is to map sub-module working onto the other object types (blobs and trees). The user would be unable to edit the trimmed files/directories anyway, so its sha1 value can't change, allowing it to be included in the next commit in the branch series.

Philip
> --
> Duy
> --
Duy Nguyen· Jul 24, 2013, 01:57 UTC · re: Philip Oakley · lore

Re: [PATCH v2 00/16] First class shallow clone

On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley <philipoakley@iee.org> wrote:
> In some sense a project with a sub-module is a narrow clone, split at a
> 'commit' object.

Yes, except narrow clone is more flexible. You have to decide the split boundary at commit time for sub-module, while you decide the same at clone time for narrow clone.

Show 5 quoted lines
> There have been comments on the git-user list about the
> problem of accidental adding of large files which then make the repo's foot
> print pretty large as one use case [Git is consuming very much RAM]. The
> bigFileThreshold being one way of spotting such files as separate objects,
> and 'trimming' them.

I think rewriting history to remove those accidents is better than working around it (the same for accidentally committing password). We might be able to spot problems early, maybe warn user at commit time that they have added an exceptionally large blob, maybe before push time..

The "Git is consuming very much RAM" part is not right. We try to keep memory usage under a limit regardless of the size of a blob. There may be some cases we haven't fixed yet. Reports are welcome.

-- 
Duy
Philip Oakley· Jul 24, 2013, 07:38 UTC · re: Duy Nguyen · lore

Re: [PATCH v2 00/16] First class shallow clone

From: "Duy Nguyen" <pclouds@gmail.com>
Sent: Wednesday, July 24, 2013 2:57 AM
Show 10 quoted lines
> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley <philipoakley@iee.org> 
> wrote:
>> In some sense a project with a sub-module is a narrow clone, split at 
>> a
>> 'commit' object.
>
> Yes, except narrow clone is more flexible. You have to decide the
> split boundary at commit time for sub-module, while you decide the
> same at clone time for narrow clone.
>
True. It was the thought experiment part I was referring to.
Show 14 quoted lines
>> There have been comments on the git-user list about the
>> problem of accidental adding of large files which then make the 
>> repo's foot
>> print pretty large as one use case [Git is consuming very much RAM]. 
>> The
>> bigFileThreshold being one way of spotting such files as separate 
>> objects,
>> and 'trimming' them.
>
> I think rewriting history to remove those accidents is better than
> working around it (the same for accidentally committing password). We
> might be able to spot problems early, maybe warn user at commit time
> that they have added an exceptionally large blob, maybe before push
> time..

Again, it was a thought experiment which related to a recent git-user list comment. I'd expect a real use case could be a team where one member who is preparing documentation adds a [large] video to his branch and others then get a bit concerned when they try to track it / pull it as they really don't want it yet. The guy may have many versions on the central repo before a final rebase has a single compressed version. Colleagues may want to review the text surrounding it but not pull the video itself. (remembering 50 % of 'idiots' are twice as dumb as the average... ;-)

>
> The "Git is consuming very much RAM" part is not right. We try to keep
> memory usage under a limit regardless of the size of a blob. There may
> be some cases we haven't fixed yet. Reports are welcome.

I think this was a Windows user, but reports do pop up every now and again. Some times its disc pressure, or just perceived slowness (from others)

> -- 
> Duy
Philip
Piotr Krukowiecki· Jul 24, 2013, 08:30 UTC · re: Duy Nguyen · lore

Re: [PATCH v2 00/16] First class shallow clone

(resending, as my phone mail client decided to send it in html, sorry about that)

On Wed, Jul 24, 2013 at 3:57 AM, Duy Nguyen <pclouds@gmail.com> wrote:
Show 12 quoted lines
> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley <philipoakley@iee.org> wrote:
>> There have been comments on the git-user list about the
>> problem of accidental adding of large files which then make the repo's foot
>> print pretty large as one use case [Git is consuming very much RAM]. The
>> bigFileThreshold being one way of spotting such files as separate objects,
>> and 'trimming' them.
>
> I think rewriting history to remove those accidents is better than
> working around it (the same for accidentally committing password). We
> might be able to spot problems early, maybe warn user at commit time
> that they have added an exceptionally large blob, maybe before push
> time..

I can imagine a situation where large files were part of the project at some point in history (they were required to build/use it) and later were removed because build/project has changed.

It would be useful to have the history for log/blame/etc even if you could not build/use old versions. A warning when checking out/branching such incomplete tree would be needed.

-- 
Piotr Krukowiecki
Duy Nguyen· Jul 24, 2013, 10:35 UTC · re: Piotr Krukowiecki · lore

Re: [PATCH v2 00/16] First class shallow clone

On Wed, Jul 24, 2013 at 3:30 PM, Piotr Krukowiecki <piotr.krukowiecki@gmail.com> wrote:

Show 24 quoted lines
> (resending, as my phone mail client decided to send it in html, sorry
> about that)
>
> On Wed, Jul 24, 2013 at 3:57 AM, Duy Nguyen <pclouds@gmail.com> wrote:
>> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley <philipoakley@iee.org> wrote:
>>> There have been comments on the git-user list about the
>>> problem of accidental adding of large files which then make the repo's foot
>>> print pretty large as one use case [Git is consuming very much RAM]. The
>>> bigFileThreshold being one way of spotting such files as separate objects,
>>> and 'trimming' them.
>>
>> I think rewriting history to remove those accidents is better than
>> working around it (the same for accidentally committing password). We
>> might be able to spot problems early, maybe warn user at commit time
>> that they have added an exceptionally large blob, maybe before push
>> time..
>
> I can imagine a situation where large files were part of the project
> at some point in history (they were required to build/use it) and
> later were removed because build/project has changed.
>
> It would be useful to have the history for log/blame/etc even if you
> could not build/use old versions. A warning when checking
> out/branching such incomplete tree would be needed.

That's what shallow clone is for. You fetch the latest (not including old large blobs) and work on top. For archaeology, make a full clone. Or do you mean log/blame/etc other paths that don't touch big blobs, and the clone is still incomplete? -- Duy

Piotr Krukowiecki· Jul 24, 2013, 16:50 UTC · re: Duy Nguyen · lore

Re: [PATCH v2 00/16] First class shallow clone

Duy Nguyen <pclouds@gmail.com> napisał:
Show 37 quoted lines
>On Wed, Jul 24, 2013 at 3:30 PM, Piotr Krukowiecki
><piotr.krukowiecki@gmail.com> wrote:
>> (resending, as my phone mail client decided to send it in html, sorry
>> about that)
>>
>> On Wed, Jul 24, 2013 at 3:57 AM, Duy Nguyen <pclouds@gmail.com>
>wrote:
>>> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley
><philipoakley@iee.org> wrote:
>>>> There have been comments on the git-user list about the
>>>> problem of accidental adding of large files which then make the
>repo's foot
>>>> print pretty large as one use case [Git is consuming very much
>RAM]. The
>>>> bigFileThreshold being one way of spotting such files as separate
>objects,
>>>> and 'trimming' them.
>>>
>>> I think rewriting history to remove those accidents is better than
>>> working around it (the same for accidentally committing password).
>We
>>> might be able to spot problems early, maybe warn user at commit time
>>> that they have added an exceptionally large blob, maybe before push
>>> time..
>>
>> I can imagine a situation where large files were part of the project
>> at some point in history (they were required to build/use it) and
>> later were removed because build/project has changed.
>>
>> It would be useful to have the history for log/blame/etc even if you
>> could not build/use old versions. A warning when checking
>> out/branching such incomplete tree would be needed.
>
>That's what shallow clone is for. You fetch the latest (not including
>old large blobs) and work on top. For archaeology, make a full clone.
>Or do you mean log/blame/etc other paths that don't touch big blobs,
>and the clone is still incomplete?
Yes, for example if large files were removed recently the last-n-commits-shallow would be useless from blame/log POV. 

← back to recent threads