{"thread":{"id":"43284","subject":"Re: [PATCH 3/5] allow cloning a repository \"shallowly\"","startedAt":"2006-10-30T19:09:29Z","lastAt":"2006-11-15T10:57:38Z","messageCount":5,"participants":["Junio C Hamano","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"296614","messageId":"Pine.LNX.4.63.0610302009160.26682@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43284","inReplyTo":null,"subject":"[PATCH 3/5] allow cloning a repository \"shallowly\"","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-10-30T19:09:29Z","receivedAt":"2006-10-30T19:09:29Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"\nBy specifying a depth, you can now clone a repository such that\nall fetched ancestor-chains' length is at most \"depth\". For example,\nif the upstream repository has only 2 branches (\"A\" and \"B\"), which\nare linear, and you specify depth 3, you will get A, A~1, A~2, A~3,\nB, B~1, B~2, and B~3. The ends are automatically made shallow\ncommits.\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n fetch-pack.c  |   61 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n git-clone.sh  |   19 +++++++++++++++--\n upload-pack.c |   21 ++++++++++++++++++-\n 3 files changed, 96 insertions(+), 5 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 488adc9..9619d6e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -8,8 +8,9 @@ static int keep_pack;\n static int quiet;\n static int verbose;\n static int fetch_all;\n+static int depth;\n static const char fetch_pack_usage[] =\n-\"git-fetch-pack [--all] [-q] [-v] [-k] [--thin] [--exec=upload-pack] [host:]directory <refs>...\";\n+\"git-fetch-pack [--all] [-q] [-v] [-k] [--thin] [--exec=upload-pack] [--depth=<n>] [host:]directory <refs>...\";\n static const char *exec = \"git-upload-pack\";\n \n #define COMPLETE\t(1U << 0)\n@@ -179,10 +180,29 @@ static int find_common(int fd[2], unsign\n \t}\n \tif (is_repository_shallow())\n \t\twrite_shallow_commits(fd[1], 1);\n+\tif (depth > 0)\n+\t\tpacket_write(fd[1], \"deepen %d\", depth);\n \tpacket_flush(fd[1]);\n \tif (!fetching)\n \t\treturn 1;\n \n+\tif (depth >  0) {\n+\t\tchar line[1024];\n+\t\tunsigned char sha1[20];\n+\t\tint len;\n+\n+\t\twhile ((len = packet_read_line(fd[0], line, sizeof(line)))) {\n+\t\t\tif (!strncmp(\"shallow \", line, 8)) {\n+\t\t\t\tif (get_sha1_hex(line + 8, sha1))\n+\t\t\t\t\tdie(\"invalid shallow line: %s\", line);\n+\t\t\t\t/* no need making it shallow if we have it already */\n+\t\t\t\tif (lookup_object(sha1))\n+\t\t\t\t\tcontinue;\n+\t\t\t\tregister_shallow(sha1);\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \tflushes = 0;\n \tretval = -1;\n \twhile ((sha1 = get_rev())) {\n@@ -480,6 +500,8 @@ int main(int argc, char **argv)\n \tchar *dest = NULL, **heads;\n \tint fd[2];\n \tpid_t pid;\n+\tstruct stat st;\n+\tstruct lock_file lock;\n \n \tsetup_git_directory();\n \n@@ -513,6 +535,12 @@ int main(int argc, char **argv)\n \t\t\t\tverbose = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strncmp(\"--depth=\", arg, 8)) {\n+\t\t\t\tdepth = strtol(arg + 8, NULL, 0);\n+\t\t\t\tif (stat(git_path(\"shallow\"), &st))\n+\t\t\t\t\tst.st_mtime = 0;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tusage(fetch_pack_usage);\n \t\t}\n \t\tdest = arg;\n@@ -522,6 +550,8 @@ int main(int argc, char **argv)\n \t}\n \tif (!dest)\n \t\tusage(fetch_pack_usage);\n+\tif (is_repository_shallow() && depth > 0)\n+\t\tdie(\"Deepening of a shallow repository not yet supported!\");\n \tpid = git_connect(fd, dest, exec);\n \tif (pid < 0)\n \t\treturn 1;\n@@ -543,5 +573,34 @@ int main(int argc, char **argv)\n \t\t\t}\n \t}\n \n+\tif (!ret && depth > 0) {\n+\t\tstruct cache_time mtime;\n+\t\tchar *shallow = git_path(\"shallow\");\n+\t\tint fd;\n+\n+\t\tmtime.sec = st.st_mtime;\n+#ifdef USE_NSEC\n+\t\tmtime.usec = st.st_mtim.usec;\n+#endif\n+\t\tif (stat(shallow, &st)) {\n+\t\t\tif (mtime.sec)\n+\t\t\t\tdie(\"shallow file was removed during fetch\");\n+\t\t} else if (st.st_mtime != mtime.sec\n+#ifdef USE_NSEC\n+\t\t\t\t|| st.st_mtim.usec != mtime.usec\n+#endif\n+\t\t\t  )\n+\t\t\tdie(\"shallow file was changed during fetch\");\n+\n+\t\tfd = hold_lock_file_for_update(&lock, shallow, 1);\n+\t\tif (!write_shallow_commits(fd, 0)) {\n+\t\t\tunlink(lock.filename);\n+\t\t\trollback_lock_file(&lock);\n+\t\t} else {\n+\t\t\tclose(fd);\n+\t\t\tcommit_lock_file(&lock);\n+\t\t}\n+\t}\n+\n \treturn !!ret;\n }\ndiff --git a/git-clone.sh b/git-clone.sh\nindex 3f006d1..595c070 100755\n--- a/git-clone.sh\n+++ b/git-clone.sh\n@@ -14,7 +14,7 @@ die() {\n }\n \n usage() {\n-\tdie \"Usage: $0 [--template=<template_directory>] [--use-separate-remote] [--reference <reference-repo>] [--bare] [-l [-s]] [-q] [-u <upload-pack>] [--origin <name>] [-n] <repo> [<dir>]\"\n+\tdie \"Usage: $0 [--template=<template_directory>] [--use-separate-remote] [--reference <reference-repo>] [--bare] [-l [-s]] [-q] [-u <upload-pack>] [--origin <name>] [--depth <n>] [-n] <repo> [<dir>]\"\n }\n \n get_repo_base() {\n@@ -116,6 +116,7 @@ reference=\n origin=\n origin_override=\n use_separate_remote=\n+depth=\n while\n \tcase \"$#,$1\" in\n \t0,*) break ;;\n@@ -158,6 +159,10 @@ while\n \t*,-u|*,--upload-pack)\n \t\tshift\n \t\tupload_pack=\"--exec=$1\" ;;\n+\t1,--depth) usage;;\n+\t*,--depth)\n+\t\tshift\n+\t\tdepth=\"--depth=$1\";;\n \t*,-*) usage ;;\n \t*) break ;;\n \tesac\n@@ -265,6 +270,10 @@ yes,yes)\n *)\n \tcase \"$repo\" in\n \trsync://*)\n+\t\tcase \"$depth\" in\n+\t\t\"\") ;;\n+\t\t*) die \"shallow over rsync not supported\" ;;\n+\t\tesac\n \t\trsync $quiet -av --ignore-existing  \\\n \t\t\t--exclude info \"$repo/objects/\" \"$GIT_DIR/objects/\" ||\n \t\texit\n@@ -293,6 +302,10 @@ yes,yes)\n \t\tgit-ls-remote \"$repo\" >\"$GIT_DIR/CLONE_HEAD\" || exit 1\n \t\t;;\n \thttps://*|http://*|ftp://*)\n+\t\tcase \"$depth\" in\n+\t\t\"\") ;;\n+\t\t*) die \"shallow over http or ftp not supported\" ;;\n+\t\tesac\n \t\tif test -z \"@@NO_CURL@@\"\n \t\tthen\n \t\t\tclone_dumb_http \"$repo\" \"$D\"\n@@ -302,8 +315,8 @@ yes,yes)\n \t\t;;\n \t*)\n \t\tcase \"$upload_pack\" in\n-\t\t'') git-fetch-pack --all -k $quiet \"$repo\" ;;\n-\t\t*) git-fetch-pack --all -k $quiet \"$upload_pack\" \"$repo\" ;;\n+\t\t'') git-fetch-pack --all -k $quiet $depth \"$repo\" ;;\n+\t\t*) git-fetch-pack --all -k $quiet \"$upload_pack\" $depth \"$repo\" ;;\n \t\tesac >\"$GIT_DIR/CLONE_HEAD\" ||\n \t\t\tdie \"fetch-pack from '$repo' failed.\"\n \t\t;;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 8dd6121..ebe1e5a 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -488,7 +488,7 @@ static void receive_needs(void)\n {\n \tstruct object_array shallows = {0, 0, NULL};\n \tstatic char line[1000];\n-\tint len;\n+\tint len, depth = 0;\n \n \tfor (;;) {\n \t\tstruct object *o;\n@@ -509,6 +509,13 @@ static void receive_needs(void)\n \t\t\tadd_object_array(object, NULL, &shallows);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strncmp(\"deepen \", line, 7)) {\n+\t\t\tchar *end;\n+\t\t\tdepth = strtol(line + 7, &end, 0);\n+\t\t\tif (end == line + 7 || depth <= 0)\n+\t\t\t\tdie(\"Invalid deepen: %s\", line);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (strncmp(\"want \", line, 5) ||\n \t\t    get_sha1_hex(line+5, sha1_buf))\n \t\t\tdie(\"git-upload-pack: protocol error, \"\n@@ -540,6 +547,18 @@ static void receive_needs(void)\n \t\t\tadd_object_array(o, NULL, &want_obj);\n \t\t}\n \t}\n+\tif (depth > 0) {\n+\t\tstruct commit_list *result, *backup;\n+\t\tif (shallows.nr > 0)\n+\t\t\tdie(\"Deepening a shallow repository not yet supported\");\n+\t\tbackup = result = get_shallow_commits(&want_obj, depth);\n+\t\twhile (result) {\n+\t\t\tpacket_write(1, \"shallow %s\",\n+\t\t\t\t\tsha1_to_hex(result->item->object.sha1));\n+\t\t\tresult = result->next;\n+\t\t}\n+\t\tfree_commit_list(backup);\n+\t}\n \tif (shallows.nr > 0) {\n \t\tint i;\n \t\tfor (i = 0; i < shallows.nr; i++)\n-- \n1.4.3.3.gca42\n"},{"id":"294802","messageId":"7v4pt21oe9.fsf@assigned-by-dhcp.cox.net","threadId":"43284","inReplyTo":"Pine.LNX.4.63.0610302009160.26682@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 3/5] allow cloning a repository \"shallowly\"","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-11-14T07:24:46Z","receivedAt":"2006-11-14T07:24:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> By specifying a depth, you can now clone a repository such that\n> all fetched ancestor-chains' length is at most \"depth\". For example,\n> if the upstream repository has only 2 branches (\"A\" and \"B\"), which\n> are linear, and you specify depth 3, you will get A, A~1, A~2, A~3,\n> B, B~1, B~2, and B~3. The ends are automatically made shallow\n> commits.\n>\n> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n> ---\n>  fetch-pack.c  |   61 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n>  git-clone.sh  |   19 +++++++++++++++--\n>  upload-pack.c |   21 ++++++++++++++++++-\n>  3 files changed, 96 insertions(+), 5 deletions(-)\n>\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 488adc9..9619d6e 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n>...\n> +\t\twhile ((len = packet_read_line(fd[0], line, sizeof(line)))) {\n> +\t\t\tif (!strncmp(\"shallow \", line, 8)) {\n> +\t\t\t\tif (get_sha1_hex(line + 8, sha1))\n> +\t\t\t\t\tdie(\"invalid shallow line: %s\", line);\n> +\t\t\t\t/* no need making it shallow if we have it already */\n> +\t\t\t\tif (lookup_object(sha1))\n> +\t\t\t\t\tcontinue;\n> +\t\t\t\tregister_shallow(sha1);\n> +\t\t\t}\n> +\t\t}\n> +\t}\n\nI understand \"no need making it shallow\", but I am not sure if a\nnon-NULL return from lookup_object() tells us that.\n\nBefore this part of the code, we have scanned the local refs in\neverything_local (which parses their refs).\n\nI think register_shallow() can take commits that are already\nshallow() so maybe we can remove this \"if() continue\"?\n"},{"id":"298405","messageId":"Pine.LNX.4.63.0611141145390.13772@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43284","inReplyTo":"7v4pt21oe9.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 3/5] allow cloning a repository \"shallowly\"","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-11-14T10:49:55Z","receivedAt":"2006-11-14T10:49:55Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 13 Nov 2006, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> > By specifying a depth, you can now clone a repository such that\n> > all fetched ancestor-chains' length is at most \"depth\". For example,\n> > if the upstream repository has only 2 branches (\"A\" and \"B\"), which\n> > are linear, and you specify depth 3, you will get A, A~1, A~2, A~3,\n> > B, B~1, B~2, and B~3. The ends are automatically made shallow\n> > commits.\n> >\n> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n> > ---\n> >  fetch-pack.c  |   61 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n> >  git-clone.sh  |   19 +++++++++++++++--\n> >  upload-pack.c |   21 ++++++++++++++++++-\n> >  3 files changed, 96 insertions(+), 5 deletions(-)\n> >\n> > diff --git a/fetch-pack.c b/fetch-pack.c\n> > index 488adc9..9619d6e 100644\n> > --- a/fetch-pack.c\n> > +++ b/fetch-pack.c\n> >...\n> > +\t\twhile ((len = packet_read_line(fd[0], line, sizeof(line)))) {\n> > +\t\t\tif (!strncmp(\"shallow \", line, 8)) {\n> > +\t\t\t\tif (get_sha1_hex(line + 8, sha1))\n> > +\t\t\t\t\tdie(\"invalid shallow line: %s\", line);\n> > +\t\t\t\t/* no need making it shallow if we have it already */\n> > +\t\t\t\tif (lookup_object(sha1))\n> > +\t\t\t\t\tcontinue;\n> > +\t\t\t\tregister_shallow(sha1);\n> > +\t\t\t}\n> > +\t\t}\n> > +\t}\n> \n> I understand \"no need making it shallow\", but I am not sure if a\n> non-NULL return from lookup_object() tells us that.\n\nYou are probably right, how about has_sha1_file()?\n\n> I think register_shallow() can take commits that are already shallow() \n> so maybe we can remove this \"if() continue\"?\n\nYes, it can, but that is not necessarily correct: since .git/shallow is \nconstructed from the registered shallow commits, we would make a commit \nshallow which is really not shallow.\n\nSo, how about\n\n> > +\t\t\t\tif (lookup_object(sha1) || has_sha1_file(sha1))\n> > +\t\t\t\t\tcontinue;\n\nCiao,\nDscho\n"},{"id":"294223","messageId":"7vpsbpzzse.fsf@assigned-by-dhcp.cox.net","threadId":"43284","inReplyTo":"Pine.LNX.4.63.0611141145390.13772@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 3/5] allow cloning a repository \"shallowly\"","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-11-14T17:46:57Z","receivedAt":"2006-11-14T17:46:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n>> I understand \"no need making it shallow\", but I am not sure if a\n>> non-NULL return from lookup_object() tells us that.\n>\n> You are probably right, how about has_sha1_file()?\n>\n>> I think register_shallow() can take commits that are already shallow() \n>> so maybe we can remove this \"if() continue\"?\n>\n> Yes, it can, but that is not necessarily correct: since .git/shallow is \n> constructed from the registered shallow commits, we would make a commit \n> shallow which is really not shallow.\n>\n> So, how about\n>\n>> > +\t\t\t\tif (lookup_object(sha1) || has_sha1_file(sha1))\n>> > +\t\t\t\t\tcontinue;\n\nIf I understand the code correctly, this loop is reading what\nthe other side thinks your shallows should be (based on your\nearlier \"deepen\" request or if this is initial fetch based on\nyour depth).  Even if we already have that object, if that\nobject _is_ shallow on our end, don't we need to keep it marked\nas shallow?  Will we get ancestors of this commit from the other\nend (and \"shallow\" lines for some of them to properly cauterize\nthe chain)?\n\n\n"},{"id":"294533","messageId":"Pine.LNX.4.63.0611151153490.13772@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43284","inReplyTo":"7vpsbpzzse.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 3/5] allow cloning a repository \"shallowly\"","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-11-15T10:57:38Z","receivedAt":"2006-11-15T10:57:38Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 14 Nov 2006, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> >> I understand \"no need making it shallow\", but I am not sure if a\n> >> non-NULL return from lookup_object() tells us that.\n> >\n> > You are probably right, how about has_sha1_file()?\n> >\n> >> I think register_shallow() can take commits that are already shallow() \n> >> so maybe we can remove this \"if() continue\"?\n> >\n> > Yes, it can, but that is not necessarily correct: since .git/shallow is \n> > constructed from the registered shallow commits, we would make a commit \n> > shallow which is really not shallow.\n> >\n> > So, how about\n> >\n> >> > +\t\t\t\tif (lookup_object(sha1) || has_sha1_file(sha1))\n> >> > +\t\t\t\t\tcontinue;\n> \n> If I understand the code correctly, this loop is reading what\n> the other side thinks your shallows should be (based on your\n> earlier \"deepen\" request or if this is initial fetch based on\n> your depth).  Even if we already have that object, if that\n> object _is_ shallow on our end, don't we need to keep it marked\n> as shallow?  Will we get ancestors of this commit from the other\n> end (and \"shallow\" lines for some of them to properly cauterize\n> the chain)?\n\nMy thinking was: if we have that object already, and it is _not_ marked as \nshallow, we probably have either all its ancestors, or at least all \nancestors up until shallow commit(s).\n\nThinking about it again, it seems very fragile: on purpose, we only trust \nto have the objects which are reachable by at least one ref, so that an \ninterrupted fetch does not corrupt the repository.\n\nSo yes, I agree, that \"if() continue;\" should go away. Even if all of a \nsudden, commits we already have are no longer reachable.\n\nCiao,\nDscho\n"}]}