{"thread":{"id":"24537","subject":"[PATCH 0/2] Subtree clone?","startedAt":"2010-07-26T23:36:08Z","lastAt":"2010-07-28T02:00:28Z","messageCount":10,"participants":["Nguyễn Thái Ngọc Duy","Ævar Arnfjörð Bjarmason","Shawn O. Pearce","Avery Pennarun","Nguyen Thai Ngoc Duy","Elijah Newren"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"146474","messageId":"1280187370-23675-1-git-send-email-pclouds@gmail.com","threadId":"24537","inReplyTo":null,"subject":"[PATCH 0/2] Subtree clone?","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-26T23:36:08Z","receivedAt":"2010-07-26T23:36:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This idea sounds quite nice to me. That is, instead of modifying git\ncore to support narrow/partial clone, {upload,fetch}-pack is modified\nto give clients enough objects of so it can reconstruct a valid tree.\nUsers are free to do whatever they want on that tree. When they want to\npush changes back, Git client creates proper commit/tree objects and\npush.\n\nThe two patches in this series allow git to send objects of a subtree\nto client, or just a barebone subtree without blobs. The client can\nrewrite commits and throw the old commits.\n\nI don't want to add much computation to server side, subtree looks\nlike a good fit (i.e. simply prefixcmp). Sparse checkout can then be\nused to shape worktree if subtree is not good enough.\n\nAll the hard work is at client side, and git-subtree is a good\ncandidate. Well, the idea is inspired by recent discussions of\ngit-subtree vs git-submodule anyway.\n\nLazy clone does something similar. However lazy clone requires\nconnectivity to upstream. Lazy clone also exposes a security issue,\nallowing client to get any object it wants.\n\nComments?\n\nNguyễn Thái Ngọc Duy (2):\n  upload-pack: support subtree packing\n  fetch-pack: support --subtree and --commit-subtree options\n\n builtin/fetch-pack.c |   15 +++++++++++++++\n upload-pack.c        |   31 ++++++++++++++++++++++++++++++-\n 2 files changed, 45 insertions(+), 1 deletions(-)\n"},{"id":"146475","messageId":"1280187370-23675-2-git-send-email-pclouds@gmail.com","threadId":"24537","inReplyTo":"1280187370-23675-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 1/2] upload-pack: support subtree packing","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-26T23:36:09Z","receivedAt":"2010-07-26T23:36:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This patch adds a new capability \"subtree\", which supports two new\nrequests \"subtree\" and \"commit-subtree\".\n\n\"subtree\" asks upload-pack to create a pack that contains only blobs\nfrom the given tree prefix (and necessary commits/trees to reach\nthose blobs).\n\n\"commit-tree\" asks upload-pack to create a pack that contains trees of\nthe given prefix (and necessary commits/trees to reach those trees)\n\nWith \"subtree\" request, Git client may then rewrite commits to create\na valid commit tree again, so that users can work on it independently.\nWhen users want to push from such a tree, \"commit-tree\" may then be\nused to re-match what users have and what is in upstream, recreate\nproper push commits.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n upload-pack.c |   31 ++++++++++++++++++++++++++++++-\n 1 files changed, 30 insertions(+), 1 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex dc464d7..f97296a 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -41,6 +41,8 @@ static int use_sideband;\n static int debug_fd;\n static int advertise_refs;\n static int stateless_rpc;\n+static char *subtree;\n+static int commit_subtree;\n \n static void reset_timeout(void)\n {\n@@ -89,6 +91,17 @@ static void show_object(struct object *obj, const struct name_path *path, const\n \t */\n \tconst char *name = path_name(path, component);\n \tconst char *ep = strchr(name, '\\n');\n+\tif (subtree) {\n+\t\tint len = strlen(name);\n+\t\t/* parent trees should always be kept */\n+\t\tif (obj->type == OBJ_TREE && !prefixcmp(subtree, name) && subtree[len] == '/')\n+\t\t\t; /* in */\n+\t\telse if (commit_subtree)\n+\t\t\tgoto out;\n+\t\telse if (prefixcmp(name, subtree))\n+\t\t\tgoto out;\n+\t}\n+\n \tif (ep) {\n \t\tfprintf(pack_pipe, \"%s %.*s\\n\", sha1_to_hex(obj->sha1),\n \t\t       (int) (ep - name),\n@@ -97,6 +110,7 @@ static void show_object(struct object *obj, const struct name_path *path, const\n \telse\n \t\tfprintf(pack_pipe, \"%s %s\\n\",\n \t\t\t\tsha1_to_hex(obj->sha1), name);\n+out:\n \tfree((char *)name);\n }\n \n@@ -504,6 +518,21 @@ static void receive_needs(void)\n \t\tif (debug_fd)\n \t\t\twrite_in_full(debug_fd, line, len);\n \n+\t\tif (!prefixcmp(line, \"subtree \")) {\n+\t\t\tint len = strlen(line+8);\n+\t\t\tsubtree = malloc(len+1);\n+\t\t\tmemcpy(subtree, line+8, len-1);\n+\t\t\tsubtree[len-1] = '\\0'; /* \\n */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!prefixcmp(line, \"commit-subtree \")) {\n+\t\t\tint len = strlen(line+15);\n+\t\t\tsubtree = malloc(len+1);\n+\t\t\tmemcpy(subtree, line+15, len-1);\n+\t\t\tsubtree[len-1] = '\\0'; /* \\n */\n+\t\t\tcommit_subtree = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!prefixcmp(line, \"shallow \")) {\n \t\t\tunsigned char sha1[20];\n \t\t\tstruct object *object;\n@@ -623,7 +652,7 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow no-progress\"\n-\t\t\" include-tag multi_ack_detailed\";\n+\t\t\" include-tag multi_ack_detailed subtree\";\n \tstruct object *o = parse_object(sha1);\n \n \tif (!o)\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146476","messageId":"1280187370-23675-3-git-send-email-pclouds@gmail.com","threadId":"24537","inReplyTo":"1280187370-23675-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/2] fetch-pack: support --subtree and --commit-subtree options","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-26T23:36:10Z","receivedAt":"2010-07-26T23:36:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"These options are simply turned to upload-pack's \"subtree\" and \"commit-tree\" requests, respectively.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n builtin/fetch-pack.c |   15 +++++++++++++++\n 1 files changed, 15 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex dbd8b7b..0bc7f6d 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -14,6 +14,8 @@ static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n static int unpack_limit = 100;\n static int prefer_ofs_delta = 1;\n+static const char *subtree;\n+static int commit_subtree;\n static struct fetch_pack_args args = {\n \t/* .uploadpack = */ \"git-upload-pack\",\n };\n@@ -237,6 +239,8 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tfor_each_ref(rev_list_insert_ref, NULL);\n \n \tfetching = 0;\n+\tif (subtree)\n+\t\tpacket_buf_write(&req_buf, \"%s %s\\n\", commit_subtree ? \"commit-subtree\" : \"subtree\", subtree);\n \tfor ( ; refs ; refs = refs->next) {\n \t\tunsigned char *remote = refs->old_sha1;\n \t\tconst char *remote_hex;\n@@ -692,6 +696,8 @@ static struct ref *do_fetch_pack(int fd[2],\n \n \tif (is_repository_shallow() && !server_supports(\"shallow\"))\n \t\tdie(\"Server does not support shallow clients\");\n+\tif (subtree && !server_supports(\"subtree\"))\n+\t\tdie(\"Server does not support subtree\");\n \tif (server_supports(\"multi_ack_detailed\")) {\n \t\tif (args.verbose)\n \t\t\tfprintf(stderr, \"Server supports multi_ack_detailed\\n\");\n@@ -860,6 +866,15 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tpack_lockfile_ptr = &pack_lockfile;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!prefixcmp(arg, \"--subtree=\")) {\n+\t\t\t\tsubtree = arg + 10;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!prefixcmp(arg, \"--commit-subtree=\")) {\n+\t\t\t\tsubtree = arg + 17;\n+\t\t\t\tcommit_subtree = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tusage(fetch_pack_usage);\n \t\t}\n \t\tdest = (char *)arg;\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146477","messageId":"AANLkTinQeEpb8BjySdPsZpwYo4O7sHUMZ-ARuA7_y5aU@mail.gmail.com","threadId":"24537","inReplyTo":"1280187370-23675-2-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 1/2] upload-pack: support subtree packing","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-27T13:15:59Z","receivedAt":"2010-07-27T13:15:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"2010/7/26 Nguyễn Thái Ngọc Duy <pclouds@gmail.com>:\n> +               int len = strlen(name);\n\nDon't you mean \"size_t len = strlen(name)\" .., or to use a cast?\n"},{"id":"146481","messageId":"20100727144605.GA25268@spearce.org","threadId":"24537","inReplyTo":"1280187370-23675-2-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 1/2] upload-pack: support subtree packing","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-07-27T14:46:05Z","receivedAt":"2010-07-27T14:46:05Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n> This patch adds a new capability \"subtree\", which supports two new\n> requests \"subtree\" and \"commit-subtree\".\n> \n> \"subtree\" asks upload-pack to create a pack that contains only blobs\n> from the given tree prefix (and necessary commits/trees to reach\n> those blobs).\n> \n> \"commit-tree\" asks upload-pack to create a pack that contains trees of\n> the given prefix (and necessary commits/trees to reach those trees)\n> \n> With \"subtree\" request, Git client may then rewrite commits to create\n> a valid commit tree again, so that users can work on it independently.\n> When users want to push from such a tree, \"commit-tree\" may then be\n> used to re-match what users have and what is in upstream, recreate\n> proper push commits.\n\nI disagree with a lot of this... but the idea is quite cool.\n\nI like the \"subtree\" command, being able to clone down only part of\nthe repository is a nice feature, and the implementation of subtree\nseems simple enough for the server.  It only has to emit some of\nthe paths, but the entire commit DAG.  This is pretty simple to\nimplement server side and is very lightweight.\n\n\nBut I disagree with the client rewriting the commits in order to\nwork with them locally.  Doing so means you can't take a commit\nfrom your team's issue tracker and look it up.  And any commit\nyou create can't be pushed back to the server without rewriting.\nIts messy for the end-user to work with.\n\nI would prefer doing something more like what we do with shallow\non the client side.  Record in a magic file the path(s) that we\ndid actually obtain.  During fsck, rev-list, or read-tree the\nclient skips over any paths that don't match that file's listing.\nThen we can keep the same commit SHA-1s, but we won't complain that\nthere are objects missing.\n\nThe downside is, a lot of the client code is impacted, and that\nis why nobody has done it yet.  Tools like rebase or cherry-pick\nstart to behave funny.  What does it mean to rebase or cherry-pick\na commit that has deltas outside of the area you don't have cloned?\nIt probably should abort and refuse to execute.  But `git show`\nshould still work, which implies you need a way to toggle the\ndiff code to either skip or fail on deltas outside of the shallow\npath space.\n \n-- \nShawn.\n"},{"id":"146510","messageId":"20100727185127.GD25124@worldvisions.ca","threadId":"24537","inReplyTo":"20100727144605.GA25268@spearce.org","subject":"Configurable callbacks for missing objects (we Re: upload-pack: support subtree packing)","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-27T18:51:27Z","receivedAt":"2010-07-27T18:51:27Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Tue, Jul 27, 2010 at 07:46:05AM -0700, Shawn O. Pearce wrote:\n\n> But I disagree with the client rewriting the commits in order to\n> work with them locally.  Doing so means you can't take a commit\n> from your team's issue tracker and look it up.  And any commit\n> you create can't be pushed back to the server without rewriting.\n> Its messy for the end-user to work with.\n\nYeah, that doesn't sound ideal.  And I wrote git-subtree, which does exactly\nthat, so I should know :)\n\n> I would prefer doing something more like what we do with shallow\n> on the client side.  Record in a magic file the path(s) that we\n> did actually obtain.  During fsck, rev-list, or read-tree the\n> client skips over any paths that don't match that file's listing.\n> Then we can keep the same commit SHA-1s, but we won't complain that\n> there are objects missing.\n\nDisclaimer: I've never looked at any of the fetch code.\n\nBut I've been thinking that a really elegant way to solve the problem could\nbe to have a user-configurable \"get the missing objects\" callback.  If any\npart of git that *needs* an object can't find it, it calls this callback to\ngo try to retrieve it (either just that one object, or it can request to\ndownload the object recursively, ie. everything it points to).\n\nThen shallow clones could just auto-fill themselves if you really need a\nprior version, for example.\n\nIt's also conceivable that we could limit this just to blobs: downloading\nthe complete set of commit objects, and probably even the complete set of\ntree objects, is probably not that expensive.  And that would allow someone\nto do virtually all operations (other than three-way merges to resolve blob\nconflicts) without having the entire repo.\n\nI say it could be user-configurable because that's where you could plugin a\ngittorrent, or a tool that just tries fetching from a series of repositories in\nturn, etc.\n\nHave fun,\n\nAvery\n"},{"id":"146546","messageId":"AANLkTin+S9Y2+DtWWETJjFKaVAesL8M6K3D=XrDPqvMY@mail.gmail.com","threadId":"24537","inReplyTo":"20100727144605.GA25268@spearce.org","subject":"Re: [PATCH 1/2] upload-pack: support subtree packing","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-27T22:29:10Z","receivedAt":"2010-07-27T22:29:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 28, 2010 at 12:46 AM, Shawn O. Pearce <spearce@spearce.org> wrote:\n> Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n>> This patch adds a new capability \"subtree\", which supports two new\n>> requests \"subtree\" and \"commit-subtree\".\n>>\n>> \"subtree\" asks upload-pack to create a pack that contains only blobs\n>> from the given tree prefix (and necessary commits/trees to reach\n>> those blobs).\n>>\n>> \"commit-tree\" asks upload-pack to create a pack that contains trees of\n>> the given prefix (and necessary commits/trees to reach those trees)\n>>\n>> With \"subtree\" request, Git client may then rewrite commits to create\n>> a valid commit tree again, so that users can work on it independently.\n>> When users want to push from such a tree, \"commit-tree\" may then be\n>> used to re-match what users have and what is in upstream, recreate\n>> proper push commits.\n>\n> I disagree with a lot of this... but the idea is quite cool.\n>\n> I like the \"subtree\" command, being able to clone down only part of\n> the repository is a nice feature, and the implementation of subtree\n> seems simple enough for the server.  It only has to emit some of\n> the paths, but the entire commit DAG.  This is pretty simple to\n> implement server side and is very lightweight.\n\nAnother point is server side can disallow full clone completely and\ngive permission to clone on directory basis. Enterprise users would\nlove this.\n\n> But I disagree with the client rewriting the commits in order to\n> work with them locally.  Doing so means you can't take a commit\n> from your team's issue tracker and look it up.  And any commit\n> you create can't be pushed back to the server without rewriting.\n> Its messy for the end-user to work with.\n\nThat's what happens with git-subtree in its current form (I don't know\nmuch about git-subtree though). But I guess if they can use\ngit-subtree as it is now, they can live with subtree clone+git-subtree\njust fine.\n\n> I would prefer doing something more like what we do with shallow\n> on the client side.  Record in a magic file the path(s) that we\n> did actually obtain.  During fsck, rev-list, or read-tree the\n> client skips over any paths that don't match that file's listing.\n> Then we can keep the same commit SHA-1s, but we won't complain that\n> there are objects missing.\n\nThat's another option. With all trees, sparse checkout can be used, as\nlong as you limit your operations within a subdirectory. Full tree\ncommands like git-fsck can be taught to realize it's subtree clone and\nstop complain of non-existing objects. Download pack would be bigger\n(I don't know how much). And it also defeats the enterprise point\nabove.\n\n> The downside is, a lot of the client code is impacted, and that\n> is why nobody has done it yet.  Tools like rebase or cherry-pick\n> start to behave funny.  What does it mean to rebase or cherry-pick\n> a commit that has deltas outside of the area you don't have cloned?\n> It probably should abort and refuse to execute.  But `git show`\n> should still work, which implies you need a way to toggle the\n> diff code to either skip or fail on deltas outside of the shallow\n> path space.\n\nWhere do those deltas come from? I thought, with proper path limiting\nin upload-pack, pack-objects would never generate anything that needs\nthings outside the area?\n\nSounds like git-subtree for short term, and without git-subtree long\nterm to me :)\n-- \nDuy\n"},{"id":"146547","messageId":"AANLkTimPVzBN_KaxMpY9NoP+M_WdErh7fT0oyPWGky7m@mail.gmail.com","threadId":"24537","inReplyTo":"20100727185127.GD25124@worldvisions.ca","subject":"Re: Configurable callbacks for missing objects (we Re: upload-pack: support subtree packing)","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-27T22:32:30Z","receivedAt":"2010-07-27T22:32:30Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/7/28 Avery Pennarun <apenwarr@gmail.com>:\n> But I've been thinking that a really elegant way to solve the problem could\n> be to have a user-configurable \"get the missing objects\" callback.  If any\n> part of git that *needs* an object can't find it, it calls this callback to\n> go try to retrieve it (either just that one object, or it can request to\n> download the object recursively, ie. everything it points to).\n>\n> Then shallow clones could just auto-fill themselves if you really need a\n> prior version, for example.\n\nI think that's what lazy clone does in [1]\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/73117/focus=73935\n-- \nDuy\n"},{"id":"146565","messageId":"AANLkTimrp4z0n75Zq7BtTvdWz6nzK3M4J9ohdbE6X3t3@mail.gmail.com","threadId":"24537","inReplyTo":"20100727185127.GD25124@worldvisions.ca","subject":"Re: Configurable callbacks for missing objects (we Re: upload-pack: support subtree packing)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-07-28T01:53:59Z","receivedAt":"2010-07-28T01:53:59Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"2010/7/27 Avery Pennarun <apenwarr@gmail.com>:\n> But I've been thinking that a really elegant way to solve the problem could\n> be to have a user-configurable \"get the missing objects\" callback.  If any\n> part of git that *needs* an object can't find it, it calls this callback to\n> go try to retrieve it (either just that one object, or it can request to\n> download the object recursively, ie. everything it points to).\n>\n> Then shallow clones could just auto-fill themselves if you really need a\n> prior version, for example.\n\nWhat counts as \"needing\" an object?  Does 'git log -Sfoo' or 'git log\n--stat' need all missing blobs?  I'd personally dislike having such\ncommands automatically result in huge downloads, but I'd probably\ndislike the automatic downloading in general so perhaps I'm just a\nmisfit for the lazy clone usecase.  It's still an interesting question\nthough -- what counts as needed?\n"},{"id":"146566","messageId":"AANLkTinDvmQhPH3QtcJTD3mCuCvSjG-pmfu63f9K7Dsb@mail.gmail.com","threadId":"24537","inReplyTo":"AANLkTimrp4z0n75Zq7BtTvdWz6nzK3M4J9ohdbE6X3t3@mail.gmail.com","subject":"Re: Configurable callbacks for missing objects (we Re: upload-pack: support subtree packing)","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-28T02:00:28Z","receivedAt":"2010-07-28T02:00:28Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"2010/7/27 Elijah Newren <newren@gmail.com>:\n> 2010/7/27 Avery Pennarun <apenwarr@gmail.com>:\n>> But I've been thinking that a really elegant way to solve the problem could\n>> be to have a user-configurable \"get the missing objects\" callback.  If any\n>> part of git that *needs* an object can't find it, it calls this callback to\n>> go try to retrieve it (either just that one object, or it can request to\n>> download the object recursively, ie. everything it points to).\n>>\n>> Then shallow clones could just auto-fill themselves if you really need a\n>> prior version, for example.\n>\n> What counts as \"needing\" an object?  Does 'git log -Sfoo' or 'git log\n> --stat' need all missing blobs?  I'd personally dislike having such\n> commands automatically result in huge downloads, but I'd probably\n> dislike the automatic downloading in general so perhaps I'm just a\n> misfit for the lazy clone usecase.  It's still an interesting question\n> though -- what counts as needed?\n\nI would say no by default (unless maybe a config option is set) but\nyou'd want to be able to force it on for a particular command.\n\nHave fun,\n\nAvery\n"}]}