{"thread":{"id":"24596","subject":"[PATCH 00/16] Subtree clone proof of concept","startedAt":"2010-07-31T16:18:09Z","lastAt":"2010-08-02T22:55:51Z","messageCount":33,"participants":["Nguyễn Thái Ngọc Duy","Sverre Rabbelier","Nguyen Thai Ngoc Duy","Ævar Arnfjörð Bjarmason","Elijah Newren","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":16},"messages":[{"id":"146882","messageId":"1280593105-22015-1-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":null,"subject":"[PATCH 00/16] Subtree clone proof of concept","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:09Z","receivedAt":"2010-07-31T16:18:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Something to play with so we can evaluate which is the best strategy\nfor non-full clone (or whatever you call it).\n\nThe idea is the same: pack only enough to access a subtree, rewrite\ncommits at client side, rewrite again when pushing. However I put\ngit-replace into the mix, so at least commit SHA-1 looks as same as from\nupstream. git-subtree is not needed (although it's still an option)\n\nWith this, I can clone Documentaion/ from git.git, update and push. I\nhaven't tested it further. Space consumption is 24MB (58MB for full\nrepo). Not really impressive, but if one truely cares about disk\nspace, he/she should also use shallow clone.\n\nPerformance is impacted, due to bulk commit replacement. There is a\nsplit second delay for every command. It's the price of replacing 24k\ncommits every time. I think the delay could be improved a little bit\n(caching or mmap..)\n\nRewriting commits at clone takes time too. Doing individual object\nwriting takes lots of space and time. I put all new objects directly\nto a pack now. Rewriting time now becomes quite acceptable (a few\nseconds). Although deep subtree/repo may take longer. Rewriting on\ndemand can be considered in such cases.\n\nRepo-care commands like fsck, repack, gc are left out for now.\n\nFinally, it's more of a hack just to see how far I can go. It will\nbreak things.\n\nNguyễn Thái Ngọc Duy (16):\n Add core.subtree\n list-objects: limit traversing within the given subtree if\n core.subtree is set\n parse_object: keep sha1 even when parsing replaced one\n Allow to invalidate a commit in in-memory object store\n Hook up replace-object to allow bulk commit replacement\n upload-pack: use a separate variable to control whether internal\n rev-list is used\n upload-pack: support subtree pack\n fetch-pack: support --subtree\n subtree: rewrite incoming commits\n clone: support subtree clone with parameter --subtree\n pack-objects: add --subtree (for pushing)\n subtree: rewriting outgoing commits\n Update commit_tree() interface to take base tree too\n commit_tree(): rewriting/replacing new commits\n commit: rewrite outgoing commits\n do not use thin packs and subtree together (just a bad feeling about\n this)\n\n Makefile | 2 +\n builtin/clone.c | 10 +\n builtin/commit-tree.c | 2 +-\n builtin/commit.c | 4 +-\n builtin/fetch-pack.c | 8 +\n builtin/merge.c | 4 +-\n builtin/notes.c | 2 +-\n builtin/pack-objects.c | 4 +\n builtin/send-pack.c | 2 +\n cache.h | 1 +\n commit.c | 25 +++-\n commit.h | 4 +-\n config.c | 3 +\n environment.c | 2 +\n list-objects.c | 23 ++-\n notes-cache.c | 2 +-\n object.c | 2 +-\n replace_object.c | 5 +\n subtree.c | 534 ++++++++++++++++++++++++++++++++++++++++++++++++\n subtree.h | 4 +\n upload-pack.c | 28 ++-\n 21 files changed, 651 insertions(+), 20 deletions(-)\n create mode 100644 subtree.c\n create mode 100644 subtree.h\n"},{"id":"146883","messageId":"1280593105-22015-2-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 01/16] Add core.subtree","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:10Z","receivedAt":"2010-07-31T16:18:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This variable contains the subtree. With core_subtree being non-empty\nbehavior of git may be totally different.\n\nPerhaps this should not stay in .git/config, rather .git/subtree\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n config.c | 3 +++\n environment.c | 2 ++\n 3 files changed, 6 insertions(+), 0 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex c9fa3df..04ebe6e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -551,6 +551,7 @@ extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\n+extern const char *core_subtree;\n \n enum safe_crlf {\n \tSAFE_CRLF_FALSE = 0,\ndiff --git a/config.c b/config.c\nindex cdcf583..86ded29 100644\n--- a/config.c\n+++ b/config.c\n@@ -595,6 +595,9 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.subtree\"))\n+\t\treturn git_config_string(&core_subtree, var, value);\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 83d38d3..1365dd0 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -57,6 +57,8 @@ int core_apply_sparse_checkout;\n /* Parallel index stat data preload? */\n int core_preload_index = 0;\n \n+const char *core_subtree;\n+\n /* This is set by setup_git_dir_gently() and/or git_default_config() */\n char *git_work_tree_cfg;\n static char *work_tree;\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146884","messageId":"1280593105-22015-3-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 02/16] list-objects: limit traversing within the given subtree if core.subtree is set","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:11Z","receivedAt":"2010-07-31T16:18:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n list-objects.c | 23 +++++++++++++++++------\n 1 files changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 8953548..1b25b54 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -61,12 +61,15 @@ static void process_tree(struct rev_info *revs,\n \t\t\t struct tree *tree,\n \t\t\t show_object_fn show,\n \t\t\t struct name_path *path,\n-\t\t\t const char *name)\n+\t\t\t const char *name,\n+\t\t\t const char *subtree)\n {\n \tstruct object *obj = &tree->object;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tstruct name_path me;\n+\tconst char *slash;\n+\tint subtree_len;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -82,13 +85,21 @@ static void process_tree(struct rev_info *revs,\n \tme.elem = name;\n \tme.elem_len = strlen(name);\n \n+\tif (subtree) {\n+\t\tslash = strchr(subtree, '/');\n+\t\tsubtree_len = slash ? slash - subtree : strlen(subtree);\n+\t}\n+\n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n \n \twhile (tree_entry(&desc, &entry)) {\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tprocess_tree(revs,\n-\t\t\t\t lookup_tree(entry.sha1),\n-\t\t\t\t show, &me, entry.path);\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tif (!subtree || !strncmp(entry.path, subtree, subtree_len))\n+\t\t\t\tprocess_tree(revs,\n+\t\t\t\t\t lookup_tree(entry.sha1),\n+\t\t\t\t\t show, &me, entry.path,\n+\t\t\t\t\t slash && slash[1] ? slash+1 : NULL);\n+\t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(revs, entry.sha1,\n \t\t\t\t\tshow, &me, entry.path);\n@@ -164,7 +175,7 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t}\n \t\tif (obj->type == OBJ_TREE) {\n \t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t NULL, name);\n+\t\t\t\t NULL, name, core_subtree);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146885","messageId":"1280593105-22015-4-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 03/16] parse_object: keep sha1 even when parsing replaced one","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:12Z","receivedAt":"2010-07-31T16:18:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n object.c | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 277b3dd..7adfda7 100644\n--- a/object.c\n+++ b/object.c\n@@ -199,7 +199,7 @@ struct object *parse_object(const unsigned char *sha1)\n \t\t\treturn NULL;\n \t\t}\n \n-\t\tobj = parse_object_buffer(repl, type, size, buffer, &eaten);\n+\t\tobj = parse_object_buffer(sha1, type, size, buffer, &eaten);\n \t\tif (!eaten)\n \t\t\tfree(buffer);\n \t\treturn obj;\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146886","messageId":"1280593105-22015-5-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 04/16] Allow to invalidate a commit in in-memory object store","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:13Z","receivedAt":"2010-07-31T16:18:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is needed if replacing object happens at run time.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n commit.c | 15 +++++++++++++++\n commit.h | 2 ++\n 2 files changed, 17 insertions(+), 0 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex e9b0750..d1e30b2 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -315,6 +315,21 @@ int parse_commit(struct commit *item)\n \treturn ret;\n }\n \n+int invalidate_commit(struct commit *item)\n+{\n+\tif (!item)\n+\t\treturn -1;\n+\n+\tif (item->object.parsed) {\n+\t\titem->object.parsed = 0;\n+\t\tif (item->buffer) {\n+\t\t\tfree(item->buffer);\n+\t\t\titem->buffer = NULL;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n struct commit_list *commit_list_insert(struct commit *item, struct commit_list **list_p)\n {\n \tstruct commit_list *new_list = xmalloc(sizeof(struct commit_list));\ndiff --git a/commit.h b/commit.h\nindex eb2b8ac..d8c01ea 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -41,6 +41,8 @@ int parse_commit_buffer(struct commit *item, void *buffer, unsigned long size);\n \n int parse_commit(struct commit *item);\n \n+int invalidate_commit(struct commit *item);\n+\n struct commit_list * commit_list_insert(struct commit *item, struct commit_list **list_p);\n unsigned commit_list_count(const struct commit_list *l);\n struct commit_list * insert_by_date(struct commit *item, struct commit_list **list);\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146887","messageId":"1280593105-22015-6-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 05/16] Hook up replace-object to allow bulk commit replacement","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:14Z","receivedAt":"2010-07-31T16:18:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"$GIT_DIR/subtree contains commit mapping in subtree mode. It's quite\nlarge that putting it in $GIT_DIR/refs/replace may slow git down\nsignificantly. Even with this, there will be a split second delay for\nevery git command.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Makefile | 2 +\n replace_object.c | 5 ++\n subtree.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n subtree.h | 2 +\n 4 files changed, 126 insertions(+), 0 deletions(-)\n create mode 100644 subtree.c\n create mode 100644 subtree.h\n\ndiff --git a/Makefile b/Makefile\nindex f33648d..0d13538 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -525,6 +525,7 @@ LIB_H += sigchain.h\n LIB_H += strbuf.h\n LIB_H += string-list.h\n LIB_H += submodule.h\n+LIB_H += subtree.h\n LIB_H += tag.h\n LIB_H += transport.h\n LIB_H += tree.h\n@@ -629,6 +630,7 @@ LIB_OBJS += sigchain.o\n LIB_OBJS += strbuf.o\n LIB_OBJS += string-list.o\n LIB_OBJS += submodule.o\n+LIB_OBJS += subtree.o\n LIB_OBJS += symlinks.o\n LIB_OBJS += tag.o\n LIB_OBJS += trace.o\ndiff --git a/replace_object.c b/replace_object.c\nindex eb59604..5fe4099 100644\n--- a/replace_object.c\n+++ b/replace_object.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"sha1-lookup.h\"\n #include \"refs.h\"\n+#include \"subtree.h\"\n \n static struct replace_object {\n \tunsigned char sha1[2][20];\n@@ -82,6 +83,7 @@ static void prepare_replace_object(void)\n \tif (replace_object_prepared)\n \t\treturn;\n \n+\tprepare_subtree_commit();\n \tfor_each_replace_ref(register_replace_ref, NULL);\n \treplace_object_prepared = 1;\n }\n@@ -99,6 +101,9 @@ const unsigned char *lookup_replace_object(const unsigned char *sha1)\n \n \tprepare_replace_object();\n \n+\tif (core_subtree)\n+\t\tcur = subtree_lookup_object(cur);\n+\n \t/* Try to recursively replace the object */\n \tdo {\n \t\tif (--depth < 0)\ndiff --git a/subtree.c b/subtree.c\nnew file mode 100644\nindex 0000000..601d827\n--- /dev/null\n+++ b/subtree.c\n@@ -0,0 +1,117 @@\n+#include \"cache.h\"\n+#include \"commit.h\"\n+#include \"tree.h\"\n+#include \"diff.h\"\n+#include \"revision.h\"\n+#include \"refs.h\"\n+#include \"tag.h\"\n+#include \"progress.h\"\n+#include \"pack.h\"\n+#include \"sha1-lookup.h\"\n+#include \"csum-file.h\"\n+\n+static struct replace_object {\n+\tunsigned char sha1[2][20];\n+} **subtree_commit;\n+\n+static struct replace_object **subtree_commit, **subtree_commit_r;\n+static int subtree_commit_nr, subtree_commit_r_nr, subtree_commit_alloc;\n+\n+static const unsigned char *replace_sha1_access(size_t index, void *table)\n+{\n+\tstruct replace_object **replace = table;\n+\treturn replace[index]->sha1[0];\n+}\n+\n+static int subtree_replace_object_pos(struct replace_object **store, int nr,\n+\t\t\t\t const unsigned char *sha1)\n+{\n+\treturn sha1_pos(sha1, store, nr, replace_sha1_access);\n+}\n+\n+static int subtree_register_object(struct replace_object **store,\n+\t\t\t\t\t int *nr,\n+\t\t\t\t\t const unsigned char *sha1,\n+\t\t\t\t\t struct replace_object *replace,\n+\t\t\t\t\t int ignore_dups)\n+{\n+\tint pos = subtree_replace_object_pos(store, *nr, sha1);\n+\n+\tif (0 <= pos) {\n+\t\tif (ignore_dups)\n+\t\t\tfree(replace);\n+\t\telse {\n+\t\t\tfree(store[pos]);\n+\t\t\tstore[pos] = replace;\n+\t\t}\n+\t\treturn 1;\n+\t}\n+\tpos = -pos - 1;\n+\t(*nr)++;\n+\tif (pos < *nr)\n+\t\tmemmove(store + pos + 1,\n+\t\t\tstore + pos,\n+\t\t\t(*nr - pos - 1) *\n+\t\t\tsizeof(*store));\n+\tstore[pos] = replace;\n+\treturn 0;\n+}\n+\n+void prepare_subtree_commit()\n+{\n+\tint fd;\n+\tstruct stat stat;\n+\tstruct replace_object *ro;\n+\tint ro_size, ro_table_size;\n+\tchar *subtree, *entry;\n+\n+\tif (!core_subtree)\n+\t\treturn;\n+\n+\tfd = open(git_path(\"subtree\"), O_RDONLY);\n+\tif (fd == -1)\n+\t\treturn;\n+\n+\tif (fstat(fd, &stat))\n+\t\tdie(\"Could not stat .git/subtree\");\n+\n+\tif (stat.st_size % 82)\n+\t\tdie(\"Invalid .git/subtree size\");\n+\n+\tsubtree_commit_alloc = stat.st_size / 82;\n+\tro_size = sizeof(struct replace_object) * subtree_commit_alloc;\n+\tro_table_size = sizeof(struct replace_object*) * subtree_commit_alloc;\n+\tsubtree_commit_nr = 0;\n+\tsubtree_commit_r_nr = 0;\n+\n+\tsubtree_commit = xmalloc(ro_size + ro_table_size*2);\n+\tsubtree_commit_r = (struct replace_object **)(((char*)subtree_commit) + ro_table_size);\n+\tro = (struct replace_object *)(((char*)subtree_commit) + 2*ro_table_size);\n+\n+\tentry = subtree = xmmap(NULL, stat.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\n+\tfor (entry = subtree; subtree_commit_nr < subtree_commit_alloc; entry += 82, ro++) {\n+\t\tif (entry[40] != ' ' || entry[81] != '\\n')\n+\t\t\tdie(\"Broken .git/subtree\");\n+\n+\t\tget_sha1_hex(entry, ro->sha1[0]);\n+\t\tget_sha1_hex(entry+41, ro->sha1[1]);\n+\t\tif (subtree_register_object(subtree_commit, &subtree_commit_nr,\n+\t\t\t\t\t\t ro->sha1[0], ro, 1) ||\n+\t\t subtree_register_object(subtree_commit_r, &subtree_commit_r_nr,\n+\t\t\t\t\t\t ro->sha1[1], ro, 1))\n+\t\t\tdie(\"duplicate replace ref: %s\", sha1_to_hex(ro->sha1[0]));\n+\t}\n+\tmunmap(subtree, stat.st_size);\n+\tclose(fd);\n+}\n+\n+const unsigned char *subtree_lookup_object(const unsigned char *sha1)\n+{\n+\tint pos = subtree_replace_object_pos(subtree_commit,\n+\t\t\t\t\t subtree_commit_nr,\n+\t\t\t\t\t sha1);\n+\tif (0 <= pos)\n+\t\treturn subtree_commit[pos]->sha1[1];\n+\treturn sha1;\n+}\ndiff --git a/subtree.h b/subtree.h\nnew file mode 100644\nindex 0000000..157153a\n--- /dev/null\n+++ b/subtree.h\n@@ -0,0 +1,2 @@\n+void prepare_subtree_commit();\n+const unsigned char *subtree_lookup_object(const unsigned char *sha1);\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146888","messageId":"1280593105-22015-7-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 06/16] upload-pack: use a separate variable to control whether internal rev-list is used","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:15Z","receivedAt":"2010-07-31T16:18:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n upload-pack.c | 9 +++++----\n 1 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex dc464d7..e432e83 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -160,8 +160,9 @@ static void create_pack_file(void)\n \tssize_t sz;\n \tconst char *argv[10];\n \tint arg = 0;\n+\tint internal_rev_list = shallow_nr;\n \n-\tif (shallow_nr) {\n+\tif (internal_rev_list) {\n \t\tmemset(&rev_list, 0, sizeof(rev_list));\n \t\trev_list.proc = do_rev_list;\n \t\trev_list.out = -1;\n@@ -187,7 +188,7 @@ static void create_pack_file(void)\n \targv[arg++] = NULL;\n \n \tmemset(&pack_objects, 0, sizeof(pack_objects));\n-\tpack_objects.in = shallow_nr ? rev_list.out : -1;\n+\tpack_objects.in = internal_rev_list ? rev_list.out : -1;\n \tpack_objects.out = -1;\n \tpack_objects.err = -1;\n \tpack_objects.git_cmd = 1;\n@@ -197,7 +198,7 @@ static void create_pack_file(void)\n \t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n \n \t/* pass on revisions we (don't) want */\n-\tif (!shallow_nr) {\n+\tif (!internal_rev_list) {\n \t\tFILE *pipe_fd = xfdopen(pack_objects.in, \"w\");\n \t\tif (!create_full_pack) {\n \t\t\tint i;\n@@ -311,7 +312,7 @@ static void create_pack_file(void)\n \t\terror(\"git upload-pack: git-pack-objects died with error.\");\n \t\tgoto fail;\n \t}\n-\tif (shallow_nr && finish_async(&rev_list))\n+\tif (internal_rev_list && finish_async(&rev_list))\n \t\tgoto fail;\t/* error was already reported */\n \n \t/* flush the data */\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146889","messageId":"1280593105-22015-8-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 07/16] upload-pack: support subtree pack","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:16Z","receivedAt":"2010-07-31T16:18:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"With core_subtree turned on (capability \"subtree\", request \"subtree\"\nfrom fetch-pack), traverse_commit_list will be in \"subtree mode\",\nwhich will not go farther than the given subtree.\n\nAs the result, the pack is broken be design, only contains enough\nblobs/trees/commits to reach the given subtree.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n upload-pack.c | 18 ++++++++++++++++--\n 1 files changed, 16 insertions(+), 2 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex e432e83..9b6710a 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -160,7 +160,7 @@ static void create_pack_file(void)\n \tssize_t sz;\n \tconst char *argv[10];\n \tint arg = 0;\n-\tint internal_rev_list = shallow_nr;\n+\tint internal_rev_list = shallow_nr || core_subtree;\n \n \tif (internal_rev_list) {\n \t\tmemset(&rev_list, 0, sizeof(rev_list));\n@@ -505,6 +505,20 @@ static void receive_needs(void)\n \t\tif (debug_fd)\n \t\t\twrite_in_full(debug_fd, line, len);\n \n+\t\tif (!prefixcmp(line, \"subtree \")) {\n+\t\t\tint len;\n+\t\t\tchar *subtree;\n+\t\t\tif (core_subtree)\n+\t\t\t\tdie(\"sorry, only one subtree supported\");\n+\t\t\tlen = strlen(line+8);\n+\t\t\tsubtree = malloc(len+1);\n+\t\t\tmemcpy(subtree, line+8, len-1);\n+\t\t\tsubtree[len-1] = '\\0'; /* \\n */\n+\t\t\tif (subtree[len-2] != '/')\n+\t\t\t\tdie(\"subtree request must end with a slash\");\n+\t\t\tcore_subtree = subtree;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!prefixcmp(line, \"shallow \")) {\n \t\t\tunsigned char sha1[20];\n \t\t\tstruct object *object;\n@@ -624,7 +638,7 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow no-progress\"\n-\t\t\" include-tag multi_ack_detailed\";\n+\t\t\" include-tag multi_ack_detailed subtree\";\n \tstruct object *o = parse_object(sha1);\n \n \tif (!o)\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146890","messageId":"1280593105-22015-9-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 08/16] fetch-pack: support --subtree","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:17Z","receivedAt":"2010-07-31T16:18:17Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This options requires subtree-aware upload-pack. It simply pass the\nsubtree from command line (or from $GIT_DIR/config) to upload-pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch-pack.c | 8 ++++++++\n 1 files changed, 8 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex dbd8b7b..7460ecc 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -237,6 +237,8 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tfor_each_ref(rev_list_insert_ref, NULL);\n \n \tfetching = 0;\n+\tif (core_subtree)\n+\t\tpacket_buf_write(&req_buf, \"subtree %s\\n\", core_subtree);\n \tfor ( ; refs ; refs = refs->next) {\n \t\tunsigned char *remote = refs->old_sha1;\n \t\tconst char *remote_hex;\n@@ -692,6 +694,8 @@ static struct ref *do_fetch_pack(int fd[2],\n \n \tif (is_repository_shallow() && !server_supports(\"shallow\"))\n \t\tdie(\"Server does not support shallow clients\");\n+\tif (core_subtree && !server_supports(\"subtree\"))\n+\t\tdie(\"Server does not support subtree\");\n \tif (server_supports(\"multi_ack_detailed\")) {\n \t\tif (args.verbose)\n \t\t\tfprintf(stderr, \"Server supports multi_ack_detailed\\n\");\n@@ -860,6 +864,10 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tpack_lockfile_ptr = &pack_lockfile;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!prefixcmp(arg, \"--subtree=\")) {\n+\t\t\t\tcore_subtree = arg + 10;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tusage(fetch_pack_usage);\n \t\t}\n \t\tdest = (char *)arg;\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146891","messageId":"1280593105-22015-10-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 09/16] subtree: rewrite incoming commits","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:18Z","receivedAt":"2010-07-31T16:18:18Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This adds the main function, subtree_import(), which is intended to be\nused by \"git clone\".\n\nBecause subtree packs are not complete. They are barely usable. Git\nclient will cry out missing objects here and there... Theortically,\nclient code could be adapted to only look for objects within\nsubtree. That was painful to try.\n\nAlternatively, subtree_import() rewrites commits to have only the\nspecified subtree, sealing all broken path. Git client now happily\nworks with these new commits.\n\nHowever, users might not, because it's different commit, different\nSHA-1. They can't use those SHA-1 to communicate within their team. To\nwork around this, all original commits are replaced by new commits\nusing git-replace.\n\nOf course this is still not perfect. Users may be able to send SHA-1\naround, which is consistent. They may not do the same with tree SHA-1.\n\nRewriting/replacing commits takes time and space. For replacing _all_\ncommits, the current replace mechanism is not suitable, which is why\nsubtree_lookup_object() was introduced in previous patches.\n\nFor rewriting, writing a huge number of objects is slow. So\nsubtree_import() builds a pack for all new objects. These packs are\nnot optimized. But it does reduce wait time for rewriting.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n subtree.c | 244 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n subtree.h | 1 +\n 2 files changed, 245 insertions(+), 0 deletions(-)\n\ndiff --git a/subtree.c b/subtree.c\nindex 601d827..8c075be 100644\n--- a/subtree.c\n+++ b/subtree.c\n@@ -115,3 +115,247 @@ const unsigned char *subtree_lookup_object(const unsigned char *sha1)\n \t\treturn subtree_commit[pos]->sha1[1];\n \treturn sha1;\n }\n+\n+static unsigned long do_compress(void **pptr, unsigned long size)\n+{\n+\tz_stream stream;\n+\tvoid *in, *out;\n+\tunsigned long maxsize;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tdeflateInit(&stream, Z_DEFAULT_COMPRESSION);\n+\tmaxsize = deflateBound(&stream, size);\n+\n+\tin = *pptr;\n+\tout = xmalloc(maxsize);\n+\t*pptr = out;\n+\n+\tstream.next_in = in;\n+\tstream.avail_in = size;\n+\tstream.next_out = out;\n+\tstream.avail_out = maxsize;\n+\twhile (deflate(&stream, Z_FINISH) == Z_OK)\n+\t\t; /* nothing */\n+\tdeflateEnd(&stream);\n+\n+\treturn stream.total_out;\n+}\n+\n+static int nr_written;\n+static int add_sha1_to_pack(int fd, void *buf, unsigned long size, enum object_type type)\n+{\n+\tunsigned long datalen;\n+\tunsigned hdrlen;\n+\tunsigned char header[10];\n+\n+\tdatalen = do_compress(&buf, size);\n+\thdrlen = encode_in_pack_object_header(type, size, header);\n+\twrite(fd, header, hdrlen);\n+\twrite(fd, buf, datalen);\n+\tnr_written++;\n+\tfree(buf);\n+\treturn 0;\n+}\n+\n+/*\n+ * Take sha1 of a tree, rewrite it to only return the prefix and return\n+ * the newsha1.\n+ *\n+ * If if is zero, write to object store. If fd is greater than zero,\n+ * it's a pack file handle.\n+ */\n+static int narrow_tree(const unsigned char *sha1, unsigned char *newsha1,\n+\t\t const char *prefix, int fd)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tstruct strbuf buffer;\n+\tconst char *slash;\n+\tint subtree_len;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tchar *tree;\n+\tstruct object *obj;\n+\n+\tslash = strchr(prefix, '/');\n+\tsubtree_len = slash ? slash - prefix : strlen(prefix);\n+\n+\ttree = read_sha1_file(sha1, &type, &size);\n+\tif (type != OBJ_TREE)\n+\t\tdie(\"%s is not a tree\", sha1_to_hex(sha1));\n+\n+\tinit_tree_desc(&desc, tree, size);\n+\tstrbuf_init(&buffer, 1024);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (!S_ISDIR(entry.mode))\n+\t\t\tcontinue;\n+\n+\t\tif (subtree_len == strlen(entry.path) &&\n+\t\t !strncmp(entry.path, prefix, subtree_len)) {\n+\t\t\tunsigned char newtree_sha1[20];\n+\n+\t\t\tif (slash && slash[1]) /* trailing slash does not count */\n+\t\t\t\tnarrow_tree(entry.sha1, newtree_sha1, prefix+subtree_len+1, fd);\n+\t\t\telse\n+\t\t\t\tmemcpy(newtree_sha1, entry.sha1, 20);\n+\n+\t\t\tstrbuf_addf(&buffer, \"%o %.*s%c\", entry.mode, strlen(entry.path), entry.path, '\\0');\n+\t\t\tstrbuf_add(&buffer, newtree_sha1, 20);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tfree(tree);\n+\n+\tif (fd == 0) {\n+\t\tif (write_sha1_file(buffer.buf, buffer.len, tree_type, newsha1)) {\n+\t\t\terror(\"Could not write replaced tree for %s\", sha1_to_hex(sha1));\n+\t\t\tstrbuf_release(&buffer);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tstrbuf_release(&buffer);\n+\t\treturn 0;\n+\t}\n+\n+\thash_sha1_file(buffer.buf, buffer.len, tree_type, newsha1);\n+\tobj = (struct object *)lookup_tree(newsha1);\n+\tif (fd > 0 &&\n+\t !(obj->flags & SEEN) &&\n+\t add_sha1_to_pack(fd, buffer.buf, buffer.len, OBJ_TREE)) {\n+\t\terror(\"Could not write replaced tree for %s\", sha1_to_hex(sha1));\n+\t\tstrbuf_release(&buffer);\n+\t\treturn 1;\n+\t}\n+\tobj->flags |= SEEN;\n+\n+\tstrbuf_release(&buffer);\n+\treturn 0;\n+}\n+\n+/*\n+ * Take sha1 of a commit, rewrite its tree using narrow_tree(), then\n+ * add a replace entry to file pointer fp (which is $GIT_DIR/subtree).\n+ *\n+ * Also update replace-object database so that the given sha1 can be\n+ * replaced with the new one right after this function returns.\n+ */\n+static int shadow_commit(const unsigned char *sha1, const char *prefix, int fd, FILE *fp)\n+{\n+\tunsigned char newsha1[20], treesha1[20];\n+\tenum object_type type;\n+\tunsigned long size;\n+\tvoid *buffer;\n+\tstruct object *obj;\n+\tint saved_read_replace_refs = read_replace_refs;\n+\n+\tread_replace_refs = 0;\n+\tbuffer = read_sha1_file(sha1, &type, &size);\n+\tread_replace_refs = saved_read_replace_refs;\n+\tget_sha1_hex(buffer+5, treesha1);\n+\n+\tif (!buffer || type != OBJ_COMMIT ||\n+\t narrow_tree(treesha1, newsha1, prefix, fd)) {\n+\t\tfree(buffer);\n+\t\terror(\"Failed to narrow tree for commit %s\", sha1_to_hex(sha1));\n+\t\treturn 1;\n+\t}\n+\n+\t/* replace new tree in */\n+\tmemcpy((char*)buffer+5, sha1_to_hex(newsha1), 40);\n+\n+\tif (fd == 0) {\n+\t\tif (write_sha1_file(buffer, size, commit_type, newsha1)) {\n+\t\t\tfree(buffer);\n+\t\t\terror(\"Could not write replaced commit for %s\", sha1_to_hex(sha1));\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\telse {\n+\t\thash_sha1_file(buffer, size, commit_type, newsha1);\n+\t\tobj = (struct object *)lookup_commit(newsha1);\n+\t\tif (fd > 0 &&\n+\t\t !(obj->flags & SEEN) &&\n+\t\t add_sha1_to_pack(fd, buffer, size, OBJ_COMMIT)) {\n+\t\t\tfree(buffer);\n+\t\t\terror(\"Could not write replaced commit for %s\", sha1_to_hex(sha1));\n+\t\t\treturn 1;\n+\t\t}\n+\t\tobj->flags |= SEEN;\n+\t}\n+\n+\tif (fp) {\n+\t\tchar buf[82];\n+\t\tmemcpy(buf, sha1_to_hex(sha1), 40);\n+\t\tbuf[40] = ' ';\n+\t\tmemcpy(buf+41, sha1_to_hex(newsha1), 40);\n+\t\tbuf[81] = '\\n';\n+\t\tfwrite(buf, 82, 1, fp);\n+\t}\n+\tfree(buffer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Rewrite all reachable commits in repo using shadow_commit().\n+ * Write out the pack that contains new tree/commit objects.\n+ */\n+void subtree_import()\n+{\n+\tconst char *args[] = {\"rev-list\", \"--all\", NULL};\n+\tstruct pack_header hdr;\n+\tstruct progress *ps;\n+\tstruct rev_info revs;\n+\tstruct commit *c;\n+\tunsigned char sha1[20];\n+\tunsigned commit_nr = 0;\n+\tchar *pack_tmp_name;\n+\tchar tmpname[PATH_MAX];\n+\tint pack_fd, i;\n+\tFILE *fp;\n+\tchar cmd[1024];\n+\n+\t/* Packing */\n+\tinit_revisions(&revs, NULL);\n+\tsetup_revisions(2, args, &revs, NULL);\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(\"revision walk setup failed\");\n+\tfp = fopen(git_path(\"subtree\"), \"w+\");\n+\n+\tpack_fd = odb_mkstemp(tmpname, sizeof(tmpname), \"pack/tmp_pack_XXXXXX\");\n+\tpack_tmp_name = xstrdup(tmpname);\n+\n+\thdr.hdr_signature = htonl(PACK_SIGNATURE);\n+\thdr.hdr_version = htonl(PACK_VERSION);\n+\thdr.hdr_entries = htonl(0);\n+\twrite(pack_fd, &hdr, sizeof(hdr));\n+\n+\tps = start_progress(\"Preparing subtree commits\", 0);\n+\twhile ((c = get_revision(&revs)) != NULL) {\n+\t\tif (shadow_commit(c->object.sha1, core_subtree, pack_fd, fp))\n+\t\t\tdie(\"Failed to shadow commit %s\", c->object.sha1);\n+\t\tdisplay_progress(ps, ++commit_nr);\n+\t}\n+\tstop_progress(&ps);\n+\tfclose(fp);\n+\tfixup_pack_header_footer(pack_fd, sha1, pack_tmp_name, nr_written, NULL, 0);\n+\tclose(pack_fd);\n+\tsprintf(cmd, \"git index-pack --stdin < %s\", pack_tmp_name);\n+\tsystem(cmd);\n+\tunlink(pack_tmp_name);\n+\n+\treprepare_packed_git();\n+\tif (subtree_commit)\n+\t\tfree(subtree_commit);\n+\tprepare_subtree_commit();\n+\n+\t/* Invalidate all replaced commits */\n+\tfor (i = 0; i < subtree_commit_nr; i++) {\n+\t\t/* lookup_commit() would create new objects, we don't want that */\n+\t\tc = (struct commit *)lookup_object(subtree_commit[i]->sha1[0]);\n+\t\tif (c)\n+\t\t\tinvalidate_commit(c);\n+\t}\n+\n+\tif (revs.pending.nr)\n+\t\tfree(revs.pending.objects);\n+}\ndiff --git a/subtree.h b/subtree.h\nindex 157153a..3512e2a 100644\n--- a/subtree.h\n+++ b/subtree.h\n@@ -1,2 +1,3 @@\n void prepare_subtree_commit();\n const unsigned char *subtree_lookup_object(const unsigned char *sha1);\n+void subtree_import();\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146892","messageId":"1280593105-22015-11-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 10/16] clone: support subtree clone with parameter --subtree","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:19Z","receivedAt":"2010-07-31T16:18:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"With all the preparation work, here comes --subtree. So clone away!\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 10 ++++++++++\n 1 files changed, 10 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex efb1e6f..43bc34b 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -23,6 +23,7 @@\n #include \"branch.h\"\n #include \"remote.h\"\n #include \"run-command.h\"\n+#include \"subtree.h\"\n \n /*\n * Overall FIXMEs:\n@@ -78,6 +79,8 @@ static struct option builtin_clone_options[] = {\n \t\t \"path to git-upload-pack on the remote\"),\n \tOPT_STRING(0, \"depth\", &option_depth, \"depth\",\n \t\t \"create a shallow clone of that depth\"),\n+\tOPT_STRING(0, \"subtree\", &core_subtree, \"subtree\",\n+\t\t \"subtree clone\"),\n \n \tOPT_END()\n };\n@@ -515,6 +518,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstrbuf_reset(&value);\n \n \tif (path && !is_bundle) {\n+\t\tif (core_subtree)\n+\t\t\tdie(\"Local subtree clone does not work (now)\");\n \t\trefs = clone_local(path, git_dir);\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\n \t} else {\n@@ -623,6 +628,11 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_disconnect(transport);\n \t}\n \n+\tif (core_subtree) {\n+\t\tgit_config_set(\"core.subtree\", core_subtree);\n+\t\tsubtree_import();\n+\t}\n+\n \tif (!option_no_checkout) {\n \t\tstruct lock_file *lock_file = xcalloc(1, sizeof(struct lock_file));\n \t\tstruct unpack_trees_options opts;\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146893","messageId":"1280593105-22015-12-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 11/16] pack-objects: add --subtree (for pushing)","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:20Z","receivedAt":"2010-07-31T16:18:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 4 ++++\n 1 files changed, 4 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 0e81673..5d7b277 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2277,6 +2277,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\tgrafts_replace_parents = 0;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!prefixcmp(arg, \"--subtree=\")) {\n+\t\t\tcore_subtree = arg + 10;\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(pack_usage);\n \t}\n \n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146894","messageId":"1280593105-22015-13-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 12/16] subtree: rewriting outgoing commits","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:21Z","receivedAt":"2010-07-31T16:18:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Which is exactly the opposite of rewriting incoming commits.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n subtree.c | 173 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n subtree.h | 1 +\n 2 files changed, 174 insertions(+), 0 deletions(-)\n\ndiff --git a/subtree.c b/subtree.c\nindex 8c075be..739ff5f 100644\n--- a/subtree.c\n+++ b/subtree.c\n@@ -359,3 +359,176 @@ void subtree_import()\n \tif (revs.pending.nr)\n \t\tfree(revs.pending.objects);\n }\n+\n+/*\n+ * The opposite of narrow_tree(). Put the subtree back to the original tree.\n+ */\n+static int widen_tree(const unsigned char *sha1,\n+\t\t unsigned char *newsha1,\n+\t\t const unsigned char *subtree_sha1,\n+\t\t const char *prefix)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tstruct strbuf buffer;\n+\tconst char *slash;\n+\tint subtree_len;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tchar *tree;\n+\n+\tslash = strchr(prefix, '/');\n+\tsubtree_len = slash ? slash - prefix : strlen(prefix);\n+\n+\ttree = read_sha1_file(sha1, &type, &size);\n+\tif (type != OBJ_TREE)\n+\t\tdie(\"%s is not a tree\", sha1_to_hex(sha1));\n+\n+\tinit_tree_desc(&desc, tree, size);\n+\tstrbuf_init(&buffer, 8192);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tstrbuf_addf(&buffer, \"%o %.*s%c\", entry.mode, strlen(entry.path), entry.path, '\\0');\n+\n+\t\tif (S_ISDIR(entry.mode) &&\n+\t\t subtree_len == strlen(entry.path) &&\n+\t\t !strncmp(entry.path, prefix, subtree_len)) {\n+\t\t\tunsigned char newtree_sha1[20];\n+\n+\t\t\tif (slash && slash[1]) /* trailing slash does not count */\n+\t\t\t\twiden_tree(entry.sha1, newtree_sha1, subtree_sha1,\n+\t\t\t\t\t prefix+subtree_len+1);\n+\t\t\telse\n+\t\t\t\t/* replace the tree */\n+\t\t\t\tmemcpy(newtree_sha1, subtree_sha1, 20);\n+\n+\t\t\tstrbuf_add(&buffer, newtree_sha1, 20);\n+\t\t}\n+\t\telse\n+\t\t\tstrbuf_add(&buffer, entry.sha1, 20);\n+\t}\n+\tfree(tree);\n+\n+\tif (write_sha1_file(buffer.buf, buffer.len, tree_type, newsha1)) {\n+\t\terror(\"Could not write replaced tree for %s\", sha1_to_hex(sha1));\n+\t\tstrbuf_release(&buffer);\n+\t\treturn 1;\n+\t}\n+\tstrbuf_release(&buffer);\n+\treturn 0;\n+}\n+\n+static int find_subtree(const unsigned char *sha1, unsigned char *newsha1, const char *prefix)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tconst char *slash;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tint subtree_len;\n+\tchar *tree;\n+\n+\tslash = strchr(prefix, '/');\n+\tsubtree_len = slash ? slash - prefix : strlen(prefix);\n+\n+\ttree = read_sha1_file(sha1, &type, &size);\n+\tif (type != OBJ_TREE)\n+\t\tdie(\"%s is not a tree\", sha1_to_hex(sha1));\n+\n+\tinit_tree_desc(&desc, tree, size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tif (!S_ISDIR(entry.mode))\n+\t\t\tcontinue;\n+\n+\t\tif (subtree_len == strlen(entry.path) &&\n+\t\t !strncmp(entry.path, prefix, subtree_len)) {\n+\n+\t\t\tif (slash && slash[1]) { /* trailing slash does not count */\n+\t\t\t\tif (find_subtree(entry.sha1, newsha1, prefix+subtree_len+1))\n+\t\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\telse\n+\t\t\t\tmemcpy(newsha1, entry.sha1, 20);\n+\t\t\tfree(tree);\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\tfree(tree);\n+\n+\treturn 1;\n+}\n+\n+/* The opposite of shadow_commit() */\n+static int expose_commit(const unsigned char *sha1, unsigned char *newsha1,\n+\t\t\t const unsigned char *basesha1,\n+\t\t\t const char *prefix, FILE *fp)\n+{\n+\tunsigned char treesha1[20], subtree_sha1[20];\n+\tenum object_type type;\n+\tunsigned long size, base_size;\n+\tvoid *base_buffer, *buffer;\n+\tint saved_read_replace_refs = read_replace_refs;\n+\n+\t/* Get subtree from the new commit, sha1 */\n+\tread_replace_refs = 0;\n+\tbuffer = read_sha1_file(sha1, &type, &size);\n+\tread_replace_refs = saved_read_replace_refs;\n+\tget_sha1_hex(buffer+5, treesha1);\n+\n+\tif (!buffer || type != OBJ_COMMIT ||\n+\t find_subtree(treesha1, subtree_sha1, prefix)) {\n+\t\tfree(buffer);\n+\t\terror(\"Failed to find subtree tree in base commit %s\", sha1_to_hex(sha1));\n+\t\treturn 1;\n+\t}\n+\n+\t/* Get the old base tree from basesha1 */\n+\tread_replace_refs = 0;\n+\tbase_buffer = read_sha1_file(basesha1, &type, &base_size);\n+\tread_replace_refs = saved_read_replace_refs;\n+\tget_sha1_hex(base_buffer+5, treesha1);\n+\n+\tif (!buffer || type != OBJ_COMMIT ||\n+\t widen_tree(treesha1, newsha1, subtree_sha1, prefix)) {\n+\t\tfree(buffer);\n+\t\terror(\"Failed to widen tree for commit %s\", sha1_to_hex(sha1));\n+\t\treturn 1;\n+\t}\n+\tfree(base_buffer);\n+\n+\t/* replace new tree in */\n+\tmemcpy((char*)buffer+5, sha1_to_hex(newsha1), 40);\n+\n+\tif (write_sha1_file(buffer, size, commit_type, newsha1)) {\n+\t\tfree(buffer);\n+\t\terror(\"Could not write replaced commit for %s\", sha1_to_hex(sha1));\n+\t\treturn 1;\n+\t}\n+\n+\tif (fp) {\n+\t\tchar buf[82];\n+\t\tmemcpy(buf, sha1_to_hex(newsha1), 40);\n+\t\tbuf[40] = ' ';\n+\t\tmemcpy(buf+41, sha1_to_hex(sha1), 40);\n+\t\tbuf[81] = '\\n';\n+\t\tfwrite(buf, 82, 1, fp);\n+\t}\n+\tfree(buffer);\n+\n+\treturn 0;\n+}\n+\n+int subtree_export(unsigned char *sha1, unsigned char *basesha1, unsigned char *newsha1)\n+{\n+\tFILE *fp;\n+\n+\tfp = fopen(git_path(\"subtree\"), \"a+\");\n+\tif (expose_commit(sha1, newsha1, basesha1, core_subtree, fp))\n+\t\tdie(\"Failed to rewrite commit %s\", sha1_to_hex(sha1));\n+\tfclose(fp);\n+\n+\tif (subtree_commit)\n+\t\tfree(subtree_commit);\n+\tprepare_subtree_commit();\n+\n+\treturn 0;\n+}\ndiff --git a/subtree.h b/subtree.h\nindex 3512e2a..081838f 100644\n--- a/subtree.h\n+++ b/subtree.h\n@@ -1,3 +1,4 @@\n void prepare_subtree_commit();\n const unsigned char *subtree_lookup_object(const unsigned char *sha1);\n void subtree_import();\n+int subtree_export(unsigned char *sha1, unsigned char *basesha1, unsigned char *newsha1);\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146895","messageId":"1280593105-22015-14-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 13/16] Update commit_tree() interface to take base tree too","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:22Z","receivedAt":"2010-07-31T16:18:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In subtree mode, you work on a narrowed trees. You make narrowed\ncommits. If you want to push upstream, you would need to put your\nupdated subtree back to the full tree again. Otherwise upstream would\ncomplain you delete all trees but your subtree, not good.\n\nIn order to do that, commit_tree() now takes the base tree SHA-1. With\nthat, it can create upstream-compatible commits. It does not now,\nthough.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/commit-tree.c | 2 +-\n builtin/commit.c | 2 +-\n builtin/merge.c | 4 ++--\n builtin/notes.c | 2 +-\n commit.c | 2 +-\n commit.h | 2 +-\n notes-cache.c | 2 +-\n 7 files changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/commit-tree.c b/builtin/commit-tree.c\nindex 87f0591..88a6833 100644\n--- a/builtin/commit-tree.c\n+++ b/builtin/commit-tree.c\n@@ -56,7 +56,7 @@ int cmd_commit_tree(int argc, const char **argv, const char *prefix)\n \tif (strbuf_read(&buffer, 0, 0) < 0)\n \t\tdie_errno(\"git commit-tree: failed to read\");\n \n-\tif (!commit_tree(buffer.buf, tree_sha1, parents, commit_sha1, NULL)) {\n+\tif (!commit_tree(buffer.buf, tree_sha1, NULL, parents, commit_sha1, NULL)) {\n \t\tprintf(\"%s\\n\", sha1_to_hex(commit_sha1));\n \t\treturn 0;\n \t}\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 2bb30c0..6b4c678 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1350,7 +1350,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\texit(1);\n \t}\n \n-\tif (commit_tree(sb.buf, active_cache_tree->sha1, parents, commit_sha1,\n+\tif (commit_tree(sb.buf, active_cache_tree->sha1, NULL, parents, commit_sha1,\n \t\t\tfmt_ident(author_name, author_email, author_date,\n \t\t\t\tIDENT_ERROR_ON_NO_NAME))) {\n \t\trollback_index_files();\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 37ce4f5..8745b54 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -779,7 +779,7 @@ static int merge_trivial(void)\n \tparent->next = xmalloc(sizeof(*parent->next));\n \tparent->next->item = remoteheads->item;\n \tparent->next->next = NULL;\n-\tcommit_tree(merge_msg.buf, result_tree, parent, result_commit, NULL);\n+\tcommit_tree(merge_msg.buf, result_tree, NULL, parent, result_commit, NULL);\n \tfinish(result_commit, \"In-index merge\");\n \tdrop_save();\n \treturn 0;\n@@ -808,7 +808,7 @@ static int finish_automerge(struct commit_list *common,\n \t}\n \tfree_commit_list(remoteheads);\n \tstrbuf_addch(&merge_msg, '\\n');\n-\tcommit_tree(merge_msg.buf, result_tree, parents, result_commit, NULL);\n+\tcommit_tree(merge_msg.buf, result_tree, NULL, parents, result_commit, NULL);\n \tstrbuf_addf(&buf, \"Merge made by %s.\", wt_strategy);\n \tfinish(result_commit, buf.buf);\n \tstrbuf_release(&buf);\ndiff --git a/builtin/notes.c b/builtin/notes.c\nindex 190005f..8d574eb 100644\n--- a/builtin/notes.c\n+++ b/builtin/notes.c\n@@ -303,7 +303,7 @@ int commit_notes(struct notes_tree *t, const char *msg)\n \t\thashclr(prev_commit);\n \t\tparent = NULL;\n \t}\n-\tif (commit_tree(buf.buf + 7, tree_sha1, parent, new_commit, NULL))\n+\tif (commit_tree(buf.buf + 7, tree_sha1, NULL, parent, new_commit, NULL))\n \t\tdie(\"Failed to commit notes tree to database\");\n \n \t/* Update notes ref with new commit */\ndiff --git a/commit.c b/commit.c\nindex d1e30b2..7121631 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -811,7 +811,7 @@ static const char commit_utf8_warn[] =\n \"You may want to amend it after fixing the message, or set the config\\n\"\n \"variable i18n.commitencoding to the encoding your project uses.\\n\";\n \n-int commit_tree(const char *msg, unsigned char *tree,\n+int commit_tree(const char *msg, unsigned char *tree, unsigned char *base_tree,\n \t\tstruct commit_list *parents, unsigned char *ret,\n \t\tconst char *author)\n {\ndiff --git a/commit.h b/commit.h\nindex d8c01ea..7c34368 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -166,7 +166,7 @@ static inline int single_parent(struct commit *commit)\n \n struct commit_list *reduce_heads(struct commit_list *heads);\n \n-extern int commit_tree(const char *msg, unsigned char *tree,\n+extern int commit_tree(const char *msg, unsigned char *tree, unsigned char *base_tree,\n \t\tstruct commit_list *parents, unsigned char *ret,\n \t\tconst char *author);\n \ndiff --git a/notes-cache.c b/notes-cache.c\nindex dee6d62..96afb25 100644\n--- a/notes-cache.c\n+++ b/notes-cache.c\n@@ -56,7 +56,7 @@ int notes_cache_write(struct notes_cache *c)\n \n \tif (write_notes_tree(&c->tree, tree_sha1))\n \t\treturn -1;\n-\tif (commit_tree(c->validity, tree_sha1, NULL, commit_sha1, NULL) < 0)\n+\tif (commit_tree(c->validity, tree_sha1, NULL, NULL, commit_sha1, NULL) < 0)\n \t\treturn -1;\n \tif (update_ref(\"update notes cache\", c->tree.ref, commit_sha1, NULL,\n \t\t 0, QUIET_ON_ERR) < 0)\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146896","messageId":"1280593105-22015-15-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 14/16] commit_tree(): rewriting/replacing new commits","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:23Z","receivedAt":"2010-07-31T16:18:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n commit.c | 8 ++++++++\n 1 files changed, 8 insertions(+), 0 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 7121631..258d3fb 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -6,6 +6,7 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"notes.h\"\n+#include \"subtree.h\"\n \n int save_commit_buffer = 1;\n \n@@ -858,5 +859,12 @@ int commit_tree(const char *msg, unsigned char *tree, unsigned char *base_tree,\n \n \tresult = write_sha1_file(buffer.buf, buffer.len, commit_type, ret);\n \tstrbuf_release(&buffer);\n+\n+\tif (core_subtree && !result) {\n+\t\tunsigned char subtree_commit[20];\n+\t\tmemcpy(subtree_commit, ret, 20);\n+\t\tresult = subtree_export(subtree_commit, base_tree, ret);\n+\t}\n+\n \treturn result;\n }\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146897","messageId":"1280593105-22015-16-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 15/16] commit: rewrite outgoing commits","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:24Z","receivedAt":"2010-07-31T16:18:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/commit.c | 4 +++-\n 1 files changed, 3 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 6b4c678..c551d72 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1350,7 +1350,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\texit(1);\n \t}\n \n-\tif (commit_tree(sb.buf, active_cache_tree->sha1, NULL, parents, commit_sha1,\n+\tif (commit_tree(sb.buf, active_cache_tree->sha1,\n+\t\t\tparents ? parents->item->object.sha1 : NULL,\n+\t\t\tparents, commit_sha1,\n \t\t\tfmt_ident(author_name, author_email, author_date,\n \t\t\t\tIDENT_ERROR_ON_NO_NAME))) {\n \t\trollback_index_files();\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146898","messageId":"1280593105-22015-17-git-send-email-pclouds@gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 16/16] do not use thin packs and subtree together (just a bad feeling about this)","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-31T16:18:25Z","receivedAt":"2010-07-31T16:18:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 2 ++\n upload-pack.c | 3 +++\n 2 files changed, 5 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 481602d..fb1ad2b 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -53,6 +53,8 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \tint i;\n \n \ti = 4;\n+\tif (core_subtree)\n+\t\targs->use_thin_pack = 0;\n \tif (args->use_thin_pack)\n \t\targv[i++] = \"--thin\";\n \tif (args->use_ofs_delta)\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 9b6710a..c65a3cb 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -581,6 +581,9 @@ static void receive_needs(void)\n \tif (!use_sideband && daemon_mode)\n \t\tno_progress = 1;\n \n+\tif (core_subtree)\n+\t\tuse_thin_pack = 0;\n+\n \tif (depth == 0 && shallows.nr == 0)\n \t\treturn;\n \tif (depth > 0) {\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"146901","messageId":"AANLkTinMBAikByLfOi5sXbcRfCsYtYqMCNSTykRofQyo@mail.gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 00/16] Subtree clone proof of concept","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-08-01T04:14:09Z","receivedAt":"2010-08-01T04:14:09Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\n2010/7/31 Nguyễn Thái Ngọc Duy :\n> With this, I can clone Documentaion/ from git.git, update and push.\n\nVery nice!\n\n> Space consumption is 24MB (58MB for full\n> repo).  Not really impressive, but if one truely cares about disk\n> space, he/she should also use shallow clone.\n\nCan they be combined to create the fabled narrow checkout?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"146903","messageId":"AANLkTimEOsNvCYyLmbLbMXEoJL-EPQbjE+iD5y7prLAt@mail.gmail.com","threadId":"24596","inReplyTo":"AANLkTinMBAikByLfOi5sXbcRfCsYtYqMCNSTykRofQyo@mail.gmail.com","subject":"Re: [PATCH 00/16] Subtree clone proof of concept","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-08-01T06:58:37Z","receivedAt":"2010-08-01T06:58:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/8/1 Sverre Rabbelier :\n>> Space consumption is 24MB (58MB for full\n>> repo).  Not really impressive, but if one truely cares about disk\n>> space, he/she should also use shallow clone.\n>\n> Can they be combined to create the fabled narrow checkout?\n\nYes. For the record, --subtree=Documentation/ with --depth=1 made a pack of 5MB.\n-- \nDuy\n"},{"id":"146911","messageId":"AANLkTinXhfSzVtcjHVzpdV_GUK+mbpVEezGY2aDXRk1-@mail.gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 02/16] list-objects: limit traversing within the given subtree if core.subtree is set","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-01T11:30:25Z","receivedAt":"2010-08-01T11:30:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"> +       int subtree_len;\n\nShouldn't that be size_t? strlen returns size_t, and strncmp expects\nsize_t, not int.\n"},{"id":"146914","messageId":"AANLkTikr9ZE9s3GG9JocLDvwDQA2FT4_Gh65=dveB8KW@mail.gmail.com","threadId":"24596","inReplyTo":"AANLkTimEOsNvCYyLmbLbMXEoJL-EPQbjE+iD5y7prLAt@mail.gmail.com","subject":"Re: [PATCH 00/16] Subtree clone proof of concept","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-08-01T20:05:01Z","receivedAt":"2010-08-01T20:05:01Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Sun, Aug 1, 2010 at 01:58, Nguyen Thai Ngoc Duy wrote:\n> 2010/8/1 Sverre Rabbelier :\n>> Can they be combined to create the fabled narrow checkout?\n>\n> Yes. For the record, --subtree=Documentation/ with --depth=1 made a pack of 5MB.\n\nI hope everybody is paying attention to these patches then! :)\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"146918","messageId":"AANLkTimSY4kv_i0HnBMCAwODkeFO3sV0tdxopcQwOfK8@mail.gmail.com","threadId":"24596","inReplyTo":"AANLkTinXhfSzVtcjHVzpdV_GUK+mbpVEezGY2aDXRk1-@mail.gmail.com","subject":"Re: [PATCH 02/16] list-objects: limit traversing within the given subtree if core.subtree is set","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-08-01T23:11:33Z","receivedAt":"2010-08-01T23:11:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/8/1 Ævar Arnfjörð Bjarmason :\n>> +       int subtree_len;\n>\n> Shouldn't that be size_t? strlen returns size_t, and strncmp expects\n> size_t, not int.\n>\n\nHmm.. yeah. The compiler didn't warn me. Anyway subtree_len should be\nsmall enough (i.e. < PATH_MAX) that type does not really matters.\n-- \nDuy\n"},{"id":"146922","messageId":"AANLkTikZrD+RRMnae0OOPU_keuu97LditFZnekcpkrYZ@mail.gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 02/16] list-objects: limit traversing within the given subtree if core.subtree is set","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-08-02T04:21:17Z","receivedAt":"2010-08-02T04:21:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\n2010/7/31 Nguyễn Thái Ngọc Duy :\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n>  list-objects.c |   23 +++++++++++++++++------\n>  1 files changed, 17 insertions(+), 6 deletions(-)\n>\n> diff --git a/list-objects.c b/list-objects.c\n> index 8953548..1b25b54 100644\n> --- a/list-objects.c\n> +++ b/list-objects.c\n> @@ -61,12 +61,15 @@ static void process_tree(struct rev_info *revs,\n>                         struct tree *tree,\n>                         show_object_fn show,\n>                         struct name_path *path,\n> -                        const char *name)\n> +                        const char *name,\n> +                        const char *subtree)\n>  {\n>        struct object *obj = &tree->object;\n>        struct tree_desc desc;\n>        struct name_entry entry;\n>        struct name_path me;\n> +       const char *slash;\n> +       int subtree_len;\n\nPerhaps slash should be initialized to NULL? Otherwise I think it\nwill be used uninitialized.\n\n>\n>        if (!revs->tree_objects)\n>                return;\n> @@ -82,13 +85,21 @@ static void process_tree(struct rev_info *revs,\n>        me.elem = name;\n>        me.elem_len = strlen(name);\n>\n> +       if (subtree) {\n> +               slash = strchr(subtree, '/');\n> +               subtree_len = slash ? slash - subtree : strlen(subtree);\n> +       }\n> +\n>        init_tree_desc(&desc, tree->buffer, tree->size);\n>\n>        while (tree_entry(&desc, &entry)) {\n> -               if (S_ISDIR(entry.mode))\n> -                       process_tree(revs,\n> -                                    lookup_tree(entry.sha1),\n> -                                    show, &me, entry.path);\n> +               if (S_ISDIR(entry.mode)) {\n> +                       if (!subtree || !strncmp(entry.path, subtree, subtree_len))\n\nOnly one subdirectory allowed? What if someone wants a sparse clone\ncontaining two or more directories? (Actually, that's not so much of\na \"what if\" -- it's exactly what I want in about half my usecases for\nsparse clones.)\n\n> +                               process_tree(revs,\n> +                                            lookup_tree(entry.sha1),\n> +                                            show, &me, entry.path,\n> +                                            slash && slash[1] ? slash+1 : NULL);\n\nIf I read correctly, slash will be used uninitialized here whenever\nsubtree == NULL.\n"},{"id":"146923","messageId":"AANLkTinznVOCA-SKNZOpGVtB9L+Ns2DnDtW5aRwX-Aiy@mail.gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-7-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 06/16] upload-pack: use a separate variable to control whether internal rev-list is used","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-08-02T04:25:00Z","receivedAt":"2010-08-02T04:25:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\n2010/7/31 Nguyễn Thái Ngọc Duy :\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n>  upload-pack.c |    9 +++++----\n>  1 files changed, 5 insertions(+), 4 deletions(-)\n>\n> diff --git a/upload-pack.c b/upload-pack.c\n> index dc464d7..e432e83 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -160,8 +160,9 @@ static void create_pack_file(void)\n>        ssize_t sz;\n>        const char *argv[10];\n>        int arg = 0;\n> +       int internal_rev_list = shallow_nr;\n\n\nI've got the exact same changes in one of my in-progress-patches in my\nsparse-clone branch. That is, other than the variable name, but I\nlike yours better. Needless to say, I agree with this change. :-)\n"},{"id":"146924","messageId":"AANLkTi=jSFa0B9iSNZdEBOjDgtuECkOxOEcO2ujVeUwb@mail.gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-8-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 07/16] upload-pack: support subtree pack","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-08-02T04:27:02Z","receivedAt":"2010-08-02T04:27:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"2010/7/31 Nguyễn Thái Ngọc Duy :\n\n> diff --git a/upload-pack.c b/upload-pack.c\n> index e432e83..9b6710a 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n\n> @@ -505,6 +505,20 @@ static void receive_needs(void)\n>                if (debug_fd)\n>                        write_in_full(debug_fd, line, len);\n>\n> +               if (!prefixcmp(line, \"subtree \")) {\n> +                       int len;\n> +                       char *subtree;\n> +                       if (core_subtree)\n> +                               die(\"sorry, only one subtree supported\");\n\nI'm not sure users would understand this error message; perhaps\nsomething more like \"Fetching/cloning from a subtree-sparse repository\nnot supported\"?\n"},{"id":"146926","messageId":"AANLkTi=pgJGSY2-h2cqf2s7q49Uj0ra8tGCOWd5Do33G@mail.gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-10-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 09/16] subtree: rewrite incoming commits","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-08-02T04:37:02Z","receivedAt":"2010-08-02T04:37:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\n2010/7/31 Nguyễn Thái Ngọc Duy :\n> This adds the main function, subtree_import(), which is intended to be\n> used by \"git clone\".\n>\n> Because subtree packs are not complete. They are barely usable. Git\n> client will cry out missing objects here and there... Theortically,\n> client code could be adapted to only look for objects within\n> subtree. That was painful to try.\n\nIt may have been painful, but personally I think it's still the right\nway to do it. Of course, that's a pretty easy thing for me to say,\nsince you're pretty far ahead of me and I haven't felt your pain yet.\nMaybe I'll change my mind after trying it for a while, but I'm not\nconvinced just yet.\n\n> +/*\n> + * Take sha1 of a tree, rewrite it to only return the prefix and return\n> + * the newsha1.\n> + *\n> + * If if is zero, write to object store. If fd is greater than zero,\n> + * it's a pack file handle.\n\nShould the second word of the second paragraph be 'fd' rather than another 'if'?\n\n\n> +                       if (slash && slash[1]) /* trailing slash does not count */\n> +                               narrow_tree(entry.sha1, newtree_sha1, prefix+subtree_len+1, fd);\n> +                       else\n> +                               memcpy(newtree_sha1, entry.sha1, 20);\n> +\n> +                       strbuf_addf(&buffer, \"%o %.*s%c\", entry.mode, strlen(entry.path), entry.path, '\\0');\n\nMy compiler complains that you didn't typecast the return value from\nstrlen to an int.\n"},{"id":"146927","messageId":"AANLkTim8mzhvm8unh4SKDJu60ZK4n+WWka-9WSH7yHGF@mail.gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-13-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 12/16] subtree: rewriting outgoing commits","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-08-02T04:40:01Z","receivedAt":"2010-08-02T04:40:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"2010/7/31 Nguyễn Thái Ngọc Duy :\n\n> +       init_tree_desc(&desc, tree, size);\n> +       strbuf_init(&buffer, 8192);\n> +       while (tree_entry(&desc, &entry)) {\n> +               strbuf_addf(&buffer, \"%o %.*s%c\", entry.mode, strlen(entry.path), entry.path, '\\0');\n\nAgain, gcc here complains that \"subtree.c:390: warning: field\nprecision should have type ‘int’, but argument 4 has type ‘size_t’\" --\ntypecast the return value of strlen to int?\n"},{"id":"146928","messageId":"AANLkTims5Xq67QXaE54bCTHP-BZSNRVsJnUKM6YJjt2G@mail.gmail.com","threadId":"24596","inReplyTo":"1280593105-22015-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 00/16] Subtree clone proof of concept","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-08-02T05:18:31Z","receivedAt":"2010-08-02T05:18:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\n2010/7/31 Nguyễn Thái Ngọc Duy :\n> Something to play with so we can evaluate which is the best strategy\n> for non-full clone (or whatever you call it).\n\nVery nice, it's awesome you're working on this. I'm of the same\nopinion that Shawn stated earlier, namely that I don't like the route\nof rewriting commits on the fly like this (more on that later), but\nit's really cool to see some ideas being tried and pushed to their\nlimits.\n\n> The idea is the same: pack only enough to access a subtree, rewrite\n> commits at client side, rewrite again when pushing. However I put\n> git-replace into the mix, so at least commit SHA-1 looks as same as from\n> upstream. git-subtree is not needed (although it's still an option)\n>\n> With this, I can clone Documentaion/ from git.git, update and push. I\n\nI tried it out, but I seem to be doing something wrong. I applied\nyour patches to current master, and tried the following -- am I doing\nsomething wrong or omitting any important steps?\n\n$ git --version\ngit version 1.7.2.1.22.g236df\n\n$ git clone file://$(pwd)/git fullclone\nCloning into fullclone...\nwarning: templates not found /home/newren/share/git-core/templates\nremote: Counting objects: 96220, done.\nremote: Compressing objects: 100% (24925/24925), done.\nremote: Total 96220 (delta 70575), reused 95687 (delta 70236)\nReceiving objects: 100% (96220/96220), 18.45 MiB | 11.43 MiB/s, done.\nResolving deltas: 100% (70575/70575), done.\nfatal: unable to read tree 49374ea4780c0db6db7c604697194bc9b148f3dc\n\n$ git clone --subtree=Documentation/ file://$(pwd)/git docclone\nCloning into docclone...\nwarning: templates not found /home/newren/share/git-core/templates\nfatal: The remote end hung up unexpectedly\nfatal: early EOF\nfatal: index-pack failed\n\n\n> haven't tested it further. Space consumption is 24MB (58MB for full\n> repo).  Not really impressive, but if one truely cares about disk\n> space, he/she should also use shallow clone.\n\n58 MB for full repo? What are you counting? For me, I get 25M:\n\n$ git clone git://git.kernel.org/pub/scm/git/git.git\n$ ls -lh git/.git/objects/pack/*.pack\n-r--r--r--. 1 newren newren 25M 2010-08-01 18:05\ngit/.git/objects/pack/pack-d41d36a8f0f34d5bc647b3c83c5d6b64fbc059c8.pack\n\nAre you counting the full checkout too or something? If so, that\nvaries very wildly between systems, making it hard to compare numbers.\n (For me, 'du -hs git/' returns 44 MB.) I'd like to be able to\nduplicate your numbers and investigate further. It seems to me that\nwe ought to be able to get that lower.\n\n> Performance is impacted, due to bulk commit replacement. There is a\n> split second delay for every command. It's the price of replacing 24k\n> commits every time. I think the delay could be improved a little bit\n> (caching or mmap..)\n>\n> Rewriting commits at clone takes time too. Doing individual object\n> writing takes lots of space and time. I put all new objects directly\n> to a pack now. Rewriting time now becomes quite acceptable (a few\n> seconds). Although deep subtree/repo may take longer. Rewriting on\n> demand can be considered in such cases.\n>\n> Repo-care commands like fsck, repack, gc are left out for now.\n>\n> Finally, it's more of a hack just to see how far I can go. It will\n> break things.\n\nI think it's a pretty nifty hack. It's fun to see. :-) However, I\ndo have a number of reservations about the general strategy: As\nmentioned earlier, I'm not sure I like the on-the-fly commit\nrewriting, as mentioned by Shawn in your previous\nsubtree-for-upload-pack patch series. You did take care of the\n\"referring to commit-sha1\" issue he brought up by using the replace\nmechanism, but I'm still not sure I'm comfortable with it. The\nperformance implications also worry me (a lot of the reason for sparse\nclones was to improve performance, at least from my view), as does the\nfact that it only works on exactly one subtree (at least your current\nimplementation; most of my usecases involve multiple sibling\nsubdirectories that I'd like to get), as does the fact that it\n(currently) only handles trees and does not handle files (ruling out\nthe translator usecase I'd like to see covered, e.g. cloning just\npo/de.po and its history without all sibling files).\n\nAlso, I couldn't tell if your implementation downloaded full commit\ninformation for commits that didn't touch any of the files under the\nrelevant subtree. I think it does, but couldn't tell for sure (I\nwanted to use a clone and dig into it to find out, but ran into the\nproblems I mentioned above). If so, that also worries me a bit -- see\nhttp://article.gmane.org/gmane.comp.version-control.git/152343.\n\nYour implementation also suffers from the same limitations as current\nshallow clones. For example, you can't clone or fetch from a subtree\nclone. That limits collaboration between people needing to work on\nthe same subset of history, and was a limitation I was hoping to see\nfixed, rather than propagated to more features.\n\nI hope I'm not coming across as too critical. I'm really excited to\nsee work in this area. Hopefully I can get more time to pursue my\nroute a bit further; currently I don't have too much more than a\ndetailed idea write-up (heavily revised since the previous thread --\nthanks for the feedback, btw). Or maybe you just know how to address\nall my concerns and you beat me to the punch. That'd be awesome.\n\n\nElijah\n"},{"id":"146930","messageId":"AANLkTi=B0fDy6u2zL_SV-HnYAzfOaAvWcFn_5deL_ZQq@mail.gmail.com","threadId":"24596","inReplyTo":"AANLkTikZrD+RRMnae0OOPU_keuu97LditFZnekcpkrYZ@mail.gmail.com","subject":"Re: [PATCH 02/16] list-objects: limit traversing within the given subtree if core.subtree is set","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-08-02T06:51:53Z","receivedAt":"2010-08-02T06:51:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/8/2 Elijah Newren :\n>>\n>>        if (!revs->tree_objects)\n>>                return;\n>> @@ -82,13 +85,21 @@ static void process_tree(struct rev_info *revs,\n>>        me.elem = name;\n>>        me.elem_len = strlen(name);\n>>\n>> +       if (subtree) {\n>> +               slash = strchr(subtree, '/');\n>> +               subtree_len = slash ? slash - subtree : strlen(subtree);\n>> +       }\n>> +\n>>        init_tree_desc(&desc, tree->buffer, tree->size);\n>>\n>>        while (tree_entry(&desc, &entry)) {\n>> -               if (S_ISDIR(entry.mode))\n>> -                       process_tree(revs,\n>> -                                    lookup_tree(entry.sha1),\n>> -                                    show, &me, entry.path);\n>> +               if (S_ISDIR(entry.mode)) {\n>> +                       if (!subtree || !strncmp(entry.path, subtree, subtree_len))\n>\n> Only one subdirectory allowed?  What if someone wants a sparse clone\n> containing two or more directories?  (Actually, that's not so much of\n> a \"what if\" -- it's exactly what I want in about half my usecases for\n> sparse clones.)\n\nOne is simpler. So one first, multiple may come later.\n\n>> +                               process_tree(revs,\n>> +                                            lookup_tree(entry.sha1),\n>> +                                            show, &me, entry.path,\n>> +                                            slash && slash[1] ? slash+1 : NULL);\n>\n> If I read correctly, slash will be used uninitialized here whenever\n> subtree == NULL.\n\nYes. Thanks. Will fix.\n-- \nDuy\n"},{"id":"146931","messageId":"AANLkTinL3i8J046b=1AJQqwcHjBE3snFL7fMcApGyjVD@mail.gmail.com","threadId":"24596","inReplyTo":"AANLkTims5Xq67QXaE54bCTHP-BZSNRVsJnUKM6YJjt2G@mail.gmail.com","subject":"Re: [PATCH 00/16] Subtree clone proof of concept","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-08-02T07:10:03Z","receivedAt":"2010-08-02T07:10:03Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/8/2 Elijah Newren :\n>> The idea is the same: pack only enough to access a subtree, rewrite\n>> commits at client side, rewrite again when pushing. However I put\n>> git-replace into the mix, so at least commit SHA-1 looks as same as from\n>> upstream. git-subtree is not needed (although it's still an option)\n>>\n>> With this, I can clone Documentaion/ from git.git, update and push. I\n>\n> I tried it out, but I seem to be doing something wrong.  I applied\n> your patches to current master, and tried the following -- am I doing\n> something wrong or omitting any important steps?\n>\n> $ git --version\n> git version 1.7.2.1.22.g236df\n>\n> $ git clone file://$(pwd)/git fullclone\n> Cloning into fullclone...\n> warning: templates not found /home/newren/share/git-core/templates\n> remote: Counting objects: 96220, done.\n> remote: Compressing objects: 100% (24925/24925), done.\n> remote: Total 96220 (delta 70575), reused 95687 (delta 70236)\n> Receiving objects: 100% (96220/96220), 18.45 MiB | 11.43 MiB/s, done.\n> Resolving deltas: 100% (70575/70575), done.\n> fatal: unable to read tree 49374ea4780c0db6db7c604697194bc9b148f3dc\n\nThis one looks like the unintialized case you pointed out in\nprocess_tree(). No I did not try full clone on my patched git :-P\n\n> $ git clone --subtree=Documentation/ file://$(pwd)/git docclone\n> Cloning into docclone...\n> warning: templates not found /home/newren/share/git-core/templates\n> fatal: The remote end hung up unexpectedly\n> fatal: early EOF\n> fatal: index-pack failed\n\nNot sure. Does file:// use receive-pack/upload-pack? I tested it over\nlocal ssh. Will try again soon.\n\n>> haven't tested it further. Space consumption is 24MB (58MB for full\n>> repo).  Not really impressive, but if one truely cares about disk\n>> space, he/she should also use shallow clone.\n>\n> 58 MB for full repo?  What are you counting?  For me, I get 25M:\n>\n> $ git clone git://git.kernel.org/pub/scm/git/git.git\n> $ ls -lh git/.git/objects/pack/*.pack\n> -r--r--r--. 1 newren newren 25M 2010-08-01 18:05\n> git/.git/objects/pack/pack-d41d36a8f0f34d5bc647b3c83c5d6b64fbc059c8.pack\n>\n> Are you counting the full checkout too or something?  If so, that\n> varies very wildly between systems, making it hard to compare numbers.\n>  (For me, 'du -hs git/' returns 44 MB.)  I'd like to be able to\n> duplicate your numbers and investigate further.  It seems to me that\n> we ought to be able to get that lower.\n\nIt's my git.git, probably has more topic branches plus junk stuff. If\nyou are only interested in numbers, playing with git pack-objects is\nenough. You need changes in list-objects.c and builtin/pack-objects.c,\nthen you can\n\ngit pack-objects --stdout --subtree=foo/ > temp.pack\n\nand examine it with verify-pack.\n\n>> Finally, it's more of a hack just to see how far I can go. It will\n>> break things.\n>\n> I think it's a pretty nifty hack.  It's fun to see.  :-)  However, I\n> do have a number of reservations about the general strategy:  As\n> mentioned earlier, I'm not sure I like the on-the-fly commit\n> rewriting, as mentioned by Shawn in your previous\n> subtree-for-upload-pack patch series.  You did take care of the\n> \"referring to commit-sha1\" issue he brought up by using the replace\n> mechanism, but I'm still not sure I'm comfortable with it.  The\n> performance implications also worry me (a lot of the reason for sparse\n> clones was to improve performance, at least from my view), as does the\n> fact that it only works on exactly one subtree (at least your current\n> implementation; most of my usecases involve multiple sibling\n> subdirectories that I'd like to get), as does the fact that it\n> (currently) only handles trees and does not handle files (ruling out\n> the translator usecase I'd like to see covered, e.g. cloning just\n> po/de.po and its history without all sibling files).\n\nAnd it's also fun to try. I'd like to try it on larger repos but I\nhave quite limited network until October.\n\n> Also, I couldn't tell if your implementation downloaded full commit\n> information for commits that didn't touch any of the files under the\n> relevant subtree.  I think it does, but couldn't tell for sure (I\n> wanted to use a clone and dig into it to find out, but ran into the\n> problems I mentioned above).  If so, that also worries me a bit -- see\n> http://article.gmane.org/gmane.comp.version-control.git/152343.\n\nIt does. Yes, that's also something to think of.\n\n> Your implementation also suffers from the same limitations as current\n> shallow clones.  For example, you can't clone or fetch from a subtree\n> clone.  That limits collaboration between people needing to work on\n> the same subset of history, and was a limitation I was hoping to see\n> fixed, rather than propagated to more features.\n\nI agree. Being able to fetch from an incomplete repo is very nice.\nThough I admit I don't know how to do it. I think sparse clone would\nsuffer the same, wouldn't it?\n\n> I hope I'm not coming across as too critical.  I'm really excited to\n> see work in this area.  Hopefully I can get more time to pursue my\n> route a bit further; currently I don't have too much more than a\n> detailed idea write-up (heavily revised since the previous thread --\n> thanks for the feedback, btw).  Or maybe you just know how to address\n> all my concerns and you beat me to the punch.  That'd be awesome.\n\nLook forward to see sparse clone realized. Although I think that would\nbe painful :-)\n-- \nDuy\n"},{"id":"146980","messageId":"7v8w4olrc0.fsf@alter.siamese.dyndns.org","threadId":"24596","inReplyTo":"1280593105-22015-6-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 05/16] Hook up replace-object to allow bulk commit replacement","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-08-02T19:58:23Z","receivedAt":"2010-08-02T19:58:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I really do not like the use of \"replace\" for the purpose of narrow\nclones. While \"replace\" is about fixing a mistake by tweaking trees, a\ndesire to have a narrow clone at this moment is _not_ a mistake. You may\nwant to have wider or full clone of the project tomorrow. You may want to\npush the result of committing on top of such a narrowed clone back to a\nfull repository. My gut feeling is that that use of \"replace\" to stub out\nthe objects that you do not currently have would make it a nightmare when\nyou would want to widen (especially to widen over the wire while pushing\ninto a full repository on the other end), although I haven't looked at all\nthe patches in the series.\n\nCan you back up a bit and give us a high-level overview of how various\noperations in a narrowed clone should work, and how you achieve that\ndesign goal?\n\nLet's take an example of starting from git.git and narrow-clone only its\nDocumentation/ (as you seem to have used as a guinea-pig) subdirectory.\nFor the sake of simplicity, let's say the upstream project has only one\ncommit.\n\nOne plausible approach would be to have the commit, its top level tree\nobject, its Documentation/ tree object and all the blobs below that level,\nwhile other blobs and trees that are reachable from the top level tree\nobject are left missing, but somehow are marked so that fsck would think\nthey are OK to be missing. Your worktree would obviously be narrowed to\nthe same Documentation/ area, and unlike the narrow checkout codepath, you\ndo not widen on demand (unless you automatically fetch missing parts of\nthe tree, which I do not think you should do by default to help people who\nwork while at 30,000ft). Instead, any operation that tries to modify\noutside the \"subtree\" area should fail.\n\nWhen you build a commit that represents a Documentation patch on top of\nsuch a narrowed clone, because you have a full tree of Documentation/\narea, you can come up with the updated tree object for that part of the\nproject. If \"subtree\" mode (aka narrowed clone) rejects operation outside\nthe cloned area, your commit is guaranteed to touch only Documentation/\narea and nothing outside. You therefore should be able to compute the\ntree object for the whole repository (i.e. all the other entries in the\ntop level tree object should be the same as those from HEAD).\n\nBecause the index is a flat structure, you would need to fudge the entries\nthat are missing-but-OK in there somehow, _and_ you would need to be able\nto recompute the tree after updating Documentation/ area. E.g. you may\nknow ppc/ is tree db31c066 but may not know that it has three blobs\nunderneath it nor what their object names are, so your index operating in\nthis mode would need to record (ppc -> db31c066) mapping in order to be\nable to recreate the tree object out of it.\n\nUsing cache-tree data structure might help in doing this. It so far has\nbeen an optimization (i.e. when it says it has an up-to-date information,\nit does, but if it doesn't you can always recompute what is needed from\nthe flat index entries), but I would imagine that you can add an \"out of\ncloned area\" bit to cache-tree entries, and mark a subtree that represents\nmissing parts (e.g. 'ppc/') as such---anything that tries to invalidate\nsuch a cache-tree entry would be an error anyway, and when you need to\nwrite the index out as a tree, such cache-tree entries that record the\ntrees outside your cloned area can be reused, no?\n"},{"id":"147011","messageId":"AANLkTimMxtVGYnVhMOdnZ4oWQa1BsmUC2gUpt0ZcDspE@mail.gmail.com","threadId":"24596","inReplyTo":"7v8w4olrc0.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 05/16] Hook up replace-object to allow bulk commit replacement","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-08-02T22:42:06Z","receivedAt":"2010-08-02T22:42:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/8/3 Junio C Hamano :\n> I really do not like the use of \"replace\" for the purpose of narrow\n> clones.  While \"replace\" is about fixing a mistake by tweaking trees, a\n> desire to have a narrow clone at this moment is _not_ a mistake.  You may\n> want to have wider or full clone of the project tomorrow.  You may want to\n> push the result of committing on top of such a narrowed clone back to a\n> full repository.  My gut feeling is that that use of \"replace\" to stub out\n> the objects that you do not currently have would make it a nightmare when\n> you would want to widen (especially to widen over the wire while pushing\n> into a full repository on the other end), although I haven't looked at all\n> the patches in the series.\n\nIndeed. My intention was \"hey this repo is too big, I only need some\npieces of it. Let me grab something and do my work. (Then throw away\nthe cloned repo)\". It's best used together with shallow clone to give\nlow download/disk space, and a minimum tree to fix something quick.\n\nI'm not really sure if such repos are sustainable in long run. And no\nI did not want to widen/narrow the tree (as it was to be throw away\ntree). Now thinking of widening. The way I do narrow clone is quite\nsimilar with shallow clone. I hope the way shallow clone is deepen can\nbe applied to widening clone.\n\n> Can you back up a bit and give us a high-level overview of how various\n> operations in a narrowed clone should work, and how you achieve that\n> design goal?\n\nOperations work as normal (as the incomplete clone is augmented to\nbecome \"normal\"). In order to make it looks normal, every time a new\ncommit comes in (either from another repository, or user creates a new\none), the commit needs to be processed/replaced, so that the repo\nlooks normal from git perspective.\n\n> Let's take an example of starting from git.git and narrow-clone only its\n> Documentation/ (as you seem to have used as a guinea-pig) subdirectory.\n> For the sake of simplicity, let's say the upstream project has only one\n> commit.\n>\n> One plausible approach would be to have the commit, its top level tree\n> object, its Documentation/ tree object and all the blobs below that level,\n> while other blobs and trees that are reachable from the top level tree\n> object are left missing, but somehow are marked so that fsck would think\n> they are OK to be missing.  Your worktree would obviously be narrowed to\n> the same Documentation/ area, and unlike the narrow checkout codepath, you\n> do not widen on demand (unless you automatically fetch missing parts of\n> the tree, which I do not think you should do by default to help people who\n> work while at 30,000ft).  Instead, any operation that tries to modify\n> outside the \"subtree\" area should fail.\n\nChanges outside the subtree area are dropped on the floor now, not\nfail. But yes, it should fail.\n\n> When you build a commit that represents a Documentation patch on top of\n> such a narrowed clone, because you have a full tree of Documentation/\n> area, you can come up with the updated tree object for that part of the\n> project.  If \"subtree\" mode (aka narrowed clone) rejects operation outside\n> the cloned area, your commit is guaranteed to touch only Documentation/\n> area and nothing outside.  You therefore should be able to compute the\n> tree object for the whole repository (i.e. all the other entries in the\n> top level tree object should be the same as those from HEAD).\n\nCorrect. Except..\n\n> Because the index is a flat structure, you would need to fudge the entries\n> that are missing-but-OK in there somehow, _and_ you would need to be able\n> to recompute the tree after updating Documentation/ area.  E.g. you may\n> know ppc/ is tree db31c066 but may not know that it has three blobs\n> underneath it nor what their object names are, so your index operating in\n> this mode would need to record (ppc -> db31c066) mapping in order to be\n> able to recreate the tree object out of it.\n\nThis is where git-replace comes in. I do not want to deal with full\nflat index. Giving pointers to missing objects may make git commands\nnervous. I rewrite the commit so that now it only has Documentation/\nand nothing else (which I have all needed objects). The index is\nnarrowed too. Because the index (even narrowed) is complete (i.e. all\nentries reachable), most operations should work.\n\nThen, to hide the helper commit from user, I replace the original\n(full) commit with this new commit. So from outside git sees SHA-1 of\nthe original commit, but its content is from the helper one. These\nhelper commits guarantee git won't reach out for missing objects.\n\nIt's a trade off. Doing full index requires much more effort into git.\nUsing \"git-subtree split\", while free git developers to do other\nthings, might be inconvenient for users (without server support, full\nrepo must be downloaded, replaced SHA-1 from git-subtree cannot be\nused to communicate with coworkers..)\n\n> Using cache-tree data structure might help in doing this.  It so far has\n> been an optimization (i.e. when it says it has an up-to-date information,\n> it does, but if it doesn't you can always recompute what is needed from\n> the flat index entries), but I would imagine that you can add an \"out of\n> cloned area\" bit to cache-tree entries, and mark a subtree that represents\n> missing parts (e.g. 'ppc/') as such---anything that tries to invalidate\n> such a cache-tree entry would be an error anyway, and when you need to\n> write the index out as a tree, such cache-tree entries that record the\n> trees outside your cloned area can be reused, no?\n\nThat's just a part of the story. Repository integrity is a\nprerequisite in git from the beginning. git-merge operates directly on\ntrees so cache-tree won't help much. git-commit does sha1 existence\ncheck on every sha1 before commit, so it needs to be\nnarrow-clone-aware too. That made me wonder if has_sha1_file was used\nelsewhere. Then \"git grep has_sha1_file\" scared me off and I'm back\naway.\n-- \nDuy\n"},{"id":"147015","messageId":"AANLkTi=309R5e9TOGSRrtqe5u_5p5U2Xk7oHzez5+oOp@mail.gmail.com","threadId":"24596","inReplyTo":"AANLkTims5Xq67QXaE54bCTHP-BZSNRVsJnUKM6YJjt2G@mail.gmail.com","subject":"Re: [PATCH 00/16] Subtree clone proof of concept","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-08-02T22:55:51Z","receivedAt":"2010-08-02T22:55:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/8/2 Elijah Newren :\n>> haven't tested it further. Space consumption is 24MB (58MB for full\n>> repo).  Not really impressive, but if one truely cares about disk\n>> space, he/she should also use shallow clone.\n>\n> 58 MB for full repo?  What are you counting?  For me, I get 25M:\n\nMy number 24MB was incorrect because process_tree() leaked too many\nblobs. It should have been 16MB. Anyway I have updated my series and\nput it here (to spam git mailing less)\n\nhttp://repo.or.cz/w/git/pclouds.git/shortlog/refs/heads/subtree\n(caveat: constantly rebased tree)\n\nif you still want to play with it. For number lovers, fetching only\nDocumentation from linux-2.6.git took 94MB (full repo 366MB). Yeah\nDocumentation was an easy target.\n-- \nDuy\n"}]}