{"thread":{"id":"42947","subject":"[PATCH/RFC 0/7] Add possibility to clone specific subdirectories","startedAt":"2016-07-28T16:02:49Z","lastAt":"2016-07-29T15:52:27Z","messageCount":12,"participants":["Robin Ruede","Duy Nguyen","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"292412","messageId":"20160728160226.24018-1-r.ruede@gmail.com","threadId":"42947","inReplyTo":null,"subject":"[PATCH/RFC 0/7] Add possibility to clone specific subdirectories","fromName":"Robin Ruede","fromEmail":"r.ruede@gmail.com","sentAt":"2016-07-28T16:02:19Z","receivedAt":"2016-07-28T16:02:49Z","isPatch":true,"sender":{"key":"r.ruede@gmail.com","avatar":null},"body":"This patch series adds a `--sparse-prefix=` option to multiple commands,\nallowing fetching repository contents from only a subdirectory of a remote.\n\nThis works along with sparse-checkout, and is especially useful for repositories\nwhere a subdirectory has meaning when standing alone.\n\n* Motivation (example use cases)\n\n1. Git repositories used for managing large/binary files\n  My university has a repository containing lecture slides etc.\n  as pdfs, with a subdirectory for each lecture. The bandwith for getting the\n  whole repository (even with --depth=1) is 4GiB with significant processing\n  time, getting the complete history of a single lecture uses 25MiB and\n  completes instantly.\n2. package-manager-like repositories. Examples:\n  a) Arch Linux package build files repository [1]\n  b) Rust crates.io packages [2]\n  c) TypeScript type definitions [3]\n3. Excluding a specific directory containing e.g. large binary assets\n  Not currently possible with this patch set, but could be added\n  (see problem 2 below).\n4. Getting the history of a single file\n5. Other uses\n  As a non kernel developer, I wanted to quickly search through\n  the code of only the btrfs filesystem using the git tools, but I do not have\n  a local clone of the complete repository. Using `--depth=100` in combination\n  with `--sparse-prefix=/fs/btrfs` allows me to have little bandwidth usage\n  while still retaining some history.\n6. This is trivial in SVN, and searching on the internet, there are multiple\nquestions about this feature [4-7]\n\n* Examples usage:\n\nGetting the source of the btrfs filesystem with a bit of history:\n\n    $ git clone git@server:linux --depth=100 # shallow, not sparse\n    Receiving objects: 100% (814945/814945), 438.55 MiB | 35.21 MiB/s, done.\n    ...\n    $ git clone git@server:linux --depth=100 --sparse-prefix=/fs/btrfs # sparse and shallow\n    Receiving objects: 100% (503747/503747), 121.45 MiB | 59.75 MiB/s, done.\n    ...\n    $ cd linux && ls ./\n    fs\n    $ ls fs/\n    btrfs\n    $ git log --oneline\n    (repo behaves the same as a full clone with sparse-checkout /fs/btrfs)\n\n\n\n* Open problems:\n\n1. Currently all trees are still included. It would be possible to\ninclude only the trees relevant to the sparse files, which would significantly\nreduce the pack sizes for repositories containing a lot of small files changing\noften. For example package managers using git. Not sure in how many places all\ntrees are presumed present.\n\n2. This patch set implements it as a simple single prefix check command line\noption.\nUsing the exclude_list format (same as in sparse-checkout) might be useful.\nThe server needs to check these patterns for all files in history, so I'm not\nsure if allowing multiple/complex patterns is a good idea.\n\n3. This patch set assumes the sparse-prefix and sparse-checkout does not change.\nrunning clone and fetch both need to have the --sparse-prefix= option, otherwise\ncomplete packs will be fetched. Not sure what the best way to store the\ninformation is, possibly create a new file `.git/sparse` similar to\n`.git/shallow` containing the path(s).\n\n3. Bitmap indices cannot be used, because they do not contain the paths of the\nobjects. So for creating packs, the whole DAG has to be walked.\n\n4. Fsck complains about missing blobs. Should be fairly easy to fix.\n\n5. Tests and documentation is missing.\n\n[1]: https://git.archlinux.org/svntogit/packages.git/\n[2]: https://github.com/rust-lang/crates.io-index\n[3]: https://github.com/DefinitelyTyped/DefinitelyTyped\n[4]: https://stackoverflow.com/questions/600079/is-there-any-way-to-clone-a-git-repositorys-sub-directory-only\n[5]: https://stackoverflow.com/questions/11834386/cloning-only-a-subdirectory-with-git\n[6]: https://askubuntu.com/questions/460885/how-to-clone-git-repository-only-some-directories\n[7]: https://coderwall.com/p/o2fasg/how-to-download-a-project-subdirectory-from-github\n\nRobin Ruede (7):\n  list-objects: add sparse-prefix option to rev_info\n  pack-objects: add sparse-prefix\n  Skip checking integrity of files ignored by sparse\n  fetch-pack: add sparse prefix to smart protocol\n  fetch: add sparse-prefix option\n  clone: add sparse-prefix option\n  remote-curl: add sparse prefix\n\n builtin/clone.c        | 27 ++++++++++++++++++++++++---\n builtin/fetch-pack.c   |  6 ++++++\n builtin/fetch.c        | 19 ++++++++++++++-----\n builtin/pack-objects.c | 11 +++++++++++\n cache-tree.c           |  3 ++-\n connected.c            |  7 ++++++-\n fetch-pack.c           |  4 ++++\n fetch-pack.h           |  1 +\n list-objects.c         |  4 +++-\n remote-curl.c          | 17 ++++++++++++++++-\n revision.c             |  4 ++++\n revision.h             |  1 +\n transport.c            |  4 ++++\n transport.h            |  4 ++++\n upload-pack.c          | 15 ++++++++++++++-\n 15 files changed, 114 insertions(+), 13 deletions(-)\n\n-- \n2.9.1.283.g3ca5b4c.dirty\n\n"},{"id":"292413","messageId":"20160728160226.24018-2-r.ruede@gmail.com","threadId":"42947","inReplyTo":"20160728160226.24018-1-r.ruede@gmail.com","subject":"[PATCH/RFC 1/7] list-objects: add sparse-prefix option to rev_info","fromName":"Robin Ruede","fromEmail":"r.ruede@gmail.com","sentAt":"2016-07-28T16:02:20Z","receivedAt":"2016-07-28T16:03:12Z","isPatch":true,"sender":{"key":"r.ruede@gmail.com","avatar":null},"body":"this skips all blob objects who's path does not begin with the specified\nprefix\n\nSigned-off-by: Robin Ruede <r.ruede@gmail.com>\n---\n list-objects.c | 4 +++-\n revision.c     | 4 ++++\n revision.h     | 1 +\n 3 files changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex f3ca6aa..91a6091 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -28,7 +28,9 @@ static void process_blob(struct rev_info *revs,\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tshow(obj, path->buf, cb_data);\n+\tif (!revs->sparse_prefix || starts_with(path->buf, revs->sparse_prefix + 1)) {\n+\t\tshow(obj, path->buf, cb_data);\n+\t}\n \tstrbuf_setlen(path, pathlen);\n }\n \ndiff --git a/revision.c b/revision.c\nindex edba5b7..a36a796 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1664,6 +1664,10 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t} else if ((argcount = parse_long_opt(\"skip\", argv, &optarg))) {\n \t\trevs->skip_count = atoi(optarg);\n \t\treturn argcount;\n+\t} else if ((argcount = parse_long_opt(\"sparse-prefix\", argv, &optarg))) {\n+\t\tif(optarg[0] != '/') return error(N_(\"sparse prefix must start with /\"));\n+\t\trevs->sparse_prefix = optarg;\n+\t\treturn argcount;\n \t} else if ((*arg == '-') && isdigit(arg[1])) {\n \t\t/* accept -<digit>, like traditional \"head\" */\n \t\tif (strtol_i(arg + 1, 10, &revs->max_count) < 0 ||\ndiff --git a/revision.h b/revision.h\nindex 9fac1a6..2c7c5f2 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -113,6 +113,7 @@ struct rev_info {\n \t\t\tancestry_path:1,\n \t\t\tfirst_parent_only:1,\n \t\t\tline_level_traverse:1;\n+\tconst char *sparse_prefix;\n \n \t/* Diff flags */\n \tunsigned int\tdiff:1,\n-- \n2.9.1.283.g3ca5b4c.dirty\n\n"},{"id":"292414","messageId":"20160728160226.24018-3-r.ruede@gmail.com","threadId":"42947","inReplyTo":"20160728160226.24018-1-r.ruede@gmail.com","subject":"[PATCH/RFC 2/7] pack-objects: add sparse-prefix","fromName":"Robin Ruede","fromEmail":"r.ruede@gmail.com","sentAt":"2016-07-28T16:02:21Z","receivedAt":"2016-07-28T16:03:16Z","isPatch":true,"sender":{"key":"r.ruede@gmail.com","avatar":null},"body":"this allows creating packs that contain only the blobs relevant to a\nspecific subdirectory, e.g.\n\n    echo HEAD | git pack-objects --revs --sparse-prefix=/contrib/\n\nwill create a pack containing the complete history of HEAD, including\nall commits, all trees, and the files in the contrib directory.\n\nSigned-off-by: Robin Ruede <r.ruede@gmail.com>\n---\n builtin/pack-objects.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a2f8cfd..0674f57 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2621,6 +2621,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tstruct argv_array rp = ARGV_ARRAY_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n+\tconst char *sparse_prefix = NULL;\n+\n \tstruct option pack_objects_options[] = {\n \t\tOPT_SET_INT('q', \"quiet\", &progress,\n \t\t\t    N_(\"do not show progress meter\"), 0),\n@@ -2685,6 +2687,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n+\t\tOPT_STRING(0, \"sparse-prefix\", &sparse_prefix, N_(\"path\"),\n+\t\t  N_(\"only include blobs relevant for sparse checkout (implies --revs)\")),\n \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\n \t\t\t N_(\"ignore packs that have companion .keep file\")),\n \t\tOPT_INTEGER(0, \"compression\", &pack_compression_level,\n@@ -2725,6 +2729,13 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t} else\n \t\targv_array_push(&rp, \"--objects\");\n \n+\tif (sparse_prefix) {\n+\t\tuse_internal_rev_list = 1;\n+\t\t// with bitmaps the path of the blobs is not known\n+\t\tuse_bitmap_index = 0;\n+\t\targv_array_push(&rp, \"--sparse-prefix\");\n+\t\targv_array_push(&rp, sparse_prefix);\n+\t}\n \tif (rev_list_all) {\n \t\tuse_internal_rev_list = 1;\n \t\targv_array_push(&rp, \"--all\");\n-- \n2.9.1.283.g3ca5b4c.dirty\n\n"},{"id":"292415","messageId":"20160728160226.24018-4-r.ruede@gmail.com","threadId":"42947","inReplyTo":"20160728160226.24018-1-r.ruede@gmail.com","subject":"[PATCH/RFC 3/7] Skip checking integrity of files ignored by sparse","fromName":"Robin Ruede","fromEmail":"r.ruede@gmail.com","sentAt":"2016-07-28T16:02:22Z","receivedAt":"2016-07-28T16:03:19Z","isPatch":true,"sender":{"key":"r.ruede@gmail.com","avatar":null},"body":"(this might be wrong, not sure if this is the right place)\n\nSigned-off-by: Robin Ruede <r.ruede@gmail.com>\n---\n cache-tree.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex f28b1f4..ab01ae5 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -354,7 +354,8 @@ static int update_one(struct cache_tree *it,\n \t\t\tentlen = pathlen - baselen;\n \t\t\ti++;\n \t\t}\n-\t\tif (mode != S_IFGITLINK && !missing_ok && !has_sha1_file(sha1)) {\n+\t\tif (!ce_skip_worktree(ce) && mode != S_IFGITLINK\n+\t\t\t\t&& !missing_ok && !has_sha1_file(sha1)) {\n \t\t\tstrbuf_release(&buffer);\n \t\t\tif (expected_missing)\n \t\t\t\treturn -1;\n-- \n2.9.1.283.g3ca5b4c.dirty\n\n"},{"id":"292416","messageId":"20160728160226.24018-6-r.ruede@gmail.com","threadId":"42947","inReplyTo":"20160728160226.24018-1-r.ruede@gmail.com","subject":"[PATCH/RFC 5/7] fetch: add sparse-prefix option","fromName":"Robin Ruede","fromEmail":"r.ruede@gmail.com","sentAt":"2016-07-28T16:02:24Z","receivedAt":"2016-07-28T16:03:24Z","isPatch":true,"sender":{"key":"r.ruede@gmail.com","avatar":null},"body":"also pass sparse-prefix option from fetch to rev-list while checking\nconnectivity\n\nSigned-off-by: Robin Ruede <r.ruede@gmail.com>\n---\n builtin/fetch.c | 19 ++++++++++++++-----\n connected.c     |  7 ++++++-\n transport.c     |  4 ++++\n transport.h     |  4 ++++\n 4 files changed, 28 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex acd0cf1..b48537f 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -41,6 +41,7 @@ static int tags = TAGS_DEFAULT, unshallow, update_shallow;\n static int max_children = -1;\n static enum transport_family family;\n static const char *depth;\n+static const char *sparse_prefix;\n static const char *upload_pack;\n static struct strbuf default_rla = STRBUF_INIT;\n static struct transport *gtransport;\n@@ -117,6 +118,8 @@ static struct option builtin_fetch_options[] = {\n \tOPT_BOOL(0, \"progress\", &progress, N_(\"force progress reporting\")),\n \tOPT_STRING(0, \"depth\", &depth, N_(\"depth\"),\n \t\t   N_(\"deepen history of shallow clone\")),\n+\tOPT_STRING(0, \"sparse-prefix\", &sparse_prefix, N_(\"path-prefix\"),\n+\t\t   N_(\"only fetch blobs for the specified path-prefix\")),\n \t{ OPTION_SET_INT, 0, \"unshallow\", &unshallow, NULL,\n \t\t   N_(\"convert to a complete repository\"),\n \t\t   PARSE_OPT_NONEG | PARSE_OPT_NOARG, NULL, 1 },\n@@ -706,9 +709,11 @@ static int iterate_ref_map(void *cb_data, unsigned char sha1[20])\n \treturn 0;\n }\n \n-static int store_updated_refs(const char *raw_url, const char *remote_name,\n+static int store_updated_refs(struct transport *transport,\n \t\tstruct ref *ref_map)\n {\n+\tconst char *raw_url = transport->url;\n+\tconst char *remote_name = transport->remote->name;\n \tFILE *fp;\n \tstruct commit *commit;\n \tint url_len, i, rc = 0;\n@@ -729,7 +734,8 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\turl = xstrdup(\"foreign\");\n \n \trm = ref_map;\n-\tif (check_everything_connected(iterate_ref_map, 0, &rm)) {\n+\tif (check_everything_connected_with_transport(iterate_ref_map, 0, &rm,\n+\t\t\t\ttransport)) {\n \t\trc = error(_(\"%s did not send all necessary objects\\n\"), url);\n \t\tgoto abort;\n \t}\n@@ -885,9 +891,7 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map)\n \tif (ret)\n \t\tret = transport_fetch_refs(transport, ref_map);\n \tif (!ret)\n-\t\tret |= store_updated_refs(transport->url,\n-\t\t\t\ttransport->remote->name,\n-\t\t\t\tref_map);\n+\t\tret |= store_updated_refs(transport, ref_map);\n \ttransport_unlock_pack(transport);\n \treturn ret;\n }\n@@ -993,6 +997,11 @@ static struct transport *prepare_transport(struct remote *remote)\n \t\tset_option(transport, TRANS_OPT_KEEP, \"yes\");\n \tif (depth)\n \t\tset_option(transport, TRANS_OPT_DEPTH, depth);\n+\tif (sparse_prefix) {\n+\t\tif(sparse_prefix[0] != '/')\n+\t\t\tdie(N_(\"sparse prefix must start with /\"));\n+\t\tset_option(transport, TRANS_OPT_SPARSE_PREFIX, sparse_prefix);\n+\t}\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n \treturn transport;\ndiff --git a/connected.c b/connected.c\nindex bf1b12e..1534c5c 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -26,7 +26,7 @@ static int check_everything_connected_real(sha1_iterate_fn fn,\n \t\t\t\t\t   const char *shallow_file)\n {\n \tstruct child_process rev_list = CHILD_PROCESS_INIT;\n-\tconst char *argv[9];\n+\tconst char *argv[11];\n \tchar commit[41];\n \tunsigned char sha1[20];\n \tint err = 0, ac = 0;\n@@ -56,6 +56,11 @@ static int check_everything_connected_real(sha1_iterate_fn fn,\n \targv[ac++] = \"--stdin\";\n \targv[ac++] = \"--not\";\n \targv[ac++] = \"--all\";\n+\tif(transport && transport->smart_options && \n+\t\t\ttransport->smart_options->sparse_prefix) {\n+\t\targv[ac++] = \"--sparse-prefix\";\n+\t\targv[ac++] = transport->smart_options->sparse_prefix;\n+\t}\n \tif (quiet)\n \t\targv[ac++] = \"--quiet\";\n \targv[ac] = NULL;\ndiff --git a/transport.c b/transport.c\nindex b233e3e..ce7e2e1 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -141,6 +141,9 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_UPDATE_SHALLOW)) {\n \t\topts->update_shallow = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_SPARSE_PREFIX)) {\n+\t\topts->sparse_prefix = value;\n+\t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_DEPTH)) {\n \t\tif (!value)\n \t\t\topts->depth = 0;\n@@ -211,6 +214,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.quiet = (transport->verbose < 0);\n \targs.no_progress = !transport->progress;\n \targs.depth = data->options.depth;\n+\targs.sparse_prefix = data->options.sparse_prefix;\n \targs.check_self_contained_and_connected =\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\ndiff --git a/transport.h b/transport.h\nindex c681408..abee186 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -15,6 +15,7 @@ struct git_transport_options {\n \tint depth;\n \tconst char *uploadpack;\n \tconst char *receivepack;\n+\tconst char *sparse_prefix;\n \tstruct push_cas_option *cas;\n };\n \n@@ -179,6 +180,9 @@ int transport_restrict_protocols(void);\n /* Limit the depth of the fetch if not null */\n #define TRANS_OPT_DEPTH \"depth\"\n \n+/* Only fetch blobs whose referenced path begins with this if not null */\n+#define TRANS_OPT_SPARSE_PREFIX \"sparse-prefix\"\n+\n /* Aggressively fetch annotated tags if possible */\n #define TRANS_OPT_FOLLOWTAGS \"followtags\"\n \n-- \n2.9.1.283.g3ca5b4c.dirty\n\n"},{"id":"292417","messageId":"20160728160226.24018-7-r.ruede@gmail.com","threadId":"42947","inReplyTo":"20160728160226.24018-1-r.ruede@gmail.com","subject":"[PATCH/RFC 6/7] clone: add sparse-prefix option","fromName":"Robin Ruede","fromEmail":"r.ruede@gmail.com","sentAt":"2016-07-28T16:02:25Z","receivedAt":"2016-07-28T16:03:27Z","isPatch":true,"sender":{"key":"r.ruede@gmail.com","avatar":null},"body":"For example\n\n    git clone git@remote:repo --sparse-prefix=/contrib/\n\nwould create a repository, cloning only the relevant files for the\ncontrib subdirectory, and also setting the sparse-checkout option, so\nonly those files will be checked out.\n\nSigned-off-by: Robin Ruede <r.ruede@gmail.com>\n---\n builtin/clone.c | 27 ++++++++++++++++++++++++---\n 1 file changed, 24 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 31ea247..dc0d364 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -41,7 +41,7 @@ static const char * const builtin_clone_usage[] = {\n static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;\n static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;\n static int option_shallow_submodules;\n-static char *option_template, *option_depth;\n+static char *option_template, *option_depth, *option_sparse_prefix;\n static char *option_origin = NULL;\n static char *option_branch = NULL;\n static const char *real_git_dir;\n@@ -91,6 +91,8 @@ static struct option builtin_clone_options[] = {\n \t\t   N_(\"path to git-upload-pack on the remote\")),\n \tOPT_STRING(0, \"depth\", &option_depth, N_(\"depth\"),\n \t\t    N_(\"create a shallow clone of that depth\")),\n+\tOPT_STRING(0, \"sparse-prefix\", &option_sparse_prefix, N_(\"path-prefix\"),\n+\t            N_(\"only fetch the blobs for the specified path-prefix\")),\n \tOPT_BOOL(0, \"single-branch\", &option_single_branch,\n \t\t    N_(\"clone only one branch, HEAD or --branch\")),\n \tOPT_BOOL(0, \"shallow-submodules\", &option_shallow_submodules,\n@@ -959,9 +961,22 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \tinit_db(option_template, INIT_DB_QUIET);\n \twrite_config(&option_config);\n-\n+\tif (option_sparse_prefix) {\n+\t\tFILE *f;\n+\t\tconst char *name;\n+\t\tif (option_sparse_prefix[0] != '/')\n+\t\t\tdie(N_(\"sparse prefix must start with /\"));\n+\t\tname = mkpath(\"%s/info/sparse-checkout\", git_dir);\n+\t\tgit_config_set(\"core.sparsecheckout\", \"true\");\n+\t\tsafe_create_leading_directories_const(name);\n+\t\tf = fopen(name, \"w\");\n+\t\tif(f == NULL) die(\"Could not open %s\", name);\n+\t\tfprintf(f, \"%s\\n\", option_sparse_prefix);\n+\t\tfclose(f);\n+\t}\n \tgit_config(git_default_config, NULL);\n \n+\n \tif (option_bare) {\n \t\tif (option_mirror)\n \t\t\tsrc_ref_prefix = \"refs/\";\n@@ -977,6 +992,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tgit_config_set(key.buf, repo);\n \tstrbuf_reset(&key);\n \n+\n \tif (option_reference.nr)\n \t\tsetup_reference();\n \n@@ -995,6 +1011,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (is_local) {\n \t\tif (option_depth)\n \t\t\twarning(_(\"--depth is ignored in local clones; use file:// instead.\"));\n+\t\tif (option_sparse_prefix)\n+\t\t\twarning(_(\"--sparse-prefix is ignored in local clones; use file:// instead.\"));\n \t\tif (!access(mkpath(\"%s/shallow\", path), F_OK)) {\n \t\t\tif (option_local > 0)\n \t\t\t\twarning(_(\"source repository is shallow, ignoring --local\"));\n@@ -1013,6 +1031,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_depth)\n \t\ttransport_set_option(transport, TRANS_OPT_DEPTH,\n \t\t\t\t     option_depth);\n+\tif (option_sparse_prefix)\n+\t\ttransport_set_option(transport, TRANS_OPT_SPARSE_PREFIX,\n+\t\t                     option_sparse_prefix);\n \tif (option_single_branch)\n \t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, \"1\");\n \n@@ -1020,7 +1041,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t     option_upload_pack);\n \n-\tif (transport->smart_options && !option_depth)\n+\tif (transport->smart_options && !option_depth && !option_sparse_prefix)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n \trefs = transport_get_remote_refs(transport);\n-- \n2.9.1.283.g3ca5b4c.dirty\n\n"},{"id":"292418","messageId":"20160728160226.24018-5-r.ruede@gmail.com","threadId":"42947","inReplyTo":"20160728160226.24018-1-r.ruede@gmail.com","subject":"[PATCH/RFC 4/7] fetch-pack: add sparse prefix to smart protocol","fromName":"Robin Ruede","fromEmail":"r.ruede@gmail.com","sentAt":"2016-07-28T16:02:23Z","receivedAt":"2016-07-28T16:03:32Z","isPatch":true,"sender":{"key":"r.ruede@gmail.com","avatar":null},"body":"For example\n\n    git fetch-pack --sparse-prefix=/contrib/ origin HEAD\n\nShould fetch a pack that is generated on the remote by\n\n    echo HEAD | git pack-objects --revs --stdout --sparse-prefix=/contrib/\n\nSigned-off-by: Robin Ruede <r.ruede@gmail.com>\n---\n builtin/fetch-pack.c |  6 ++++++\n fetch-pack.c         |  4 ++++\n fetch-pack.h         |  1 +\n remote-curl.c        |  2 ++\n upload-pack.c        | 15 ++++++++++++++-\n 5 files changed, 27 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex bfd0be4..7f10001 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -64,6 +64,12 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.uploadpack = arg + 14;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (starts_with(arg, \"--sparse-prefix=\")) {\n+\t\t\targs.sparse_prefix = arg + 16;\n+\t\t\tif(args.sparse_prefix[0] != '/')\n+\t\t\t\tdie(N_(\"sparse prefix must start with /\"));\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (starts_with(arg, \"--exec=\")) {\n \t\t\targs.uploadpack = arg + 7;\n \t\t\tcontinue;\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex b501d5c..8571b02 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -326,6 +326,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\treturn 1;\n \t}\n \n+\tif(args->sparse_prefix)\n+\t\tpacket_buf_write(&req_buf, \"sparse-prefix %s\", args->sparse_prefix);\n \tif (is_repository_shallow())\n \t\twrite_shallow_commits(&req_buf, 1, NULL);\n \tif (args->depth > 0)\n@@ -811,6 +813,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \n \tif ((args->depth > 0 || is_repository_shallow()) && !server_supports(\"shallow\"))\n \t\tdie(\"Server does not support shallow clients\");\n+\tif (args->sparse_prefix && !server_supports(\"sparse-prefix\"))\n+\t\tdie(\"Server does not support sparse prefix\");\n \tif (server_supports(\"multi_ack_detailed\")) {\n \t\tif (args->verbose)\n \t\t\tfprintf(stderr, \"Server supports multi_ack_detailed\\n\");\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex bb7fd76..8f36ef4 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -8,6 +8,7 @@ struct sha1_array;\n \n struct fetch_pack_args {\n \tconst char *uploadpack;\n+\tconst char *sparse_prefix;\n \tint unpacklimit;\n \tint depth;\n \tunsigned quiet:1;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 6b83b77..e181e62 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -727,6 +727,8 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n \tALLOC_ARRAY(targets, nr_heads);\n \tif (options.depth)\n \t\tdie(\"dumb http transport does not support --depth\");\n+\tif (options.sparse_prefix)\n+\t\tdie(\"dumb http transport does not support --sparse-prefix\");\n \tfor (i = 0; i < nr_heads; i++)\n \t\ttargets[i] = xstrdup(oid_to_hex(&to_fetch[i]->old_oid));\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex d4cc414..56d8c1a 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -57,6 +57,7 @@ static int use_sideband;\n static int advertise_refs;\n static int stateless_rpc;\n static const char *pack_objects_hook;\n+static char *sparse_prefix;\n \n static void reset_timeout(void)\n {\n@@ -125,6 +126,12 @@ static void create_pack_file(void)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n+\tif (sparse_prefix) {\n+\t\targv_array_push(&pack_objects.args, \"--sparse-prefix\");\n+\t\targv_array_push(&pack_objects.args, sparse_prefix);\n+\t\tfree(sparse_prefix);\n+\t\tsparse_prefix = NULL;\n+\t}\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n@@ -582,6 +589,12 @@ static void receive_needs(void)\n \t\t\t\tdie(\"Invalid deepen: %s\", line);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (starts_with(line, \"sparse-prefix \")) {\n+\t\t\tif(sparse_prefix)\n+\t\t\t\tdie(\"Only single sparse-prefix is allowed\");\n+\t\t\tsparse_prefix = xstrdup(line + 14);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!starts_with(line, \"want \") ||\n \t\t    get_sha1_hex(line+5, sha1_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n@@ -730,7 +743,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow no-progress\"\n-\t\t\" include-tag multi_ack_detailed\";\n+\t\t\" include-tag multi_ack_detailed sparse-prefix\";\n \tconst char *refname_nons = strip_namespace(refname);\n \tstruct object_id peeled;\n \n-- \n2.9.1.283.g3ca5b4c.dirty\n\n"},{"id":"292419","messageId":"20160728160226.24018-8-r.ruede@gmail.com","threadId":"42947","inReplyTo":"20160728160226.24018-1-r.ruede@gmail.com","subject":"[PATCH/RFC 7/7] remote-curl: add sparse prefix","fromName":"Robin Ruede","fromEmail":"r.ruede@gmail.com","sentAt":"2016-07-28T16:02:26Z","receivedAt":"2016-07-28T16:03:34Z","isPatch":true,"sender":{"key":"r.ruede@gmail.com","avatar":null},"body":"Signed-off-by: Robin Ruede <r.ruede@gmail.com>\n---\n remote-curl.c | 15 ++++++++++++++-\n 1 file changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex e181e62..b9f7cf1 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -20,6 +20,7 @@ static struct strbuf url = STRBUF_INIT;\n struct options {\n \tint verbosity;\n \tunsigned long depth;\n+\tconst char *sparse_prefix;\n \tunsigned progress : 1,\n \t\tcheck_self_contained_and_connected : 1,\n \t\tcloning : 1,\n@@ -60,6 +61,10 @@ static int set_option(const char *name, const char *value)\n \t\toptions.depth = v;\n \t\treturn 0;\n \t}\n+\telse if (!strcmp(name, \"sparse-prefix\")) {\n+\t\toptions.sparse_prefix = xstrdup(value);\n+\t\treturn 0;\n+\t}\n \telse if (!strcmp(name, \"followtags\")) {\n \t\tif (!strcmp(value, \"true\"))\n \t\t\toptions.followtags = 1;\n@@ -754,8 +759,9 @@ static int fetch_git(struct discovery *heads,\n \tstruct rpc_state rpc;\n \tstruct strbuf preamble = STRBUF_INIT;\n \tchar *depth_arg = NULL;\n+\tchar *sparse_arg = NULL;\n \tint argc = 0, i, err;\n-\tconst char *argv[17];\n+\tconst char *argv[19];\n \n \targv[argc++] = \"fetch-pack\";\n \targv[argc++] = \"--stateless-rpc\";\n@@ -783,6 +789,12 @@ static int fetch_git(struct discovery *heads,\n \t\tdepth_arg = strbuf_detach(&buf, NULL);\n \t\targv[argc++] = depth_arg;\n \t}\n+\tif (options.sparse_prefix) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"--sparse-prefix=%s\", options.sparse_prefix);\n+\t\tsparse_arg = strbuf_detach(&buf, NULL);\n+\t\targv[argc++] = sparse_arg;\n+\t}\n \targv[argc++] = url.buf;\n \targv[argc++] = NULL;\n \n@@ -807,6 +819,7 @@ static int fetch_git(struct discovery *heads,\n \tstrbuf_release(&rpc.result);\n \tstrbuf_release(&preamble);\n \tfree(depth_arg);\n+\tfree(sparse_arg);\n \treturn err;\n }\n \n-- \n2.9.1.283.g3ca5b4c.dirty\n\n"},{"id":"292428","messageId":"CACsJy8AW3Z+C81F6w7WiZXXvcLRv9PB4=Wjbze6eE_MPgikU8A@mail.gmail.com","threadId":"42947","inReplyTo":"20160728160226.24018-1-r.ruede@gmail.com","subject":"Re: [PATCH/RFC 0/7] Add possibility to clone specific subdirectories","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-28T16:59:16Z","receivedAt":"2016-07-28T16:59:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jul 28, 2016 at 6:02 PM, Robin Ruede <r.ruede@gmail.com> wrote:\n> This patch series adds a `--sparse-prefix=` option to multiple commands,\n> allowing fetching repository contents from only a subdirectory of a remote.\n>\n> This works along with sparse-checkout, and is especially useful for repositories\n> where a subdirectory has meaning when standing alone.\n\nAh.. this is what I call narrow checkout [1] (but gmane is down at the moment)\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/155427\n\n> * Motivation (example use cases)\n>\n> ...\n\nnods nods.. all good stuff\n\n> * Open problems:\n>\n> 1. Currently all trees are still included. It would be possible to\n> include only the trees relevant to the sparse files, which would significantly\n> reduce the pack sizes for repositories containing a lot of small files changing\n> often. For example package managers using git. Not sure in how many places all\n> trees are presumed present.\n\nYou can limit some trees by passing a pathspec to \"git rev-list\" (in\nyour \"list-objects\" patch). All trees completely outside sub/dir will\nbe excluded. Trees leading to it (e.g. root tree and \"sub\") are still\nincluded. Not having all trees open up a new set of problems.. This is\nwhat I did in narrow clone: pass some directories (as pathspec) to\nrev-list on the server side, then deal with lack of trees on client\nside.\n\n> 2. This patch set implements it as a simple single prefix check command line\n> option.\n> Using the exclude_list format (same as in sparse-checkout) might be useful.\n> The server needs to check these patterns for all files in history, so I'm not\n> sure if allowing multiple/complex patterns is a good idea.\n\nI would go with something else than sparse-checkout, which I call\nnarrow checkout: instead of flattening the entire tree in index and\nkeep only files there, we keep trees that we don't have as trees.\nThose trees have the same \"sparse checkout\" attributes, e.g. ignore\nworktree and some of submodules e.g. don't bother checking the\nassociated hash. This approach [2] eliminates changes in cache-tree.c\n(i.e. 3/7).\n\nAnd you would need something like that, when you don't have all the\ntrees (from open problem 1), because you just can't flatten trees when\nyou don't have them.\n\n[2] https://github.com/pclouds/git/commits/lanh/narrow-checkout (I\nthink core functionality is in place, but narrow operation still needs\nmore work)\n\n> 3. This patch set assumes the sparse-prefix and sparse-checkout does not change.\n> running clone and fetch both need to have the --sparse-prefix= option, otherwise\n> complete packs will be fetched. Not sure what the best way to store the\n> information is, possibly create a new file `.git/sparse` similar to\n> `.git/shallow` containing the path(s).\n\nSomething like .git/shallow, yes. It's similar in nature anyway\n(shallow cuts depth, you cut the side)\n\n> 3. Bitmap indices cannot be used, because they do not contain the paths of the\n> objects. So for creating packs, the whole DAG has to be walked.\n\nAnd shallow clones have this same problem. Something to be sorted out :)\n\n> 4. Fsck complains about missing blobs. Should be fairly easy to fix.\n\nNot really. You'll have to associate path information with blobs\nbefore you decide that a blob should exist or not. Sparse patterns are\njust not designed for that (tree walking). If you narrow (heh) down to\njust path prefix not full blown sparse patterns, then it's feasible to\nwalk tree and filter. A subset of pathspec would be good because we\ncan already filter by pathspec, but I would not go full pathspec at\nthe first step.\n\n> 5. Tests and documentation is missing.\n\nPersonally I would go with my narrow clone approach, but the ability\nto selectively exclude some large blobs is still good, I think.\nHowever, another approach to excluding some blobs is the external\nobject database [3]. It gives you what you need with a lot less code\nimpact (but you will not be able to work offline 100% the time like\nwhat you can now with git)\n\n[3] https://public-inbox.org/git/20160613085546.11784-1-chriscool%40tuxfamily.org/\n-- \nDuy\n"},{"id":"292429","messageId":"CACsJy8D0bTE_EvW-wixYDf0f2RFBY3VzLdYnUa-G-9QNydu77Q@mail.gmail.com","threadId":"42947","inReplyTo":"CACsJy8AW3Z+C81F6w7WiZXXvcLRv9PB4=Wjbze6eE_MPgikU8A@mail.gmail.com","subject":"Re: [PATCH/RFC 0/7] Add possibility to clone specific subdirectories","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-28T17:03:12Z","receivedAt":"2016-07-28T17:03:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Corrections..\n\nOn Thu, Jul 28, 2016 at 6:59 PM, Duy Nguyen <pclouds@gmail.com> wrote:\n> Ah.. this is what I call narrow checkout [1] (but gmane is down at the moment)\n\ns/checkout/clone/\n\n> [2] https://github.com/pclouds/git/commits/lanh/narrow-checkout\n\ns,lanh/,,\n-- \nDuy\n"},{"id":"292451","messageId":"xmqqlh0lqz8y.fsf@gitster.mtv.corp.google.com","threadId":"42947","inReplyTo":"CACsJy8AW3Z+C81F6w7WiZXXvcLRv9PB4=Wjbze6eE_MPgikU8A@mail.gmail.com","subject":"Re: [PATCH/RFC 0/7] Add possibility to clone specific subdirectories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-28T20:33:17Z","receivedAt":"2016-07-28T20:33:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n>> 4. Fsck complains about missing blobs. Should be fairly easy to fix.\n>\n> Not really. You'll have to associate path information with blobs\n> before you decide that a blob should exist or not.\n\nAlso the same blob or the tree can exist both inside and outside the\nnarrowed area, as people reorganize their trees all the time.  I am\nnot quite convinced a path-based approach (either yours or Robin's)\nis workable in the longer term.\n"},{"id":"292519","messageId":"CACsJy8AeWmDh+24shCTi1SjYAxnkNJsi=Npq7mYA0_c5ZQ740g@mail.gmail.com","threadId":"42947","inReplyTo":"xmqqlh0lqz8y.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH/RFC 0/7] Add possibility to clone specific subdirectories","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-29T15:51:52Z","receivedAt":"2016-07-29T15:52:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jul 28, 2016 at 10:33 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Duy Nguyen <pclouds@gmail.com> writes:\n>\n>>> 4. Fsck complains about missing blobs. Should be fairly easy to fix.\n>>\n>> Not really. You'll have to associate path information with blobs\n>> before you decide that a blob should exist or not.\n>\n> Also the same blob or the tree can exist both inside and outside the\n> narrowed area, as people reorganize their trees all the time.  I am\n> not quite convinced a path-based approach (either yours or Robin's)\n> is workable in the longer term.\n\nI think it should be ok. What I meant was when we travel the trees to\nfind connected blobs, if a tree points to paths outside the narrowed\narea, we do not add those blobs to our fsck list. Trees inside the\nnarrow area work as usually so those shared blobs are added to fsck\nlist anyway. Object islands are going to be problem (because we can't\nassign paths to them)...\n-- \nDuy\n"}]}