This patch series adds a `--sparse-prefix=` option to multiple commands, allowing fetching repository contents from only a subdirectory of a remote.
This works along with sparse-checkout, and is especially useful for repositories where a subdirectory has meaning when standing alone.
* Motivation (example use cases)
1. Git repositories used for managing large/binary files My university has a repository containing lecture slides etc. as pdfs, with a subdirectory for each lecture. The bandwith for getting the whole repository (even with --depth=1) is 4GiB with significant processing time, getting the complete history of a single lecture uses 25MiB and completes instantly. 2. package-manager-like repositories. Examples: a) Arch Linux package build files repository [1] b) Rust crates.io packages [2] c) TypeScript type definitions [3] 3. Excluding a specific directory containing e.g. large binary assets Not currently possible with this patch set, but could be added (see problem 2 below). 4. Getting the history of a single file 5. Other uses As a non kernel developer, I wanted to quickly search through the code of only the btrfs filesystem using the git tools, but I do not have a local clone of the complete repository. Using `--depth=100` in combination with `--sparse-prefix=/fs/btrfs` allows me to have little bandwidth usage while still retaining some history. 6. This is trivial in SVN, and searching on the internet, there are multiple questions about this feature [4-7]
* Examples usage:
Getting the source of the btrfs filesystem with a bit of history:
$ git clone git@server:linux --depth=100 # shallow, not sparse
Receiving objects: 100% (814945/814945), 438.55 MiB | 35.21 MiB/s, done.
...
$ git clone git@server:linux --depth=100 --sparse-prefix=/fs/btrfs # sparse and shallow
Receiving objects: 100% (503747/503747), 121.45 MiB | 59.75 MiB/s, done.
...
$ cd linux && ls ./
fs
$ ls fs/
btrfs
$ git log --oneline
(repo behaves the same as a full clone with sparse-checkout /fs/btrfs)* Open problems:
1. Currently all trees are still included. It would be possible to include only the trees relevant to the sparse files, which would significantly reduce the pack sizes for repositories containing a lot of small files changing often. For example package managers using git. Not sure in how many places all trees are presumed present.
2. This patch set implements it as a simple single prefix check command line option. Using the exclude_list format (same as in sparse-checkout) might be useful. The server needs to check these patterns for all files in history, so I'm not sure if allowing multiple/complex patterns is a good idea.
3. This patch set assumes the sparse-prefix and sparse-checkout does not change. running clone and fetch both need to have the --sparse-prefix= option, otherwise complete packs will be fetched. Not sure what the best way to store the information is, possibly create a new file `.git/sparse` similar to `.git/shallow` containing the path(s).
3. Bitmap indices cannot be used, because they do not contain the paths of the objects. So for creating packs, the whole DAG has to be walked.
4. Fsck complains about missing blobs. Should be fairly easy to fix.
5. Tests and documentation is missing.
[1]: https://git.archlinux.org/svntogit/packages.git/ [2]: https://github.com/rust-lang/crates.io-index [3]: https://github.com/DefinitelyTyped/DefinitelyTyped [4]: https://stackoverflow.com/questions/600079/is-there-any-way-to-clone-a-git-repositorys-sub-directory-only [5]: https://stackoverflow.com/questions/11834386/cloning-only-a-subdirectory-with-git [6]: https://askubuntu.com/questions/460885/how-to-clone-git-repository-only-some-directories [7]: https://coderwall.com/p/o2fasg/how-to-download-a-project-subdirectory-from-github
Robin Ruede (7): list-objects: add sparse-prefix option to rev_info pack-objects: add sparse-prefix Skip checking integrity of files ignored by sparse fetch-pack: add sparse prefix to smart protocol fetch: add sparse-prefix option clone: add sparse-prefix option remote-curl: add sparse prefix
builtin/clone.c | 27 ++++++++++++++++++++++++--- builtin/fetch-pack.c | 6 ++++++ builtin/fetch.c | 19 ++++++++++++++----- builtin/pack-objects.c | 11 +++++++++++ cache-tree.c | 3 ++- connected.c | 7 ++++++- fetch-pack.c | 4 ++++ fetch-pack.h | 1 + list-objects.c | 4 +++- remote-curl.c | 17 ++++++++++++++++- revision.c | 4 ++++ revision.h | 1 + transport.c | 4 ++++ transport.h | 4 ++++ upload-pack.c | 15 ++++++++++++++- 15 files changed, 114 insertions(+), 13 deletions(-)
-- 2.9.1.283.g3ca5b4c.dirty
