{"thread":{"id":"24979","subject":"[RFC PATCH 00/15] Sparse clones","startedAt":"2010-09-05T00:13:52Z","lastAt":"2010-09-07T03:06:13Z","messageCount":41,"participants":["Elijah Newren","Nguyen Thai Ngoc Duy","Nguyễn Thái Ngọc Duy","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":15},"messages":[{"id":"149902","messageId":"1283645647-1891-1-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":null,"subject":"[RFC PATCH 00/15] Sparse clones","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:13:52Z","receivedAt":"2010-09-05T00:13:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series implements some basics for sparse clones, which I\ndefine as a clone where not all blob, tree, or commit objects are\ndownloaded. The idea is to include sparseness both relative to span\nof files/directories and depth of history, though currently I've only\nput effort into span of paths.\n\nThis patch is built on pu, because it requires\nen/object-list-with-pathspec.\n\nWhat works:\n * all operations on non-sparse clones (full testsuite passes)\n * clone\n * read-tree\n * ls-files\n * cat-file\n * ls-tree\n * checkout\n * diff\n * status\n * log\n * add (except for not giving errors for paths outside the sparse limits)\n * commit\nWhat doesn't work, yet:\n * Probably everything not tested in the new t572*.sh tests :-)\n Notable examples of things missing from t572*.sh tests:\n * fetch\n * push\n * merge\n * rebase\n * thin packs (need to modify pack-objects to only delta against\n objects within the sparse limits)\n * densify command (to make a sparse repository non-sparse)\n * \"missing\" commits (see README file in PATCH1)\n\nCursory comparison with Nguyễn Thái Ngọc Duy's subtree clone (he's probably\nmade progress since his last submission, so this may be outdated):\n * His series supports fetch, mine doesn't (yet).\n * His series supports push, mine doesn't (yet).\n * His series supports merge, mine doesn't (yet).\n * His handling of subtree request over clone/fetch via capabilities\n is probably the right way; I'm pretty sure my adding of sparse\n limits are extra arguments to upload-pack would break backward\n compatibility and be bad.\n * He supports just one selected subtree (though he mentioned he's\n working on extending that); I support arbitrary number of subtrees\n or subfiles.\n * He modifies index format (bumping to header version 4); I don't.\n Perhaps it's necessary for merge handling as I haven't implemented\n that, but at an early glance I don't think it's necessary.\n * While there are some similarities in the low-level details of how\n we've modified the git to avoid missing objects, there are many\n differences as well. I'm hoping to provoke some good discussion.\n\nElijah Newren (15):\n\n P1- README-sparse-clone: Add a basic writeup of my ideas for sparse clones\n\nJust a big old write-up. Not everything in it is implemented yet, but it\ngives you the high-level picture.\n\n P2- Add tests for client handling in a sparse repository\n\nTests! Yaay!\n\n P3- Read sparse limiting args from $GIT_DIR/sparse-limit\n\nWhen a sparse clone is created, limiting paths will be stored.\n\n P4- When unpacking in a sparse repository, avoid traversing missing\n trees/blobs\n P5- read_tree_recursive: Avoid missing blobs and trees in a sparse\n repository\n P6- Automatically reuse sparse limiting arguments in revision walking\n P7- cache_tree_update(): Capability to handle tree entries missing from\n index\n P8- cache_tree_update(): Require relevant tree to be passed\n\nAvoiding missing trees/blobs. \n\n P9- Add tests for communication dealing with sparse repositories\n\nTests for clone/fetch/push/etc. Just clone so far.\n\n P10- sparse-repo: Provide a function to record sparse limiting arguments\n\nCan't just read from $GIT_DIR/sparse-limit; gotta write to it too.\n\n P11- builtin-clone: Accept paths for sparse clone\n P12- Pass extra (rev-list) args on, at least in some cases\n P13- upload-pack: Handle extra rev-list arguments being passed\n P14- EVIL COMMIT: Include all commits\n P15- clone: Ensure sparse limiting arguments are used in subsequent\n operations\n\nI like the changes to how clone accepts additional rev-list arguments\nto limit what is downloaded, but I'm not too happy with how these\npatches pass those rev-list arguments on to upload-pack. So don't\nbother looking too closely at these.\n\n\n Makefile | 2 +\n README-sparse-clone | 284 ++++++++++++++++++++++++++++\n builtin/archive.c | 2 +-\n builtin/checkout.c | 2 +-\n builtin/clone.c | 39 +++-\n builtin/commit.c | 15 +-\n builtin/fetch-pack.c | 3 +-\n builtin/merge.c | 19 +-\n builtin/revert.c | 7 +-\n builtin/send-pack.c | 3 +-\n builtin/write-tree.c | 6 +-\n cache-tree.c | 92 +++++++++-\n cache-tree.h | 4 +-\n cache.h | 5 +-\n connect.c | 9 +-\n diff.h | 1 -\n environment.c | 2 +\n merge-recursive.c | 6 +-\n merge-recursive.h | 2 +-\n revision.c | 21 ++-\n revision.h | 3 +-\n setup.c | 2 +\n sparse-repo.c | 84 ++++++++\n sparse-repo.h | 4 +\n t/sparse-lib.sh | 38 ++++\n t/t5601-clone.sh | 14 --\n t/t5720-sparse-repository-basics.sh | 130 +++++++++++++\n t/t5721-sparse-repository-communication.sh | 106 +++++++++++\n test-dump-cache-tree.c | 3 +-\n transport-helper.c | 5 +-\n transport.c | 13 +-\n transport.h | 9 +-\n tree-diff.c | 4 +-\n tree-walk.c | 48 ++++-\n tree-walk.h | 3 +\n tree.c | 5 +\n upload-pack.c | 45 +++--\n 37 files changed, 952 insertions(+), 88 deletions(-)\n create mode 100644 README-sparse-clone\n create mode 100644 sparse-repo.c\n create mode 100644 sparse-repo.h\n create mode 100644 t/sparse-lib.sh\n create mode 100755 t/t5720-sparse-repository-basics.sh\n create mode 100755 t/t5721-sparse-repository-communication.sh\n\n-- \n1.7.2.3.541.g94cc33\n"},{"id":"149917","messageId":"1283645647-1891-2-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 01/15] README-sparse-clone: Add a basic writeup of my ideas for sparse clones","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:13:53Z","receivedAt":"2010-09-05T00:13:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This write-up just has basic ideas, strategies, notes of what needs to be\ndone, etc. It needs to be pruned, cleaned up, corrected as I learn more,\nmoved elsewhere, etc.\n\nSigned-off-by: Elijah Newren \n---\n README-sparse-clone | 283 +++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 283 insertions(+), 0 deletions(-)\n create mode 100644 README-sparse-clone\n\ndiff --git a/README-sparse-clone b/README-sparse-clone\nnew file mode 100644\nindex 0000000..cfeeef3\n--- /dev/null\n+++ b/README-sparse-clone\n@@ -0,0 +1,283 @@\n+This is my set of notes on implementing sparse clones, which I define\n+as a clone where not all blob, tree, or commit objects are downloaded.\n+This includes sparseness both relative to span of directories and\n+depth of history.\n+\n+(Note: This project has work-in-progress patches -- no promises about\n+quality, speed of implementation, promises not to rebase, etc. etc.)\n+\n+*** Summary ***\n+\n+ Basic Idea:\n+ 0) Only relevant blobs, trees, and commits (+ ancestry) are downloaded.\n+ User View:\n+ U1) A user controls sparseness by passing rev-list arguments to clone.\n+ U2) \"Densifying\" a sparse clone can be done (with new rev-list arguments)\n+ U3) Cloning-from/fetching-from/pushing-to sparse clones is supported.\n+ U4) Operations that need unavailable data simply error out\n+ U5) Old style shallow clones (--depth argument to clone) are obsolete\n+ U6) Miscellaneous notes\n+ Internals:\n+ I1) The limiting rev-list arguments passed to clone are stored.\n+ I2) All revision-walking operations automatically use the limiting args.\n+ I3) The index only contains paths matching the sparse limits\n+ I4) Loading a missing commit results in a fake commit being created\n+ I5) In sparse clones, a special merge strategy must be used\n+ I6) Miscellaneous notes\n+\n+*** Basic Idea ***\n+\n+0) Only relevant blobs, trees, and commits (+ ancestry) are downloaded.\n+\n+Only the relevant blobs, trees, and commits are downloaded.\n+Irrelevant blobs and trees are left out entirely (see items I2 & I3\n+for how we avoid accessing these).\n+\n+To ensure minimum necessary connectivity, we also download basic\n+information from otherwise excluded commits\n+ * parents of these commits\n+ * trees matching the specified sparse path(s)\n+but, for security and space reasons, do not download\n+ * author\n+ * author date\n+ * committer\n+ * committer date\n+ * log message\n+Such commits are still considered \"missing\" (see item I4 for more\n+details about how we handle \"missing\" commits).\n+\n+Tags/branches are downloaded if specified (or, if no branch/tag is\n+specified, all tags/branches are downloaded).\n+\n+Security note: No modifications are done to existing trees, meaning\n+that sparse clones will download the name of \"irrelevant\" blobs/trees\n+with their type, mode, and sha1sum if (and only if) such blobs/trees\n+are siblings of a relevant blob/tree. It is assumed that such\n+information is okay to be transmitted and need not remain private; if\n+such information does need to remain private, an alternate mechanism\n+involving rewriting commits will be necessary (such as git-subtree).\n+\n+*** User View ***\n+\n+U1) A user controls sparseness by passing rev-list arguments to clone.\n+\n+This allows a user to control sparseness both in terms of span of\n+content (files/directories) and depth of history. It can also be used\n+to limit to a subset of refs (cloning just one or two branches instead\n+of all branches and tags). For example,\n+ $ git clone ssh://repo.git dst -- Documentation/\n+ $ git clone ssh://repo.git dst master~6..master\n+ $ git clone ssh://repo.git dst -3\n+(Note that the destination argument becomes mandatory for those doing\n+a sparse clone in order to disambiguate it from rev-list options.)\n+\n+This method also means users don't need much training to learn how to\n+use sparse clones -- they just use syntax they've already learned with\n+log, and clone will pass this info on to upload-pack.\n+\n+There is a difference due to inclusive revision specifications\n+(master, master~6, v4.15.6) vs. exclusive ones (-3, ^master,\n+^master~6). Inclusive revisions must be branch or tag names\n+(e.g. stable or v1.8, but not master~6 or v4.18.2~1 or sha1sum or\n+:/)[1]. \"HEAD --all\"\n+are assumed if no inclusive revisions are specified. (Note: Avery\n+seems to suggest always assuming \"HEAD --all\", at least at first.)\n+\n+[1] This limitation on inclusive revisions could be relaxed in the\n+future for specifications derived from branch names, as long as each\n+branch has no more than one associated derived revision specification.\n+For example, master~6 would mean to clone a copy of the master branch\n+on the remote side, excluding the last 6 commits, so that you start\n+out \"6 commits behind\" the remote. Obviously, it wouldn't make sense\n+to have both \"master^1\" and \"master^2\" specified, since we then\n+wouldn't know where master should point in the clone.\n+\n+U2) \"Densifying\" a sparse clone can be done (with new rev-list arguments)\n+\n+One can fetch a new pack, replace the original limiting rev-list args\n+with the new choice (see item I1), and update the working copy to\n+reflect the changes. As users wouldn't expect a \"fetch\" or a \"merge\"\n+to un-sparsify a checkout, there's a special operation for performing\n+all three operations.\n+\n+[First cut will be to just redownload everything, instead of just the\n+necessary data. I'm thinking it won't be a common operation, and it\n+could always be improved later.]\n+\n+U3) Cloning-from/fetching-from/pushing-to sparse clones is supported.\n+\n+This allows people who need to operate on a subset of the repository\n+(e.g. translators, technical writers, etc.) to collaborate on that\n+subset. I think one simple rule should enable this:\n+\n+ * The receiving repository specifies the limiting rev-list arguments\n+ to use (if the sending repository does not have the relevant data,\n+ it will naturally error out)\n+\n+By having the receving side specify the limiting rev-list arguments,\n+it ensures that any data it receives fulfills its needs. The sending\n+side then uses this information when creating a pack to determine the\n+necessary objects to send, ignoring anything outside the paths/ranges\n+specified in those limits. If the sending side is a sparse clone that\n+does not have the necessary data specified by the receiver, then\n+pack-objects will hit a nasty low-level missing object error, aborting\n+the operation. In the future, we could maybe add a nicer error\n+message.\n+\n+One special case:\n+ * When cloning a repository, if the user did not specify any\n+ limiting rev-list arguments, use those from the repository being\n+ cloned. (Don't require the user to type out all the paths every\n+ time; e.g. 'git clone URL DEST -- PATH1 PATH2 PATH3 PATH4...')\n+\n+U4) Operations that need unavailable data simply error out\n+\n+Although no normal git command should be disabled entirely, there will\n+be cases when some git commands cannot function without more data.\n+\n+Examples:\n+ * merge, cherry-pick, rebase (if unavailable files needed)\n+ * upload-pack (if more data requested than available in a sparse clone)\n+\n+Merge, cherry-pick, and rebase deserve special consideration to\n+operate in sparse clones (see item I5), since merge strategies\n+normally require full trees.\n+\n+U5) Old style shallow clones (--depth argument to clone) are obsolete\n+\n+Since one can pass \"-3\" to get a \"shallow\" clone, old-style shallow\n+clones are obsolete. New style shallow/sparse clones will also be\n+more capable, since one can\n+ * exclude based on commit (e.g. ^master~10) in addition to depth\n+ * clone/push/pull from/to shallow clones\n+\n+What to do with old style shallow clones? Probably deprecate them,\n+make the --depth argument to clone print an error message suggesting\n+the new syntax, and then gut the related code at some point in the\n+future.\n+\n+U6) Miscellaneous notes\n+ * fsck & status should print a notice when working on a sparse clone\n+ * paths in limiting rev-list args *must* follow '--' (current or\n+ future remote repo may be bare, meaning setup_revisions will\n+ complain about nonexistent paths specified without a preceding\n+ '--'). Having all paths folow a '--' will also make it easier to\n+ find them and pass them on to diff machinery (see item I2).\n+ * notes hierarchy may also need to be made sparse in a way that only\n+ notes pointing downloaded objects should be downloaded. This\n+ implies missing blobs/trees, and maybe even \"missing\" commits.\n+ But how do I avoid traversing the wrong notes on the client side?\n+ Ouch. Maybe just include all notes? Or exclude all notes?\n+\n+*** Internals ***\n+\n+I1) The limiting rev-list arguments passed to clone are stored.\n+\n+However, relative arguments such as \"-3\" or \"^master~6\" first need to\n+be translated into one or more exclude ranges written as \"^\".\n+\n+I2) All revision-walking operations automatically use the limiting args.\n+\n+This should be a simple code change, and would enable rev-list, log,\n+diff (which also uses the revision walking machinery), etc. to avoid\n+missing blobs/trees/commits and thus enable them to work with sparse\n+clones. fsck would take a bit more work, since it doesn't use the\n+setup_revisions() and revision.h walking machinery, but shouldn't be\n+too bad (I hope).\n+\n+Also, the pathspecs (or the diff options they generate) are available\n+easily for operations that need them (see I3).\n+\n+I3) The index only contains paths matching the sparse limits\n+\n+Since not all trees are downloaded, not all files can even be\n+referenced in the index. Further, in some cases, the only thing that\n+can be referenced is a tree rather than a file. We only want paths\n+matching the relevant sparse limits to be included in the index. This\n+means two things:\n+ * When extracting entries from trees into the index, the sparse limits\n+ need to be taken into consideration\n+ * Whenever writing trees, using the index is no longer sufficient.\n+ Instead, the files in the index are used to record\n+ sha1sums/modes/filenames for paths within the sparse limits, and\n+ another tree (typically from HEAD) is used to record\n+ sha1sums/modes/filenames/types for paths outside the sparse\n+ limits.\n+\n+Note that writing trees from the index can occur with commit, merge,\n+checkout (-m), revert/cherry-pick --no-commit, and write-tree. All\n+need to be updated to either provide a relevant tree or error out when\n+run from a sparse clone.\n+\n+I4) Loading a missing commit results in a fake commit being created\n+\n+Fake commits have correct parentage and an appropriate (sparse) tree\n+(since those pieces of information are available), but blank author &\n+committer, 0 for times & timezones, and a commit log message such as\n+the following:\n+ This commit is missing from this sparse clone. You can use the\n+ densify command to download missing commits and files.\n+\n+This allows the following to work:\n+ * git commit (which needs tree/file sha1sums that were not modified,\n+ though if a given tree is unmodified, no subtree/subfile sha1s are\n+ needed)\n+ * tags & branches (which can correctly point at missing commits)\n+ * git show (with a branch/tag/commit)\n+ * git prune (missing objects correctly reference their parent(s))\n+ * git fsck (missing commits still referenced)\n+\n+Extra notes:\n+ * Stored in a file using multiple lines of: ...\n+ * Only referenced when git would otherwise die\n+\n+I5) In sparse clones, a special merge strategy must be used\n+\n+Most merge strategies work at the file/content level. Since many\n+files and even whole trees will be unavailable, a special strategy\n+that works with tree-level items is necessary. It should only perform\n+trivial merges when forced to operate at the tree-level (modified on\n+at most one side of history, and probably no rename handling at least\n+at first). When such trivial merges are not possible, it should fail\n+with a helpful error message noting the needed tree contents.\n+\n+For non-missing blobs, standard merge strategies may be used.\n+\n+I6) Miscellaneous notes\n+ * thin-packs: git pack-objects needs to be told to only delta\n+ against objects that match the sparse limits, otherwise the\n+ receiving side will not be able to use the resulting pack.\n+\n+----------------------------------------------------------------------\n+\n+Testcases needed:\n+ * basics: checkout, status, diff, log (w/ options!), add, commit\n+ * extras: blame, apply, bisect, branch, tag, grep, reset\n+ * maintainence: fsck, prune, gc/repack, verify-pack\n+ * plumbing: {read,write,ls,commit,merge,tar,diff}-tree, mktree\n+ * direct: cat-file, show (esp. missing obj. or tag/branch of such)\n+ * merge strat.: merge, cherry-pick/revert, rebase\n+ * communication: pull, push, fetch, clone, bundle, archive\n+ * protocols: http, ssh, git, rsync\n+ * rewrite: filter-branch, fast-{export, import}\n+ * notes: ?\n+\n+ General:\n+ 'clone NON-BARE-REPO dst PATHS' should fail (needs double dash)!\n+ git rev-list master should show subset of available commits\n+ Keep Index sparse:\n+ git add for not in git_sparse_pathspec should error out\n+ update-index on not in git_sparse_pathspec should error out\n+ Sparse Index Handling:\n+ merge into branch yet to be born, revert\n+ checkout -m (to real branch, from valid or yet-to-be born branch)\n+\n+ Major TODOs:\n+ * fetch\n+ * push\n+ * don't pass revlist arguments on command line to upload pack; use protocol\n+ * densify command\n+ * missing commits\n+ * fix thin packs to only delta against objects within sparse limits\n+ * lots more testcases\n+ * cleanup FIXMEs\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149915","messageId":"1283645647-1891-3-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 02/15] Add tests for client handling in a sparse repository","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:13:54Z","receivedAt":"2010-09-05T00:13:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"\nSigned-off-by: Elijah Newren \n---\n t/sparse-lib.sh | 38 ++++++++++\n t/t5720-sparse-repository-basics.sh | 130 +++++++++++++++++++++++++++++++++++\n 2 files changed, 168 insertions(+), 0 deletions(-)\n create mode 100644 t/sparse-lib.sh\n create mode 100755 t/t5720-sparse-repository-basics.sh\n\ndiff --git a/t/sparse-lib.sh b/t/sparse-lib.sh\nnew file mode 100644\nindex 0000000..0b779c6\n--- /dev/null\n+++ b/t/sparse-lib.sh\n@@ -0,0 +1,38 @@\n+#!/bin/sh\n+\n+make_sparse() {\n+\t# We only want loose objects\n+\tmv .git/objects/pack/* . &&\n+\tfor i in $(ls *.pack); do\n+\t\tgit unpack-objects -q < $i\n+\tdone &&\n+\trm -f *.pack *.idx &&\n+\n+\tcd .git/objects &&\n+\n+\t# Find the objects need for the specified paths\n+\tfor i in $(git rev-list master); do\n+\t\techo $i;\n+\t\tgit rev-parse $i^{tree};\n+\t\tgit ls-tree -rt $i -- \"$@\" | awk '{print $3}';\n+\tdone | sort | uniq > ../wanted &&\n+\n+\t# Find all other objects and delete them\n+\tfind . -type f | sed -e s#[\\./]##g \\\n+\t\t | grep -v -F \"$(cat ../wanted)\" > ../bad &&\n+\tfor i in $(cat ../bad); do\n+\t\trm -f ./${i:0:2}/${i:2};\n+\tdone &&\n+\n+\t# Record the sparse limits\n+\tcd .. &&\n+\techo -n \"'--'\" > sparse-limits &&\n+\tfor i in \"$@\"; do\n+\t\techo -n \" '$i'\" >> sparse-limits\n+\tdone &&\n+\techo >> sparse-limits &&\n+\n+\t# Trim the index while we're at it\n+\tcd .. &&\n+\tgit reset --hard HEAD\n+}\ndiff --git a/t/t5720-sparse-repository-basics.sh b/t/t5720-sparse-repository-basics.sh\nnew file mode 100755\nindex 0000000..b8e9a3a\n--- /dev/null\n+++ b/t/t5720-sparse-repository-basics.sh\n@@ -0,0 +1,130 @@\n+#!/bin/sh\n+\n+test_description='sparse repository basics'\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/sparse-lib.sh\n+\n+test_expect_success 'setup' '\n+\trm -fr .git &&\n+\ttest_create_repo src &&\n+\t(\n+\t\tcd src &&\n+\n+\t\tmkdir -p sub/{a,b} &&\n+\t\t> sub/a/file &&\n+\t\tgit add sub/a/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m initial &&\n+\n+\t\t> sub/b/file &&\n+\t\tgit add sub/b/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m two &&\n+\n+\t\techo unique > sub/file &&\n+\t\tgit add sub/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m three &&\n+\n+\t\techo content > sub/b/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m subbfile sub/b/file &&\n+\n+\t\tcp -a sub/b sub/bcopy &&\n+\t\tgit add sub/bcopy &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m subbcopy &&\n+\n+\t\techo stuff > sub/a/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m subafile sub/a/file\n+\t)\n+'\n+\n+test_expect_failure 'make sparse repository' '\n+\tgit clone -q \"file://$(pwd)/src\" dst &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest 25 = \"$(git rev-list --objects master | wc -l)\" &&\n+\t\tmake_sparse sub/b/file &&\n+\t\ttest 0 = $(find .git/objects/pack -type f | wc -l) &&\n+\t\ttest 22 = $(find .git/objects -type f | wc -l)\n+\t)\n+'\n+\n+cd dst 2>/dev/null || test_done\n+srcgit=\"--git-dir=../src/.git\"\n+\n+test_expect_failure 'plumbing: ls-files works' '\n+\tgit ls-files > output &&\n+\ttest \"sub/b/file\" = \"$(cat output)\"\n+'\n+\n+test_expect_failure 'plumbing: rev-list works' '\n+\ttest \"$(git rev-list HEAD)\" = \\\n+\t \"$(git $srcgit rev-list HEAD -- sub/b/)\" &&\n+\ttest \"$(git rev-list --objects HEAD)\" = \\\n+\t \"$(git $srcgit rev-list --objects HEAD -- sub/b/)\"\n+'\n+\n+for i in $(git $srcgit rev-list HEAD | xargs git name-rev | cut -b 42-); do\n+\tgit $srcgit rev-parse $i:sub/b/file >/dev/null 2>&1 &&\n+\ttest_expect_success \"plumbing: We can access $i:sub/b/file\" \"\n+\t\tgit cat-file -t $i:sub/b/file\n+\t\"\n+done\n+\n+final_afile_sha=$(git $srcgit rev-parse master:sub/a/file)\n+known_objects=$(git $srcgit rev-list --objects master \\\n+\t\t| grep sub \\\n+\t\t| grep -v $final_afile_sha \\\n+\t\t| cut -b -40)\n+for i in $(git $srcgit rev-list HEAD | xargs git name-rev | cut -b 42-); do\n+\tgit $srcgit ls-tree -rt $i | grep -F \"$known_objects\" >expect &&\n+\ttest_expect_failure \"plumbing: ls-tree -rt $i works\" \"\n+\t\tgit ls-tree -rt $i 2>error >output &&\n+\t\ttest_cmp output expect\n+\t\"\n+done\n+\n+test_expect_failure 'basic: log works' '\n+\tgit log > /dev/null &&\n+\tgit log -p > /dev/null &&\n+\tgit log -Scontent > /dev/null\n+'\n+\n+test_expect_failure 'basic: diff works' '\n+\tgit diff master~3 master &&\n+\tgit diff master~3\n+'\n+\n+test_expect_failure 'basic: checkout works' '\n+\tgit checkout master~2 &&\n+\tgit checkout master\n+'\n+\n+test_expect_failure 'basic: status works with modified stuff' '\n+\tgit status &&\n+\techo more content >> sub/b/file &&\n+\techo newfile content >> sub/b/whatever &&\n+\tgit status\n+'\n+\n+test_expect_failure 'basic: add works' '\n+\tgit add sub/b/file &&\n+\tgit add sub/b/whatever\n+'\n+\n+test_expect_failure 'basic: commit works' '\n+\tgit commit -m \"Commit in a sparse clone\" &&\n+\tgit rev-parse master^{tree} &&\n+\tgit rev-parse master:sub &&\n+\tgit rev-parse master:sub/b &&\n+\tgit rev-parse master:sub/b/file &&\n+\tgit rev-parse master:sub/bcopy &&\n+\tgit rev-parse master:sub/bcopy/file &&\n+\tgit rev-parse master:sub/a &&\n+\tgit rev-parse master:sub/file\n+'\n+\n+test_done\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149912","messageId":"1283645647-1891-4-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 03/15] Read sparse limiting args from $GIT_DIR/sparse-limit","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:13:55Z","receivedAt":"2010-09-05T00:13:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Sparse clones will be created by specifying rev-list args to clone in order\nto limit what is downloaded. In order to avoid accessing unavailable\nobjects, these limiting rev-list args are recorded in $GIT_DIR/sparse-limit\nso that other operations can make use of them.\n\nSigned-off-by: Elijah Newren \n---\n Makefile | 2 +\n cache.h | 3 ++\n environment.c | 2 +\n setup.c | 2 +\n sparse-repo.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n sparse-repo.h | 3 ++\n tree-diff.c | 2 +\n 7 files changed, 78 insertions(+), 0 deletions(-)\n create mode 100644 sparse-repo.c\n create mode 100644 sparse-repo.h\n\ndiff --git a/Makefile b/Makefile\nindex d3db459..ba4b9bf 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -548,6 +548,7 @@ LIB_H += run-command.h\n LIB_H += sha1-lookup.h\n LIB_H += sideband.h\n LIB_H += sigchain.h\n+LIB_H += sparse-repo.h\n LIB_H += strbuf.h\n LIB_H += string-list.h\n LIB_H += submodule.h\n@@ -657,6 +658,7 @@ LIB_OBJS += sha1_name.o\n LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n+LIB_OBJS += sparse-repo.o\n LIB_OBJS += strbuf.o\n LIB_OBJS += string-list.o\n LIB_OBJS += submodule.o\ndiff --git a/cache.h b/cache.h\nindex ce7bdd1..6f88dbb 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -403,6 +403,9 @@ extern const char *const local_repo_env[LOCAL_REPO_ENV_SIZE + 1];\n extern int is_bare_repository_cfg;\n extern int is_bare_repository(void);\n extern int is_inside_git_dir(void);\n+extern const char **git_sparse_limits;\n+extern const char **git_sparse_pathspecs;\n+extern struct diff_options git_sparse_diffopts;\n extern char *git_work_tree_cfg;\n extern int is_inside_work_tree(void);\n extern int have_git_dir(void);\ndiff --git a/environment.c b/environment.c\nindex cc097b7..fc9d21d 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -61,6 +61,8 @@ struct startup_info *startup_info;\n int core_preload_index = 0;\n \n /* This is set by setup_git_dir_gently() and/or git_default_config() */\n+const char **git_sparse_limits = NULL;\n+const char **git_sparse_pathspecs = NULL;\n char *git_work_tree_cfg;\n static char *work_tree;\n \ndiff --git a/setup.c b/setup.c\nindex a3b76de..92f4385 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"sparse-repo.h\"\n \n static int inside_git_dir = -1;\n static int inside_work_tree = -1;\n@@ -256,6 +257,7 @@ static int check_repository_format_gently(int *nongit_ok)\n \t\t*nongit_ok = -1;\n \t\treturn -1;\n \t}\n+\tcheck_sparse_repo();\n \treturn 0;\n }\n \ndiff --git a/sparse-repo.c b/sparse-repo.c\nnew file mode 100644\nindex 0000000..18ea976\n--- /dev/null\n+++ b/sparse-repo.c\n@@ -0,0 +1,64 @@\n+#include \"sparse-repo.h\"\n+#include \"cache.h\"\n+#include \"diff.h\"\n+#include \"quote.h\"\n+\n+#define SPARSE_FILE \"sparse-limits\"\n+\n+void register_sparse_limits(const char **revlist_argv)\n+{\n+\tif (revlist_argv != git_sparse_limits) {\n+\t\tfree(git_sparse_limits);\n+\t\tgit_sparse_limits = revlist_argv;\n+\t}\n+\n+\tgit_sparse_pathspecs = git_sparse_limits;\n+\tfor (; *git_sparse_pathspecs; git_sparse_pathspecs++)\n+\t\t/*\n+\t\t * Sparse limiting args must always have a double dash before\n+\t\t * any paths, since users will likely fetch/pull from bare\n+\t\t * repositories. That makes it easy to find the pathspec\n+\t\t * portion of the sparse limits.\n+\t\t */\n+\t\tif (strcmp(*git_sparse_pathspecs, \"--\") == 0) {\n+\t\t\t/*\n+\t\t\t * Make git_sparse_limits provide quick access\n+\t\t\t * to non-pathspec rev-list arguments, and\n+\t\t\t * git_sparse_pathspec provide access to the\n+\t\t\t * rest.\n+\t\t\t */\n+\t\t\t*git_sparse_pathspecs = NULL;\n+\t\t\tgit_sparse_pathspecs++;\n+\t\t\tbreak;\n+\t\t}\n+\tdiff_tree_setup_paths(git_sparse_pathspecs, &git_sparse_diffopts);\n+}\n+\n+void register_quoted_sparse_limits(char *quoted_argv)\n+{\n+\tint nr = 0, alloc = 0;\n+\n+\tsq_dequote_to_argv(strdup(quoted_argv), &git_sparse_limits, &nr, &alloc);\n+\tALLOC_GROW(git_sparse_limits, nr+1, alloc);\n+\tgit_sparse_limits[nr] = NULL;\n+\n+\tregister_sparse_limits(git_sparse_limits);\n+}\n+\n+void check_sparse_repo(void)\n+{\n+\tstruct stat statbuf;\n+\tstruct strbuf buffer = STRBUF_INIT;\n+\n+\tchar *sparse_file = git_path(SPARSE_FILE);\n+\tif (stat(sparse_file, &statbuf))\n+\t\treturn;\n+\n+\tif (git_sparse_limits)\n+\t\tdie(\"tried to initialize sparse limits twice!\\n\");\n+\n+\tif (strbuf_read_file(&buffer, sparse_file, statbuf.st_size) < 0)\n+\t\tdie_errno(\"could not read %s\", sparse_file);\n+\tregister_quoted_sparse_limits(buffer.buf);\n+\tstrbuf_release(&buffer);\n+}\ndiff --git a/sparse-repo.h b/sparse-repo.h\nnew file mode 100644\nindex 0000000..ed9018a\n--- /dev/null\n+++ b/sparse-repo.h\n@@ -0,0 +1,3 @@\n+void register_sparse_limits(const char **revlist_argv);\n+void register_quoted_sparse_limits(char *quoted_argv);\n+void check_sparse_repo(void);\ndiff --git a/tree-diff.c b/tree-diff.c\nindex d976bdf..951b53b 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -6,6 +6,8 @@\n #include \"diffcore.h\"\n #include \"tree.h\"\n \n+struct diff_options git_sparse_diffopts;\n+\n static char *malloc_base(const char *base, int baselen, const char *path, int pathlen)\n {\n \tchar *newbase = xmalloc(baselen + pathlen + 2);\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149903","messageId":"1283645647-1891-5-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 04/15] When unpacking in a sparse repository, avoid traversing missing trees/blobs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:13:56Z","receivedAt":"2010-09-05T00:13:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"\nSigned-off-by: Elijah Newren \n---\n diff.h | 1 -\n t/t5720-sparse-repository-basics.sh | 10 +++---\n tree-diff.c | 2 +-\n tree-walk.c | 48 ++++++++++++++++++++++++++++++----\n tree-walk.h | 3 ++\n 5 files changed, 51 insertions(+), 13 deletions(-)\n\ndiff --git a/diff.h b/diff.h\nindex 986a015..e17383c 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -171,7 +171,6 @@ extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,\n \t\t\t const char *base, struct diff_options *opt);\n extern int diff_root_tree_sha1(const unsigned char *new, const char *base,\n struct diff_options *opt);\n-extern int tree_entry_interesting(struct tree_desc *desc, const char *base, int baselen, struct diff_options *opt);\n \n struct combine_diff_path {\n \tstruct combine_diff_path *next;\ndiff --git a/t/t5720-sparse-repository-basics.sh b/t/t5720-sparse-repository-basics.sh\nindex b8e9a3a..b946c23 100755\n--- a/t/t5720-sparse-repository-basics.sh\n+++ b/t/t5720-sparse-repository-basics.sh\n@@ -41,7 +41,7 @@ test_expect_success 'setup' '\n \t)\n '\n \n-test_expect_failure 'make sparse repository' '\n+test_expect_success 'make sparse repository' '\n \tgit clone -q \"file://$(pwd)/src\" dst &&\n \t(\n \t\tcd dst &&\n@@ -55,7 +55,7 @@ test_expect_failure 'make sparse repository' '\n cd dst 2>/dev/null || test_done\n srcgit=\"--git-dir=../src/.git\"\n \n-test_expect_failure 'plumbing: ls-files works' '\n+test_expect_success 'plumbing: ls-files works' '\n \tgit ls-files > output &&\n \ttest \"sub/b/file\" = \"$(cat output)\"\n '\n@@ -98,19 +98,19 @@ test_expect_failure 'basic: diff works' '\n \tgit diff master~3\n '\n \n-test_expect_failure 'basic: checkout works' '\n+test_expect_success 'basic: checkout works' '\n \tgit checkout master~2 &&\n \tgit checkout master\n '\n \n-test_expect_failure 'basic: status works with modified stuff' '\n+test_expect_success 'basic: status works with modified stuff' '\n \tgit status &&\n \techo more content >> sub/b/file &&\n \techo newfile content >> sub/b/whatever &&\n \tgit status\n '\n \n-test_expect_failure 'basic: add works' '\n+test_expect_success 'basic: add works' '\n \tgit add sub/b/file &&\n \tgit add sub/b/whatever\n '\ndiff --git a/tree-diff.c b/tree-diff.c\nindex 951b53b..95e956e 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -261,7 +261,7 @@ static void show_entry(struct diff_options *opt, const char *prefix, struct tree\n \t}\n }\n \n-static void skip_uninteresting(struct tree_desc *t, const char *base, int baselen, struct diff_options *opt, int *all_interesting)\n+void skip_uninteresting(struct tree_desc *t, const char *base, int baselen, struct diff_options *opt, int *all_interesting)\n {\n \twhile (t->size) {\n \t\tint show = tree_entry_interesting(t, base, baselen, opt);\ndiff --git a/tree-walk.c b/tree-walk.c\nindex a9bbf4e..a584dd8 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -66,8 +66,19 @@ static void entry_clear(struct name_entry *a)\n \tmemset(a, 0, sizeof(*a));\n }\n \n-static void entry_extract(struct tree_desc *t, struct name_entry *a)\n+struct tree_sparse_info {\n+\tint all_interesting;\n+\tconst char *base;\n+\tint baselen;\n+\tstruct diff_options *diffopt; /* really is const too */\n+};\n+\n+static void entry_extract(struct tree_desc *t, struct name_entry *a,\n+\t\t\t struct tree_sparse_info *i)\n {\n+\tif (!i->all_interesting)\n+\t\tskip_uninteresting(t, i->base, i->baselen,\n+\t\t\t\t i->diffopt, &i->all_interesting);\n \t*a = t->entry;\n }\n \n@@ -219,7 +230,8 @@ static int check_entry_match(const char *a, int a_len, const char *b, int b_len)\n static void extended_entry_extract(struct tree_desc_x *t,\n \t\t\t\t struct name_entry *a,\n \t\t\t\t const char *first,\n-\t\t\t\t int first_len)\n+\t\t\t\t int first_len,\n+\t\t\t\t struct tree_sparse_info *i)\n {\n \tconst char *path;\n \tint len;\n@@ -235,7 +247,7 @@ static void extended_entry_extract(struct tree_desc_x *t,\n \t\t\tentry_clear(a);\n \t\t\tbreak; /* not found */\n \t\t}\n-\t\tentry_extract(&t->d, a);\n+\t\tentry_extract(&t->d, a, i);\n \t\tfor (skip = t->skip; skip; skip = skip->prev)\n \t\t\tif (a->path == skip->ptr)\n \t\t\t\tbreak; /* found */\n@@ -268,7 +280,7 @@ static void extended_entry_extract(struct tree_desc_x *t,\n \t */\n \tprobe = t->d;\n \twhile (probe.size) {\n-\t\tentry_extract(&probe, a);\n+\t\tentry_extract(&probe, a, i);\n \t\tpath = a->path;\n \t\tlen = tree_entry_len(a->path, a->sha1);\n \t\tswitch (check_entry_match(first, first_len, path, len)) {\n@@ -315,6 +327,28 @@ int traverse_trees(int n, struct tree_desc *t, struct traverse_info *info)\n \tstruct name_entry *entry = xmalloc(n*sizeof(*entry));\n \tint i;\n \tstruct tree_desc_x *tx = xcalloc(n, sizeof(*tx));\n+\tchar prefix[PATH_MAX];\n+\tstruct tree_sparse_info sparse_info;\n+\tif (!git_sparse_pathspecs)\n+\t\tsparse_info.all_interesting = 1;\n+\telse {\n+\t\tif (info->prev)\n+\t\t\tmake_traverse_path(prefix, info->prev, &info->name);\n+\t\telse\n+\t\t\tstrcpy(prefix, info->name.path);\n+\t\tsparse_info.baselen = strlen(prefix);\n+\n+\t\t/* prefix must be slash terminated if non-empty */\n+\t\tif (sparse_info.baselen) {\n+\t\t\tsparse_info.baselen += 1;\n+\t\t\tprefix[sparse_info.baselen-1] = '/';\n+\t\t\tprefix[sparse_info.baselen] = '\\0';\n+\t\t}\n+\n+\t\tsparse_info.all_interesting = 0;\n+\t\tsparse_info.base = prefix;\n+\t\tsparse_info.diffopt = &git_sparse_diffopts;\n+\t}\n \n \tfor (i = 0; i < n; i++)\n \t\ttx[i].d = t[i];\n@@ -328,7 +362,7 @@ int traverse_trees(int n, struct tree_desc *t, struct traverse_info *info)\n \n \t\tfor (i = 0; i < n; i++) {\n \t\t\te = entry + i;\n-\t\t\textended_entry_extract(tx + i, e, NULL, 0);\n+\t\t\textended_entry_extract(tx + i, e, NULL, 0, &sparse_info);\n \t\t}\n \n \t\t/*\n@@ -356,7 +390,9 @@ int traverse_trees(int n, struct tree_desc *t, struct traverse_info *info)\n \t\tif (first) {\n \t\t\tfor (i = 0; i < n; i++) {\n \t\t\t\te = entry + i;\n-\t\t\t\textended_entry_extract(tx + i, e, first, first_len);\n+\t\t\t\textended_entry_extract(tx + i, e,\n+\t\t\t\t\t\t first, first_len,\n+\t\t\t\t\t\t &sparse_info);\n \t\t\t\t/* Cull the ones that are not the earliest */\n \t\t\t\tif (!e->path)\n \t\t\t\t\tcontinue;\ndiff --git a/tree-walk.h b/tree-walk.h\nindex 7e3e0b5..eae1997 100644\n--- a/tree-walk.h\n+++ b/tree-walk.h\n@@ -60,4 +60,7 @@ static inline int traverse_path_len(const struct traverse_info *info, const stru\n \treturn info->pathlen + tree_entry_len(n->path, n->sha1);\n }\n \n+void skip_uninteresting(struct tree_desc *t, const char *base, int baselen, struct diff_options *opt, int *all_interesting);\n+int tree_entry_interesting(struct tree_desc *desc, const char *base, int baselen, struct diff_options *opt);\n+\n #endif\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149904","messageId":"1283645647-1891-6-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 05/15] read_tree_recursive: Avoid missing blobs and trees in a sparse repository","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:13:57Z","receivedAt":"2010-09-05T00:13:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"\nSigned-off-by: Elijah Newren \n---\n t/t5720-sparse-repository-basics.sh | 2 +-\n tree.c | 5 +++++\n 2 files changed, 6 insertions(+), 1 deletions(-)\n\ndiff --git a/t/t5720-sparse-repository-basics.sh b/t/t5720-sparse-repository-basics.sh\nindex b946c23..b11c5ab 100755\n--- a/t/t5720-sparse-repository-basics.sh\n+++ b/t/t5720-sparse-repository-basics.sh\n@@ -81,7 +81,7 @@ known_objects=$(git $srcgit rev-list --objects master \\\n \t\t| cut -b -40)\n for i in $(git $srcgit rev-list HEAD | xargs git name-rev | cut -b 42-); do\n \tgit $srcgit ls-tree -rt $i | grep -F \"$known_objects\" >expect &&\n-\ttest_expect_failure \"plumbing: ls-tree -rt $i works\" \"\n+\ttest_expect_success \"plumbing: ls-tree -rt $i works\" \"\n \t\tgit ls-tree -rt $i 2>error >output &&\n \t\ttest_cmp output expect\n \t\"\ndiff --git a/tree.c b/tree.c\nindex 5ab90af..5f9e37a 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -119,6 +119,11 @@ int read_tree_recursive(struct tree *tree,\n \t\tdefault:\n \t\t\treturn -1;\n \t\t}\n+\n+\t\tif (git_sparse_pathspecs &&\n+\t\t sha1_object_info(entry.sha1, NULL) <= 0)\n+\t\t\tcontinue;\n+\n \t\tif (S_ISDIR(entry.mode)) {\n \t\t\tint retval;\n \t\t\tchar *newbase;\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149916","messageId":"1283645647-1891-7-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 06/15] Automatically reuse sparse limiting arguments in revision walking","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:13:58Z","receivedAt":"2010-09-05T00:13:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"In a sparse repository, by automatically making use of sparse limits\nspecified at clone time, we can avoid walking uninteresting commits and\nprevent access to missing trees and blobs. Note that this means that if\nyou created a sparse clone with\n git clone -- PATH1 PATH2 PATH3 PATH4 PATH5\nthen\n git log\nimplicitly runs as though you had manually specified\n git log -- PATH1 PATH2 PATH3 PATH4 PATH5\nSimilarly, running\n git diff\nimplicitly runs as though you had manually specified\n git diff -- PATH1 PATH2 PATH3 PATH4 PATH5\n\nThis is necessary for proper operation of git diff in a sparse clone to\navoid accessing missing objects. In the case of a plain git log, this\nmerely serves as an additional convenience, but for more complicated log\noperations (e.g. when passing -p or -S options) it becomes necessary.\n\nSigned-off-by: Elijah Newren \n---\n revision.c | 16 +++++++++++++++-\n t/t5720-sparse-repository-basics.sh | 6 +++---\n 2 files changed, 18 insertions(+), 4 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex a962b77..67b1a1d 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1492,6 +1492,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \tconst char *submodule = NULL;\n \tconst char *optarg;\n \tint argcount;\n+\tconst char *arg;\n \n \tif (opt)\n \t\tsubmodule = opt->submodule;\n@@ -1514,7 +1515,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \tflags = 0;\n \tread_from_stdin = 0;\n \tfor (left = i = 1; i < argc; i++) {\n-\t\tconst char *arg = argv[i];\n+\t\targ = argv[i];\n \t\tif (*arg == '-') {\n \t\t\tint opts;\n \n@@ -1624,6 +1625,19 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\t\tgot_rev_arg = 1;\n \t}\n \n+\t/*\n+\t * FIXME: This disallows things like --all, --stdin, --not, although\n+\t * some of those like --stdin may not make sense anyway.\n+\t */\n+\tfor (arg = git_sparse_limits ? *git_sparse_limits : NULL; arg; arg++) {\n+\t\tconst int flags = 0;\n+\t\tconst int cant_be_filename = 1;\n+\t\tif (handle_revision_arg(arg, revs, flags, cant_be_filename))\n+\t\t\tdie(\"bad revision '%s'\", arg);\n+\t}\n+\tif (git_sparse_pathspecs)\n+\t\tappend_prune_data(&prune_data, git_sparse_pathspecs);\n+\n \tif (prune_data)\n \t\trevs->prune_data = get_pathspec(revs->prefix, prune_data);\n \ndiff --git a/t/t5720-sparse-repository-basics.sh b/t/t5720-sparse-repository-basics.sh\nindex b11c5ab..d04e171 100755\n--- a/t/t5720-sparse-repository-basics.sh\n+++ b/t/t5720-sparse-repository-basics.sh\n@@ -60,7 +60,7 @@ test_expect_success 'plumbing: ls-files works' '\n \ttest \"sub/b/file\" = \"$(cat output)\"\n '\n \n-test_expect_failure 'plumbing: rev-list works' '\n+test_expect_success 'plumbing: rev-list works' '\n \ttest \"$(git rev-list HEAD)\" = \\\n \t \"$(git $srcgit rev-list HEAD -- sub/b/)\" &&\n \ttest \"$(git rev-list --objects HEAD)\" = \\\n@@ -87,13 +87,13 @@ for i in $(git $srcgit rev-list HEAD | xargs git name-rev | cut -b 42-); do\n \t\"\n done\n \n-test_expect_failure 'basic: log works' '\n+test_expect_success 'basic: log works' '\n \tgit log > /dev/null &&\n \tgit log -p > /dev/null &&\n \tgit log -Scontent > /dev/null\n '\n \n-test_expect_failure 'basic: diff works' '\n+test_expect_success 'basic: diff works' '\n \tgit diff master~3 master &&\n \tgit diff master~3\n '\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149906","messageId":"1283645647-1891-8-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 07/15] cache_tree_update(): Capability to handle tree entries missing from index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:13:59Z","receivedAt":"2010-09-05T00:13:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"cache_tree_update() will write trees using the index. With sparse clones,\nthe index will only contain entries matching the sparse limits, meaning\nthat the index does not provide enough information to write complete tree\nobjects. Having cache_tree_update() take a tree (typically HEAD), will\nallow new complete trees to be constructed by using entries from the\nspecified tree outside the sparse limits together with the index.\n\nOf course, in the non-sparse clone case, cache_tree_update() needs no tree\nto be provided since the index contains all relevant information. So\nproviding NULL as the tree is supported as another way of getting the\ntraditional behavior.\n\nThis patch only provides the new capability without invoking it; subsequent\npatches will update callers of cache_tree_update() to provide a relevant\ntree.\n\nSigned-off-by: Elijah Newren \n---\n cache-tree.c | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 files changed, 87 insertions(+), 1 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex c60cf91..2ba6a76 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -2,6 +2,7 @@\n #include \"tree.h\"\n #include \"tree-walk.h\"\n #include \"cache-tree.h\"\n+#include \"diff.h\" /* FIXME: for tree_entry_interesting; maybe it should be in tree-walk.h? */\n \n #ifndef DEBUG\n #define DEBUG 0\n@@ -232,15 +233,91 @@ int cache_tree_fully_valid(struct cache_tree *it)\n \treturn 1;\n }\n \n+static void setup_tree_desc(struct tree_desc *desc, struct tree* tree)\n+{\n+\tif (git_sparse_pathspecs && tree)\n+\t\tinit_tree_desc(desc, tree->buffer, tree->size);\n+\telse\n+\t\tinit_tree_desc(desc, NULL, 0);\n+}\n+\n+static struct tree* find_non_sparse_subtree(struct tree_desc *desc,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t int pathlen)\n+{\n+\tstruct name_entry entry;\n+\n+\tif (!git_sparse_pathspecs)\n+\t\treturn NULL; /* No tree needed or useful */\n+\n+\twhile (tree_entry(desc, &entry))\n+\t\t if (!memcmp(entry.path, path, pathlen) &&\n+\t\t strlen(entry.path) == pathlen) {\n+\t\t\tstruct tree *subtree = lookup_tree(entry.sha1);\n+\t\t\tif (parse_tree(subtree) < 0)\n+\t\t\t\tdie(\"bad tree %s\", sha1_to_hex(entry.sha1));\n+\t\t\treturn subtree;\n+\t\t }\n+\treturn NULL;\n+}\n+\n+static void add_missing_paths_before(const char *path,\n+\t\t\t\t int pathlen,\n+\t\t\t\t int baselen,\n+\t\t\t\t struct tree_desc *desc,\n+\t\t\t\t struct strbuf *buffer)\n+{\n+\tif (!git_sparse_pathspecs)\n+\t\treturn; /* No paths are missing */\n+\n+\tfor (; desc->size; update_tree_entry(desc)) {\n+\t\tstruct name_entry entry = desc->entry;\n+\t\tint entlen = strlen(entry.path);\n+\n+\t\t/* We only want paths before path */\n+\t\tif (path)\n+\t\t\t/*\n+\t\t\t * FIXME: if entlen < pathlen, do I need to\n+\t\t\t * use strncmp instead? Does entry.path being\n+\t\t\t * NUL-terminated ensure memcmp exits\n+\t\t\t * early?\n+\t\t\t */\n+\t\t\tif (memcmp(entry.path,\n+\t\t\t\t path+baselen,\n+\t\t\t\t pathlen-baselen) >= 0)\n+\t\t\t\tbreak;\n+\n+\t\t/* We only want paths \"missing\" from index due to sparsity */\n+\t\tif (path) {\n+\t\t\tint show = tree_entry_interesting(desc,\n+\t\t\t\t\t\t\t path,\n+\t\t\t\t\t\t\t baselen,\n+\t\t\t\t\t\t\t &git_sparse_diffopts);\n+\t\t\tif (show == 2) {\n+\t\t\t\tdesc->size = 0;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\telse if (show > 0)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tstrbuf_grow(buffer, entlen + 100);\n+\t\tstrbuf_addf(buffer, \"%o %.*s%c\", entry.mode, entlen, entry.path, '\\0');\n+\t\tstrbuf_add(buffer, entry.sha1, 20);\n+\t}\n+}\n+\n static int update_one(struct cache_tree *it,\n \t\t struct cache_entry **cache,\n \t\t int entries,\n+\t\t struct tree *non_sparse_tree,\n \t\t const char *base,\n \t\t int baselen,\n \t\t int missing_ok,\n \t\t int dryrun)\n {\n \tstruct strbuf buffer;\n+\tstruct tree_desc desc;\n \tint i;\n \n \tif (0 <= it->entry_count && has_sha1_file(it->sha1))\n@@ -257,9 +334,11 @@ static int update_one(struct cache_tree *it,\n \t/*\n \t * Find the subtrees and update them.\n \t */\n+\tsetup_tree_desc(&desc, non_sparse_tree);\n \tfor (i = 0; i < entries; i++) {\n \t\tstruct cache_entry *ce = cache[i];\n \t\tstruct cache_tree_sub *sub;\n+\t\tstruct tree *subtree;\n \t\tconst char *path, *slash;\n \t\tint pathlen, sublen, subcnt;\n \n@@ -280,8 +359,10 @@ static int update_one(struct cache_tree *it,\n \t\tsub = find_subtree(it, path + baselen, sublen, 1);\n \t\tif (!sub->cache_tree)\n \t\t\tsub->cache_tree = cache_tree();\n+\t\tsubtree = find_non_sparse_subtree(&desc, path+baselen, sublen);\n \t\tsubcnt = update_one(sub->cache_tree,\n \t\t\t\t cache + i, entries - i,\n+\t\t\t\t subtree,\n \t\t\t\t path,\n \t\t\t\t baselen + sublen + 1,\n \t\t\t\t missing_ok,\n@@ -298,6 +379,7 @@ static int update_one(struct cache_tree *it,\n \t * Then write out the tree object for this level.\n \t */\n \tstrbuf_init(&buffer, 8192);\n+\tsetup_tree_desc(&desc, non_sparse_tree);\n \n \tfor (i = 0; i < entries; i++) {\n \t\tstruct cache_entry *ce = cache[i];\n@@ -309,6 +391,8 @@ static int update_one(struct cache_tree *it,\n \n \t\tpath = ce->name;\n \t\tpathlen = ce_namelen(ce);\n+\t\tadd_missing_paths_before(path, pathlen, baselen,\n+\t\t\t\t\t &desc, &buffer);\n \t\tif (pathlen <= baselen || memcmp(base, path, baselen))\n \t\t\tbreak; /* at the end of this level */\n \n@@ -346,6 +430,7 @@ static int update_one(struct cache_tree *it,\n \t\t\tmode, entlen, path + baselen);\n #endif\n \t}\n+\tadd_missing_paths_before(NULL, 0, 0, &desc, &buffer);\n \n \tif (dryrun)\n \t\thash_sha1_file(buffer.buf, buffer.len, tree_type, it->sha1);\n@@ -374,7 +459,8 @@ int cache_tree_update(struct cache_tree *it,\n \ti = verify_cache(cache, entries);\n \tif (i)\n \t\treturn i;\n-\ti = update_one(it, cache, entries, \"\", 0, missing_ok, dryrun);\n+\n+\ti = update_one(it, cache, entries, NULL, \"\", 0, missing_ok, dryrun);\n \tif (i < 0)\n \t\treturn i;\n \treturn 0;\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149905","messageId":"1283645647-1891-9-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 08/15] cache_tree_update(): Require relevant tree to be passed","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:14:00Z","receivedAt":"2010-09-05T00:14:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This updates the callers of cache_tree_update() to provide it a tree, so\nthat cache_tree_update() can use it for paths outside the sparse limits\ntogether with index entries to write out a new set of trees. In cases\nwhere there is no obvious tree to provide: if in a sparse repository, the\ncode dies; if in a non-sparse repository, a NULL tree is provided (in the\nnon-sparse case, the tree provided is irrelevant since the index is\ncomplete).\n\nFIXME: The changes to buildin/merge.c and merge-recursive.[ch] are probably\nwrong; a new tree needs to be generated via some trivial merge algorithm\nand then used, rather than just using the HEAD commit.\n\nSigned-off-by: Elijah Newren \n---\n builtin/checkout.c | 2 +-\n builtin/commit.c | 15 +++++++++++----\n builtin/merge.c | 19 ++++++++++---------\n builtin/revert.c | 7 ++++++-\n builtin/write-tree.c | 6 +++++-\n cache-tree.c | 6 ++++--\n cache-tree.h | 4 ++--\n merge-recursive.c | 6 +++---\n merge-recursive.h | 2 +-\n t/t5720-sparse-repository-basics.sh | 2 +-\n test-dump-cache-tree.c | 3 ++-\n 11 files changed, 46 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex b10e8a2..3299aeb 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -454,7 +454,7 @@ static int merge_working_tree(struct checkout_opts *opts,\n \t\t\t */\n \t\t\tinit_merge_options(&o);\n \t\t\to.verbosity = 0;\n-\t\t\twork = write_tree_from_memory(&o);\n+\t\t\twork = write_tree_from_memory(&o, old->commit->tree);\n \n \t\t\tret = reset_tree(new->commit->tree, opts, 1);\n \t\t\tif (ret)\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 66fdd22..ad6ecb2 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -550,8 +550,8 @@ static int ends_rfc2822_footer(struct strbuf *sb)\n \treturn 1;\n }\n \n-static int prepare_to_commit(const char *index_file, const char *prefix,\n-\t\t\t struct wt_status *s)\n+static int prepare_to_commit(const char *index_file, struct tree *head,\n+\t\t\t const char *prefix, struct wt_status *s)\n {\n \tstruct stat statbuf;\n \tint commitable, saved_color_setting;\n@@ -733,7 +733,7 @@ static int prepare_to_commit(const char *index_file, const char *prefix,\n \tif (!active_cache_tree)\n \t\tactive_cache_tree = cache_tree();\n \tif (cache_tree_update(active_cache_tree,\n-\t\t\t active_cache, active_nr, 0, 0) < 0) {\n+\t\t\t active_cache, active_nr, head, 0, 0) < 0) {\n \t\terror(\"Error building trees\");\n \t\treturn 0;\n \t}\n@@ -1246,6 +1246,7 @@ static int run_rewrite_hook(const unsigned char *oldsha1,\n int cmd_commit(int argc, const char **argv, const char *prefix)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n+\tstruct tree *head_tree = NULL;\n \tconst char *index_file, *reflog_msg;\n \tchar *nl, *p;\n \tunsigned char commit_sha1[20];\n@@ -1273,7 +1274,13 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \n \t/* Set up everything for writing the commit object. This includes\n \t running hooks, writing the trees, and interacting with the user. */\n-\tif (!prepare_to_commit(index_file, prefix, &s)) {\n+\tif (git_sparse_pathspecs && !initial_commit) {\n+\t\thead_tree = parse_tree_indirect(head_sha1);\n+\t\tif (!head_tree)\n+\t\t\tdie(\"failed to unpack HEAD tree object\");\n+\t\tparse_tree(head_tree);\n+\t}\n+\tif (!prepare_to_commit(index_file, head_tree, prefix, &s)) {\n \t\trollback_index_files();\n \t\treturn 1;\n \t}\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex cbb6c0d..f6fe51e 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -557,9 +557,9 @@ static int read_tree_trivial(unsigned char *common, unsigned char *head,\n \treturn 0;\n }\n \n-static void write_tree_trivial(unsigned char *sha1)\n+static void write_tree_trivial(unsigned char *sha1, struct tree *head_tree)\n {\n-\tif (write_cache_as_tree(sha1, 0, NULL))\n+\tif (write_cache_as_tree(sha1, 0, head_tree, NULL))\n \t\tdie(\"git write-tree failed to write a tree\");\n }\n \n@@ -778,12 +778,12 @@ static void add_strategies(const char *string, unsigned attr)\n \n }\n \n-static int merge_trivial(void)\n+static int merge_trivial(struct tree *head_tree)\n {\n \tunsigned char result_tree[20], result_commit[20];\n \tstruct commit_list *parent = xmalloc(sizeof(*parent));\n \n-\twrite_tree_trivial(result_tree);\n+\twrite_tree_trivial(result_tree, head_tree);\n \tprintf(\"Wonderful.\\n\");\n \tparent->item = lookup_commit(head);\n \tparent->next = xmalloc(sizeof(*parent->next));\n@@ -901,6 +901,7 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n \tconst char *head_arg;\n \tint flag, head_invalid = 0, i;\n \tint best_cnt = -1, merge_was_ok = 0, automerge_was_ok = 0;\n+\tstruct commit *head_commit;\n \tstruct commit_list *common = NULL;\n \tconst char *best_strategy = NULL, *wt_strategy = NULL;\n \tstruct commit_list **remotes = &remoteheads;\n@@ -1056,12 +1057,12 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n \t\t\tallow_trivial = 0;\n \t}\n \n+\thead_commit = lookup_commit(head);\n \tif (!remoteheads->next)\n-\t\tcommon = get_merge_bases(lookup_commit(head),\n-\t\t\t\tremoteheads->item, 1);\n+\t\tcommon = get_merge_bases(head_commit, remoteheads->item, 1);\n \telse {\n \t\tstruct commit_list *list = remoteheads;\n-\t\tcommit_list_insert(lookup_commit(head), &list);\n+\t\tcommit_list_insert(head_commit, &list);\n \t\tcommon = get_octopus_merge_bases(list);\n \t\tfree(list);\n \t}\n@@ -1127,7 +1128,7 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n \t\t\tprintf(\"Trying really trivial in-index merge...\\n\");\n \t\t\tif (!read_tree_trivial(common->item->object.sha1,\n \t\t\t\t\thead, remoteheads->item->object.sha1))\n-\t\t\t\treturn merge_trivial();\n+\t\t\t\treturn merge_trivial(head_commit->tree);\n \t\t\tprintf(\"Nope.\\n\");\n \t\t}\n \t} else {\n@@ -1232,7 +1233,7 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n \t\t}\n \n \t\t/* Automerge succeeded. */\n-\t\twrite_tree_trivial(result_tree);\n+\t\twrite_tree_trivial(result_tree, head_commit->tree);\n \t\tautomerge_was_ok = 1;\n \t\tbreak;\n \t}\ndiff --git a/builtin/revert.c b/builtin/revert.c\nindex 4b47ace..4e95589 100644\n--- a/builtin/revert.c\n+++ b/builtin/revert.c\n@@ -404,7 +404,12 @@ static int do_pick_commit(void)\n \t\t * that represents the \"current\" state for merge-recursive\n \t\t * to work on.\n \t\t */\n-\t\tif (write_cache_as_tree(head, 0, NULL))\n+\t\tstruct tree *non_sparse_tree;\n+\t\tif (get_sha1(\"HEAD\", head))\n+\t\t\tnon_sparse_tree = NULL;\n+\t\telse\n+\t\t\tnon_sparse_tree = lookup_commit(head)->tree;\n+\t\tif (write_cache_as_tree(head, 0, non_sparse_tree, NULL))\n \t\t\tdie (\"Your index file is unmerged.\");\n \t} else {\n \t\tif (get_sha1(\"HEAD\", head))\ndiff --git a/builtin/write-tree.c b/builtin/write-tree.c\nindex b223af4..1e459b6 100644\n--- a/builtin/write-tree.c\n+++ b/builtin/write-tree.c\n@@ -19,6 +19,7 @@ int cmd_write_tree(int argc, const char **argv, const char *unused_prefix)\n \tint flags = 0, ret;\n \tconst char *prefix = NULL;\n \tunsigned char sha1[20];\n+\tstruct tree *non_sparse_tree = NULL;\n \tconst char *me = \"git-write-tree\";\n \tstruct option write_tree_options[] = {\n \t\tOPT_BIT(0, \"missing-ok\", &flags, \"allow missing objects\",\n@@ -37,7 +38,10 @@ int cmd_write_tree(int argc, const char **argv, const char *unused_prefix)\n \targc = parse_options(argc, argv, unused_prefix, write_tree_options,\n \t\t\t write_tree_usage, 0);\n \n-\tret = write_cache_as_tree(sha1, flags, prefix);\n+\tif (git_sparse_pathspecs)\n+\t\t/* FIXME: Let user specify non_sparse_tree? Just use HEAD? */\n+\t\tdie(\"Error: write-tree in a sparse repository would clip paths outside sparse region.\");\n+\tret = write_cache_as_tree(sha1, flags, non_sparse_tree, prefix);\n \tswitch (ret) {\n \tcase 0:\n \t\tprintf(\"%s\\n\", sha1_to_hex(sha1));\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 2ba6a76..4309fa8 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -452,6 +452,7 @@ static int update_one(struct cache_tree *it,\n int cache_tree_update(struct cache_tree *it,\n \t\t struct cache_entry **cache,\n \t\t int entries,\n+\t\t struct tree *head,\n \t\t int missing_ok,\n \t\t int dryrun)\n {\n@@ -460,7 +461,7 @@ int cache_tree_update(struct cache_tree *it,\n \tif (i)\n \t\treturn i;\n \n-\ti = update_one(it, cache, entries, NULL, \"\", 0, missing_ok, dryrun);\n+\ti = update_one(it, cache, entries, head, \"\", 0, missing_ok, dryrun);\n \tif (i < 0)\n \t\treturn i;\n \treturn 0;\n@@ -629,7 +630,7 @@ static struct cache_tree *cache_tree_find(struct cache_tree *it, const char *pat\n \treturn it;\n }\n \n-int write_cache_as_tree(unsigned char *sha1, int flags, const char *prefix)\n+int write_cache_as_tree(unsigned char *sha1, int flags, struct tree *head, const char *prefix)\n {\n \tint entries, was_valid, newfd;\n \tstruct lock_file *lock_file;\n@@ -657,6 +658,7 @@ int write_cache_as_tree(unsigned char *sha1, int flags, const char *prefix)\n \n \t\tif (cache_tree_update(active_cache_tree,\n \t\t\t\t active_cache, active_nr,\n+\t\t\t\t head,\n \t\t\t\t missing_ok, 0) < 0)\n \t\t\treturn WRITE_TREE_UNMERGED_INDEX;\n \t\tif (0 <= newfd) {\ndiff --git a/cache-tree.h b/cache-tree.h\nindex 3df641f..85cc016 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -29,7 +29,7 @@ void cache_tree_write(struct strbuf *, struct cache_tree *root);\n struct cache_tree *cache_tree_read(const char *buffer, unsigned long size);\n \n int cache_tree_fully_valid(struct cache_tree *);\n-int cache_tree_update(struct cache_tree *, struct cache_entry **, int, int, int);\n+int cache_tree_update(struct cache_tree *, struct cache_entry **, int, struct tree*, int, int);\n \n /* bitmasks to write_cache_as_tree flags */\n #define WRITE_TREE_MISSING_OK 1\n@@ -40,7 +40,7 @@ int cache_tree_update(struct cache_tree *, struct cache_entry **, int, int, int)\n #define WRITE_TREE_UNMERGED_INDEX (-2)\n #define WRITE_TREE_PREFIX_ERROR (-3)\n \n-int write_cache_as_tree(unsigned char *sha1, int flags, const char *prefix);\n+int write_cache_as_tree(unsigned char *sha1, int flags, struct tree *head, const char *prefix);\n void prime_cache_tree(struct cache_tree **, struct tree *);\n \n extern int cache_tree_matches_traversal(struct cache_tree *, struct name_entry *ent, struct traverse_info *info);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 325a97b..b74c29f 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -191,7 +191,7 @@ static int git_merge_trees(int index_only,\n \treturn rc;\n }\n \n-struct tree *write_tree_from_memory(struct merge_options *o)\n+struct tree *write_tree_from_memory(struct merge_options *o, struct tree *head)\n {\n \tstruct tree *result = NULL;\n \n@@ -212,7 +212,7 @@ struct tree *write_tree_from_memory(struct merge_options *o)\n \n \tif (!cache_tree_fully_valid(active_cache_tree) &&\n \t cache_tree_update(active_cache_tree,\n-\t\t\t active_cache, active_nr, 0, 0) < 0)\n+\t\t\t active_cache, active_nr, head, 0, 0) < 0)\n \t\tdie(\"error building trees\");\n \n \tresult = lookup_tree(active_cache_tree->sha1);\n@@ -1361,7 +1361,7 @@ int merge_trees(struct merge_options *o,\n \t\tclean = 1;\n \n \tif (o->call_depth)\n-\t\t*result = write_tree_from_memory(o);\n+\t\t*result = write_tree_from_memory(o, head);\n \n \treturn clean;\n }\ndiff --git a/merge-recursive.h b/merge-recursive.h\nindex 2eb5d1a..26110e2 100644\n--- a/merge-recursive.h\n+++ b/merge-recursive.h\n@@ -51,7 +51,7 @@ int merge_recursive_generic(struct merge_options *o,\n \t\t\t struct commit **result);\n \n void init_merge_options(struct merge_options *o);\n-struct tree *write_tree_from_memory(struct merge_options *o);\n+struct tree *write_tree_from_memory(struct merge_options *o, struct tree *head);\n \n int parse_merge_opt(struct merge_options *out, const char *s);\n \ndiff --git a/t/t5720-sparse-repository-basics.sh b/t/t5720-sparse-repository-basics.sh\nindex d04e171..88f40ab 100755\n--- a/t/t5720-sparse-repository-basics.sh\n+++ b/t/t5720-sparse-repository-basics.sh\n@@ -115,7 +115,7 @@ test_expect_success 'basic: add works' '\n \tgit add sub/b/whatever\n '\n \n-test_expect_failure 'basic: commit works' '\n+test_expect_success 'basic: commit works' '\n \tgit commit -m \"Commit in a sparse clone\" &&\n \tgit rev-parse master^{tree} &&\n \tgit rev-parse master:sub &&\ndiff --git a/test-dump-cache-tree.c b/test-dump-cache-tree.c\nindex 1f73f1e..ef0644f 100644\n--- a/test-dump-cache-tree.c\n+++ b/test-dump-cache-tree.c\n@@ -57,8 +57,9 @@ static int dump_cache_tree(struct cache_tree *it,\n int main(int ac, char **av)\n {\n \tstruct cache_tree *another = cache_tree();\n+\tstruct tree *non_sparse_tree = NULL; /* We don't test sparse clones */\n \tif (read_cache() < 0)\n \t\tdie(\"unable to read index file\");\n-\tcache_tree_update(another, active_cache, active_nr, 0, 1);\n+\tcache_tree_update(another, active_cache, active_nr, non_sparse_tree, 0, 1);\n \treturn dump_cache_tree(active_cache_tree, another, \"\");\n }\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149908","messageId":"1283645647-1891-10-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 09/15] Add tests for communication dealing with sparse repositories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:14:01Z","receivedAt":"2010-09-05T00:14:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"\nSigned-off-by: Elijah Newren \n---\n t/t5721-sparse-repository-communication.sh | 106 ++++++++++++++++++++++++++++\n 1 files changed, 106 insertions(+), 0 deletions(-)\n create mode 100755 t/t5721-sparse-repository-communication.sh\n\ndiff --git a/t/t5721-sparse-repository-communication.sh b/t/t5721-sparse-repository-communication.sh\nnew file mode 100755\nindex 0000000..35cb6a2\n--- /dev/null\n+++ b/t/t5721-sparse-repository-communication.sh\n@@ -0,0 +1,106 @@\n+#!/bin/sh\n+\n+test_description='sparse repository communication'\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/sparse-lib.sh\n+\n+test_expect_success 'setup' '\n+\trm -fr .git &&\n+\ttest_create_repo src &&\n+\t(\n+\t\tcd src &&\n+\n+\t\tmkdir -p sub/{a,b} &&\n+\t\t> sub/a/file &&\n+\t\tgit add sub/a/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m initial &&\n+\n+\t\t> sub/b/file &&\n+\t\tgit add sub/b/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m two &&\n+\n+\t\techo unique > sub/file &&\n+\t\tgit add sub/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m three &&\n+\n+\t\techo content > sub/b/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m subbfile sub/b/file &&\n+\n+\t\tcp -a sub/b sub/bcopy &&\n+\t\tgit add sub/bcopy &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m subbcopy &&\n+\n+\t\techo stuff > sub/a/file &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m subafile sub/a/file\n+\t)\n+'\n+\n+test_expect_success 'make comparison sparse repository' '\n+\tgit clone -q \"file://$(pwd)/src\" compare_dst &&\n+\t(\n+\t\tcd compare_dst &&\n+\t\ttest 25 = \"$(git rev-list --objects master | wc -l)\" &&\n+\t\tmake_sparse sub/b/ &&\n+\t\ttest 0 = $(find .git/objects/pack -type f | wc -l) &&\n+\t\ttest 22 = $(find .git/objects -type f | wc -l)\n+\t)\n+'\n+\n+test_expect_failure 'basic sparse clone succeeds' '\n+\trm -fr dst &&\n+\tgit clone \"file://$(pwd)/src\" dst -- sub/b/ &&\n+\t(\n+\t\tcd dst &&\n+\t\tgit rev-parse master^{tree} &&\n+\t\tgit rev-parse master:sub &&\n+\t\tgit rev-parse master:sub/b &&\n+\t\tgit rev-parse master:sub/b/file &&\n+\t\tgit rev-parse master:sub/bcopy &&\n+\t\tgit rev-parse master:sub/bcopy/file &&\n+\t\tgit rev-parse master:sub/a &&\n+\t\tgit rev-parse master:sub/file &&\n+\n+\t\tgit cat-file -p master:sub &&\n+\t\tgit cat-file -p master:sub/b &&\n+\t\tgit cat-file -p master:sub/b/file &&\n+\t\tgit cat-file -p master:sub/bcopy &&\n+\t\tgit cat-file -p master:sub/bcopy/file &&\n+\t\ttest_must_fail git cat-file -p master:sub/file &&\n+\t\ttest_must_fail git cat-file -p master:sub/a &&\n+\t\ttest_must_fail git rev-parse master:sub/a/file &&\n+\n+\t\ttest -f sub/b/file &&\n+\t\t! test -d sub/a &&\n+\t\t! test -d sub/bcopy &&\n+\t\t! test -f sub/file\n+\t)\n+'\n+\n+test_expect_failure 'basic sparse clone guts match expectations' '\n+\t(\n+\t\t# Loose objects only, to facilitate comparison\n+\t\tcd dst &&\n+\t\tmv .git/objects/pack/* . &&\n+\t\tfor i in $(ls *.pack); do\n+\t\t\tgit unpack-objects -q < $i\n+\t\tdone &&\n+\t\trm -f *.pack *.idx\n+\t) &&\n+\t(\n+\t\t# Get the names of all loose objects\n+\t\tcd dst &&\n+\t\tfind .git/objects -type f > ../dst-objects &&\n+\t\tcd ../compare_dst &&\n+\t\tfind .git/objects -type f > ../compare_dst-objects\n+\t) &&\n+\ttest_cmp dst-objects compare_dst-objects &&\n+\ttest_cmp dst/.git/sparse-limits compare_dst/.git/sparse-limits\n+'\n+\n+test_done\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149909","messageId":"1283645647-1891-11-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 10/15] sparse-repo: Provide a function to record sparse limiting arguments","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:14:02Z","receivedAt":"2010-09-05T00:14:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"\nSigned-off-by: Elijah Newren \n---\n sparse-repo.c | 20 ++++++++++++++++++++\n sparse-repo.h | 1 +\n 2 files changed, 21 insertions(+), 0 deletions(-)\n\ndiff --git a/sparse-repo.c b/sparse-repo.c\nindex 18ea976..c983096 100644\n--- a/sparse-repo.c\n+++ b/sparse-repo.c\n@@ -5,6 +5,26 @@\n \n #define SPARSE_FILE \"sparse-limits\"\n \n+void record_sparse_limits(const char *quoted_argv)\n+{\n+\tint fd, len;\n+\tchar *sparse_file = git_path(SPARSE_FILE);\n+\n+\t/* FIXME: sq_quote_argv puts a leading space in front of quoted\n+\t * arguments, but sq_dequote_to_argv chokes on it. Maybe sq_quote_argv\n+\t * should be fixed?\n+\t */\n+\tif (*quoted_argv == ' ')\n+\t\tquoted_argv++;\n+\tlen = strlen(quoted_argv);\n+\n+\tfd = open(sparse_file, O_WRONLY | O_CREAT, 0666);\n+\tif (fd < 0 || write_in_full(fd, quoted_argv, len) != len)\n+\t\tdie_errno(\"could not write %s\", sparse_file);\n+\txwrite(fd, \"\\n\", 1);\n+\tclose(fd);\n+}\n+\n void register_sparse_limits(const char **revlist_argv)\n {\n \tif (revlist_argv != git_sparse_limits) {\ndiff --git a/sparse-repo.h b/sparse-repo.h\nindex ed9018a..2df175f 100644\n--- a/sparse-repo.h\n+++ b/sparse-repo.h\n@@ -1,3 +1,4 @@\n+void record_sparse_limits(const char *quoted_argv);\n void register_sparse_limits(const char **revlist_argv);\n void register_quoted_sparse_limits(char *quoted_argv);\n void check_sparse_repo(void);\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149907","messageId":"1283645647-1891-12-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 11/15] builtin-clone: Accept paths for sparse clone","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:14:03Z","receivedAt":"2010-09-05T00:14:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"\nSigned-off-by: Elijah Newren \n---\n builtin/clone.c | 28 ++++++++++++++++++++--------\n t/t5601-clone.sh | 14 --------------\n transport.h | 3 +++\n 3 files changed, 23 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 19ed640..de0fb66 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -23,6 +23,7 @@\n #include \"branch.h\"\n #include \"remote.h\"\n #include \"run-command.h\"\n+#include \"quote.h\"\n \n /*\n * Overall FIXMEs:\n@@ -33,7 +34,7 @@\n *\n */\n static const char * const builtin_clone_usage[] = {\n-\t\"git clone [options] [--] []\",\n+\t\"git clone [options] [--] [] [-- PATH1 PATH2...]\",\n \tNULL\n };\n \n@@ -375,6 +376,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstruct transport *transport = NULL;\n \tchar *src_ref_prefix = \"refs/heads/\";\n \tint err = 0;\n+\tint rest_argc = 0;\n \n \tstruct refspec *refspec;\n \tconst char *fetch_pattern;\n@@ -382,11 +384,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tjunk_pid = getpid();\n \n \targc = parse_options(argc, argv, prefix, builtin_clone_options,\n-\t\t\t builtin_clone_usage, 0);\n-\n-\tif (argc > 2)\n-\t\tusage_msg_opt(\"Too many arguments.\",\n-\t\t\tbuiltin_clone_usage, builtin_clone_options);\n+\t\t\t builtin_clone_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (argc == 0)\n \t\tusage_msg_opt(\"You must specify a repository to clone.\",\n@@ -418,10 +416,13 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (is_local && option_depth)\n \t\twarning(\"--depth is ignored in local clones; use file:// instead.\");\n \n-\tif (argc == 2)\n+\tif (argc >= 2 && strcmp(argv[1], \"--\") != 0) {\n \t\tdir = xstrdup(argv[1]);\n-\telse\n+\t\trest_argc = 2;\n+\t} else {\n \t\tdir = guess_dir_name(repo_name, is_bundle, option_bare);\n+\t\trest_argc = 1;\n+\t}\n \tstrip_trailing_slashes(dir);\n \n \tdest_exists = !stat(dir, &buf);\n@@ -539,6 +540,17 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t\t option_upload_pack);\n \n+\t\tif (argc > rest_argc) {\n+\t\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\t\tint ret;\n+\n+\t\t\tsq_quote_argv(&buf, &argv[rest_argc], 0);\n+\t\t\tret = transport_set_option(transport, TRANS_OPT_SPARSE,\n+\t\t\t\t\t\t strbuf_detach(&buf, NULL));\n+\t\t\tif (ret)\n+\t\t\t\twarning (\"Sparse clone not supported!\\n\");\n+\t\t}\n+\n \t\trefs = transport_get_remote_refs(transport);\n \t\tif (refs) {\n \t\t\tmapped_refs = wanted_peer_refs(refs, refspec);\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 987e0c8..17c6a30 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -17,20 +17,6 @@ test_expect_success setup '\n \n '\n \n-test_expect_success 'clone with excess parameters (1)' '\n-\n-\trm -fr dst &&\n-\ttest_must_fail git clone -n src dst junk\n-\n-'\n-\n-test_expect_success 'clone with excess parameters (2)' '\n-\n-\trm -fr dst &&\n-\ttest_must_fail git clone -n \"file://$(pwd)/src\" dst junk\n-\n-'\n-\n test_expect_success 'output from clone' '\n \trm -fr dst &&\n \tgit clone -n \"file://$(pwd)/src\" dst >output &&\ndiff --git a/transport.h b/transport.h\nindex e803c0e..41e347a 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -127,6 +127,9 @@ struct transport *transport_get(struct remote *, const char *);\n /* Aggressively fetch annotated tags if possible */\n #define TRANS_OPT_FOLLOWTAGS \"followtags\"\n \n+/* Fetch only certain paths */\n+#define TRANS_OPT_SPARSE \"sparse\"\n+\n /**\n * Returns 0 if the option was used, non-zero otherwise. Prints a\n * message to stderr if the option is not used.\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149910","messageId":"1283645647-1891-13-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 12/15] Pass extra (rev-list) args on, at least in some cases","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:14:04Z","receivedAt":"2010-09-05T00:14:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"\nSigned-off-by: Elijah Newren \n---\n builtin/archive.c | 2 +-\n builtin/clone.c | 2 +-\n builtin/fetch-pack.c | 3 ++-\n builtin/send-pack.c | 3 ++-\n cache.h | 2 +-\n connect.c | 9 ++++++++-\n transport-helper.c | 5 ++++-\n transport.c | 13 +++++++++----\n transport.h | 8 +++++---\n 9 files changed, 33 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/archive.c b/builtin/archive.c\nindex 6a887f5..018d2b6 100644\n--- a/builtin/archive.c\n+++ b/builtin/archive.c\n@@ -35,7 +35,7 @@ static int run_remote_archiver(int argc, const char **argv,\n \tif (!_remote->url[0])\n \t\tdie(\"git archive: Remote with no URL\");\n \ttransport = transport_get(_remote, _remote->url[0]);\n-\ttransport_connect(transport, \"git-upload-archive\", exec, fd);\n+\ttransport_connect(transport, \"git-upload-archive\", exec, NULL, fd);\n \n \tfor (i = 1; i < argc; i++)\n \t\tpacket_write(fd[1], \"argument %s\\n\", argv[i]);\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex de0fb66..5c0f594 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -545,7 +545,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\tint ret;\n \n \t\t\tsq_quote_argv(&buf, &argv[rest_argc], 0);\n-\t\t\tret = transport_set_option(transport, TRANS_OPT_SPARSE,\n+\t\t\tret = transport_set_option(transport, TRANS_OPT_REVLIST_ARGS,\n \t\t\t\t\t\t strbuf_detach(&buf, NULL));\n \t\t\tif (ret)\n \t\t\t\twarning (\"Sparse clone not supported!\\n\");\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex dbd8b7b..e550f3d 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -875,7 +875,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\tfd[0] = 0;\n \t\tfd[1] = 1;\n \t} else {\n-\t\tconn = git_connect(fd, (char *)dest, args.uploadpack,\n+\t\terror(\"Should pass extra_args to git_connect!\");\n+\t\tconn = git_connect(fd, (char *)dest, args.uploadpack, NULL,\n \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 481602d..c3ae328 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -478,7 +478,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\tfd[0] = 0;\n \t\tfd[1] = 1;\n \t} else {\n-\t\tconn = git_connect(fd, dest, receivepack,\n+\t\terror(\"Should pass extra_args to git_connect!\");\n+\t\tconn = git_connect(fd, dest, receivepack, NULL,\n \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \ndiff --git a/cache.h b/cache.h\nindex 6f88dbb..84c0969 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -937,7 +937,7 @@ extern struct ref *find_ref_by_name(const struct ref *list, const char *name);\n \n #define CONNECT_VERBOSE (1u << 0)\n extern char *git_getpass(const char *prompt);\n-extern struct child_process *git_connect(int fd[2], const char *url, const char *prog, int flags);\n+extern struct child_process *git_connect(int fd[2], const char *url, const char *prog, const char *extra_args, int flags);\n extern int finish_connect(struct child_process *conn);\n extern int path_match(const char *path, int nr, char **match);\n struct extra_have_objects {\ndiff --git a/connect.c b/connect.c\nindex 3450cab..42135c8 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -449,7 +449,8 @@ static struct child_process no_fork;\n * the connection failed).\n */\n struct child_process *git_connect(int fd[2], const char *url_orig,\n-\t\t\t\t const char *prog, int flags)\n+\t\t\t\t const char *prog, const char *extra_args,\n+\t\t\t\t int flags)\n {\n \tchar *url;\n \tchar *host, *path;\n@@ -550,6 +551,8 @@ struct child_process *git_connect(int fd[2], const char *url_orig,\n \t\t * Note: Do not add any other headers here! Doing so\n \t\t * will cause older git-daemon servers to crash.\n \t\t */\n+\t\tif (extra_args)\n+\t\t\terror(\"What to do with extra_args?!?\\n\");\n \t\tpacket_write(fd[1],\n \t\t\t \"%s %s%chost=%s%c\",\n \t\t\t prog, path, 0,\n@@ -567,6 +570,10 @@ struct child_process *git_connect(int fd[2], const char *url_orig,\n \tstrbuf_addstr(&cmd, prog);\n \tstrbuf_addch(&cmd, ' ');\n \tsq_quote_buf(&cmd, path);\n+\tif (extra_args) {\n+\t\tstrbuf_addch(&cmd, ' ');\n+\t\tstrbuf_addstr(&cmd, extra_args);\n+\t}\n \tif (cmd.len >= MAX_CMD_LEN)\n \t\tdie(\"command line too long\");\n \ndiff --git a/transport-helper.c b/transport-helper.c\nindex af81fe1..c2b6db3 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -509,7 +509,7 @@ static int process_connect(struct transport *transport,\n }\n \n static int connect_helper(struct transport *transport, const char *name,\n-\t\t const char *exec, int fd[2])\n+\t\t\t const char *exec, const char *extra_args, int fd[2])\n {\n \tstruct helper_data *data = transport->data;\n \n@@ -518,6 +518,9 @@ static int connect_helper(struct transport *transport, const char *name,\n \tif (!data->connect)\n \t\tdie(\"Operation not supported by protocol.\");\n \n+\tif (extra_args)\n+\t\tdie(\"extra_args not handled!!!! OH NOES!!!\");\n+\n \tif (!process_connect_service(transport, name, exec))\n \t\tdie(\"Can't connect to subservice %s.\", name);\n \ndiff --git a/transport.c b/transport.c\nindex 4dba6f8..1d3cab3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -475,6 +475,9 @@ static int set_git_option(struct git_transport_options *opts,\n \t\telse\n \t\t\topts->depth = atoi(value);\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_REVLIST_ARGS)) {\n+\t\topts->revlist_args = value;\n+\t\treturn 0;\n \t}\n \treturn 1;\n }\n@@ -489,6 +492,7 @@ static int connect_setup(struct transport *transport, int for_push, int verbose)\n \tdata->conn = git_connect(data->fd, transport->url,\n \t\t\t\t for_push ? data->options.receivepack :\n \t\t\t\t data->options.uploadpack,\n+\t\t\t\t data->options.revlist_args,\n \t\t\t\t verbose ? CONNECT_VERBOSE : 0);\n \n \treturn 0;\n@@ -805,11 +809,12 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n }\n \n static int connect_git(struct transport *transport, const char *name,\n-\t\t const char *executable, int fd[2])\n+\t\t const char *executable, const char *extra_args,\n+\t\t int fd[2])\n {\n \tstruct git_transport_data *data = transport->data;\n \tdata->conn = git_connect(data->fd, transport->url,\n-\t\t\t\t executable, 0);\n+\t\t\t\t executable, extra_args, 0);\n \tfd[0] = data->fd[0];\n \tfd[1] = data->fd[1];\n \treturn 0;\n@@ -1124,10 +1129,10 @@ void transport_unlock_pack(struct transport *transport)\n }\n \n int transport_connect(struct transport *transport, const char *name,\n-\t\t const char *exec, int fd[2])\n+\t\t const char *exec, const char *extra_args, int fd[2])\n {\n \tif (transport->connect)\n-\t\treturn transport->connect(transport, name, exec, fd);\n+\t\treturn transport->connect(transport, name, exec, extra_args, fd);\n \telse\n \t\tdie(\"Operation not supported by protocol\");\n }\ndiff --git a/transport.h b/transport.h\nindex 41e347a..2669e84 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -11,6 +11,7 @@ struct git_transport_options {\n \tint depth;\n \tconst char *uploadpack;\n \tconst char *receivepack;\n+\tconst char *revlist_args;\n };\n \n struct transport {\n@@ -71,7 +72,8 @@ struct transport {\n \tint (*push_refs)(struct transport *transport, struct ref *refs, int flags);\n \tint (*push)(struct transport *connection, int refspec_nr, const char **refspec, int flags);\n \tint (*connect)(struct transport *connection, const char *name,\n-\t\t const char *executable, int fd[2]);\n+\t\t const char *executable, const char *extra_args,\n+\t\t int fd[2]);\n \n \t/** get_refs_list(), fetch(), and push_refs() can keep\n \t * resources (such as a connection) reserved for futher\n@@ -128,7 +130,7 @@ struct transport *transport_get(struct remote *, const char *);\n #define TRANS_OPT_FOLLOWTAGS \"followtags\"\n \n /* Fetch only certain paths */\n-#define TRANS_OPT_SPARSE \"sparse\"\n+#define TRANS_OPT_REVLIST_ARGS \"revlist_args\"\n \n /**\n * Returns 0 if the option was used, non-zero otherwise. Prints a\n@@ -153,7 +155,7 @@ void transport_take_over(struct transport *transport,\n \t\t\t struct child_process *child);\n \n int transport_connect(struct transport *transport, const char *name,\n-\t\t const char *exec, int fd[2]);\n+\t\t const char *exec, const char *extra_args, int fd[2]);\n \n /* Transport methods defined outside transport.c */\n int transport_helper_init(struct transport *transport, const char *name);\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149911","messageId":"1283645647-1891-14-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 13/15] upload-pack: Handle extra rev-list arguments being passed","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:14:05Z","receivedAt":"2010-09-05T00:14:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"\nSigned-off-by: Elijah Newren \n---\n upload-pack.c | 44 ++++++++++++++++++++++++++++----------------\n 1 files changed, 28 insertions(+), 16 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex ce27a1a..721197b 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -110,6 +110,8 @@ static int do_rev_list(int in, int out, void *user_data)\n {\n \tint i;\n \tstruct rev_info revs;\n+\tint revlist_argc = 0;\n+\tconst char **revlist_argv = user_data;\n \n \tpack_pipe = xfdopen(out, \"w\");\n \tinit_revisions(&revs, NULL);\n@@ -130,7 +132,10 @@ static int do_rev_list(int in, int out, void *user_data)\n \t\to->flags |= UNINTERESTING;\n \t\tadd_pending_object(&revs, o, NULL);\n \t}\n-\tsetup_revisions(0, NULL, &revs, NULL);\n+\tif (revlist_argv)\n+\t\tfor (; revlist_argv[revlist_argc]; revlist_argc++)\n+\t\t\t;\n+\tsetup_revisions(revlist_argc, revlist_argv, &revs, NULL);\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tmark_edges_uninteresting(revs.commits, &revs, show_edge);\n@@ -144,7 +149,7 @@ static int do_rev_list(int in, int out, void *user_data)\n \treturn 0;\n }\n \n-static void create_pack_file(void)\n+static void create_pack_file(const char **revlist_argv)\n {\n \tstruct async rev_list;\n \tstruct child_process pack_objects;\n@@ -156,11 +161,14 @@ static void create_pack_file(void)\n \tssize_t sz;\n \tconst char *argv[10];\n \tint arg = 0;\n+\tint run_revlist = shallow_nr || revlist_argv;\n \n-\tif (shallow_nr) {\n+\tif (run_revlist) {\n \t\tmemset(&rev_list, 0, sizeof(rev_list));\n \t\trev_list.proc = do_rev_list;\n \t\trev_list.out = -1;\n+\t\tif (revlist_argv)\n+\t\t\trev_list.data = revlist_argv;\n \t\tif (start_async(&rev_list))\n \t\t\tdie(\"git upload-pack: unable to fork git-rev-list\");\n \t\targv[arg++] = \"pack-objects\";\n@@ -183,7 +191,7 @@ static void create_pack_file(void)\n \targv[arg++] = NULL;\n \n \tmemset(&pack_objects, 0, sizeof(pack_objects));\n-\tpack_objects.in = shallow_nr ? rev_list.out : -1;\n+\tpack_objects.in = run_revlist ? rev_list.out : -1;\n \tpack_objects.out = -1;\n \tpack_objects.err = -1;\n \tpack_objects.git_cmd = 1;\n@@ -193,7 +201,7 @@ static void create_pack_file(void)\n \t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n \n \t/* pass on revisions we (don't) want */\n-\tif (!shallow_nr) {\n+\tif (!run_revlist) {\n \t\tFILE *pipe_fd = xfdopen(pack_objects.in, \"w\");\n \t\tif (!create_full_pack) {\n \t\t\tint i;\n@@ -307,7 +315,7 @@ static void create_pack_file(void)\n \t\terror(\"git upload-pack: git-pack-objects died with error.\");\n \t\tgoto fail;\n \t}\n-\tif (shallow_nr && finish_async(&rev_list))\n+\tif (run_revlist && finish_async(&rev_list))\n \t\tgoto fail;\t/* error was already reported */\n \n \t/* flush the data */\n@@ -656,7 +664,7 @@ static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag\n \treturn 0;\n }\n \n-static void upload_pack(void)\n+static void upload_pack(const char **revlist_argv)\n {\n \tif (advertise_refs || !stateless_rpc) {\n \t\treset_timeout();\n@@ -673,13 +681,13 @@ static void upload_pack(void)\n \treceive_needs();\n \tif (want_obj.nr) {\n \t\tget_common_commits();\n-\t\tcreate_pack_file();\n+\t\tcreate_pack_file(revlist_argv);\n \t}\n }\n \n-int main(int argc, char **argv)\n+int main(int argc, const char **argv)\n {\n-\tchar *dir;\n+\tchar dir[4096];\n \tint i;\n \tint strict = 0;\n \n@@ -689,7 +697,7 @@ int main(int argc, char **argv)\n \tread_replace_refs = 0;\n \n \tfor (i = 1; i < argc; i++) {\n-\t\tchar *arg = argv[i];\n+\t\tconst char *arg = argv[i];\n \n \t\tif (arg[0] != '-')\n \t\t\tbreak;\n@@ -705,6 +713,9 @@ int main(int argc, char **argv)\n \t\t\tstrict = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--help\")) {\n+\t\t\tusage(upload_pack_usage);\n+\t\t}\n \t\tif (!prefixcmp(arg, \"--timeout=\")) {\n \t\t\ttimeout = atoi(arg+10);\n \t\t\tdaemon_mode = 1;\n@@ -716,12 +727,9 @@ int main(int argc, char **argv)\n \t\t}\n \t}\n \n-\tif (i != argc-1)\n-\t\tusage(upload_pack_usage);\n-\n \tsetup_path();\n \n-\tdir = argv[i];\n+\tstrcpy(dir, argv[i]); /* enter-repo smudges its argument */\n \n \tif (!enter_repo(dir, strict))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n@@ -729,6 +737,10 @@ int main(int argc, char **argv)\n \t\tdie(\"attempt to fetch/clone from a shallow repository\");\n \tif (getenv(\"GIT_DEBUG_SEND_PACK\"))\n \t\tdebug_fd = atoi(getenv(\"GIT_DEBUG_SEND_PACK\"));\n-\tupload_pack();\n+\t/*\n+\t * revlist arguments for revision walking start at argv[i+1], but\n+\t * setup_revisions() ignores the first argument.\n+\t */\n+\tupload_pack(argc > i+1 ? &argv[i] : NULL);\n \treturn 0;\n }\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149913","messageId":"1283645647-1891-15-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 14/15] EVIL COMMIT: Include all commits","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:14:06Z","receivedAt":"2010-09-05T00:14:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"The correct fix is to include the *relevant pieces* of all commits. But\nthis is a quick stopgap that lets me test some stuff.\n\nSigned-off-by: Elijah Newren \n---\n revision.c | 5 +++--\n revision.h | 3 ++-\n upload-pack.c | 1 +\n 3 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 67b1a1d..c411bda 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -335,7 +335,7 @@ static int rev_compare_tree(struct rev_info *revs, struct commit *parent, struct\n \ttree_difference = REV_TREE_SAME;\n \tDIFF_OPT_CLR(&revs->pruning, HAS_CHANGES);\n \tif (diff_tree_sha1(t1->object.sha1, t2->object.sha1, \"\",\n-\t\t\t &revs->pruning) < 0)\n+\t\t\t &revs->pruning) < 0 || revs->sparse_traversal)\n \t\treturn REV_TREE_DIFFERENT;\n \treturn tree_difference;\n }\n@@ -380,7 +380,8 @@ static void try_to_simplify_commit(struct rev_info *revs, struct commit *commit)\n \t\treturn;\n \n \tif (!commit->parents) {\n-\t\tif (rev_same_tree_as_empty(revs, commit))\n+\t\tif (rev_same_tree_as_empty(revs, commit) &&\n+\t\t !revs->sparse_traversal)\n \t\t\tcommit->object.flags |= TREESAME;\n \t\treturn;\n \t}\ndiff --git a/revision.h b/revision.h\nindex ba879c9..17865e8 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -85,7 +85,8 @@ struct rev_info {\n \t\t\tignore_merges:1,\n \t\t\tcombine_merges:1,\n \t\t\tdense_combined_merges:1,\n-\t\t\talways_show_header:1;\n+\t\t\talways_show_header:1,\n+\t\t\tsparse_traversal:1;\n \n \t/* Format info */\n \tunsigned int\tshown_one:1,\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 721197b..a58b350 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -120,6 +120,7 @@ static int do_rev_list(int in, int out, void *user_data)\n \trevs.blob_objects = 1;\n \tif (use_thin_pack)\n \t\trevs.edge_hint = 1;\n+\trevs.sparse_traversal = 1;\n \n \tfor (i = 0; i < want_obj.nr; i++) {\n \t\tstruct object *o = want_obj.objects[i].item;\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149914","messageId":"1283645647-1891-16-git-send-email-newren@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-1-git-send-email-newren@gmail.com","subject":"[RFC PATCH 15/15] clone: Ensure sparse limiting arguments are used in subsequent operations","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T00:14:07Z","receivedAt":"2010-09-05T00:14:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"After fetching, register sparse args for subsequent checkout portion of\nclone, and record sparse args for any later commands run in the clone.\n\nSigned-off-by: Elijah Newren \n---\n builtin/clone.c | 17 +++++++++++++----\n t/t5721-sparse-repository-communication.sh | 4 ++--\n 2 files changed, 15 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 5c0f594..4f55ab0 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -24,6 +24,7 @@\n #include \"remote.h\"\n #include \"run-command.h\"\n #include \"quote.h\"\n+#include \"sparse-repo.h\"\n \n /*\n * Overall FIXMEs:\n@@ -370,7 +371,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tconst struct ref *refs, *remote_head;\n \tconst struct ref *remote_head_points_at;\n \tconst struct ref *our_head_points_at;\n-\tstruct ref *mapped_refs;\n+\tstruct ref *mapped_refs = NULL;\n \tstruct strbuf key = STRBUF_INIT, value = STRBUF_INIT;\n \tstruct strbuf branch_top = STRBUF_INIT, reflog_msg = STRBUF_INIT;\n \tstruct transport *transport = NULL;\n@@ -519,9 +520,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstrbuf_reset(&value);\n \n \tif (is_local) {\n+\t\tif (argc > rest_argc)\n+\t\t\tdie(\"Sparse clones of local repositories must be done using the file:// protocol.\");\n \t\trefs = clone_local(path, git_dir);\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\n \t} else {\n+\t\tstruct strbuf quoted_sparse_args = STRBUF_INIT;\n \t\tstruct remote *remote = remote_get(option_origin);\n \t\ttransport = transport_get(remote, remote->url[0]);\n \n@@ -541,12 +545,11 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t option_upload_pack);\n \n \t\tif (argc > rest_argc) {\n-\t\t\tstruct strbuf buf = STRBUF_INIT;\n \t\t\tint ret;\n \n-\t\t\tsq_quote_argv(&buf, &argv[rest_argc], 0);\n+\t\t\tsq_quote_argv("ed_sparse_args, &argv[rest_argc], 0);\n \t\t\tret = transport_set_option(transport, TRANS_OPT_REVLIST_ARGS,\n-\t\t\t\t\t\t strbuf_detach(&buf, NULL));\n+\t\t\t\t\t\t quoted_sparse_args.buf);\n \t\t\tif (ret)\n \t\t\t\twarning (\"Sparse clone not supported!\\n\");\n \t\t}\n@@ -556,6 +559,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\tmapped_refs = wanted_peer_refs(refs, refspec);\n \t\t\ttransport_fetch_refs(transport, mapped_refs);\n \t\t}\n+\n+\t\tif (argc > rest_argc) {\n+\t\t\tregister_sparse_limits(&argv[rest_argc]);\n+\t\t\trecord_sparse_limits(quoted_sparse_args.buf);\n+\t\t}\n+\t\tstrbuf_release("ed_sparse_args);\n \t}\n \n \tif (refs) {\ndiff --git a/t/t5721-sparse-repository-communication.sh b/t/t5721-sparse-repository-communication.sh\nindex 35cb6a2..61ce8c6 100755\n--- a/t/t5721-sparse-repository-communication.sh\n+++ b/t/t5721-sparse-repository-communication.sh\n@@ -52,7 +52,7 @@ test_expect_success 'make comparison sparse repository' '\n \t)\n '\n \n-test_expect_failure 'basic sparse clone succeeds' '\n+test_expect_success 'basic sparse clone succeeds' '\n \trm -fr dst &&\n \tgit clone \"file://$(pwd)/src\" dst -- sub/b/ &&\n \t(\n@@ -82,7 +82,7 @@ test_expect_failure 'basic sparse clone succeeds' '\n \t)\n '\n \n-test_expect_failure 'basic sparse clone guts match expectations' '\n+test_expect_success 'basic sparse clone guts match expectations' '\n \t(\n \t\t# Loose objects only, to facilitate comparison\n \t\tcd dst &&\n-- \n1.7.2.2.140.gd06af\n"},{"id":"149921","messageId":"AANLkTik1E=7Uawq-fGXppac3ur-oohq104A5wEgxv8cL@mail.gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-7-git-send-email-newren@gmail.com","subject":"Re: [RFC PATCH 06/15] Automatically reuse sparse limiting arguments in revision walking","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-05T01:58:08Z","receivedAt":"2010-09-05T01:58:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n> In a sparse repository, by automatically making use of sparse limits\n> specified at clone time, we can avoid walking uninteresting commits and\n> prevent access to missing trees and blobs.  Note that this means that if\n> you created a sparse clone with\n>  git clone -- PATH1 PATH2 PATH3 PATH4 PATH5\n> then\n>  git log\n> implicitly runs as though you had manually specified\n>  git log -- PATH1 PATH2 PATH3 PATH4 PATH5\n> Similarly, running\n>  git diff\n> implicitly runs as though you had manually specified\n>  git diff -- PATH1 PATH2 PATH3 PATH4 PATH5\n>\n> This is necessary for proper operation of git diff in a sparse clone to\n> avoid accessing missing objects.  In the case of a plain git log, this\n> merely serves as an additional convenience, but for more complicated log\n> operations (e.g. when passing -p or -S options) it becomes necessary.\n\nI have a nicer approach here. Instead of modifying setup_revisions()\nand similar functions, I update get_pathspec() to rewrite the\npathspecs from command line in narrow/shallow repos. get_pathspec()\ncurrently does some form of rewriting already (prepending prefix).\n\nSo if you do \"git log\", get_pathspec() would return \"git log -- PATH1\nPATH2...\". Will repost my series soon, or I can send that particular\npatch to you.\n-- \nDuy\n"},{"id":"149922","messageId":"AANLkTinkz1YG45t3njtzj--LcPs3MPbGMTj9cWG_Nj6B@mail.gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-6-git-send-email-newren@gmail.com","subject":"Re: [RFC PATCH 05/15] read_tree_recursive: Avoid missing blobs and trees in a sparse repository","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-05T02:00:24Z","receivedAt":"2010-09-05T02:00:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n> @@ -119,6 +119,11 @@ int read_tree_recursive(struct tree *tree,\n>                default:\n>                        return -1;\n>                }\n> +\n> +               if (git_sparse_pathspecs &&\n> +                   sha1_object_info(entry.sha1, NULL) <= 0)\n> +                       continue;\n> +\n\nI suppose this is temporary and the final solution would have stricter\nchecks (i.e. only ignore if those entries are outside sparse zone)?\nThis opens a door for broken repo.\n-- \nDuy\n"},{"id":"149926","messageId":"AANLkTikx89M+JcOcabU3TazGB=k8x39QLbVe7sH7Vvaa@mail.gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-2-git-send-email-newren@gmail.com","subject":"Re: [RFC PATCH 01/15] README-sparse-clone: Add a basic writeup of my ideas for sparse clones","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-05T03:01:06Z","receivedAt":"2010-09-05T03:01:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n> +To ensure minimum necessary connectivity, we also download basic\n> +information from otherwise excluded commits\n> +  * parents of these commits\n> +  * trees matching the specified sparse path(s)\n> +but, for security and space reasons, do not download\n> +  * author\n> +  * author date\n> +  * committer\n> +  * committer date\n> +  * log message\n> +Such commits are still considered \"missing\" (see item I4 for more\n> +details about how we handle \"missing\" commits).\n\nJust an observation. When I ran pack-objects with irrelevant commits\nremoved (i.e. try_to_simplify_commit) on Documentation/, I got a 6MB\npack. When I ran it without commit simplification, I got 16MB pack.\nThat's 10MB larger.\n\nNow I don't how much of that 10MB share is commit messages, authors,\ncommitters and trees but I suspect trees take a large part in it.\nMaybe you can just fake the trees in those fake commits as well, to\navoid downloading more trees.\n-- \nDuy\n"},{"id":"149928","messageId":"AANLkTikxCwSOBVREuRc7sShJahKR5FXWdaW79f_K36bU@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTikx89M+JcOcabU3TazGB=k8x39QLbVe7sH7Vvaa@mail.gmail.com","subject":"Re: [RFC PATCH 01/15] README-sparse-clone: Add a basic writeup of my ideas for sparse clones","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T03:13:13Z","receivedAt":"2010-09-05T03:13:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Sep 4, 2010 at 9:01 PM, Nguyen Thai Ngoc Duy wrote:\n> On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n>> +To ensure minimum necessary connectivity, we also download basic\n>> +information from otherwise excluded commits\n>> +  * parents of these commits\n>> +  * trees matching the specified sparse path(s)\n>> +but, for security and space reasons, do not download\n>> +  * author\n>> +  * author date\n>> +  * committer\n>> +  * committer date\n>> +  * log message\n>> +Such commits are still considered \"missing\" (see item I4 for more\n>> +details about how we handle \"missing\" commits).\n>\n> Just an observation. When I ran pack-objects with irrelevant commits\n> removed (i.e. try_to_simplify_commit) on Documentation/, I got a 6MB\n> pack. When I ran it without commit simplification, I got 16MB pack.\n> That's 10MB larger.\n\nHmm. I get 22MB pack for a full clone of git.git and 13 MB for a\nsparse clone with Documentation/; that's including all commits too. I\nwonder why our numbers differ by 3 MB. Weird.\n\n> Now I don't how much of that 10MB share is commit messages, authors,\n> committers and trees but I suspect trees take a large part in it.\n> Maybe you can just fake the trees in those fake commits as well, to\n> avoid downloading more trees.\n\nI originally planned to do that, but that makes working with tags and\nbranches difficult. For example, documenters could clone a repository\nbut be unable to make new commits on top of maint or master since we\nprobably wouldn't have trees for the tips of those branches.\n\nSo I really think trees are needed. Of course, for someone making a\nsparse clone of \"just\" the Documentation directory will need the\ntoplevel trees, but they won't need trees under t/, for example. So\nthey do get some savings.\n"},{"id":"149929","messageId":"AANLkTimsqDbtXArL_EQc+ZCqOd=tYWJ1ZXZpa7T5qUiH@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTinkz1YG45t3njtzj--LcPs3MPbGMTj9cWG_Nj6B@mail.gmail.com","subject":"Re: [RFC PATCH 05/15] read_tree_recursive: Avoid missing blobs and trees in a sparse repository","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T03:16:38Z","receivedAt":"2010-09-05T03:16:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Sep 4, 2010 at 8:00 PM, Nguyen Thai Ngoc Duy wrote:\n> On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n>> @@ -119,6 +119,11 @@ int read_tree_recursive(struct tree *tree,\n>>                default:\n>>                        return -1;\n>>                }\n>> +\n>> +               if (git_sparse_pathspecs &&\n>> +                   sha1_object_info(entry.sha1, NULL) <= 0)\n>> +                       continue;\n>> +\n>\n> I suppose this is temporary and the final solution would have stricter\n> checks (i.e. only ignore if those entries are outside sparse zone)?\n> This opens a door for broken repo.\n\nYes, good catch. Looks like I somehow missed that one, but I agree,\nthere should be an \"&& !tree_entry_interesting(...)\" in there.\n"},{"id":"149934","messageId":"AANLkTimH_iMvvOadoCx_DJybDpzS3eNe92-6ywzdgUUk@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTimsqDbtXArL_EQc+ZCqOd=tYWJ1ZXZpa7T5qUiH@mail.gmail.com","subject":"Re: [RFC PATCH 05/15] read_tree_recursive: Avoid missing blobs and trees in a sparse repository","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T04:31:33Z","receivedAt":"2010-09-05T04:31:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Sep 4, 2010 at 9:16 PM, Elijah Newren wrote:\n> On Sat, Sep 4, 2010 at 8:00 PM, Nguyen Thai Ngoc Duy wrote:\n>> On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n>>> @@ -119,6 +119,11 @@ int read_tree_recursive(struct tree *tree,\n>>>                default:\n>>>                        return -1;\n>>>                }\n>>> +\n>>> +               if (git_sparse_pathspecs &&\n>>> +                   sha1_object_info(entry.sha1, NULL) <= 0)\n>>> +                       continue;\n>>> +\n>>\n>> I suppose this is temporary and the final solution would have stricter\n>> checks (i.e. only ignore if those entries are outside sparse zone)?\n>> This opens a door for broken repo.\n>\n> Yes, good catch.  Looks like I somehow missed that one, but I agree,\n> there should be an \"&& !tree_entry_interesting(...)\" in there.\n\nSorry, now I remember why that isn't in there and can't be. I did\nhave it there at one point. However, base & baselen in\nread_tree_recursive do not necessarily correspond to the relative path\nfrom the toplevel of the repository, though the sparse limit pathspecs\nalways will. For example, running\n git ls-tree master:Documentation/technical\nor, equivalently (for current git.git)\n git ls-tree da0ae7c59bb0df4c13554fd840e1a563cde659ea\nthen base will be \"\" for paths under Documentation/technical rather\nthan \"Documenation/technical/\". And there's really no way of\ndetermining the \"real base\" either in order to apply matching to the\nsparse limit pathspecs (well...I guess you could do an exhaustive walk\nof all history each time, so long as the given sha1sum only appears as\none particular directory, but that's unrealistic and slow and leaves\nopen what to do when multiple directories at different paths happen to\nhave the same sha1sum at some point(s) in their history). Note that\nthis affects cat-file -p as well, since it calls ls-tree and thus this\ncode.\n\nI really don't see how one can change this. However, if it's any\nconsolation, sha1_object_info() will print out a warning message\nwhenever it's asked for a sha1sum that cannot be found in the object\nstore. For example, in a sparse clone:\n\n$ git ls-tree -rt master\n040000 tree f98bf35e9a746ebbd5a706591fe1ea4942942bad sub\n040000 tree 436913a91c5648a6ed8fa23719fbd6e3052e2597 sub/a\nerror: unable to find 436913a91c5648a6ed8fa23719fbd6e3052e2597\n040000 tree 07753f428765ac1afe2020b24e40785869bd4a85 sub/b\n100644 blob d95f3ad14dee633a758d2e331151e950dd13e4ed sub/b/file\n040000 tree 07753f428765ac1afe2020b24e40785869bd4a85 sub/bcopy\n100644 blob d95f3ad14dee633a758d2e331151e950dd13e4ed sub/bcopy/file\n100644 blob 4b7b65e07a8641bcd14375ebddf5c8a7fc002a30 sub/file\n\nIt can't traverse into sub/a because it doesn't have the necessary\ninformation. I figured the warning message was useful to the end user\nas a reminder that they are in a sparse clone. (Such warning messages\ndon't affect cat-file -p, because its callbacks don't return\nREAD_TREE_RECURSIVE, making sure it doesn't trigger this part of the\ncode.)\n"},{"id":"149935","messageId":"AANLkTi=ywOZfkxfr1Gt3AMMbWDphKjTBudG=E4+xer-P@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTik1E=7Uawq-fGXppac3ur-oohq104A5wEgxv8cL@mail.gmail.com","subject":"Re: [RFC PATCH 06/15] Automatically reuse sparse limiting arguments in revision walking","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T04:50:34Z","receivedAt":"2010-09-05T04:50:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Sep 4, 2010 at 7:58 PM, Nguyen Thai Ngoc Duy wrote:\n> On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n>> In a sparse repository, by automatically making use of sparse limits\n>> specified at clone time, we can avoid walking uninteresting commits and\n\n>\n> I have a nicer approach here. Instead of modifying setup_revisions()\n> and similar functions, I update get_pathspec() to rewrite the\n> pathspecs from command line in narrow/shallow repos. get_pathspec()\n> currently does some form of rewriting already (prepending prefix).\n>\n> So if you do \"git log\", get_pathspec() would return \"git log -- PATH1\n> PATH2...\". Will repost my series soon, or I can send that particular\n> patch to you.\n\nOoh, that's clever. I like it. Do you also do verification that any\npaths specified by the user are a subset of the paths the sparse clone\nis limited to? That'd be really nice. Much better than my simple\ndumb \"just append\" logic.\n\nHow do you reconcile conflicting needs, though? For most cases,\nwhenever the user specifies paths, get_pathspec should make sure those\npaths are a subset of the sparse paths (throwing an error if they're\nnot) and then just use the user-specified ones. However, I think\ndoing this in all cases would break ls-files/cat-file -p, as the\npathspecs passed to those are not necessarily rooted at the toplevel\nof the repository (and the prefix doesn't correct for that either).\n\nAlso, when no paths are specified by the user, the correct answer can\nstill vary by command. ls-tree should be given no pathspecs in such a\ncase, even in a sparse clone. Nor should commit (it wouldn't hurt too\nbad to send it the sparse pathspecs, though it'd send commit through\nthe \"partial commit\" section of the code base each time rather than\nthe \"as-is commit\" section, which seems suboptimal). And on the other\nend of the spectrum, log/rev-list/diff should all take the sparse\npathspecs in such a case.\n"},{"id":"149958","messageId":"AANLkTimw89nkui5UdJPu-s5yQ=mP4wtw0tKTZxsq2L4W@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTi=ywOZfkxfr1Gt3AMMbWDphKjTBudG=E4+xer-P@mail.gmail.com","subject":"Re: [RFC PATCH 06/15] Automatically reuse sparse limiting arguments in revision walking","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-05T07:12:07Z","receivedAt":"2010-09-05T07:12:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Sep 5, 2010 at 2:50 PM, Elijah Newren wrote:\n> On Sat, Sep 4, 2010 at 7:58 PM, Nguyen Thai Ngoc Duy wrote:\n>> On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n>>> In a sparse repository, by automatically making use of sparse limits\n>>> specified at clone time, we can avoid walking uninteresting commits and\n>\n>>\n>> I have a nicer approach here. Instead of modifying setup_revisions()\n>> and similar functions, I update get_pathspec() to rewrite the\n>> pathspecs from command line in narrow/shallow repos. get_pathspec()\n>> currently does some form of rewriting already (prepending prefix).\n>>\n>> So if you do \"git log\", get_pathspec() would return \"git log -- PATH1\n>> PATH2...\". Will repost my series soon, or I can send that particular\n>> patch to you.\n>\n> Ooh, that's clever.  I like it.  Do you also do verification that any\n> paths specified by the user are a subset of the paths the sparse clone\n> is limited to?  That'd be really nice.  Much better than my simple\n> dumb \"just append\" logic.\n\nI do.\n\n> How do you reconcile conflicting needs, though?  For most cases,\n> whenever the user specifies paths, get_pathspec should make sure those\n> paths are a subset of the sparse paths (throwing an error if they're\n> not) and then just use the user-specified ones.  However, I think\n> doing this in all cases would break ls-files/cat-file -p, as the\n> pathspecs passed to those are not necessarily rooted at the toplevel\n> of the repository (and the prefix doesn't correct for that either).\n\nYes, some commands will just skip the pathspec rewrite. It's up to\nthose commands to rewrite pathspec themselves.\n\nSo get_pathspec() will have pathspec rewrite by default. Other\ncommands will use another variant of it (i.e. get_pathspec_narrow()),\nwhich allows to skip the rewrite completely.\n-- \nDuy\n"},{"id":"149961","messageId":"AANLkTi=ijbaATavcujUY-WnEfKFKrNue_kP6vSngKSvQ@mail.gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-8-git-send-email-newren@gmail.com","subject":"Re: [RFC PATCH 07/15] cache_tree_update(): Capability to handle tree entries missing from index","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-05T07:54:54Z","receivedAt":"2010-09-05T07:54:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n> cache_tree_update() will write trees using the index.  With sparse clones,\n> the index will only contain entries matching the sparse limits, meaning\n> that the index does not provide enough information to write complete tree\n> objects.  Having cache_tree_update() take a tree (typically HEAD), will\n> allow new complete trees to be constructed by using entries from the\n> specified tree outside the sparse limits together with the index.\n\nYou are moving it closer to the index (from my view because I changed\nin commit_tree()). This makes me think, why don't you move the base\ntree into the index itself?\n\nThe index is supposed to save the image of full worktree. While you\ndon't have all path names, you have the clue to all of them, the base\ntree. To me, that means it belongs to the index. That would reduce\ncode change to\n - cache-tree.c (generate new tree from the base tree and index)\n - read-cache.c (new sparse-clone index extension)\n - index writing operations (save the base tree in index): read-tree and merge\n-- \nDuy\n"},{"id":"150000","messageId":"AANLkTikK9vGgtzrrAAyqptPVHM2wjOjK_cH6GB0rgewP@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTi=ijbaATavcujUY-WnEfKFKrNue_kP6vSngKSvQ@mail.gmail.com","subject":"Re: [RFC PATCH 07/15] cache_tree_update(): Capability to handle tree entries missing from index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-05T21:09:24Z","receivedAt":"2010-09-05T21:09:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"n Sun, Sep 5, 2010 at 1:54 AM, Nguyen Thai Ngoc Duy wrote:\n> On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n>> cache_tree_update() will write trees using the index.  With sparse clones,\n>> the index will only contain entries matching the sparse limits, meaning\n>> that the index does not provide enough information to write complete tree\n>> objects.  Having cache_tree_update() take a tree (typically HEAD), will\n>> allow new complete trees to be constructed by using entries from the\n>> specified tree outside the sparse limits together with the index.\n>\n> You are moving it closer to the index (from my view because I changed\n> in commit_tree()). This makes me think, why don't you move the base\n> tree into the index itself?\n>\n> The index is supposed to save the image of full worktree. While you\n> don't have all path names, you have the clue to all of them, the base\n> tree. To me, that means it belongs to the index. That would reduce\n> code change to\n>  - cache-tree.c (generate new tree from the base tree and index)\n>  - read-cache.c (new sparse-clone index extension)\n>  - index writing operations (save the base tree in index): read-tree and merge\n\nThat's a really good idea. I'll look into that.\n"},{"id":"150029","messageId":"AANLkTimtOH65n_XNdCHPmUS77Xxk5MAuQDqbh401wgmb@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTikxCwSOBVREuRc7sShJahKR5FXWdaW79f_K36bU@mail.gmail.com","subject":"Re: [RFC PATCH 01/15] README-sparse-clone: Add a basic writeup of my ideas for sparse clones","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-06T03:14:16Z","receivedAt":"2010-09-06T03:14:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Sep 5, 2010 at 1:13 PM, Elijah Newren wrote:\n>> Just an observation. When I ran pack-objects with irrelevant commits\n>> removed (i.e. try_to_simplify_commit) on Documentation/, I got a 6MB\n>> pack. When I ran it without commit simplification, I got 16MB pack.\n>> That's 10MB larger.\n>\n> Hmm.  I get 22MB pack for a full clone of git.git and 13 MB for a\n> sparse clone with Documentation/; that's including all commits too.  I\n> wonder why our numbers differ by 3 MB.  Weird.\n\nHmm.. just got a fresh alt-git.git from repo.or.cz and did\nnarrow/sparse clone from that. \"git clone\" reported 16.27MB for both\nnarrow/sparse version.\n-- \nDuy\n"},{"id":"150032","messageId":"AANLkTinufi3TwnPZ+smq5FE5S3zZdQSzKpqYv=hVfcLV@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTikK9vGgtzrrAAyqptPVHM2wjOjK_cH6GB0rgewP@mail.gmail.com","subject":"Re: [RFC PATCH 07/15] cache_tree_update(): Capability to handle tree entries missing from index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-06T04:42:53Z","receivedAt":"2010-09-06T04:42:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 5, 2010 at 3:09 PM, Elijah Newren wrote:\n> n Sun, Sep 5, 2010 at 1:54 AM, Nguyen Thai Ngoc Duy wrote:\n>> On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n>>> cache_tree_update() will write trees using the index.  With sparse clones,\n>>> the index will only contain entries matching the sparse limits, meaning\n>>> that the index does not provide enough information to write complete tree\n>>> objects.  Having cache_tree_update() take a tree (typically HEAD), will\n>>> allow new complete trees to be constructed by using entries from the\n>>> specified tree outside the sparse limits together with the index.\n>>\n>> You are moving it closer to the index (from my view because I changed\n>> in commit_tree()). This makes me think, why don't you move the base\n>> tree into the index itself?\n>>\n>> The index is supposed to save the image of full worktree. While you\n>> don't have all path names, you have the clue to all of them, the base\n>> tree. To me, that means it belongs to the index. That would reduce\n>> code change to\n>>  - cache-tree.c (generate new tree from the base tree and index)\n>>  - read-cache.c (new sparse-clone index extension)\n>>  - index writing operations (save the base tree in index): read-tree and merge\n>\n> That's a really good idea.  I'll look into that.\n\nHmm..maybe before I get ahead of myself, I should back up for a\nsecond. Which way do we think is better -- handling this in\ncache_tree_update() or doing a fixup in commit_tree()? If the latter,\nthen I should just drop my 7/15 and 8/15 for your 13/17 and 14/17. If\nthe former, then it makes sense to look into this change you suggest.\nIn that case, we'd probably keep my 7/15 but drop 8/15 for patch(es)\nthat implement your idea above. But you're more familiar with the\nindex format than I am and it's your idea, so you'd probably be the\nbetter one to implement it.\n\nThoughts?\n\nElijah\n"},{"id":"150033","messageId":"1283748429-31076-1-git-send-email-pclouds@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-8-git-send-email-newren@gmail.com","subject":"[PATCH 0/4] en/object-list-with-pathspec update","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-06T04:47:05Z","receivedAt":"2010-09-06T04:47:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n> diff --git a/cache-tree.c b/cache-tree.c\n> index c60cf91..2ba6a76 100644\n> --- a/cache-tree.c\n> +++ b/cache-tree.c\n> @@ -2,6 +2,7 @@\n> #include \"tree.h\"\n> #include \"tree-walk.h\"\n> #include \"cache-tree.h\"\n> +#include \"diff.h\" /* FIXME: for tree_entry_interesting; maybe it should be in tree-walk.h? */\n\nYes, please. And I'd rather see it done sooner than later, before\ntree_entry_interesting() usage is spread over the place.\n\nElijah Newren (2):\n Add testcases showing how pathspecs are ignored with rev-list\n --objects\n Make rev-list --objects work together with pathspecs\n\nNguyễn Thái Ngọc Duy (2):\n tree-walk: copy tree_entry_interesting() as-is from tree-diff.c\n tree-walk: actually move tree_entry_interesting() to tree-walk.c\n\n list-objects.c | 25 ++++++++++\n revision.c | 8 ++-\n revision.h | 3 +-\n t/t6000-rev-list-misc.sh | 51 +++++++++++++++++++\n tree-diff.c | 121 ++--------------------------------------------\n tree-walk.c | 110 +++++++++++++++++++++++++++++++++++++++++\n tree-walk.h | 3 +\n 7 files changed, 201 insertions(+), 120 deletions(-)\n create mode 100755 t/t6000-rev-list-misc.sh\n"},{"id":"150035","messageId":"1283748429-31076-2-git-send-email-pclouds@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-8-git-send-email-newren@gmail.com","subject":"[PATCH 1/4] Add testcases showing how pathspecs are ignored with rev-list --objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-06T04:47:06Z","receivedAt":"2010-09-06T04:47:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \nSigned-off-by: Junio C Hamano \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n No changes.\n\n t/t6000-rev-list-misc.sh | 51 ++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 51 insertions(+), 0 deletions(-)\n create mode 100755 t/t6000-rev-list-misc.sh\n\ndiff --git a/t/t6000-rev-list-misc.sh b/t/t6000-rev-list-misc.sh\nnew file mode 100755\nindex 0000000..b3c1dd8\n--- /dev/null\n+++ b/t/t6000-rev-list-misc.sh\n@@ -0,0 +1,51 @@\n+#!/bin/sh\n+\n+test_description='miscellaneous rev-list tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success setup '\n+\techo content1 >wanted_file &&\n+\techo content2 >unwanted_file &&\n+\tgit add wanted_file unwanted_file &&\n+\tgit commit -m one\n+'\n+\n+test_expect_failure 'rev-list --objects heeds pathspecs' '\n+\tgit rev-list --objects HEAD -- wanted_file >output &&\n+\tgrep wanted_file output &&\n+\t! grep unwanted_file output\n+'\n+\n+test_expect_failure 'rev-list --objects with pathspecs and deeper paths' '\n+\tmkdir foo &&\n+\t>foo/file &&\n+\tgit add foo/file &&\n+\tgit commit -m two &&\n+\n+\tgit rev-list --objects HEAD -- foo >output &&\n+\tgrep foo/file output &&\n+\n+\tgit rev-list --objects HEAD -- foo/file >output &&\n+\tgrep foo/file output &&\n+\t! grep unwanted_file output\n+'\n+\n+test_expect_failure 'rev-list --objects with pathspecs and copied files' '\n+\tgit checkout --orphan junio-testcase &&\n+\tgit rm -rf . &&\n+\n+\tmkdir two &&\n+\techo frotz >one &&\n+\tcp one two/three &&\n+\tgit add one two/three &&\n+\ttest_tick &&\n+\tgit commit -m that &&\n+\n+\tONE=$(git rev-parse HEAD:one)\n+\tgit rev-list --objects HEAD two >output &&\n+\tgrep \"$ONE two/three\" output &&\n+\t! grep one output\n+'\n+\n+test_done\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"150034","messageId":"1283748429-31076-3-git-send-email-pclouds@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-8-git-send-email-newren@gmail.com","subject":"[PATCH 2/4] tree-walk: copy tree_entry_interesting() as is from tree-diff.c","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-06T04:47:07Z","receivedAt":"2010-09-06T04:47:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Just a straight copy. The function is not used anywhere. It is to\nseparate changes that will be made to this function in the next patch.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n tree-walk.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 115 insertions(+), 0 deletions(-)\n\ndiff --git a/tree-walk.c b/tree-walk.c\nindex a9bbf4e..bc83fa3 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -2,6 +2,7 @@\n #include \"tree-walk.h\"\n #include \"unpack-trees.h\"\n #include \"tree.h\"\n+#include \"diff.h\"\n \n static const char *get_mode(const char *str, unsigned int *modep)\n {\n@@ -455,3 +456,117 @@ int get_tree_entry(const unsigned char *tree_sha1, const char *name, unsigned ch\n \tfree(tree);\n \treturn retval;\n }\n+\n+/*\n+ * Is a tree entry interesting given the pathspec we have?\n+ *\n+ * Return:\n+ * - 2 for \"yes, and all subsequent entries will be\"\n+ * - 1 for yes\n+ * - zero for no\n+ * - negative for \"no, and no subsequent entries will be either\"\n+ */\n+static int tree_entry_interesting(struct tree_desc *desc, const char *base, int baselen, struct diff_options *opt)\n+{\n+\tconst char *path;\n+\tconst unsigned char *sha1;\n+\tunsigned mode;\n+\tint i;\n+\tint pathlen;\n+\tint never_interesting = -1;\n+\n+\tif (!opt->nr_paths)\n+\t\treturn 1;\n+\n+\tsha1 = tree_entry_extract(desc, &path, &mode);\n+\n+\tpathlen = tree_entry_len(path, sha1);\n+\n+\tfor (i = 0; i < opt->nr_paths; i++) {\n+\t\tconst char *match = opt->paths[i];\n+\t\tint matchlen = opt->pathlens[i];\n+\t\tint m = -1; /* signals that we haven't called strncmp() */\n+\n+\t\tif (baselen >= matchlen) {\n+\t\t\t/* If it doesn't match, move along... */\n+\t\t\tif (strncmp(base, match, matchlen))\n+\t\t\t\tcontinue;\n+\n+\t\t\t/*\n+\t\t\t * If the base is a subdirectory of a path which\n+\t\t\t * was specified, all of them are interesting.\n+\t\t\t */\n+\t\t\tif (!matchlen ||\n+\t\t\t base[matchlen] == '/' ||\n+\t\t\t match[matchlen - 1] == '/')\n+\t\t\t\treturn 2;\n+\n+\t\t\t/* Just a random prefix match */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* Does the base match? */\n+\t\tif (strncmp(base, match, baselen))\n+\t\t\tcontinue;\n+\n+\t\tmatch += baselen;\n+\t\tmatchlen -= baselen;\n+\n+\t\tif (never_interesting) {\n+\t\t\t/*\n+\t\t\t * We have not seen any match that sorts later\n+\t\t\t * than the current path.\n+\t\t\t */\n+\n+\t\t\t/*\n+\t\t\t * Does match sort strictly earlier than path\n+\t\t\t * with their common parts?\n+\t\t\t */\n+\t\t\tm = strncmp(match, path,\n+\t\t\t\t (matchlen < pathlen) ? matchlen : pathlen);\n+\t\t\tif (m < 0)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/*\n+\t\t\t * If we come here even once, that means there is at\n+\t\t\t * least one pathspec that would sort equal to or\n+\t\t\t * later than the path we are currently looking at.\n+\t\t\t * In other words, if we have never reached this point\n+\t\t\t * after iterating all pathspecs, it means all\n+\t\t\t * pathspecs are either outside of base, or inside the\n+\t\t\t * base but sorts strictly earlier than the current\n+\t\t\t * one. In either case, they will never match the\n+\t\t\t * subsequent entries. In such a case, we initialized\n+\t\t\t * the variable to -1 and that is what will be\n+\t\t\t * returned, allowing the caller to terminate early.\n+\t\t\t */\n+\t\t\tnever_interesting = 0;\n+\t\t}\n+\n+\t\tif (pathlen > matchlen)\n+\t\t\tcontinue;\n+\n+\t\tif (matchlen > pathlen) {\n+\t\t\tif (match[pathlen] != '/')\n+\t\t\t\tcontinue;\n+\t\t\tif (!S_ISDIR(mode))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (m == -1)\n+\t\t\t/*\n+\t\t\t * we cheated and did not do strncmp(), so we do\n+\t\t\t * that here.\n+\t\t\t */\n+\t\t\tm = strncmp(match, path, pathlen);\n+\n+\t\t/*\n+\t\t * If common part matched earlier then it is a hit,\n+\t\t * because we rejected the case where path is not a\n+\t\t * leading directory and is shorter than match.\n+\t\t */\n+\t\tif (!m)\n+\t\t\treturn 1;\n+\t}\n+\treturn never_interesting; /* No matches */\n+}\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"150036","messageId":"1283748429-31076-4-git-send-email-pclouds@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-8-git-send-email-newren@gmail.com","subject":"[PATCH 3/4] tree-walk: actually move tree_entry_interesting() to tree-walk.c","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-06T04:47:08Z","receivedAt":"2010-09-06T04:47:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This function can be potentially used in more places than just\ntree-diff.c. This patches removes struct diff_options dependency from\nthe function, and moves it to tree-walk.c.\n\nNo functionality change intended.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n tree-diff.c | 121 ++---------------------------------------------------------\n tree-walk.c | 25 +++++-------\n tree-walk.h | 3 +\n 3 files changed, 17 insertions(+), 132 deletions(-)\n\ndiff --git a/tree-diff.c b/tree-diff.c\nindex cd659c6..332f0c8 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -82,120 +82,6 @@ static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2, const\n \treturn 0;\n }\n \n-/*\n- * Is a tree entry interesting given the pathspec we have?\n- *\n- * Return:\n- * - 2 for \"yes, and all subsequent entries will be\"\n- * - 1 for yes\n- * - zero for no\n- * - negative for \"no, and no subsequent entries will be either\"\n- */\n-static int tree_entry_interesting(struct tree_desc *desc, const char *base, int baselen, struct diff_options *opt)\n-{\n-\tconst char *path;\n-\tconst unsigned char *sha1;\n-\tunsigned mode;\n-\tint i;\n-\tint pathlen;\n-\tint never_interesting = -1;\n-\n-\tif (!opt->nr_paths)\n-\t\treturn 1;\n-\n-\tsha1 = tree_entry_extract(desc, &path, &mode);\n-\n-\tpathlen = tree_entry_len(path, sha1);\n-\n-\tfor (i = 0; i < opt->nr_paths; i++) {\n-\t\tconst char *match = opt->paths[i];\n-\t\tint matchlen = opt->pathlens[i];\n-\t\tint m = -1; /* signals that we haven't called strncmp() */\n-\n-\t\tif (baselen >= matchlen) {\n-\t\t\t/* If it doesn't match, move along... */\n-\t\t\tif (strncmp(base, match, matchlen))\n-\t\t\t\tcontinue;\n-\n-\t\t\t/*\n-\t\t\t * If the base is a subdirectory of a path which\n-\t\t\t * was specified, all of them are interesting.\n-\t\t\t */\n-\t\t\tif (!matchlen ||\n-\t\t\t base[matchlen] == '/' ||\n-\t\t\t match[matchlen - 1] == '/')\n-\t\t\t\treturn 2;\n-\n-\t\t\t/* Just a random prefix match */\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\t/* Does the base match? */\n-\t\tif (strncmp(base, match, baselen))\n-\t\t\tcontinue;\n-\n-\t\tmatch += baselen;\n-\t\tmatchlen -= baselen;\n-\n-\t\tif (never_interesting) {\n-\t\t\t/*\n-\t\t\t * We have not seen any match that sorts later\n-\t\t\t * than the current path.\n-\t\t\t */\n-\n-\t\t\t/*\n-\t\t\t * Does match sort strictly earlier than path\n-\t\t\t * with their common parts?\n-\t\t\t */\n-\t\t\tm = strncmp(match, path,\n-\t\t\t\t (matchlen < pathlen) ? matchlen : pathlen);\n-\t\t\tif (m < 0)\n-\t\t\t\tcontinue;\n-\n-\t\t\t/*\n-\t\t\t * If we come here even once, that means there is at\n-\t\t\t * least one pathspec that would sort equal to or\n-\t\t\t * later than the path we are currently looking at.\n-\t\t\t * In other words, if we have never reached this point\n-\t\t\t * after iterating all pathspecs, it means all\n-\t\t\t * pathspecs are either outside of base, or inside the\n-\t\t\t * base but sorts strictly earlier than the current\n-\t\t\t * one. In either case, they will never match the\n-\t\t\t * subsequent entries. In such a case, we initialized\n-\t\t\t * the variable to -1 and that is what will be\n-\t\t\t * returned, allowing the caller to terminate early.\n-\t\t\t */\n-\t\t\tnever_interesting = 0;\n-\t\t}\n-\n-\t\tif (pathlen > matchlen)\n-\t\t\tcontinue;\n-\n-\t\tif (matchlen > pathlen) {\n-\t\t\tif (match[pathlen] != '/')\n-\t\t\t\tcontinue;\n-\t\t\tif (!S_ISDIR(mode))\n-\t\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (m == -1)\n-\t\t\t/*\n-\t\t\t * we cheated and did not do strncmp(), so we do\n-\t\t\t * that here.\n-\t\t\t */\n-\t\t\tm = strncmp(match, path, pathlen);\n-\n-\t\t/*\n-\t\t * If common part matched earlier then it is a hit,\n-\t\t * because we rejected the case where path is not a\n-\t\t * leading directory and is shorter than match.\n-\t\t */\n-\t\tif (!m)\n-\t\t\treturn 1;\n-\t}\n-\treturn never_interesting; /* No matches */\n-}\n-\n /* A whole sub-tree went away or appeared */\n static void show_tree(struct diff_options *opt, const char *prefix, struct tree_desc *desc, const char *base, int baselen)\n {\n@@ -206,8 +92,8 @@ static void show_tree(struct diff_options *opt, const char *prefix, struct tree_\n \t\tif (all_interesting)\n \t\t\tshow = 1;\n \t\telse {\n-\t\t\tshow = tree_entry_interesting(desc, base, baselen,\n-\t\t\t\t\t\t opt);\n+\t\t\tshow = tree_entry_interesting(&desc->entry, base, baselen,\n+\t\t\t\t\t\t opt->paths, opt->pathlens, opt->nr_paths);\n \t\t\tif (show == 2)\n \t\t\t\tall_interesting = 1;\n \t\t}\n@@ -266,7 +152,8 @@ static void skip_uninteresting(struct tree_desc *t, const char *base, int basele\n \t\tif (all_interesting)\n \t\t\tshow = 1;\n \t\telse {\n-\t\t\tshow = tree_entry_interesting(t, base, baselen, opt);\n+\t\t\tshow = tree_entry_interesting(&t->entry, base, baselen,\n+\t\t\t\t\t\t opt->paths, opt->pathlens, opt->nr_paths);\n \t\t\tif (show == 2)\n \t\t\t\tall_interesting = 1;\n \t\t}\ndiff --git a/tree-walk.c b/tree-walk.c\nindex bc83fa3..c665e35 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -2,7 +2,6 @@\n #include \"tree-walk.h\"\n #include \"unpack-trees.h\"\n #include \"tree.h\"\n-#include \"diff.h\"\n \n static const char *get_mode(const char *str, unsigned int *modep)\n {\n@@ -466,25 +465,21 @@ int get_tree_entry(const unsigned char *tree_sha1, const char *name, unsigned ch\n * - zero for no\n * - negative for \"no, and no subsequent entries will be either\"\n */\n-static int tree_entry_interesting(struct tree_desc *desc, const char *base, int baselen, struct diff_options *opt)\n+int tree_entry_interesting(const struct name_entry *entry, const char *base, int baselen,\n+\t\t\t const char **paths, int *pathlens, int nr_paths)\n {\n-\tconst char *path;\n-\tconst unsigned char *sha1;\n-\tunsigned mode;\n \tint i;\n \tint pathlen;\n \tint never_interesting = -1;\n \n-\tif (!opt->nr_paths)\n+\tif (!nr_paths)\n \t\treturn 1;\n \n-\tsha1 = tree_entry_extract(desc, &path, &mode);\n-\n-\tpathlen = tree_entry_len(path, sha1);\n+\tpathlen = tree_entry_len(entry->path, entry->sha1);\n \n-\tfor (i = 0; i < opt->nr_paths; i++) {\n-\t\tconst char *match = opt->paths[i];\n-\t\tint matchlen = opt->pathlens[i];\n+\tfor (i = 0; i < nr_paths; i++) {\n+\t\tconst char *match = paths[i];\n+\t\tint matchlen = pathlens[i];\n \t\tint m = -1; /* signals that we haven't called strncmp() */\n \n \t\tif (baselen >= matchlen) {\n@@ -522,7 +517,7 @@ static int tree_entry_interesting(struct tree_desc *desc, const char *base, int\n \t\t\t * Does match sort strictly earlier than path\n \t\t\t * with their common parts?\n \t\t\t */\n-\t\t\tm = strncmp(match, path,\n+\t\t\tm = strncmp(match, entry->path,\n \t\t\t\t (matchlen < pathlen) ? matchlen : pathlen);\n \t\t\tif (m < 0)\n \t\t\t\tcontinue;\n@@ -549,7 +544,7 @@ static int tree_entry_interesting(struct tree_desc *desc, const char *base, int\n \t\tif (matchlen > pathlen) {\n \t\t\tif (match[pathlen] != '/')\n \t\t\t\tcontinue;\n-\t\t\tif (!S_ISDIR(mode))\n+\t\t\tif (!S_ISDIR(entry->mode))\n \t\t\t\tcontinue;\n \t\t}\n \n@@ -558,7 +553,7 @@ static int tree_entry_interesting(struct tree_desc *desc, const char *base, int\n \t\t\t * we cheated and did not do strncmp(), so we do\n \t\t\t * that here.\n \t\t\t */\n-\t\t\tm = strncmp(match, path, pathlen);\n+\t\t\tm = strncmp(match, entry->path, pathlen);\n \n \t\t/*\n \t\t * If common part matched earlier then it is a hit,\ndiff --git a/tree-walk.h b/tree-walk.h\nindex 88ea7e9..b20f3c4 100644\n--- a/tree-walk.h\n+++ b/tree-walk.h\n@@ -57,4 +57,7 @@ static inline int traverse_path_len(const struct traverse_info *info, const stru\n \treturn info->pathlen + tree_entry_len(n->path, n->sha1);\n }\n \n+extern int tree_entry_interesting(const struct name_entry *entry, const char *base, int baselen,\n+\t\t\t\t const char **paths, int *pathlens, int nr_paths);\n+\n #endif\n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"150037","messageId":"1283748429-31076-5-git-send-email-pclouds@gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-8-git-send-email-newren@gmail.com","subject":"[PATCH 4/4] Make rev-list --objects work together with pathspecs","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-06T04:47:09Z","receivedAt":"2010-09-06T04:47:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"From: Elijah Newren \n\nWhen traversing commits, the selection of commits would heed the list of\npathspecs passed, but subsequent walking of the trees of those commits\nwould not. This resulted in 'rev-list --objects HEAD -- '\ndisplaying objects at unwanted paths.\n\nHave process_tree() call tree_entry_interesting() to determine which paths\nare interesting and should be walked.\n\nNaturally, this change can provide a large speedup when paths are specified\ntogether with --objects, since many tree entries are now correctly ignored.\nInterestingly, though, this change also gives me a small (~1%) but\nrepeatable speedup even when no paths are specified with --objects.\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Junio C Hamano \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n The \"while (tree_entry(..))\" loop makes a come back.\n tree_entry_interesting() call is updated.\n Nothing else is changed\n\n list-objects.c | 25 +++++++++++++++++++++++++\n revision.c | 8 ++++++--\n revision.h | 3 ++-\n t/t6000-rev-list-misc.sh | 6 +++---\n 4 files changed, 36 insertions(+), 6 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 8953548..8ea1822 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -67,6 +67,9 @@ static void process_tree(struct rev_info *revs,\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tstruct name_path me;\n+\tint all_interesting = (revs->diffopt.nr_paths == 0);\n+\tchar *full_prefix = NULL;\n+\tint full_prefix_len = 0;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -82,9 +85,30 @@ static void process_tree(struct rev_info *revs,\n \tme.elem = name;\n \tme.elem_len = strlen(name);\n \n+\tif (!all_interesting) {\n+\t\tfull_prefix = path_name_impl(path, name, 1);\n+\t\tfull_prefix_len = strlen(full_prefix);\n+\t}\n+\n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n \n \twhile (tree_entry(&desc, &entry)) {\n+\t\tif (!all_interesting) {\n+\t\t\tint showit = tree_entry_interesting(&entry,\n+\t\t\t\t\t\t\t full_prefix,\n+\t\t\t\t\t\t\t full_prefix_len,\n+\t\t\t\t\t\t\t revs->diffopt.paths,\n+\t\t\t\t\t\t\t revs->diffopt.pathlens,\n+\t\t\t\t\t\t\t revs->diffopt.nr_paths);\n+\n+\t\t\tif (showit < 0)\n+\t\t\t\tbreak;\n+\t\t\telse if (!showit)\n+\t\t\t\tcontinue;\n+\t\t\telse if (showit == 2)\n+\t\t\t\tall_interesting = 1;\n+\t\t}\n+\n \t\tif (S_ISDIR(entry.mode))\n \t\t\tprocess_tree(revs,\n \t\t\t\t lookup_tree(entry.sha1),\n@@ -97,6 +121,7 @@ static void process_tree(struct rev_info *revs,\n \t\t\t\t lookup_blob(entry.sha1),\n \t\t\t\t show, &me, entry.path);\n \t}\n+\tfree(full_prefix);\n \tfree(tree->buffer);\n \ttree->buffer = NULL;\n }\ndiff --git a/revision.c b/revision.c\nindex b1c1890..55c4586 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -16,7 +16,7 @@\n \n volatile show_early_output_fn_t show_early_output;\n \n-char *path_name(const struct name_path *path, const char *name)\n+char *path_name_impl(const struct name_path *path, const char *name, int isdir)\n {\n \tconst struct name_path *p;\n \tchar *n, *m;\n@@ -27,7 +27,7 @@ char *path_name(const struct name_path *path, const char *name)\n \t\tif (p->elem_len)\n \t\t\tlen += p->elem_len + 1;\n \t}\n-\tn = xmalloc(len);\n+\tn = xmalloc(len + !!isdir);\n \tm = n + len - (nlen + 1);\n \tstrcpy(m, name);\n \tfor (p = path; p; p = p->up) {\n@@ -37,6 +37,10 @@ char *path_name(const struct name_path *path, const char *name)\n \t\t\tm[p->elem_len] = '/';\n \t\t}\n \t}\n+\tif (isdir && len > 1) {\n+\t\tn[len-1] = '/';\n+\t\tn[len] = '\\0';\n+\t}\n \treturn n;\n }\n \ndiff --git a/revision.h b/revision.h\nindex 05659c6..92f4feb 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -173,7 +173,8 @@ struct name_path {\n \tconst char *elem;\n };\n \n-char *path_name(const struct name_path *path, const char *name);\n+char *path_name_impl(const struct name_path *path, const char *name, int isdir);\n+#define path_name(path, name) path_name_impl(path, name, 0)\n \n extern void add_object(struct object *obj,\n \t\t struct object_array *p,\ndiff --git a/t/t6000-rev-list-misc.sh b/t/t6000-rev-list-misc.sh\nindex b3c1dd8..b10685a 100755\n--- a/t/t6000-rev-list-misc.sh\n+++ b/t/t6000-rev-list-misc.sh\n@@ -11,13 +11,13 @@ test_expect_success setup '\n \tgit commit -m one\n '\n \n-test_expect_failure 'rev-list --objects heeds pathspecs' '\n+test_expect_success 'rev-list --objects heeds pathspecs' '\n \tgit rev-list --objects HEAD -- wanted_file >output &&\n \tgrep wanted_file output &&\n \t! grep unwanted_file output\n '\n \n-test_expect_failure 'rev-list --objects with pathspecs and deeper paths' '\n+test_expect_success 'rev-list --objects with pathspecs and deeper paths' '\n \tmkdir foo &&\n \t>foo/file &&\n \tgit add foo/file &&\n@@ -31,7 +31,7 @@ test_expect_failure 'rev-list --objects with pathspecs and deeper paths' '\n \t! grep unwanted_file output\n '\n \n-test_expect_failure 'rev-list --objects with pathspecs and copied files' '\n+test_expect_success 'rev-list --objects with pathspecs and copied files' '\n \tgit checkout --orphan junio-testcase &&\n \tgit rm -rf . &&\n \n-- \n1.7.1.rc1.69.g24c2f7\n"},{"id":"150038","messageId":"AANLkTikvJoVdUWAbQw0xT=JOiry4FaA0jsOMTNf=iZqH@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTinufi3TwnPZ+smq5FE5S3zZdQSzKpqYv=hVfcLV@mail.gmail.com","subject":"Re: [RFC PATCH 07/15] cache_tree_update(): Capability to handle tree entries missing from index","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-06T05:02:50Z","receivedAt":"2010-09-06T05:02:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Sep 6, 2010 at 2:42 PM, Elijah Newren wrote:\n> On Sun, Sep 5, 2010 at 3:09 PM, Elijah Newren wrote:\n>> n Sun, Sep 5, 2010 at 1:54 AM, Nguyen Thai Ngoc Duy wrote:\n>>> On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n>>>> cache_tree_update() will write trees using the index.  With sparse clones,\n>>>> the index will only contain entries matching the sparse limits, meaning\n>>>> that the index does not provide enough information to write complete tree\n>>>> objects.  Having cache_tree_update() take a tree (typically HEAD), will\n>>>> allow new complete trees to be constructed by using entries from the\n>>>> specified tree outside the sparse limits together with the index.\n>>>\n>>> You are moving it closer to the index (from my view because I changed\n>>> in commit_tree()). This makes me think, why don't you move the base\n>>> tree into the index itself?\n>>>\n>>> The index is supposed to save the image of full worktree. While you\n>>> don't have all path names, you have the clue to all of them, the base\n>>> tree. To me, that means it belongs to the index. That would reduce\n>>> code change to\n>>>  - cache-tree.c (generate new tree from the base tree and index)\n>>>  - read-cache.c (new sparse-clone index extension)\n>>>  - index writing operations (save the base tree in index): read-tree and merge\n>>\n>> That's a really good idea.  I'll look into that.\n>\n> Hmm..maybe before I get ahead of myself, I should back up for a\n> second.  Which way do we think is better -- handling this in\n> cache_tree_update() or doing a fixup in commit_tree()?  If the latter,\n> then I should just drop my 7/15 and 8/15 for your 13/17 and 14/17.  If\n> the former, then it makes sense to look into this change you suggest.\n> In that case, we'd probably keep my 7/15 but drop 8/15 for patch(es)\n> that implement your idea above.  But you're more familiar with the\n> index format than I am and it's your idea, so you'd probably be the\n> better one to implement it.\n>\n> Thoughts?\n\nThe former makes more sense. I still keep an eye on remote merge\nsupport. Think of this case: you request a remote merge and have a\ncompletely new base tree, different from HEAD. Then you get some\nconflicts inside narrow/sparse area. By the time you resolve all\nconflicts and are about to commit, where do you get the base tree?\n\nIt could follow the same way merge does, store it in a file in\n$GIT_DIR, similar to $GIT_DIR/MERGE_HEAD, and make \"git commit\" pick\nit up. Or just store it in index. You would need to (or should) change\nthe index anyway, to prevent naive git implementation from using it.\nIt makes more sense to put some more in index rather than a separate\nfile. Some commands (especially \"git commit\") make use of temporary\nindex. I think putting the base tree in index is a good point here,\nbut I'm not quite sure.\n\nAnyway if you want to play with it, apply this on top of my 04/17\n(then change index_state.narrow_base to something suitable for sparse\nclone)\n\ndiff --git a/cache.h b/cache.h\nindex 9c014ef..c7d626d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -293,6 +293,7 @@ struct index_state {\n \tstruct cache_tree *cache_tree;\n \tstruct cache_time timestamp;\n \tchar *narrow_prefix;\n+\tunsigned char narrow_base[20];\n \tvoid *alloc;\n \tunsigned name_hash_initialized : 1,\n \t\t initialized : 1;\ndiff --git a/read-cache.c b/read-cache.c\nindex 250013c..15c1c9e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1191,7 +1191,8 @@ static int read_index_extension(struct\nindex_state *istate,\n \t\tistate->resolve_undo = resolve_undo_read(data, sz);\n \t\tbreak;\n \tcase CACHE_EXT_NARROW:\n-\t\tistate->narrow_prefix = xstrdup(data);\n+\t\thashcpy(istate->narrow_base, data);\n+\t\tistate->narrow_prefix = xstrdup(data+sizeof(istate->narrow_base));\n \t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n@@ -1396,6 +1397,7 @@ int discard_index(struct index_state *istate)\n \tistate->alloc = NULL;\n \tistate->initialized = 0;\n \tfree(istate->narrow_prefix);\n+\thashcpy(istate->narrow_base, (const unsigned char *)EMPTY_TREE_SHA1_BIN);\n \tistate->narrow_prefix = NULL;\n\n \t/* no need to throw away allocated active_cache */\n@@ -1627,11 +1629,15 @@ int write_index(struct index_state *istate, int newfd)\n \t\t\treturn -1;\n \t}\n \tif (get_narrow_prefix()) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n \t\tchar *buf = get_narrow_string();\n-\t\tint len = strlen(buf)+1;\n-\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_NARROW, len) < 0 ||\n-\t\t\tce_write(&c, newfd, buf, len) < 0;\n+\t\tint len = 20+strlen(buf)+1;\n+\t\tstrbuf_add(&sb, istate->narrow_base, 20);\n+\t\tstrbuf_addstr(&sb, buf);\n \t\tfree(buf);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_NARROW, len) < 0 ||\n+\t\t\tce_write(&c, newfd, sb.buf, len) < 0;\n+\t\tstrbuf_release(&sb);\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n-- \nDuy\n"},{"id":"150086","messageId":"AANLkTikmTcKdEjwX+3+BZoV25cJQk=XDC4V03kc5G44s@mail.gmail.com","threadId":"24979","inReplyTo":"1283748429-31076-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 2/4] tree-walk: copy tree_entry_interesting() as is from tree-diff.c","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-06T15:22:38Z","receivedAt":"2010-09-06T15:22:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"2010/9/5 Nguyễn Thái Ngọc Duy :\n> Just a straight copy. The function is not used anywhere. It is to\n> separate changes that will be made to this function in the next patch.\n\nWhy copy instead of move?\n"},{"id":"150088","messageId":"AANLkTinV+Bo_EijJU2q5GVG=jr9GFAuQGo5KQpvf0K-V@mail.gmail.com","threadId":"24979","inReplyTo":"1283748429-31076-4-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 3/4] tree-walk: actually move tree_entry_interesting() to tree-walk.c","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-06T15:31:57Z","receivedAt":"2010-09-06T15:31:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"2010/9/5 Nguyễn Thái Ngọc Duy :\n> This function can be potentially used in more places than just\n> tree-diff.c. This patches removes struct diff_options dependency from\n> the function, and moves it to tree-walk.c.\n>\n> No functionality change intended.\n\nThanks for working on this. I like having the declaration of\ntree_entry_interesting() moved to tree_walk.h, at the very least. The\nchange to make tree_entry_interesting() take an entry instead of a\ntree_desc makes sense too.\n\nI'm unsure about replacing the diff_options with paths + pathlens +\nnr_paths -- that might be exposing too much implementation detail in\nthe API. In particular, I'm worried that if we try to add support for\nnegated pathspecs or globs or regexes to tree_entry_interesting(),\nthen we'll need to pass different data to this function and update an\nawful lot of callers.\n\nPerhaps we should make a new struct containing paths + pathlens +\nnr_paths, make tree_entry_interesting() take such a struct, modify\ndiff_options have such a struct instead of the current three paths,\npathlens, and nr_paths fields, and modify diff_tree_setup_path()s to\ntake such a struct instead of a diff_options* (and perhaps move\ndiff_tree_setup_paths() out of diff.h and tree-diff.c into some other\nfile(s)?).\n\nThoughts?\n\nElijah\n"},{"id":"150142","messageId":"AANLkTimG1JzOtSDKh_BwAMj92iHWCA2bHThVcT2k4bZn@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTikmTcKdEjwX+3+BZoV25cJQk=XDC4V03kc5G44s@mail.gmail.com","subject":"Re: [PATCH 2/4] tree-walk: copy tree_entry_interesting() as is from tree-diff.c","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-06T22:09:56Z","receivedAt":"2010-09-06T22:09:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/9/7 Elijah Newren :\n> 2010/9/5 Nguyễn Thái Ngọc Duy :\n>> Just a straight copy. The function is not used anywhere. It is to\n>> separate changes that will be made to this function in the next patch.\n>\n> Why copy instead of move?\n>\n\nRight. I wanted to keep it in buildable state at anytime so I made\nzero changes to tree-diff.c. But you're right, that should be a move\nand make sure the code builds.\n-- \nDuy\n"},{"id":"150145","messageId":"AANLkTi=+iga89znGPgOV14+Z9AZ0qbzJJbN1wNAum=qX@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTinV+Bo_EijJU2q5GVG=jr9GFAuQGo5KQpvf0K-V@mail.gmail.com","subject":"Re: [PATCH 3/4] tree-walk: actually move tree_entry_interesting() to tree-walk.c","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-06T22:20:06Z","receivedAt":"2010-09-06T22:20:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/9/7 Elijah Newren :\n> 2010/9/5 Nguyễn Thái Ngọc Duy :\n>> This function can be potentially used in more places than just\n>> tree-diff.c. This patches removes struct diff_options dependency from\n>> the function, and moves it to tree-walk.c.\n>>\n>> No functionality change intended.\n>\n> Thanks for working on this.  I like having the declaration of\n> tree_entry_interesting() moved to tree_walk.h, at the very least.  The\n> change to make tree_entry_interesting() take an entry instead of a\n> tree_desc makes sense too.\n>\n> I'm unsure about replacing the diff_options with paths + pathlens +\n> nr_paths -- that might be exposing too much implementation detail in\n> the API.  In particular, I'm worried that if we try to add support for\n> negated pathspecs or globs or regexes to tree_entry_interesting(),\n> then we'll need to pass different data to this function and update an\n> awful lot of callers.\n>\n> Perhaps we should make a new struct containing paths + pathlens +\n> nr_paths, make tree_entry_interesting() take such a struct, modify\n> diff_options have such a struct instead of the current three paths,\n> pathlens, and nr_paths fields, and modify diff_tree_setup_path()s to\n> take such a struct instead of a diff_options* (and perhaps move\n> diff_tree_setup_paths() out of diff.h and tree-diff.c into some other\n> file(s)?).\n>\n> Thoughts?\n\nYou're right again. Perhaps something like struct exclude_list from dir.h?\n\nstruct pathspec_list {\n\tint nr;\n\tint alloc;\n\tstruct pathspec {\n\t\tconst char *path;\n\t\tint pathlen;\n\t\tint flags;\n\t} **paths;\n};\n\nHmm.. or just use struct exclude_list, with diff_options.path{s,len}\nbecoming exclude_list.base{,len}. exclude_list.pattern can be used for\nglob/regex matching (if we are ever going to support that).\n-- \nDuy\n"},{"id":"150156","messageId":"7vtym2sa20.fsf@alter.siamese.dyndns.org","threadId":"24979","inReplyTo":"AANLkTi=+iga89znGPgOV14+Z9AZ0qbzJJbN1wNAum=qX@mail.gmail.com","subject":"Re: [PATCH 3/4] tree-walk: actually move tree_entry_interesting() to tree-walk.c","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-09-06T23:53:59Z","receivedAt":"2010-09-06T23:53:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy writes:\n\n> Hmm.. or just use struct exclude_list, with diff_options.path{s,len}\n> becoming exclude_list.base{,len}. exclude_list.pattern can be used for\n> glob/regex matching (if we are ever going to support that).\n\nIf you are changing the API and structures around this area, you really\nshould aim to help us to support the globbing in diff-tree family just\nlike we do in ls-files family, so yes.\n"},{"id":"150166","messageId":"AANLkTinACDSruJXon0DkbU8V=ko55KzpVZmCRKA7i=5f@mail.gmail.com","threadId":"24979","inReplyTo":"1283645647-1891-8-git-send-email-newren@gmail.com","subject":"Re: [RFC PATCH 07/15] cache_tree_update(): Capability to handle tree entries missing from index","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-09-07T01:28:10Z","receivedAt":"2010-09-07T01:28:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n> +static void add_missing_paths_before(const char *path,\n> +                                    int pathlen,\n> +                                    int baselen,\n> +                                    struct tree_desc *desc,\n> +                                    struct strbuf *buffer)\n> +{\n> +       if (!git_sparse_pathspecs)\n> +               return; /* No paths are missing */\n> +\n> +       for (; desc->size; update_tree_entry(desc)) {\n> +               struct name_entry entry = desc->entry;\n> +               int entlen = strlen(entry.path);\n> +\n> +               /* We only want paths before path */\n> +               if (path)\n> +                       /*\n> +                        * FIXME: if entlen < pathlen, do I need to\n> +                        * use strncmp instead? Does entry.path being\n> +                        * NUL-terminated ensure memcmp exits\n> +                        * early?\n> +                        */\n> +                       if (memcmp(entry.path,\n> +                                  path+baselen,\n> +                                  pathlen-baselen) >= 0)\n> +                               break;\n\nIf you do \"break;\" here, the current entry remains in \"desc\" and will\nbe used to regenerate a duplicate tree in\nadd_missing_paths_before(NULL, 0, 0, ..);. That entry is added the\nfirst time by update_one itself before the second add_missing() call.\nThis fixes it for me\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 79c28cc..1655738 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -278,17 +278,19 @@ static void add_missing_paths_before(const char *path,\n \t\tint entlen = strlen(entry.path);\n\n \t\t/* We only want paths before path */\n-\t\tif (path)\n+\t\tif (path) {\n \t\t\t/*\n \t\t\t * FIXME: if entlen < pathlen, do I need to\n \t\t\t * use strncmp instead? Does entry.path being\n \t\t\t * NUL-terminated ensure memcmp exits\n \t\t\t * early?\n \t\t\t */\n-\t\t\tif (memcmp(entry.path,\n-\t\t\t\t path+baselen,\n-\t\t\t\t pathlen-baselen) >= 0)\n+\t\t\tint result = memcmp(entry.path, path+baselen, pathlen-baselen);\n+\t\t\tif (result > 0)\n \t\t\t\tbreak;\n+\t\t\tif (!result)\n+\t\t\t\tcontinue;\n+\t\t}\n\n \t\t/* We only want paths \"missing\" from index due to sparsity */\n \t\tif (path) {\n-- \nDuy\n"},{"id":"150178","messageId":"AANLkTinPyXCGWXH6VUtvTx6Cb4F5Bn238HQRE7tDZDtJ@mail.gmail.com","threadId":"24979","inReplyTo":"AANLkTinACDSruJXon0DkbU8V=ko55KzpVZmCRKA7i=5f@mail.gmail.com","subject":"Re: [RFC PATCH 07/15] cache_tree_update(): Capability to handle tree entries missing from index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-09-07T03:06:13Z","receivedAt":"2010-09-07T03:06:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Sep 6, 2010 at 7:28 PM, Nguyen Thai Ngoc Duy wrote:\n> On Sun, Sep 5, 2010 at 10:13 AM, Elijah Newren wrote:\n>> +static void add_missing_paths_before(const char *path,\n>> +                                    int pathlen,\n>> +                                    int baselen,\n>> +                                    struct tree_desc *desc,\n>> +                                    struct strbuf *buffer)\n>> +{\n>> +       if (!git_sparse_pathspecs)\n>> +               return; /* No paths are missing */\n>> +\n>> +       for (; desc->size; update_tree_entry(desc)) {\n>> +               struct name_entry entry = desc->entry;\n>> +               int entlen = strlen(entry.path);\n>> +\n>> +               /* We only want paths before path */\n>> +               if (path)\n>> +                       /*\n>> +                        * FIXME: if entlen < pathlen, do I need to\n>> +                        * use strncmp instead? Does entry.path being\n>> +                        * NUL-terminated ensure memcmp exits\n>> +                        * early?\n>> +                        */\n>> +                       if (memcmp(entry.path,\n>> +                                  path+baselen,\n>> +                                  pathlen-baselen) >= 0)\n>> +                               break;\n>\n> If you do \"break;\" here, the current entry remains in \"desc\" and will\n> be used to regenerate a duplicate tree in\n> add_missing_paths_before(NULL, 0, 0, ..);. That entry is added the\n> first time by update_one itself before the second add_missing() call.\n> This fixes it for me\n>\n> diff --git a/cache-tree.c b/cache-tree.c\n> index 79c28cc..1655738 100644\n> --- a/cache-tree.c\n> +++ b/cache-tree.c\n> @@ -278,17 +278,19 @@ static void add_missing_paths_before(const char *path,\n>                int entlen = strlen(entry.path);\n>\n>                /* We only want paths before path */\n> -               if (path)\n> +               if (path) {\n>                        /*\n>                         * FIXME: if entlen < pathlen, do I need to\n>                         * use strncmp instead? Does entry.path being\n>                         * NUL-terminated ensure memcmp exits\n>                         * early?\n>                         */\n> -                       if (memcmp(entry.path,\n> -                                  path+baselen,\n> -                                  pathlen-baselen) >= 0)\n> +                       int result = memcmp(entry.path, path+baselen, pathlen-baselen);\n> +                       if (result > 0)\n>                                break;\n> +                       if (!result)\n> +                               continue;\n> +               }\n>\n>                /* We only want paths \"missing\" from index due to sparsity */\n>                if (path) {\n\nYes, good catch and thanks for the fix.\n"}]}