RFC patch, 9 partsNarrow/Sparse checkout round 3: "easy mode"

The implementation with insights from Junio turns out smaller and better (and I was thinking about applying it for huge maildir).

So there is a bit of changes since the last round. This time I follow "assume unchanged" code path and relax the rules a bit. There are issues I will mention later.

From user POV, we can now checkout a single file or a subdirectory (checking out subdirectory non-recursively is possible too). You may start with a narrow clone like:

git clone --path="Documentation/*" git.git

or start from a full checkout and narrow it (very much like the last round):

git checkout --path="Documentation/*"

However narrow spec is now using wildcards, not prefix, so you can checkout a single file, or just header files, etc.

git checkout --add-path|--remove-path has been aded to update narrow areas. But you don't have to use those. Narrow areas will be widened as needed when you do something outside of it (e.g "git checkout foo" or "git add foo"...)

Another difference from the last round is "narrow rules" will not be preserved when switching branches. When you switch branch with no option, you will get full checkout. You may want to use --path|--add-path|--remove-path when switching branches to have narrow checkout again.

Now back to technical POV. I did not reuse CE_VALID (assume unchanged) bit because it has been used for core.ignorestat. So I took another bit, which seems to be the last on-disk bit in ce_flags.

I call this mode "easy mode" because most of constraints have been eliminated. Now your "narrow rules" are like "I don't like those files, remove them if they are not needed". If some operations need those files on workdir again, they will be checked out. Those may include:

 - "git checkout foo" or "git apply"
 - git add foo (even if foo is marked no-checkout)
 - conflict files after merge
 - new files after merge

"Strict mode" may be added later but then it must clearly define which operation is allowed to checkout files. There's a problem with strict mode if it wants to limit checking out new files after merge. Because we don't save "narrow rules" anymore (we applied the rules immediately in checkout/clone stage, then update narrow areas over time), we will not know how to deal with new files. Adding [--path|--add-path|--remove-path] to git merge commands, and apply "narrow rules" again, looks too cumbersome to me. Comments?

Last bit. "Narrow rules" for --path|--add-path|--remove-path is currently wildcards separated by colons. While it does the job, it does not allow to checkout easily a subdirectory non-recusively. I was thinking about '*' as "match everything except slashes" and '**' as "match everything even slashes". Any ideas?

Oh.. and "git grep" may not work correctly (or "as expected") with narrow checkout. Haven't checked it yet.

Nguyễn Thái Ngọc Duy (9):
  Introduce CE_NO_CHECKOUT bit
  update-index: add --checkout/--no-checkout options to update
    CE_NO_CHECKOUT bit
  ls-files: add --checkout option to show checked out files
  Prevent diff machinery from examining worktree outside narrow
    checkout
  Clear CE_NO_CHECKOUT on checked out entries.
  Add support for narrow checkout in unpack_trees()
  ls-files: add --narrow-match=spec option to test narrow matching
  clone: support narrow checkout with --path option
  checkout: add new options to support narrow checkout

 builtin-checkout.c     |   41 ++++++++++++++++++
 builtin-clone.c        |   13 ++++++
 builtin-ls-files.c     |   23 +++++++++--
 builtin-update-index.c |   40 +++++++++++-------
 cache.h                |   11 +++++
 diff-lib.c             |    5 +-
 diff.c                 |    4 +-
 entry.c                |    1 +
 read-cache.c           |    6 +-
 unpack-trees.c         |  106 ++++++++++++++++++++++++++++++++++++++++++++++++
 unpack-trees.h         |    4 ++
 11 files changed, 229 insertions(+), 25 deletions(-)

Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:

I think I can agree with the general direction this is taking, except that we would need to think about the transition plans. Note that I haven't really read through the full series yet.

You could save the "narrow rules" in the extension section of the index. If the final form of this series needs to use a separate CE_NO_CHECKOUT bit (which would make the resulting index incompatible with the current git), the narrow rules section can be marked as "your git must understand this" class of extension to make sure that people do not mistakenly access an index written by this new version of git with the current or older git.

I am not sure what's the relation between these two.

On 8/16/08, Junio C Hamano <gitster@pobox.com> wrote:

The problem is "narrow rules" may change over time in a way that git
may handle it wrong. Assume that you have a directory with two files:
a and b. You first narrow checkout a (which would save the rule
"checkout a"). Then you do "git checkout b". When you update HEAD,
what should happen?
 - consider only a and b in narrow area (new files not counted)
 - consider the whole directory in narrow area (new files counted)

This does not matter until we implement strict mode that only checkout new files inside narrow area (the usage is similar to submodule).

Because the usage is different? When you "git update-index foo" with core.ignorestat=1, it will mark it CE_VALID. And if the same bit is used for narrow checkout, the file is considered not existing in workdir. I'd expect foo is still in my narrow area after "git update-index foo".

"Nguyen Thai Ngoc Duy" <pclouds@gmail.com> writes:

I'd expect that this sequence:

	git checkout --set-no-checkout arch include
        git checkout arch/x86 include/asm-x86

to set up narrowing rules to (1) exclude everything in arch/ and include/ area by default, but (2) allow checking out everything in arch/x86/ and include/asm-x86/ that currently exist _and_ will exist in different commits when we switch to.

On the other hand, if I did this after the above two-command sequence:

	git checkout include/Kbuild

then I'd expect only that file to be added to the checkout set. I think you can record list of pathspecs (with positive and negative) to implement that semantics, no?

Ok.  We would need to use an extra bit for this.

The bit 0x4000 is the last one available, so we would want to use it as "this index entry uses more bits than the traditional format" bit, and define a backward incompatible on-disk index entry format to actually record CE_NO_CHECKOUT and other flags we will invent in the future.

Perhaps ondisk_cache_entry structure will have an extra "unsigned int flags2" after "flags" when that bit is on, and we can have 31 more bits in flags2, with the highest bit of flags2 signalling the presense of flags3 word in the future, or something like that.

By the way, "uint" and "ushort" in struct ondisk_cache_entry are 4-byte and 2-byte network byte order integers; should we write them as uint32_t and uint16_t instead in the longer run?

Junio C Hamano <gitster@pobox.com> writes:

By the way, I was just mentioning the index extension area as a means to store the rules if _you wanted to_. I do not insist you to actually store the rules, and in fact, I do not know if it is even a good idea to do so.

It might make sense to do this first as a futureproof, if we really want to go this route. We can ensure that an index that does use the new flag bits won't be misinterpreted by older git.

-- >8 --
From: Junio C Hamano <gitster@pobox.com>
Date: Sat, 16 Aug 2008 23:02:08 -0700
Subject: [PATCH] index: future proof for "extended" index entries

We do not have any more bits in the on-disk index flags word, but we would need to have more in the future. Use the last remaining bits as a signal to tell us that the index entry we are looking at is an extended one.

Since we do not understand the extended format yet, we will just error out when we see it.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 cache.h      |    1 +
 read-cache.c |    4 ++++
 2 files changed, 5 insertions(+), 0 deletions(-)

s/but we would need to have/but we may need/ in the commit message?

Eric Raible <raible@gmail.com> writes:

Yeah, strictly speaking, perhaps.

One thing that I refuse to believe is we will need only one more bit and after assigning the 0x4000 bit to whatever that single purpose the index will stay that way forever. So we would need to reserve that bit as the extension bit in any case. If we do not have any extension forever, that means any index entry with the bit set is corrupt, so erroring out would be the right thing to do anyway ;-).

On 8/17/08, Junio C Hamano <gitster@pobox.com> wrote:

I was more worried about those rules getting out of control because git-checkout is not the only command that can change narrow rules. After enough commands, the rules can become a mess that you don't even want to look at them. I don't do negative rules now, but yes that's possible.

The patch is fine. Still we need to do something to prevent older git from using new index format.

Nguyễn Thái Ngọc Duy <pclouds <at> gmail.com> writes:

Is there any reason for the change in terminology from "sparse" to "narrow"? I understand the difference between "partial" and "sparse", but I can't tell if there's any difference between "narrow" and "sparse". If they are the same thing, then I think "sparse" is the better term.

I like this idea - it would make this much more intuitive to use, since '*' and '**' would work the same as they do in the shell (for shells that support '**' at least). I tried the patch in it's current form, and it took me a while to figure out that paths were non-recursive and '*' was matching everything, including slashes.

James

On 8/20/08, James Pickens <jepicken@gmail.com> wrote:

I have no particular preference. It's up to the community to choose the name.

Tried the last few days but it was not easy, needed to duplicate fnmatch code. I may come up with a less powerful syntax for recusive/non-recursive '*'.