{"thread":{"id":"15021","subject":"[RFC PATCH 0/9] Narrow/Sparse checkout round 3: \"easy mode\"","startedAt":"2008-08-15T14:24:39Z","lastAt":"2008-08-30T09:21:50Z","messageCount":10,"participants":["Nguyễn Thái Ngọc Duy","Junio C Hamano","Nguyen Thai Ngoc Duy","Eric Raible","James Pickens"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"87305","messageId":"20080815142439.GA10609@laptop","threadId":"15021","inReplyTo":null,"subject":"[RFC PATCH 0/9] Narrow/Sparse checkout round 3: \"easy mode\"","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2008-08-15T14:24:39Z","receivedAt":"2008-08-15T14:24:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The implementation with insights from Junio turns out smaller and better\n(and I was thinking about applying it for huge maildir).\n\nSo there is a bit of changes since the last round. This time I follow\n\"assume unchanged\" code path and relax the rules a bit. There are issues\nI will mention later.\n\nFrom user POV, we can now checkout a single file or a subdirectory (checking\nout subdirectory non-recursively is possible too). You may start with a narrow\nclone like:\n\ngit clone --path=\"Documentation/*\" git.git\n\nor start from a full checkout and narrow it (very much like the last round):\n\ngit checkout --path=\"Documentation/*\"\n\nHowever narrow spec is now using wildcards, not prefix, so you can checkout\na single file, or just header files, etc.\n\ngit checkout --add-path|--remove-path has been aded to update narrow areas.\nBut you don't have to use those. Narrow areas will be widened as needed when\nyou do something outside of it (e.g \"git checkout foo\" or \"git add foo\"...)\n\nAnother difference from the last round is \"narrow rules\" will not be preserved\nwhen switching branches. When you switch branch with no option, you will get\nfull checkout. You may want to use --path|--add-path|--remove-path when\nswitching branches to have narrow checkout again.\n\nNow back to technical POV. I did not reuse CE_VALID (assume unchanged) bit\nbecause it has been used for core.ignorestat. So I took another bit, which\nseems to be the last on-disk bit in ce_flags.\n\nI call this mode \"easy mode\" because most of constraints have been eliminated.\nNow your \"narrow rules\" are like \"I don't like those files, remove them if\nthey are not needed\". If some operations\nneed those files on workdir again, they will be checked out. Those may include:\n\n - \"git checkout foo\" or \"git apply\"\n - git add foo (even if foo is marked no-checkout)\n - conflict files after merge\n - new files after merge\n\n\"Strict mode\" may be added later but then it must clearly define which\noperation is allowed to checkout files. There's a problem with strict mode if\nit wants to limit checking out new files after merge. Because we don't save\n\"narrow rules\" anymore (we applied the rules immediately in checkout/clone\nstage, then update narrow areas over time), we will not know how to deal with\nnew files. Adding [--path|--add-path|--remove-path] to git merge commands, and\napply \"narrow rules\" again, looks too cumbersome to me. Comments?\n\nLast bit. \"Narrow rules\" for --path|--add-path|--remove-path is currently\nwildcards separated by colons. While it does the job, it does not allow to\ncheckout easily a subdirectory non-recusively. I was thinking about '*' as\n\"match everything except slashes\" and '**' as \"match everything even slashes\".\nAny ideas?\n\nOh.. and \"git grep\" may not work correctly (or \"as expected\") with narrow\ncheckout. Haven't checked it yet.\n\nNguyễn Thái Ngọc Duy (9):\n Introduce CE_NO_CHECKOUT bit\n update-index: add --checkout/--no-checkout options to update\n CE_NO_CHECKOUT bit\n ls-files: add --checkout option to show checked out files\n Prevent diff machinery from examining worktree outside narrow\n checkout\n Clear CE_NO_CHECKOUT on checked out entries.\n Add support for narrow checkout in unpack_trees()\n ls-files: add --narrow-match=spec option to test narrow matching\n clone: support narrow checkout with --path option\n checkout: add new options to support narrow checkout\n\n builtin-checkout.c | 41 ++++++++++++++++++\n builtin-clone.c | 13 ++++++\n builtin-ls-files.c | 23 +++++++++--\n builtin-update-index.c | 40 +++++++++++-------\n cache.h | 11 +++++\n diff-lib.c | 5 +-\n diff.c | 4 +-\n entry.c | 1 +\n read-cache.c | 6 +-\n unpack-trees.c | 106 ++++++++++++++++++++++++++++++++++++++++++++++++\n unpack-trees.h | 4 ++\n 11 files changed, 229 insertions(+), 25 deletions(-)\n"},{"id":"87373","messageId":"7v1w0pdze0.fsf@gitster.siamese.dyndns.org","threadId":"15021","inReplyTo":"20080815142439.GA10609@laptop","subject":"Re: [RFC PATCH 0/9] Narrow/Sparse checkout round 3: \"easy mode\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-08-16T10:31:51Z","receivedAt":"2008-08-16T10:31:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> The implementation with insights from Junio turns out smaller and better\n> (and I was thinking about applying it for huge maildir).\n\nI think I can agree with the general direction this is taking, except that\nwe would need to think about the transition plans. Note that I haven't\nreally read through the full series yet.\n\n> Another difference from the last round is \"narrow rules\" will not be preserved\n> when switching branches. When you switch branch with no option, you will get\n> full checkout. You may want to use --path|--add-path|--remove-path when\n> switching branches to have narrow checkout again.\n\nYou could save the \"narrow rules\" in the extension section of the index.\nIf the final form of this series needs to use a separate CE_NO_CHECKOUT\nbit (which would make the resulting index incompatible with the current\ngit), the narrow rules section can be marked as \"your git must understand\nthis\" class of extension to make sure that people do not mistakenly access\nan index written by this new version of git with the current or older git.\n\n> Now back to technical POV. I did not reuse CE_VALID (assume unchanged) bit\n> because it has been used for core.ignorestat.\n\nI am not sure what's the relation between these two.\n"},{"id":"87407","messageId":"fcaeb9bf0808162212y756465a5ib7ce3a8849765033@mail.gmail.com","threadId":"15021","inReplyTo":"7v1w0pdze0.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFC PATCH 0/9] Narrow/Sparse checkout round 3: \"easy mode\"","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2008-08-17T05:12:50Z","receivedAt":"2008-08-17T05:12:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On 8/16/08, Junio C Hamano wrote:\n> Nguyen Thai Ngoc Duy writes:\n> > Another difference from the last round is \"narrow rules\" will not be preserved\n> > when switching branches. When you switch branch with no option, you will get\n> > full checkout. You may want to use --path|--add-path|--remove-path when\n> > switching branches to have narrow checkout again.\n>\n>\n> You could save the \"narrow rules\" in the extension section of the index.\n> If the final form of this series needs to use a separate CE_NO_CHECKOUT\n> bit (which would make the resulting index incompatible with the current\n> git), the narrow rules section can be marked as \"your git must understand\n> this\" class of extension to make sure that people do not mistakenly access\n> an index written by this new version of git with the current or older git.\n\nThe problem is \"narrow rules\" may change over time in a way that git\nmay handle it wrong. Assume that you have a directory with two files:\na and b. You first narrow checkout a (which would save the rule\n\"checkout a\"). Then you do \"git checkout b\". When you update HEAD,\nwhat should happen?\n - consider only a and b in narrow area (new files not counted)\n - consider the whole directory in narrow area (new files counted)\n\nThis does not matter until we implement strict mode that only checkout\nnew files inside narrow area (the usage is similar to submodule).\n\n> > Now back to technical POV. I did not reuse CE_VALID (assume unchanged) bit\n> > because it has been used for core.ignorestat.\n>\n>\n> I am not sure what's the relation between these two.\n\nBecause the usage is different? When you \"git update-index foo\" with\ncore.ignorestat=1, it will mark it CE_VALID. And if the same bit is\nused for narrow checkout, the file is considered not existing in\nworkdir. I'd expect foo is still in my narrow area after \"git\nupdate-index foo\".\n-- \nDuy\n"},{"id":"87411","messageId":"7vbpzs9om7.fsf@gitster.siamese.dyndns.org","threadId":"15021","inReplyTo":"fcaeb9bf0808162212y756465a5ib7ce3a8849765033@mail.gmail.com","subject":"Re: [RFC PATCH 0/9] Narrow/Sparse checkout round 3: \"easy mode\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-08-17T05:50:24Z","receivedAt":"2008-08-17T05:50:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Nguyen Thai Ngoc Duy\" writes:\n\n> On 8/16/08, Junio C Hamano wrote:\n>> Nguyen Thai Ngoc Duy writes:\n>> > Another difference from the last round is \"narrow rules\" will not be preserved\n>> > when switching branches. When you switch branch with no option, you will get\n>> > full checkout. You may want to use --path|--add-path|--remove-path when\n>> > switching branches to have narrow checkout again.\n>>\n>>\n>> You could save the \"narrow rules\" in the extension section of the index.\n>> If the final form of this series needs to use a separate CE_NO_CHECKOUT\n>> bit (which would make the resulting index incompatible with the current\n>> git), the narrow rules section can be marked as \"your git must understand\n>> this\" class of extension to make sure that people do not mistakenly access\n>> an index written by this new version of git with the current or older git.\n>\n> The problem is \"narrow rules\" may change over time in a way that git\n> may handle it wrong. Assume that you have a directory with two files:\n> a and b. You first narrow checkout a (which would save the rule\n> \"checkout a\"). Then you do \"git checkout b\". When you update HEAD,\n> what should happen?\n\nI'd expect that this sequence:\n\n\tgit checkout --set-no-checkout arch include\n git checkout arch/x86 include/asm-x86\n\nto set up narrowing rules to (1) exclude everything in arch/ and include/\narea by default, but (2) allow checking out everything in arch/x86/ and\ninclude/asm-x86/ that currently exist _and_ will exist in different commits\nwhen we switch to.\n\nOn the other hand, if I did this after the above two-command sequence:\n\n\tgit checkout include/Kbuild\n\nthen I'd expect only that file to be added to the checkout set. I think\nyou can record list of pathspecs (with positive and negative) to implement\nthat semantics, no?\n\n>> > Now back to technical POV. I did not reuse CE_VALID (assume unchanged) bit\n>> > because it has been used for core.ignorestat.\n>>\n>> I am not sure what's the relation between these two.\n>\n> Because the usage is different? When you \"git update-index foo\" with\n> core.ignorestat=1, it will mark it CE_VALID. And if the same bit is\n> used for narrow checkout, the file is considered not existing in\n> workdir. I'd expect foo is still in my narrow area after \"git\n> update-index foo\".\n\nOk. We would need to use an extra bit for this.\n\nThe bit 0x4000 is the last one available, so we would want to use it as\n\"this index entry uses more bits than the traditional format\" bit, and\ndefine a backward incompatible on-disk index entry format to actually\nrecord CE_NO_CHECKOUT and other flags we will invent in the future.\n\nPerhaps ondisk_cache_entry structure will have an extra \"unsigned int\nflags2\" after \"flags\" when that bit is on, and we can have 31 more bits in\nflags2, with the highest bit of flags2 signalling the presense of flags3\nword in the future, or something like that.\n\nBy the way, \"uint\" and \"ushort\" in struct ondisk_cache_entry are 4-byte\nand 2-byte network byte order integers; should we write them as uint32_t\nand uint16_t instead in the longer run?\n"},{"id":"87413","messageId":"7v3al49nos.fsf@gitster.siamese.dyndns.org","threadId":"15021","inReplyTo":"7vbpzs9om7.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFC PATCH 0/9] Narrow/Sparse checkout round 3: \"easy mode\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-08-17T06:10:27Z","receivedAt":"2008-08-17T06:10:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> \"Nguyen Thai Ngoc Duy\" writes:\n> ...\n>> The problem is \"narrow rules\" may change over time in a way that git\n>> may handle it wrong. Assume that you have a directory with two files:\n>> a and b. You first narrow checkout a (which would save the rule\n>> \"checkout a\"). Then you do \"git checkout b\". When you update HEAD,\n>> what should happen?\n>\n> I'd expect that this sequence:\n> ...\n> you can record list of pathspecs (with positive and negative) to implement\n> that semantics, no?\n\nBy the way, I was just mentioning the index extension area as a means to\nstore the rules if _you wanted to_. I do not insist you to actually store\nthe rules, and in fact, I do not know if it is even a good idea to do so.\n\n> Ok. We would need to use an extra bit for this.\n>\n> The bit 0x4000 is the last one available, so we would want to use it as\n> \"this index entry uses more bits than the traditional format\" bit, and\n> define a backward incompatible on-disk index entry format to actually\n> record CE_NO_CHECKOUT and other flags we will invent in the future.\n>\n> Perhaps ondisk_cache_entry structure will have an extra \"unsigned int\n> flags2\" after \"flags\" when that bit is on, and we can have 31 more bits in\n> flags2, with the highest bit of flags2 signalling the presense of flags3\n> word in the future, or something like that.\n\nIt might make sense to do this first as a futureproof, if we really want\nto go this route. We can ensure that an index that does use the new flag\nbits won't be misinterpreted by older git.\n\n-- >8 --\nFrom: Junio C Hamano \nDate: Sat, 16 Aug 2008 23:02:08 -0700\nSubject: [PATCH] index: future proof for \"extended\" index entries\n\nWe do not have any more bits in the on-disk index flags word, but we would\nneed to have more in the future. Use the last remaining bits as a signal\nto tell us that the index entry we are looking at is an extended one.\n\nSince we do not understand the extended format yet, we will just error out\nwhen we see it.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 1 +\n read-cache.c | 4 ++++\n 2 files changed, 5 insertions(+), 0 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 2475de9..7b5cc83 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -126,6 +126,7 @@ struct cache_entry {\n \n #define CE_NAMEMASK (0x0fff)\n #define CE_STAGEMASK (0x3000)\n+#define CE_EXTENDED (0x4000)\n #define CE_VALID (0x8000)\n #define CE_STAGESHIFT 12\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 2c03ec3..f0ba224 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1118,6 +1118,10 @@ static void convert_from_disk(struct ondisk_cache_entry *ondisk, struct cache_en\n \tce->ce_size = ntohl(ondisk->size);\n \t/* On-disk flags are just 16 bits */\n \tce->ce_flags = ntohs(ondisk->flags);\n+\n+\t/* For future extension: we do not understand this entry yet */\n+\tif (ce->ce_flags & CE_EXTENDED)\n+\t\tdie(\"Unknown index entry format\");\n \thashcpy(ce->sha1, ondisk->sha1);\n \n \tlen = ce->ce_flags & CE_NAMEMASK;\n-- \n1.6.0.rc3.18.g20157\n"},{"id":"87426","messageId":"loom.20080817T094637-285@post.gmane.org","threadId":"15021","inReplyTo":"7v3al49nos.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFC PATCH 0/9] Narrow/Sparse checkout round 3:","fromName":"Eric Raible","fromEmail":"raible@gmail.com","sentAt":"2008-08-17T09:49:33Z","receivedAt":"2008-08-17T09:49:33Z","isPatch":true,"sender":{"key":"raible@gmail.com","avatar":null},"body":"s/but we would need to have/but we may need/\nin the commit message?\n"},{"id":"87443","messageId":"fcaeb9bf0808170636j3c273ef9sfa55c8e432ab1293@mail.gmail.com","threadId":"15021","inReplyTo":"7v3al49nos.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFC PATCH 0/9] Narrow/Sparse checkout round 3: \"easy mode\"","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2008-08-17T13:36:28Z","receivedAt":"2008-08-17T13:36:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On 8/17/08, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n> > \"Nguyen Thai Ngoc Duy\" writes:\n>\n> > ...\n>\n> >> The problem is \"narrow rules\" may change over time in a way that git\n> >> may handle it wrong. Assume that you have a directory with two files:\n> >> a and b. You first narrow checkout a (which would save the rule\n> >> \"checkout a\"). Then you do \"git checkout b\". When you update HEAD,\n> >> what should happen?\n> >\n> > I'd expect that this sequence:\n>\n> > ...\n>\n> > you can record list of pathspecs (with positive and negative) to implement\n> > that semantics, no?\n>\n> By the way, I was just mentioning the index extension area as a means to\n> store the rules if _you wanted to_. I do not insist you to actually store\n> the rules, and in fact, I do not know if it is even a good idea to do so.\n\nI was more worried about those rules getting out of control because\ngit-checkout is not the only command that can change narrow rules.\nAfter enough commands, the rules can become a mess that you don't even\nwant to look at them. I don't do negative rules now, but yes that's\npossible.\n\n> > Ok. We would need to use an extra bit for this.\n> >\n> > The bit 0x4000 is the last one available, so we would want to use it as\n> > \"this index entry uses more bits than the traditional format\" bit, and\n> > define a backward incompatible on-disk index entry format to actually\n> > record CE_NO_CHECKOUT and other flags we will invent in the future.\n> >\n> > Perhaps ondisk_cache_entry structure will have an extra \"unsigned int\n> > flags2\" after \"flags\" when that bit is on, and we can have 31 more bits in\n> > flags2, with the highest bit of flags2 signalling the presense of flags3\n> > word in the future, or something like that.\n>\n>\n> It might make sense to do this first as a futureproof, if we really want\n> to go this route. We can ensure that an index that does use the new flag\n> bits won't be misinterpreted by older git.\n\nThe patch is fine. Still we need to do something to prevent older git\nfrom using new index format.\n-- \nDuy\n"},{"id":"87672","messageId":"7viqtxcr1i.fsf@gitster.siamese.dyndns.org","threadId":"15021","inReplyTo":"loom.20080817T094637-285@post.gmane.org","subject":"Re: [RFC PATCH 0/9] Narrow/Sparse checkout round 3:","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-08-19T09:06:33Z","receivedAt":"2008-08-19T09:06:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Raible writes:\n\n> s/but we would need to have/but we may need/\n> in the commit message?\n\nYeah, strictly speaking, perhaps.\n\nOne thing that I refuse to believe is we will need only one more bit and\nafter assigning the 0x4000 bit to whatever that single purpose the index\nwill stay that way forever. So we would need to reserve that bit as the\nextension bit in any case. If we do not have any extension forever, that\nmeans any index entry with the bit set is corrupt, so erroring out would\nbe the right thing to do anyway ;-).\n"},{"id":"87755","messageId":"loom.20080819T210550-149@post.gmane.org","threadId":"15021","inReplyTo":"20080815142439.GA10609@laptop","subject":"Re: [RFC PATCH 0/9] Narrow/Sparse checkout round 3: \"easy mode\"","fromName":"James Pickens","fromEmail":"jepicken@gmail.com","sentAt":"2008-08-19T21:10:03Z","receivedAt":"2008-08-19T21:10:03Z","isPatch":true,"sender":{"key":"jepicken@gmail.com","avatar":null},"body":"Nguyễn Thái Ngọc Duy gmail.com> writes:\n\n> From user POV, we can now checkout a single file or a\n> subdirectory (checking out subdirectory non-recursively is\n> possible too). You may start with a narrow clone like:\n\nIs there any reason for the change in terminology from \"sparse\"\nto \"narrow\"? I understand the difference between \"partial\"\nand \"sparse\", but I can't tell if there's any difference\nbetween \"narrow\" and \"sparse\". If they are the same thing, then\nI think \"sparse\" is the better term.\n\n> Last bit. \"Narrow rules\" for --path|--add-path|--remove-path is\n> currently wildcards separated by colons. While it does the job,\n> it does not allow to checkout easily a subdirectory\n> non-recusively. I was thinking about '*' as \"match everything\n> except slashes\" and '**' as \"match everything even slashes\".\n\nI like this idea - it would make this much more intuitive to use,\nsince '*' and '**' would work the same as they do in the\nshell (for shells that support '**' at least). I tried the patch\nin it's current form, and it took me a while to figure out that\npaths were non-recursive and '*' was matching everything,\nincluding slashes.\n\nJames\n"},{"id":"89196","messageId":"fcaeb9bf0808300221q12d1d984mdf78c81363a909d1@mail.gmail.com","threadId":"15021","inReplyTo":"loom.20080819T210550-149@post.gmane.org","subject":"Re: [RFC PATCH 0/9] Narrow/Sparse checkout round 3: \"easy mode\"","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2008-08-30T09:21:50Z","receivedAt":"2008-08-30T09:21:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On 8/20/08, James Pickens wrote:\n> Nguyễn Thái Ngọc Duy gmail.com> writes:\n>\n> > From user POV, we can now checkout a single file or a\n> > subdirectory (checking out subdirectory non-recursively is\n> > possible too). You may start with a narrow clone like:\n>\n>\n> Is there any reason for the change in terminology from \"sparse\"\n> to \"narrow\"? I understand the difference between \"partial\"\n> and \"sparse\", but I can't tell if there's any difference\n> between \"narrow\" and \"sparse\". If they are the same thing, then\n> I think \"sparse\" is the better term.\n\nI have no particular preference. It's up to the community to choose the name.\n\n> > Last bit. \"Narrow rules\" for --path|--add-path|--remove-path is\n> > currently wildcards separated by colons. While it does the job,\n> > it does not allow to checkout easily a subdirectory\n> > non-recusively. I was thinking about '*' as \"match everything\n> > except slashes\" and '**' as \"match everything even slashes\".\n>\n>\n> I like this idea - it would make this much more intuitive to use,\n> since '*' and '**' would work the same as they do in the\n> shell (for shells that support '**' at least). I tried the patch\n> in it's current form, and it took me a while to figure out that\n> paths were non-recursive and '*' was matching everything,\n> including slashes.\n\nTried the last few days but it was not easy, needed to duplicate\nfnmatch code. I may come up with a less powerful syntax for\nrecusive/non-recursive '*'.\n-- \nDuy\n"}]}