{"thread":{"id":"51992","subject":"Why is \"Sparse checkout leaves no entry on working directory\" a fatal error?","startedAt":"2019-10-08T06:50:15Z","lastAt":"2019-10-09T09:40:12Z","messageCount":3,"participants":["Josef Wolf","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"383656","messageId":"20191008064538.GB30443@raven.inka.de","threadId":"51992","inReplyTo":null,"subject":"Why is \"Sparse checkout leaves no entry on working directory\" a fatal error?","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2019-10-08T06:45:38Z","receivedAt":"2019-10-08T06:50:15Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Hello,\n\nThis is a repost, since the original message seems to have been lost somehow.\n\n\nI am trying to add a file to an arbitrary branch without touching the current\nworktree with as little overhead as possible. This should work no matter in\nwhich state the current worktree is in. And it should not touch the current WT\nin any way.\n\nFor this, the sparse-checkout feature in conjuntion with the \"shared\nrepository\" feature seems to be perfect.\n\nThe basic idea goes like this:\n\n\n   TMP=`mktemp -d /var/tmp/test-XXXXXXXXX`\n   GD=$TMP/git\n   WD=$TMP/wd\n   \n   git --work-tree $WD --git-dir $GD clone -qns -n . $GD\n   git --work-tree $WD --git-dir $GD config core.sparsecheckout true\n   echo path/of/file/which/I/want/to/create >>$GD/info/sparse-checkout\n   \n   git --work-tree $WD --git-dir $GD checkout -b some-branch remotes/origin/some-branch  # !!!\n   \n   ( cd $WD\n     mkdir -p path/of/file/which/I/want/to\n     echo huhuh >path/of/file/which/I/want/to/create\n     git --work-tree $WD --git-dir $GD add path/of/file/which/I/want/to/create\n     git --work-tree $WD --git-dir $GD commit\n     git --work-tree $WD --git-dir $GD push\n   )\n   \n   rm -rf $TMP\n\n\nUnfortunately, the marked command errors out with\n\n   \"error: Sparse checkout leaves no entry on working directory\"\n\nand won't create/switch to the branch that is to be modified.\n\nWhy is this an error? Since there are no matching files, an empty worktree\nis EXACTLY what I wanted. Why will the \"git checkout -b\" command error out?\n\n\nStrange enough, I have some repositories at this machine where the\n.git/info/sparse-checkout file contains only non-existing files and git\nhappily executes this \"git checkout -b XXX remotes/origin/XXX\" command leaving\nthe working tree totally empty all the time.\n\nSomeone understands this inconsistent behaviour?\n\nThanks,\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"383686","messageId":"CABPp-BE8YmVTS=4UWy5jvxBwr3EOcqzbnpWf2Wc78Kv6YScKgQ@mail.gmail.com","threadId":"51992","inReplyTo":"20191008064538.GB30443@raven.inka.de","subject":"Re: Why is \"Sparse checkout leaves no entry on working directory\" a fatal error?","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-08T16:14:27Z","receivedAt":"2019-10-08T16:14:41Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 11:52 PM Josef Wolf <jw@raven.inka.de> wrote:\n>\n> Hello,\n>\n> This is a repost, since the original message seems to have been lost somehow.\n>\n>\n> I am trying to add a file to an arbitrary branch without touching the current\n> worktree with as little overhead as possible. This should work no matter in\n> which state the current worktree is in. And it should not touch the current WT\n> in any way.\n>\n> For this, the sparse-checkout feature in conjuntion with the \"shared\n> repository\" feature seems to be perfect.\n\nI can see the logical progression that a sparse worktree would be less\noverhead than a full worktree, and that a bare worktree would be even\nbetter.  But you're still dealing with unnecessary overhead; you don't\nneed a worktree at all to achieve what you want.\n\nTraditionally, if you wanted to modify another branch without touching\nthe worktree at all, you would use a combination of hash-object,\nmktree, commit-tree, and update-ref.  That would be a better solution\nto your problem than trying to approximate it with a sparse checkout.\nHowever, that's at least four invocations of git, and you said as\nlittle overhead as possible, so I'd recommend you use fast-import.\n\nBut, since you asked some other questions about sparse checkouts...\n\n> The basic idea goes like this:\n>\n>\n>    TMP=`mktemp -d /var/tmp/test-XXXXXXXXX`\n>    GD=$TMP/git\n>    WD=$TMP/wd\n>\n>    git --work-tree $WD --git-dir $GD clone -qns -n . $GD\n>    git --work-tree $WD --git-dir $GD config core.sparsecheckout true\n>    echo path/of/file/which/I/want/to/create >>$GD/info/sparse-checkout\n>\n>    git --work-tree $WD --git-dir $GD checkout -b some-branch remotes/origin/some-branch  # !!!\n>\n>    ( cd $WD\n>      mkdir -p path/of/file/which/I/want/to\n>      echo huhuh >path/of/file/which/I/want/to/create\n>      git --work-tree $WD --git-dir $GD add path/of/file/which/I/want/to/create\n>      git --work-tree $WD --git-dir $GD commit\n>      git --work-tree $WD --git-dir $GD push\n>    )\n>\n>    rm -rf $TMP\n>\n>\n> Unfortunately, the marked command errors out with\n>\n>    \"error: Sparse checkout leaves no entry on working directory\"\n>\n> and won't create/switch to the branch that is to be modified.\n>\n> Why is this an error? Since there are no matching files, an empty worktree\n> is EXACTLY what I wanted. Why will the \"git checkout -b\" command error out?\n\nIt is very easy to mess up the sparse specifications.  We can't check\nfor all errors, but a pretty obvious one is when people specify\nrestrictions that match no path.  We can at least give an error in\nthat case.  There are times when folks might intentionally specify\npaths that don't match anything, but they are quite rare.  The ones I\ncan think of:\n\n1) When they are doing something exotic where they are just trying to\napproximate something else rather than actual use sparse checkouts as\nintended.\n2) When they've learned about sparse checkouts and just want to test\nwhat things are like in extreme situations.\n\nCase 1 consists of stuff like what you are doing here, for which there\nare better solutions, or when I was attempting to simulate the\nperformance issues microsoft folks were having with a really large\nrepo and knowing they used sparse checkouts as part of VFS-for-git (I\ncreated a very large index and had no entries checked out at first,\nbut then ran into these errors, and added one file to the index and\nhad a sparse specification match it.)\n\nFor case 2, people learn that an empty working tree is a too extreme\nsituation that we'll throw an error at and so they adjust and make\nsure to match at least one path.\n\n> Strange enough, I have some repositories at this machine where the\n> .git/info/sparse-checkout file contains only non-existing files and git\n> happily executes this \"git checkout -b XXX remotes/origin/XXX\" command leaving\n> the working tree totally empty all the time.\n\nI can't reproduce:\n\n$ git config core.sparseCheckout true\n$ echo 'non-existent' > .git/info/sparse-checkout\n$ git checkout -b next origin/next\nerror: Sparse checkout leaves no entry on working directory\n\nCan you provide any more details about how you get into this state?\n\n> Someone understands this inconsistent behaviour?\n\nNo, but I wouldn't be surprised if there are bugs and edge cases.  I\nthink I ran into one or two when testing things out, didn't take good\nenough notes, and had trouble reproducing later.  The sparse checkout\nstuff has been under-tested and not well documented, something Stolee\nis trying to fix right now.\n"},{"id":"383740","messageId":"20191009093719.GC30443@raven.inka.de","threadId":"51992","inReplyTo":"CABPp-BE8YmVTS=4UWy5jvxBwr3EOcqzbnpWf2Wc78Kv6YScKgQ@mail.gmail.com","subject":"Re: Why is \"Sparse checkout leaves no entry on working directory\" a fatal error?","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2019-10-09T09:37:19Z","receivedAt":"2019-10-09T09:40:12Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Thanks for your comprehensive answer, Elijah!\n\nOn Di, Okt 08, 2019 at 09:14:27 -0700, Elijah Newren wrote:\n> On Mon, Oct 7, 2019 at 11:52 PM Josef Wolf <jw@raven.inka.de> wrote:\n> >\n> > I am trying to add a file to an arbitrary branch without touching the current\n> > worktree with as little overhead as possible.\n>\n> I can see the logical progression that a sparse worktree would be less\n> overhead than a full worktree, and that a bare worktree would be even\n> better.  But you're still dealing with unnecessary overhead; you don't\n> need a worktree at all to achieve what you want.\n\nWell, the \"as little overhead as possible\" must be seen from the context. This\nis a repository with roundabout 10GB and more than 6200 files. Shared-clones\nwith sparse-worktree is a BIG BIG BIG improvement here, which reduces\noperations from \"minutes\" to \"withhin a second\".\n\n> Traditionally, if you wanted to modify another branch without touching\n> the worktree at all, you would use a combination of hash-object,\n> mktree, commit-tree, and update-ref.  That would be a better solution\n> to your problem than trying to approximate it with a sparse checkout.\n> However, that's at least four invocations of git, and you said as\n> little overhead as possible, so I'd recommend you use fast-import.\n\nI have taken a look into the commands you are recommending, and indeed, they\nseem to be better suited. Especially fast-import looks very\npromising. Unfortunately, those commands require intimate knowledge about git\ninternals. I'll take a closer look into this!\n\n> It is very easy to mess up the sparse specifications.  We can't check\n> for all errors, but a pretty obvious one is when people specify\n> restrictions that match no path.\n\nBut why erroring out only on completely empty tree? Why not requiring that\n_every_ line in .git/info/sparse-checkout should match at least one file?\nWould make no sense, right?\n\n> We can at least give an error in that case.\n\nWhy must this be a fatal error? Wouldn't a warning suffice?\n\n> 2) When they've learned about sparse checkouts and just want to test\n> what things are like in extreme situations.\n[ ... ]\n> For case 2, people learn that an empty working tree is a too extreme\n> situation that we'll throw an error at and so they adjust and make\n> sure to match at least one path.\n\nWhen I am trying to learn how a new feature works, I tend to double-check the\nresults. If I expect contens but end up with an empty WT, I'd go and double\ncheck the specifications I've given anyway.\n\nI can easily understand that a warning might be desirable. But erroring out\nand failing to honor the \"-b\" flag is a bit too drastic, IMHO.\n\n> > Strange enough, I have some repositories at this machine where the\n> > .git/info/sparse-checkout file contains only non-existing files and git\n> > happily executes this \"git checkout -b XXX remotes/origin/XXX\" command leaving\n> > the working tree totally empty all the time.\n> \n> I can't reproduce:\n> \n> $ git config core.sparseCheckout true\n> $ echo 'non-existent' > .git/info/sparse-checkout\n> $ git checkout -b next origin/next\n> error: Sparse checkout leaves no entry on working directory\n> \n> Can you provide any more details about how you get into this state?\n\nUnfortunately not.\n\nHonestly, I have tried to reproduce for several days, since I tried\nto find a way how to work around that fatal error. Unfortunately, I could not\nfind how to reproduce it. The only thing I can say is: threre are several\nclones on my disk which happily switch branches with an empty WT and without\nany complaints.\n\n> > Someone understands this inconsistent behaviour?\n> \n> No, but I wouldn't be surprised if there are bugs and edge cases.  I\n> think I ran into one or two when testing things out, didn't take good\n> enough notes, and had trouble reproducing later.  The sparse checkout\n> stuff has been under-tested and not well documented, something Stolee\n> is trying to fix right now.\n\nYes, I've seen the work on the ML. But I am only a user of git and have a very\nhard time to understand what is going on there.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"}]}