{"thread":{"id":"60486","subject":"Error when \"git mv\" file in a sparsed checkout","startedAt":"2023-11-07T13:32:02Z","lastAt":"2023-11-10T20:11:57Z","messageCount":4,"participants":["Josef Wolf","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"484529","messageId":"20231107130303.GS7041@raven.inka.de","threadId":"60486","inReplyTo":null,"subject":"Error when \"git mv\" file in a sparsed checkout","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2023-11-07T13:03:03Z","receivedAt":"2023-11-07T13:32:02Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Hello,\n\nI have used the procedure described below for many years. In fact,\nthis procedure is part of a script which I am using for about 10 years.\nThis procedure was definitely working with git-2-25-1 and git-2.26.2.\n\nNow, with git-2.34.1 (on a freshly installed ubuntu-22.04), this\nprocedure fails.\n\nHere is what I do:\n\nI want to rename a file on a branch which is not currently checked out\nwithout messing/touching my current working directory.\n\nFor this, I first create a clone of the repo with shared git-directory:\n\n  $ SANDBOX=/var/tmp/manage-scans-X1pKZQiey\n  $ WT=$SANDBOX/wt\n  $ GIT=$SANDBOX/git\n\n  $ mkdir -p $SANDBOX\n  $ git --work-tree $WT --git-dir $GIT clone -qns -n ~/upstream-repo $GIT\n\nThen, I do a sparse checkout in this clone, containing only the file\nthat is to be renamed:\n\n  $ cd $WT\n  $ echo 'path/to/old-filename' >>$GIT/info/sparse-checkout\n  $ git --work-tree $WT --git-dir $GIT config core.sparsecheckout true\n  $ git --work-tree $WT --git-dir $GIT checkout -b the-branch remotes/origin/the-branch\n  Switched to a new branch 'the-branch'\n\nNext step would be to \"git mv\" the file:\n\n  $ mkdir -p /path/to  # already exists, but should do no harm\n  $ git --work-tree $WT --git-dir $GIT mv path/to/old-filename path/to/new-filename\n  The following paths and/or pathspecs matched paths that exist\n  outside of your sparse-checkout definition, so will not be\n  updated in the index:\n  path/to/new-filename\n  hint: If you intend to update such entries, try one of the following:\n  hint: * Use the --sparse option.\n  hint: * Disable or modify the sparsity rules.\n  hint: Disable this message with \"git config advice.updateSparsePath false\"\n\nThis error is something I have not expected.\n\nError message suggests, there already exists a file named \"new-filename\". This\nis not true at all. There is no file named \"new-filename\" in the entire\nrepository. Not in any directory of any branch.\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"484553","messageId":"CABPp-BEAU8rPeNHphut0ZxcLdH0pzjh+Z_CF+rg2uhvVZoZfxg@mail.gmail.com","threadId":"60486","inReplyTo":"20231107130303.GS7041@raven.inka.de","subject":"Re: Error when \"git mv\" file in a sparsed checkout","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-11-08T02:21:00Z","receivedAt":"2023-11-08T02:22:48Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Tue, Nov 7, 2023 at 5:32 AM Josef Wolf <jw@raven.inka.de> wrote:\n>\n> Hello,\n>\n> I have used the procedure described below for many years. In fact,\n> this procedure is part of a script which I am using for about 10 years.\n> This procedure was definitely working with git-2-25-1 and git-2.26.2.\n>\n> Now, with git-2.34.1 (on a freshly installed ubuntu-22.04), this\n> procedure fails.\n>\n> Here is what I do:\n>\n> I want to rename a file on a branch which is not currently checked out\n> without messing/touching my current working directory.\n>\n> For this, I first create a clone of the repo with shared git-directory:\n>\n>   $ SANDBOX=/var/tmp/manage-scans-X1pKZQiey\n>   $ WT=$SANDBOX/wt\n>   $ GIT=$SANDBOX/git\n>\n>   $ mkdir -p $SANDBOX\n>   $ git --work-tree $WT --git-dir $GIT clone -qns -n ~/upstream-repo $GIT\n>\n> Then, I do a sparse checkout in this clone, containing only the file\n> that is to be renamed:\n>\n>   $ cd $WT\n>   $ echo 'path/to/old-filename' >>$GIT/info/sparse-checkout\n>   $ git --work-tree $WT --git-dir $GIT config core.sparsecheckout true\n>   $ git --work-tree $WT --git-dir $GIT checkout -b the-branch remotes/origin/the-branch\n>   Switched to a new branch 'the-branch'\n>\n> Next step would be to \"git mv\" the file:\n>\n>   $ mkdir -p /path/to  # already exists, but should do no harm\n>   $ git --work-tree $WT --git-dir $GIT mv path/to/old-filename path/to/new-filename\n\nsparse checkouts are designed such that only files matching the\npatterns in the sparse-checkout file should be present in the working\ntree, so renaming to a path that should not be present is problematic.\nWe could possibly have \"git-mv\" immediately remove the path from the\nworking tree (while leaving the new pathname in the index), but that's\nproblematic in that users often overlook the index and only look at\nthe working tree and might think the file was deleted instead of\nrenamed.  Not immediately removing it is potentially even worse,\nbecause any subsequent operation (particularly ones like checkout,\nreset, merge, rebase, etc.) are likely to nuke the file from the\nworking tree and the fact that the removal is delayed makes it much\nharder for users to understand and diagnose.\n\nSo, Stolee fixed this to make it throw an error; see\nhttps://lore.kernel.org/git/pull.1018.v4.git.1632497954.gitgitgadget@gmail.com/\nfor details.  His description did focus on cone mode, but you'll note\nthat none of my explanation here did.  The logic for making this an\nerror fully applies to non-cone mode for all the same reasons.\n\nIf you want to interact with `path/to/new-filename` as a path within\nyour sparse checkout (as suggested by your git-mv command), then that\npath should actually be part of your sparse checkout.  In other words,\nyou should add `path/to/new-filename` to $GIT/info/sparse-checkout and\ndo so _before_ attempting your `git mv` command.  If you don't like\nthat for some reason, you are allowed to instead ignore the\nproblematic consequences of renaming outside the sparse-checkout by\nproviding the `--sparse` flag.  Both of these possibilities are\ndocumented in the hints provided along with the error message you\nshowed below:\n\n>   The following paths and/or pathspecs matched paths that exist\n>   outside of your sparse-checkout definition, so will not be\n>   updated in the index:\n>   path/to/new-filename\n>   hint: If you intend to update such entries, try one of the following:\n>   hint: * Use the --sparse option.\n>   hint: * Disable or modify the sparsity rules.\n>   hint: Disable this message with \"git config advice.updateSparsePath false\"\n>\n> This error is something I have not expected.\n>\n> Error message suggests, there already exists a file named \"new-filename\". This\n> is not true at all. There is no file named \"new-filename\" in the entire\n> repository. Not in any directory of any branch.\n\nYou are correct; the wording of the error message here is suboptimal\nand seems to have been focused more on the git-add case (the error\nmessage is shared by git-add, git-mv, and git-rm).  Thanks for\npointing it out!  We could improve that wording, perhaps with\nsomething like:\n\n    The following paths and/or pathspecs match paths that are\n    outside of your sparse-checkout definition, so will not be\n    updated:\n\nWhich is still slightly slanted towards git-add and git-rm cases, but\nI hope it works better than the current message.  Thoughts?\n"},{"id":"484572","messageId":"20231108113636.GT7041@raven.inka.de","threadId":"60486","inReplyTo":"CABPp-BEAU8rPeNHphut0ZxcLdH0pzjh+Z_CF+rg2uhvVZoZfxg@mail.gmail.com","subject":"Re: Error when \"git mv\" file in a sparsed checkout","fromName":"Josef Wolf","fromEmail":"jw@raven.inka.de","sentAt":"2023-11-08T11:36:36Z","receivedAt":"2023-11-08T11:38:18Z","isPatch":false,"sender":{"key":"jw@raven.inka.de","avatar":null},"body":"Thanks for the reply, Elijah!\n\nOn Tue, Nov 07, 2023 at 06:21:00PM -0800, Elijah Newren wrote:\n> On Tue, Nov 7, 2023 at 5:32 AM Josef Wolf <jw@raven.inka.de> wrote:\n> > I have used the procedure described below for many years. In fact,\n> > this procedure is part of a script which I am using for about 10 years.\n> > This procedure was definitely working with git-2-25-1 and git-2.26.2.\n> >\n> > Now, with git-2.34.1 (on a freshly installed ubuntu-22.04), this\n> > procedure fails.\n> >\n> > Here is what I do:\n> >\n> > I want to rename a file on a branch which is not currently checked out\n> > without messing/touching my current working directory.\n> >\n> > For this, I first create a clone of the repo with shared git-directory:\n> >\n> >   $ SANDBOX=/var/tmp/manage-scans-X1pKZQiey\n> >   $ WT=$SANDBOX/wt\n> >   $ GIT=$SANDBOX/git\n> >\n> >   $ mkdir -p $SANDBOX\n> >   $ git --work-tree $WT --git-dir $GIT clone -qns -n ~/upstream-repo $GIT\n> >\n> > Then, I do a sparse checkout in this clone, containing only the file\n> > that is to be renamed:\n> >\n> >   $ cd $WT\n> >   $ echo 'path/to/old-filename' >>$GIT/info/sparse-checkout\n> >   $ git --work-tree $WT --git-dir $GIT config core.sparsecheckout true\n> >   $ git --work-tree $WT --git-dir $GIT checkout -b the-branch remotes/origin/the-branch\n> >   Switched to a new branch 'the-branch'\n> >\n> > Next step would be to \"git mv\" the file:\n> >\n> >   $ mkdir -p /path/to  # already exists, but should do no harm\n> >   $ git --work-tree $WT --git-dir $GIT mv path/to/old-filename path/to/new-filename\n> \n> sparse checkouts are designed such that only files matching the\n> patterns in the sparse-checkout file should be present in the working\n> tree, so renaming to a path that should not be present is problematic.\n> We could possibly have \"git-mv\" immediately remove the path from the\n> working tree (while leaving the new pathname in the index), but that's\n> problematic in that users often overlook the index and only look at\n> the working tree and might think the file was deleted instead of\n> renamed.  Not immediately removing it is potentially even worse,\n> because any subsequent operation (particularly ones like checkout,\n> reset, merge, rebase, etc.) are likely to nuke the file from the\n> working tree and the fact that the removal is delayed makes it much\n> harder for users to understand and diagnose.\n> \n> So, Stolee fixed this to make it throw an error; see\n> https://lore.kernel.org/git/pull.1018.v4.git.1632497954.gitgitgadget@gmail.com/\n> for details.  His description did focus on cone mode, but you'll note\n> that none of my explanation here did.  The logic for making this an\n> error fully applies to non-cone mode for all the same reasons.\n> \n> If you want to interact with `path/to/new-filename` as a path within\n> your sparse checkout (as suggested by your git-mv command), then that\n> path should actually be part of your sparse checkout.  In other words,\n> you should add `path/to/new-filename` to $GIT/info/sparse-checkout and\n> do so _before_ attempting your `git mv` command.  If you don't like\n> that for some reason, you are allowed to instead ignore the\n> problematic consequences of renaming outside the sparse-checkout by\n> providing the `--sparse` flag.  Both of these possibilities are\n> documented in the hints provided along with the error message you\n> showed below:\n> \n> >   The following paths and/or pathspecs matched paths that exist\n> >   outside of your sparse-checkout definition, so will not be\n> >   updated in the index:\n> >   path/to/new-filename\n> >   hint: If you intend to update such entries, try one of the following:\n> >   hint: * Use the --sparse option.\n> >   hint: * Disable or modify the sparsity rules.\n> >   hint: Disable this message with \"git config advice.updateSparsePath false\"\n> >\n> > This error is something I have not expected.\n> >\n> > Error message suggests, there already exists a file named \"new-filename\". This\n> > is not true at all. There is no file named \"new-filename\" in the entire\n> > repository. Not in any directory of any branch.\n> \n> You are correct; the wording of the error message here is suboptimal\n> and seems to have been focused more on the git-add case (the error\n> message is shared by git-add, git-mv, and git-rm).  Thanks for\n> pointing it out!  We could improve that wording, perhaps with\n> something like:\n> \n>     The following paths and/or pathspecs match paths that are\n>     outside of your sparse-checkout definition, so will not be\n>     updated:\n> \n> Which is still slightly slanted towards git-add and git-rm cases, but\n> I hope it works better than the current message.  Thoughts?\n\nYes, the wording was pretty much confusing me, since i could not find a file\nnamed \"new-file\" anywhere in the repo.\n\n\nThere are more things confusing concerning sparse mode:\n\n- It is not clear from git-sparse-checkout(1) when changes to\n  $GIT_DIR/info/sparse-checkout are catched up. In my case: would it be enough\n  to add the new pathname just before git-mv or would a fresh git-checkout be\n  needed after modifying $GIT_DIR/info/sparse-checkout? You have clarified\n  this in your response, but shouldn't this be clear from the manpage?\n\n- git-sparse-checkout(1) refers to \"skip-worktree bit\". This concept is\n  potentially not very familiar to the average git user which uses mostly\n  porcelain. Thus, edge cases remain to be unclear.\n\n- The pathspecs refers to .gitignore (which by itself is not very clear). But\n  there are differences:\n  1. giignore is relative to containing directoy, which don't seem to make\n     much sense for sparse mode\n  2. sparse specs are the opposite of gitignore, which seems to have different\n     meaning in some edge-cases.\n\n- For cone, it is not clear how the two \"accepted patterns\" look like what the\n  semantics are. I understand that specifying a directory adds siblings\n  recursively. But what does the \"Parent\" mode mean exactly and when/how is\n  this recognized? I guess, this is just a mis-namer? IMHO, parent of /a/b/c would\n  be /a/b and not /a/b/c/* (as git-sparse-chekout(1) suggests).\n\nI guess all this is very clear to the core-developers. But for the occasional\nuser like myself, all this is pretty much confusing.\n\n\n-- \nJosef Wolf\njw@raven.inka.de\n"},{"id":"484711","messageId":"CABPp-BGRycdj5Z_YzPLvQ1CqBdz5Px797gHD6P8sf8mNfxghrQ@mail.gmail.com","threadId":"60486","inReplyTo":"20231108113636.GT7041@raven.inka.de","subject":"Re: Error when \"git mv\" file in a sparsed checkout","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-11-10T20:11:40Z","receivedAt":"2023-11-10T20:11:57Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Wed, Nov 8, 2023 at 3:38 AM Josef Wolf <jw@raven.inka.de> wrote:\n>\n> Thanks for the reply, Elijah!\n>\n[...]\n> > > Error message suggests, there already exists a file named \"new-filename\". This\n> > > is not true at all. There is no file named \"new-filename\" in the entire\n> > > repository. Not in any directory of any branch.\n> >\n> > You are correct; the wording of the error message here is suboptimal\n> > and seems to have been focused more on the git-add case (the error\n> > message is shared by git-add, git-mv, and git-rm).  Thanks for\n> > pointing it out!  We could improve that wording, perhaps with\n> > something like:\n> >\n> >     The following paths and/or pathspecs match paths that are\n> >     outside of your sparse-checkout definition, so will not be\n> >     updated:\n> >\n> > Which is still slightly slanted towards git-add and git-rm cases, but\n> > I hope it works better than the current message.  Thoughts?\n>\n> Yes, the wording was pretty much confusing me, since i could not find a file\n> named \"new-file\" anywhere in the repo.\n>\n>\n> There are more things confusing concerning sparse mode:\n\nSweet, thanks for taking the time to write these up.  It certainly\nhelps confirm some of the directions we picked and changes we made to\nmake things a little clearer, and helps us continue working in that\ndirection.  Some comments below on individual points...\n\n> - It is not clear from git-sparse-checkout(1) when changes to\n>   $GIT_DIR/info/sparse-checkout are catched up. In my case: would it be enough\n>   to add the new pathname just before git-mv or would a fresh git-checkout be\n>   needed after modifying $GIT_DIR/info/sparse-checkout? You have clarified\n>   this in your response, but shouldn't this be clear from the manpage?\n\nI believe at least part of this confusion is due to using the old\nstyle of handling sparse checkouts; namely, by actually editing\n$GIT_DIR/info/sparse-checkout.  We have taken pains to guide people\naway from that workflow, because it is both more work, and leads to\nmore confusion.  If you instead do a\n   git sparse-checkout set --no-cone <pattern1> <pattern2> ... <patternN>\nor a\n   git sparse-checkout add <another-pattern>\nthen the sparse-checkout command handles populating the\n$GIT_DIR/info/sparse-checkout for you as well as any needed checkout,\nmeaning that there isn't a \"catch up\" step as there traditionally was.\nAnd it makes it a bit clearer that if you add some path to your\nsparse-checkout, then your sparse-checkout is ready to handle the\nadditional path right away.\n\n> - git-sparse-checkout(1) refers to \"skip-worktree bit\". This concept is\n>   potentially not very familiar to the average git user which uses mostly\n>   porcelain. Thus, edge cases remain to be unclear.\n\nI totally agree that the \"skip-worktree bit\" is something we should\navoid exposing to the user.  I called it out previously:\n\"\"\"\nMost sparse checkout users are unaware of this implementation\ndetail, and the term should generally be avoided in user-facing\ndescriptions and command flags.  Unfortunately, prior to the\n`sparse-checkout` subcommand this low-level detail was exposed,\nand as of time of writing, is still exposed in various places.\n\"\"\"\n\nHowever, we should also note that you reported using v2.34.1, which is\nreally quite old.  The current git-sparse-checkout(1) has been almost\ncompletely overhauled in the meantime:\n\n$ git show v2.34.1:Documentation/git-sparse-checkout.txt | wc -l\n263\n$ git diff --stat v2.34.1 v2.43.0-rc1 -- Documentation/git-sparse-checkout.txt\n Documentation/git-sparse-checkout.txt | 446 +++++++++++++++++++++++++---------\n 1 file changed, 333 insertions(+), 113 deletions(-)\n\nAnd, in particular, \"skip-worktree\" doesn't appear until quite a bit\nlater in the file, and then only appears in a section labelled\n\"INTERNALS -- SPARSE CHECKOUT\".\n\n> - The pathspecs refers to .gitignore (which by itself is not very clear). But\n>   there are differences:\n>   1. giignore is relative to containing directoy, which don't seem to make\n>      much sense for sparse mode\n>   2. sparse specs are the opposite of gitignore, which seems to have different\n>      meaning in some edge-cases.\n\nYeah, copying .gitignore syntax and merely referring to the gitignore\nmanual for specification of the patterns was a huge design mistake.\nI've hated it since the beginning.  The internals actually managed to\nmake it *even more* confusing for quite some time as it referred to\neverything as \"excludes\", regardless of whether used for gitignore or\nsparse-checkout.  But yeah, these and other inherent problems with\nnon-cone mode are called out in the \"INTERNALS -- NON-CONE PROBLEMS\"\nsection of the manual, and is a big piece of why we recommend users\nmigrate away from it if possible.\n\n> - For cone, it is not clear how the two \"accepted patterns\" look like what the\n>   semantics are.\n\nYeah, it is a bit complex, and I advocated for just using a different\ncontrol file for cone mode to help us step away from the blight of\n$GIT_DIR/info/sparse-checkout and its inherent tie to gitignore.  I\ndidn't win that one.\n\nHowever, why does it actually matter?  You shouldn't be bothering with\nthe patterns or editing the $GIT_DIR/info/sparse-checkout file for\ncone mode.  You just\n    git sparse-checkout set <directory1> <directory2> ... <directoryN>\nand then the file will be set up for you.\n\n>  I understand that specifying a directory adds siblings\n>   recursively. But what does the \"Parent\" mode mean exactly and when/how is\n>   this recognized? I guess, this is just a mis-namer? IMHO, parent of /a/b/c would\n>   be /a/b and not /a/b/c/* (as git-sparse-chekout(1) suggests).\n\nNo, /a/b/c/* is not the parent, that's the portion that says that all\nthings below a/b/c (i.e. all descendant files) should be included.\n\nThe parent parts would be where it adds /a/b/ and /a/ as patterns to\nensure things directly under a/ and directly under a/b/ are included.\n\nI think the newer version of the manual might explain this a little\nbetter, but honestly, attempting to explain it is a losing battle.\nUsers shouldn't read or edit the sparse-checkout file in cone mode.\nWe let them, but we strongly recommend against it.  Just pass the\nactual directory names to `git sparse-checkout {set,add}` and let it\ntake care of the patterns for you.\n"}]}