{"thread":{"id":"59281","subject":"Parallel worktree checkouts result in index.lock exists","startedAt":"2023-02-21T17:33:34Z","lastAt":"2023-03-07T17:30:27Z","messageCount":3,"participants":["Raul E Rangel","brian m. carlson","Raul Rangel"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"472385","messageId":"Y/UAaC4oBPIby4kV@google.com","threadId":"59281","inReplyTo":null,"subject":"Parallel worktree checkouts result in index.lock exists","fromName":"Raul E Rangel","fromEmail":"rrangel@chromium.org","sentAt":"2023-02-21T17:33:28Z","receivedAt":"2023-02-21T17:33:34Z","isPatch":false,"sender":{"key":"rrangel@chromium.org","avatar":null},"body":"Hello,\nI'm trying to extract multiple trees in parallel so I can create a\ntarball of the trees. I can't use `git archive` since it doesn't\ncurrently produce hermetic output, and I need to support older git\nversions.\n\nIn essence what I'm trying to do is:\n\n    git --work-tree ~/tmp/bob1 checkout ff27f5415797ead8bc775518a08f3a4ab24abd53 -- . &\n    git --work-tree ~/tmp/bob2 checkout e70ebd7c76b9f9ad44b59e3002a5c57be5b9dc12 -- . &\n\nWhen I do this though, I get the following error:\n    [1] 4027482\n    [2] 4027483\n    fatal: Unable to create '/home/rrangel/cros-bazel/.repo/project-objects/chromiumos/platform/vboot_reference.git/./index.lock': File exists.\n    \n    Another git process seems to be running in this repository, e.g.\n    an editor opened by 'git commit'. Please make sure all processes\n    are terminated then try again. If it still fails, a git process\n    may have crashed in this repository earlier:\n    remove the file manually to continue.\n\nIs this expected? I'm not sure why the index is coming into play here.\nIs there another method I should be using to extract a tree into a\ndirectory?\n\nIf the index.lock isn't actually protecting me from anything, I was\nthinking of creating a symlink clone of the .git directory for each\nparallel invocation. This way the each index.lock gets written to its\nown directory:\n\n    mkdir ../vboot_reference.git.1\n    find . -mindepth 1 -maxdepth 1 -exec echo ln -s '../vboot_reference.git/{}' ../vboot_reference.git.1/ \\;\n    \n    mkdir ../vboot_reference.git.r2\n    find . -mindepth 1 -maxdepth 1 -exec echo ln -s '../vboot_reference.git/{}' ../vboot_reference.git.2/ \\;\n\n    git -C ../vboot_reference.git.1 --work-tree ~/tmp/bob1 checkout ff27f5415797ead8bc775518a08f3a4ab24abd53 -- . &\n    git -C ../vboot_reference.git.2 --work-tree ~/tmp/bob2 checkout e70ebd7c76b9f9ad44b59e3002a5c57be5b9dc12 -- . &\n\nThough if there is another way to handle this it would be great.\n\nThanks,\nRaul\n"},{"id":"472419","messageId":"Y/VcbWogtHrpXU8Z@tapette.crustytoothpaste.net","threadId":"59281","inReplyTo":"Y/UAaC4oBPIby4kV@google.com","subject":"Re: Parallel worktree checkouts result in index.lock exists","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-02-22T00:06:05Z","receivedAt":"2023-02-22T00:06:12Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-02-21 at 17:33:28, Raul E Rangel wrote:\n> Hello,\n> I'm trying to extract multiple trees in parallel so I can create a\n> tarball of the trees. I can't use `git archive` since it doesn't\n> currently produce hermetic output, and I need to support older git\n> versions.\n> \n> In essence what I'm trying to do is:\n> \n>     git --work-tree ~/tmp/bob1 checkout ff27f5415797ead8bc775518a08f3a4ab24abd53 -- . &\n>     git --work-tree ~/tmp/bob2 checkout e70ebd7c76b9f9ad44b59e3002a5c57be5b9dc12 -- . &\n> \n> When I do this though, I get the following error:\n>     [1] 4027482\n>     [2] 4027483\n>     fatal: Unable to create '/home/rrangel/cros-bazel/.repo/project-objects/chromiumos/platform/vboot_reference.git/./index.lock': File exists.\n>     \n>     Another git process seems to be running in this repository, e.g.\n>     an editor opened by 'git commit'. Please make sure all processes\n>     are terminated then try again. If it still fails, a git process\n>     may have crashed in this repository earlier:\n>     remove the file manually to continue.\n> \n> Is this expected? I'm not sure why the index is coming into play here.\n> Is there another method I should be using to extract a tree into a\n> directory?\n\nThis is expected because when you do a checkout, the timestamps and\nother metadata of the files are written into the index.  This is what\nmakes `git status` work quickly: if the metadata of the files hasn't\nchanged, Git doesn't have to re-read them to verify their contents.  You\ndefinitely don't want to delete that because you'll likely end up with\ncorrupt data.\n\nIf you want to create multiple worktrees, use `git worktree add`, which\ncan create multiple worktrees that each have their own index, but share\nthe object store.  When you're done with it, run `git worktree remove`,\nand everything will be cleaned up.\n\nNote that with worktrees, you can have at most one worktree with a given\nbranch (or detached head) checked out at a time.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"473128","messageId":"CAHQZ30BrwiW6ia3v31gto9D8oejkj2UrqK93MjPFNwBJuOBK0Q@mail.gmail.com","threadId":"59281","inReplyTo":"Y/VcbWogtHrpXU8Z@tapette.crustytoothpaste.net","subject":"Re: Parallel worktree checkouts result in index.lock exists","fromName":"Raul Rangel","fromEmail":"rrangel@chromium.org","sentAt":"2023-03-07T17:25:01Z","receivedAt":"2023-03-07T17:30:27Z","isPatch":false,"sender":{"key":"rrangel@chromium.org","avatar":null},"body":"On Tue, Feb 21, 2023 at 5:06 PM brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n>\n> On 2023-02-21 at 17:33:28, Raul E Rangel wrote:\n> > Hello,\n> > I'm trying to extract multiple trees in parallel so I can create a\n> > tarball of the trees. I can't use `git archive` since it doesn't\n> > currently produce hermetic output, and I need to support older git\n> > versions.\n> >\n> > In essence what I'm trying to do is:\n> >\n> >     git --work-tree ~/tmp/bob1 checkout ff27f5415797ead8bc775518a08f3a4ab24abd53 -- . &\n> >     git --work-tree ~/tmp/bob2 checkout e70ebd7c76b9f9ad44b59e3002a5c57be5b9dc12 -- . &\n> >\n> > When I do this though, I get the following error:\n> >     [1] 4027482\n> >     [2] 4027483\n> >     fatal: Unable to create '/home/rrangel/cros-bazel/.repo/project-objects/chromiumos/platform/vboot_reference.git/./index.lock': File exists.\n> >\n> >     Another git process seems to be running in this repository, e.g.\n> >     an editor opened by 'git commit'. Please make sure all processes\n> >     are terminated then try again. If it still fails, a git process\n> >     may have crashed in this repository earlier:\n> >     remove the file manually to continue.\n> >\n> > Is this expected? I'm not sure why the index is coming into play here.\n> > Is there another method I should be using to extract a tree into a\n> > directory?\n>\n> This is expected because when you do a checkout, the timestamps and\n> other metadata of the files are written into the index.  This is what\n> makes `git status` work quickly: if the metadata of the files hasn't\n> changed, Git doesn't have to re-read them to verify their contents.  You\n> definitely don't want to delete that because you'll likely end up with\n> corrupt data.\n>\n> If you want to create multiple worktrees, use `git worktree add`, which\n> can create multiple worktrees that each have their own index, but share\n> the object store.  When you're done with it, run `git worktree remove`,\n> and everything will be cleaned up.\n\nGreat thanks for confirming. I ended up doing the following:\n\n    git clone --shared --bare vboot_reference.git ~/tmp/repo-1 && git\n-C ~/tmp/repo-1 --work-tree ~/tmp/bob1 checkout\nff27f5415797ead8bc775518a08f3a4ab24abd53 -- . &\n    git clone --shared --bare vboot_reference.git ~/tmp/repo-2 && git\n-C ~/tmp/repo-2 --work-tree ~/tmp/bob1 checkout\ne70ebd7c76b9f9ad44b59e3002a5c57be5b9dc12 -- . &\n\nIt's a lot cleaner than my hacky `find` command. I didn't want to use\n`git worktree` since that would modify the original repo. I just\nwanted to extract the files, so the shared + bare repo works great.\n\nThanks!\n\n>\n> Note that with worktrees, you can have at most one worktree with a given\n> branch (or detached head) checked out at a time.\n> --\n> brian m. carlson (he/him or they/them)\n> Toronto, Ontario, CA\n"}]}