{"thread":{"id":"45675","subject":"","startedAt":"2017-04-12T19:53:42Z","lastAt":"2017-04-12T19:53:42Z","messageCount":1,"participants":["David Taylor"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"316699","messageId":"9305.1492026214@usendtaylorx2l","threadId":"45675","inReplyTo":null,"subject":"","fromName":"David Taylor","fromEmail":"dtaylor@emc.com","sentAt":"2017-04-12T19:43:34Z","receivedAt":"2017-04-12T19:53:42Z","isPatch":false,"sender":{"key":"dtaylor@emc.com","avatar":null},"body":"    On Tue, Apr 11, 2017 at 10:14 PM, taylor, david <David.Taylor@dell.com> wrote:\n    > We are using Git in a distributed environment.\n    >\n    > In the United States, we have the master repository in one state and a build cluster in a different state.\n    > In addition to people in the US doing builds, we have people in other countries (Ireland, India, Israel,\n    > Russia, possibly others) doing builds -- using the build cluster.\n    >\n    > The local mirror of the repository is NFS accessible.  The plan is to make builds faster through the use\n    > of work trees.  The build cluster nodes involved in the build will have a worktree in RAM -- checked out\n    > for the duration of the build.  Since the worktree is in RAM, it will not be NFS accessible.\n    >\n    > [Cloning takes 20+ minutes when the network is unloaded.  Building, with sources NFS mounted, takes\n    > 5-10 minutes.]\n\n    Using worktrees in this scenario kinda defeats the distributed nature\n    of git. Cloning takes long, yes. But what about just \"git pull\" (or\n    \"git fetch && git checkout -f\" if you want to avoid merging)?\n\nMerging isn't the issue.  Speed is an issue.  Repeatability is an issue.\nDisk space is an issue.\n\nIf someone does a build on their desktop instead of using the build\ncluster, it will take hours rather than minudes.  And if they are not in\nMassachusetts, they probably don't have access to the controlled\ntoolchain that is used to do the builds.\n\nUsers may choose to share their repository amongst several work trees\nrather than having lots of clones.  Their choice.  Such work trees would\nbe 'long lived'.\n\nI was thinking of a different use for work trees.  Work trees that would\nbe short lived -- less than, say, two hours.  Typically less than 30\nminutes.\n\nThere would be a local repository mirroring the master repository.  It\nwould be a true mirror -- it would be updated only from thhe master via\ngit fetch; there would never be any 'git commit's to it.\n\nWhen someone who is remote to Massachusetts (which is where the build\ncluster lives), wants to do a build they will invoke a script (yet to be\nwritten) that will determine two things:\n\n  . a SHA1 that exists in the master repository that is an ancestor of\n  their workspace\n\n  . their workspace differences relative to that SHA1\n\nA lightly loaded build cluster node will be selected, it will be given\nthe SHA1 and the patch file.  A short lived worktree will be created\nwhich has the SHA1 checked out and the patch applied.\n\nOnce the workspace has been recreated, it is built.\n\n[Actually, the build target is decomposed into a dozen pieces that are\nbuilt on separate build cluster nodes in parallel using separate copies\nof the workspace.]\n\nThe build deliverables will be delivered back to the requestor and the\nintermediate build products and the work tree will be deleted.\n\n[The script will also be used by Jenkins for continuous integration builds.]\n\n    > This presents a few problems.\n    >\n    > When we are done with a work tree, we want to clean up -- think: prune.  But, you cannot prune just\n    > one worktree; you have to prune the set.  And no machine has access to all the worktrees.  So, no\n    > machine knows which ones are prunable.\n\n    By \"prune one worktree\", did you mean delete one? Or delete a branch\n    the worktree uses and prune the object database?\n\nAs in:\n\n  rm -rf /path/to/top/of/work/tree\n\nand then ideally:\n\n  git worktree prune /path/to/top/of/work/tree\n\nor, alternatively, just:\n\n  git worktree prune\n\n    > There is no 'lock' option to 'add'.  If someone does a 'prune' after you do an 'add' and before you do a\n    > 'lock', then your 'add' is undone.\n    >\n    > Are there any plans to add a '[--lock]' option to 'add' to create the worktree in the locked state?  And/or\n    > plans to add a [<path>...] option to prune to say 'prune only this path / these paths'?\n\n    So this is \"git worktree prune\". Adding \"worktree add --locked\" sounds\n    reasonable (and quite simple too, because \"worktree add\" does lock the\n    worktree at creation time; we just need to stop it from releasing the\n    lock). I might be able to do it quickly (it does not mean \"available\n    in the next release\" though).\n\nYes, add the worktree, and if there are no errors, leave it in the\nlocked state.\n\n    If you need to just prune \"this path\", I think it's the equivalent of\n    \"git worktree remove\" (i.e. delete a specific worktree). Work has been\n    going on for a while to add that command. Maybe it'll be available\n    later this year.\n\nI'm not familiar with 'git worktree remove', but, yes, just 'git\nworktree prune' a specific specified path.  I was thinking do what 'git\nworktree prune' does, but do it only for the path(s) specified on the\ncommand line.\n\n[While I only anticipate giving it one path, I see no need / reason to\nlimit it to just one path on the command line.]\n\n    > If there are no plans, is the above an acceptable interface?  And if we implemented it, would it be looked\n    > upon favorably?\n\n    Speaking of this use case (and this is my own opinion) I think this is\n    stretching \"git worktree\" too much. When I created it, I imagined this\n    functionality to be used by a single person.\n\nCloning is expensive (20+ minutes, ~5 GB); checkout is cheap (seconds,\n100's of MBs).  This is / was seen as a speedup.\n\nI also noticed during my experiments that git sometimes looks at the\nrepository when I wouldn't expect it and gets an error if the directory\nabove the worktree does not exist.  These experiments were with 2.10.2;\nI haven't tried yet with 2.12.2 nor with the head of the master branch.\n\nI haven't (yet?) reported this as a bug, but what I did was:\n\n. create a directory for the tests\n\n  mkdir /home/dtaylor/worktree-tests\n\n. create a couple of worktrees (neither locked)\n\n  cd /top/of/repository\n  git worktree add -f --detach /home/dtaylor/worktree-tests/tree_1 first-branch\n  git worktree add -f --detach /home/dtaylor/worktree-tests/tree_2 second-branch\n\nDifferent computer, where /top/of/repository is accessible (it is NFS\nmounted), but /home/dtaylor/worktree-tests does not exist (/home is a\nlocal partition, actual home directories go elsewhere), do:\n\n  cd /top/of/repository\n  git fetch ==> error message about /home/dtaylor/worktree-tests not existing\n  git status ==> error message again\n\nand then the real surprise (I wouldn't expect this one to access the\nrepository):\n\n  git help ==> error message again\n\nBut, of course, if I do:\n\n  cd /\n\n(no repository in sight) and then do:\n\n  git help\n\nit works as expected.\n\nI don't know if 2.12.2 or possibly HEAD fixes this.  My testing was done\nwith 2.10.2.  I was going to wait until I knew before reporting it.\n\n    -- \n    Duy\n"}]}