{"thread":{"id":"24335","subject":"fixing workdirs","startedAt":"2010-07-08T11:08:42Z","lastAt":"2010-08-17T18:34:35Z","messageCount":10,"participants":["Pierre Habouzit","Tait","Joshua Jensen","Junio C Hamano","Avery Pennarun","Jan Hudec"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"145110","messageId":"20100708110842.GC12789@madism.org","threadId":"24335","inReplyTo":null,"subject":"fixing workdirs","fromName":"Pierre Habouzit","fromEmail":"madcoder@madism.org","sentAt":"2010-07-08T11:08:42Z","receivedAt":"2010-07-08T11:08:42Z","isPatch":false,"sender":{"key":"madcoder@madism.org","avatar":null},"body":"At work we (ab-)use workdirs a lot, though, workdirs aren't for\neverybody, and as our company grows, not everybody uses them sanely.\n\nThe two problems (that are well known to this list, and is the reason\nwhy git new-workdir is in contrib afaict) with workdirs are:\n\n - the HEAD reflogs aren't shared, which means that pruning a working\n directory may trash accessible stuff from the reflog of another one.\n\n - if two workdirs are on the same branch at the same time, really,\n really, *REALLY* bad things may happen if one isn't aware of that\n fact.\n\nI'm intending to adress those issues, though I would like to know how it\nwould be received. My current plan is this one. Have a git workdir\ncommand, with a few subcommands (create, move, rename, ...), that\naddresses both of the previous issues.\n\nfor the first one, the fix is simple: workdirs have now a name, and\ntheir HEAD reflog lives in the \"master\" git repository reflog namespace\nunder logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a\nsymlink to the masters.\n\nIn this way, all workdirs see all the reflogs of every single workdir,\nand pruning is safe again.\n\n\nFor the second one, when a workdir is created, a [workdir \"foo\"] section\nis added to the master directory, with a path configuration variable\npointing to the ... path of the working directory. My plan would be to\nteach git checkout to lean about that, and when there are workdirs\nset up, git checkout would check that no other workdir is currently \"on\nthe same branch\", and would refuse to checkout to a branch that is\nalready checkouted elsewhere.\n\n\nThe current state of my git-workdir.sh is attached, though before I\nstart diving into the checkout builtin, I wanted to be sure that's the\nway to go, and if there isn't any other issue I could have missed, plus\nif this work has any chance to enter git.git :)\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"145141","messageId":"20100708183714.GR2480@ece.pdx.edu","threadId":"24335","inReplyTo":"20100708110842.GC12789@madism.org","subject":"Re: fixing workdirs","fromName":"Tait","fromEmail":"git.git@t41t.com","sentAt":"2010-07-08T18:37:14Z","receivedAt":"2010-07-08T18:37:14Z","isPatch":false,"sender":{"key":"git.git@t41t.com","avatar":null},"body":"\nPierre Habouzit said (on 2010/07/08):\n> ... The workdir HEAD reflog is then a\n> symlink to the masters.\n\n#include \n\nOn programs (like git) pretending to be cross-platform, symlinks should\nbe avoided. They are to varying degrees, painful on non-*nix operating\nsystems.\n\nWindows is an especially compatibility-breaking example, not only on the\nprogramming side, but also in relation to user interface, and compatibility\nwith other programs. Programming-wise, documentation is sparse and\nwould require lots of platform-specific work-arounds. The user-interface\nsupport is worse than terrible. And even if git does everything right,\nthere's no guarantee a copy, backup/restore, antivirus program, etc. won't\ncome along and corrupt the environment git so carefully created. Many of\nthose other programs don't properly handle Windows reparse points. For\nthose interested, http://shell-shocked.org/article.php?id=284 gives a\nreasonable-looking overview of the details on Windows.\n"},{"id":"145144","messageId":"4C361F4F.9070505@workspacewhiz.com","threadId":"24335","inReplyTo":"20100708183714.GR2480@ece.pdx.edu","subject":"Re: fixing workdirs","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2010-07-08T18:56:15Z","receivedAt":"2010-07-08T18:56:15Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":" ----- Original Message -----\nFrom: Tait\nDate: 7/8/2010 12:37 PM\n> Pierre Habouzit said (on 2010/07/08):\n>> ... The workdir HEAD reflog is then a\n>> symlink to the masters.\n> #include\n>\n> Windows is an especially compatibility-breaking example, not only on the\n> programming side, but also in relation to user interface, and compatibility\n> with other programs. Programming-wise, documentation is sparse and\n> would require lots of platform-specific work-arounds. The user-interface\n> support is worse than terrible. And even if git does everything right,\n> there's no guarantee a copy, backup/restore, antivirus program, etc. won't\n> come along and corrupt the environment git so carefully created. Many of\n> those other programs don't properly handle Windows reparse points. For\n> those interested, http://shell-shocked.org/article.php?id=284 gives a\n> reasonable-looking overview of the details on Windows.\nI'm going to go out on a limb here and say that I suspect symbolic links \nwork fine in Windows Vista and 7, because they are used all over the \nfile system Microsoft installs. Symbolic links for files, in \nparticular, didn't exist until Vista.\n\nJosh\n"},{"id":"145158","messageId":"7v7hl5pxt0.fsf@alter.siamese.dyndns.org","threadId":"24335","inReplyTo":"20100708110842.GC12789@madism.org","subject":"Re: fixing workdirs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-07-08T19:40:11Z","receivedAt":"2010-07-08T19:40:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Pierre Habouzit writes:\n\n> for the first one, the fix is simple: workdirs have now a name, and\n> their HEAD reflog lives in the \"master\" git repository reflog namespace\n> under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a\n> symlink to the masters.\n\nI think this is a sane thing to do, except for the \"symlink\" part but that\nwould be just a minor implementation detail.\n\n> For the second one, when a workdir is created, a [workdir \"foo\"] section\n> is added to the master directory, with a path configuration variable\n> pointing to the ... path of the working directory.\n\nOk.\n\n> ... git checkout would check that no other workdir is currently \"on\n> the same branch\", and would refuse to checkout to a branch that is\n> already checkouted elsewhere.\n\nI am personally fine with this, but if there is no way to override this\nrefusal it may break some people's existing workflow. I dunno.\n"},{"id":"145159","messageId":"AANLkTil1jhqfg-WsTq6g7tMhavG_CzJf01CBzXnd_nBx@mail.gmail.com","threadId":"24335","inReplyTo":"7v7hl5pxt0.fsf@alter.siamese.dyndns.org","subject":"Re: fixing workdirs","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-08T19:56:51Z","receivedAt":"2010-07-08T19:56:51Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Thu, Jul 8, 2010 at 3:40 PM, Junio C Hamano wrote:\n> Pierre Habouzit writes:\n>> ... git checkout would check that no other workdir is currently \"on\n>> the same branch\", and would refuse to checkout to a branch that is\n>> already checkouted elsewhere.\n>\n> I am personally fine with this, but if there is no way to override this\n> refusal it may break some people's existing workflow.  I dunno.\n\nPerhaps it would be better to refuse to *update* a particular ref if\nsome other worktree has it checked out? I don't think actually having\nmultiple trees checked out to the same branch is a problem, so much as\nwhat happens when they start committing.\n\nHave fun,\n\nAvery\n"},{"id":"145196","messageId":"20100709074958.GC2304@madism.org","threadId":"24335","inReplyTo":"20100708183714.GR2480@ece.pdx.edu","subject":"Re: fixing workdirs","fromName":"Pierre Habouzit","fromEmail":"madcoder@madism.org","sentAt":"2010-07-09T07:49:58Z","receivedAt":"2010-07-09T07:49:58Z","isPatch":false,"sender":{"key":"madcoder@madism.org","avatar":null},"body":"On Thu, Jul 08, 2010 at 11:37:14AM -0700, Tait wrote:\n> \n> Pierre Habouzit said (on 2010/07/08):\n> > ... The workdir HEAD reflog is then a\n> > symlink to the masters.\n> \n> #include \n> \n> On programs (like git) pretending to be cross-platform, symlinks should\n> be avoided. They are to varying degrees, painful on non-*nix operating\n> systems.\n> \n> Windows is an especially compatibility-breaking example, not only on the\n> programming side, but also in relation to user interface, and compatibility\n> with other programs. Programming-wise, documentation is sparse and\n> would require lots of platform-specific work-arounds. The user-interface\n> support is worse than terrible. And even if git does everything right,\n> there's no guarantee a copy, backup/restore, antivirus program, etc. won't\n> come along and corrupt the environment git so carefully created. Many of\n> those other programs don't properly handle Windows reparse points. For\n> those interested, http://shell-shocked.org/article.php?id=284 gives a\n> reasonable-looking overview of the details on Windows.\n\nWell that's how git-new-workdir works, and you don't /need/ git\nnew-workdir to do actual git work. Note that git is cross compatible to\nall POSIX conformant filesystems, it's just windows that I can think of\nthat won't work with it... They don't provide any sane replacement for\nthe feature, so I don't see what we can do about it anyways.\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"145197","messageId":"20100709075617.GD2304@madism.org","threadId":"24335","inReplyTo":"7v7hl5pxt0.fsf@alter.siamese.dyndns.org","subject":"Re: fixing workdirs","fromName":"Pierre Habouzit","fromEmail":"madcoder@madism.org","sentAt":"2010-07-09T07:56:17Z","receivedAt":"2010-07-09T07:56:17Z","isPatch":false,"sender":{"key":"madcoder@madism.org","avatar":null},"body":"On Thu, Jul 08, 2010 at 12:40:11PM -0700, Junio C Hamano wrote:\n> Pierre Habouzit writes:\n> \n> > for the first one, the fix is simple: workdirs have now a name, and\n> > their HEAD reflog lives in the \"master\" git repository reflog namespace\n> > under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a\n> > symlink to the masters.\n> \n> I think this is a sane thing to do, except for the \"symlink\" part but that\n> would be just a minor implementation detail.\n\nWhat would you suggest instead of the symlink then ? (knowing that all\nthe workdir is just a full symlink farm at them moment).\n\n> > For the second one, when a workdir is created, a [workdir \"foo\"] section\n> > is added to the master directory, with a path configuration variable\n> > pointing to the ... path of the working directory.\n> \n> Ok.\n\n> > ... git checkout would check that no other workdir is currently \"on\n> > the same branch\", and would refuse to checkout to a branch that is\n> > already checkouted elsewhere.\n> \n> I am personally fine with this, but if there is no way to override this\n> refusal it may break some people's existing workflow. I dunno.\n\nWell it's probably fine to have a switch to override it of course, or\nlike it was suggested to prevent updating the reference instead. But\nthat sounds harder, and if you want to override it, it means that a lot\nof commands will have to take a new argument (update-ref, commit, reset,\nrebase, ...).\n\nI'm more in favor of having checkout refusing to checkout if the\nreference is already checkout-ed elsewhere, with a --ignore-workdirs or\nsimilar switch to override this. git-checkout would still print out a\nwarning about the fact that /maybe/ the user is doing something crazy,\nand then he'll be on his own. Plus it doesn't slow references updates\nfor the workdir case, only the branch switch which is way nicer.\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"145244","messageId":"7vy6dkl2d0.fsf@alter.siamese.dyndns.org","threadId":"24335","inReplyTo":"20100709075617.GD2304@madism.org","subject":"Re: fixing workdirs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-07-09T22:25:15Z","receivedAt":"2010-07-09T22:25:15Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Pierre Habouzit writes:\n\n> On Thu, Jul 08, 2010 at 12:40:11PM -0700, Junio C Hamano wrote:\n>> Pierre Habouzit writes:\n>> \n>> > for the first one, the fix is simple: workdirs have now a name, and\n>> > their HEAD reflog lives in the \"master\" git repository reflog namespace\n>> > under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a\n>> > symlink to the masters.\n>> \n>> I think this is a sane thing to do, except for the \"symlink\" part but that\n>> would be just a minor implementation detail.\n>\n> What would you suggest instead of the symlink then ? (knowing that all\n> the workdir is just a full symlink farm at them moment).\n\nI can imagine that we may want to have a general mechanism to help an\nobject store that belongs to one \"primary\" repository be aware of ref-like\nthings that live outside of the repoistory itself, and not just a special\npurpose hack suitable only to handle the workdirs. E.g., we have talked\nabout a \"fork\" created by \"clone -s\" wanting the forkee repository to be\naware of its refs, so that rewinding the refs in the forkee repository and\nthen running gc there won't remove the objects now unnecessary in the\nforkee but still needed by the forker repository.\n\nIt shouldn't be hard to do something similar to \"gitdir: \" support for\nthis without using a symlink, no?\n"},{"id":"145786","messageId":"20100719090248.GA10802@madism.org","threadId":"24335","inReplyTo":"7vy6dkl2d0.fsf@alter.siamese.dyndns.org","subject":"Re: fixing workdirs","fromName":"Pierre Habouzit","fromEmail":"madcoder@madism.org","sentAt":"2010-07-19T09:02:48Z","receivedAt":"2010-07-19T09:02:48Z","isPatch":false,"sender":{"key":"madcoder@madism.org","avatar":null},"body":"On Fri, Jul 09, 2010 at 03:25:15PM -0700, Junio C Hamano wrote:\n> Pierre Habouzit writes:\n> \n> > On Thu, Jul 08, 2010 at 12:40:11PM -0700, Junio C Hamano wrote:\n> >> Pierre Habouzit writes:\n> >> \n> >> > for the first one, the fix is simple: workdirs have now a name, and\n> >> > their HEAD reflog lives in the \"master\" git repository reflog namespace\n> >> > under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a\n> >> > symlink to the masters.\n> >> \n> >> I think this is a sane thing to do, except for the \"symlink\" part but that\n> >> would be just a minor implementation detail.\n> >\n> > What would you suggest instead of the symlink then ? (knowing that all\n> > the workdir is just a full symlink farm at them moment).\n> \n> I can imagine that we may want to have a general mechanism to help an\n> object store that belongs to one \"primary\" repository be aware of ref-like\n> things that live outside of the repoistory itself, and not just a special\n> purpose hack suitable only to handle the workdirs. E.g., we have talked\n> about a \"fork\" created by \"clone -s\" wanting the forkee repository to be\n> aware of its refs, so that rewinding the refs in the forkee repository and\n> then running gc there won't remove the objects now unnecessary in the\n> forkee but still needed by the forker repository.\n> \n> It shouldn't be hard to do something similar to \"gitdir: \" support for\n> this without using a symlink, no?\n\nSorry for the delay, I was on vacation.\n\nOkay, I see, this makes sense. I'll see what I can do on this path,\nthough it's probably harder than simply extending gitfiles. We still\nwant a .git/ directory as we want most of the top level stuff to be\nlocal to each repository (HEAD, ORIG_HEAD, ...) but not:\n\n - subdirectories (most of them being references, logs, ...)\n - the lock (which is kind of the weak point in my proposal atm, yours\n is nicer)\n - config (or do we want a cascading semantics: local workdir config,\n master repository config, user .gitconfig, /etc/.... ? I think not\n but ...)\n - packed-refs\n\nPlus, to make everything work, the reflogs of a given \"workdir\" (or\nshared clone) must be put in a different namespace to avoid clashes.\nThough this is probably the simplest bit.\n\nAll in all, I'm afraid to have to look at every single git script that\nfor now writes without thinking twice under .git/ :/\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"148301","messageId":"20100817183435.GA2717@efreet.light.src","threadId":"24335","inReplyTo":"20100708110842.GC12789@madism.org","subject":"Re: fixing workdirs","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2010-08-17T18:34:35Z","receivedAt":"2010-08-17T18:34:35Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Thu, Jul 08, 2010 at 13:08:42 +0200, Pierre Habouzit wrote:\n> At work we (ab-)use workdirs a lot, though, workdirs aren't for\n> everybody, and as our company grows, not everybody uses them sanely.\n> \n> The two problems (that are well known to this list, and is the reason\n> why git new-workdir is in contrib afaict) with workdirs are:\n> \n> - the HEAD reflogs aren't shared, which means that pruning a working\n> directory may trash accessible stuff from the reflog of another one.\n>\n> - if two workdirs are on the same branch at the same time, really,\n> really, *REALLY* bad things may happen if one isn't aware of that\n> fact.\n\nIt should be possible to guard against those bad things. Basically the bad\nthings happen when a symbolic HEAD's target is changed in a way the HEAD is\nnot aware. But in those, and only those, cases the HEAD's reflog is not\nupdated. So some operations could check whether HEAD@{0} resolves to the same\ncommit as HEAD and if not they would either fail or automatically detach\nHEAD.\n\nThat would serve as a safeguard not just against one worktree commiting to\na branch checked out in another worktree. It would also safeguard against\npushing to the branch provided the code in update that updates HEAD's reflog\nwas disabled for non-bare repositories.\n\n> I'm intending to adress those issues, though I would like to know how it\n> would be received. My current plan is this one. Have a git workdir\n> command, with a few subcommands (create, move, rename, ...), that\n> addresses both of the previous issues.\n> \n> for the first one, the fix is simple: workdirs have now a name, and\n> their HEAD reflog lives in the \"master\" git repository reflog namespace\n> under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a\n> symlink to the masters.\n> \n> In this way, all workdirs see all the reflogs of every single workdir,\n> and pruning is safe again.\n\nThe problem with it is that it falls apart when somebody forgets to use the\n'git worktree remove' command and deletes (or renames) the work-tree\nmanually.\n\nAlternative idea would be to add support for new layout with one repository\nand multiple worktrees. The worktrees would be restricted to siblings of the\n.git directory where the repository lives with the repository itself being\nmarked as bare. So the layout would be:\n\n somedir/\n somedir/.git/ <- the repository, makred as bare\n somedir/worktree1/ <- a worktree\n somedir/worktree1/.git/ <- worktree's .git in current format\n somedir/worktree2/ <- another worktree\n somedir/worktree2/.git/ <- worktree's .git in current format\n ...\n\nSuch scheme would exchange some flexibility for more safety. The worktrees\ncould be created with current new-workdir. They would be found as simply\n$GIT_DIR/../*/.git for purpose of solving the two problems above. On the\nother hand a method for preparing this layout (git init --multiple-trees,\ngit clone --multiple-trees or something like that) would be needed.\n\nIIRC bazaar uses scheme like this for it's multiple-worktree support.\n\n> For the second one, when a workdir is created, a [workdir \"foo\"] section\n> is added to the master directory, with a path configuration variable\n> pointing to the ... path of the working directory. My plan would be to\n> teach git checkout to lean about that, and when there are workdirs\n> set up, git checkout would check that no other workdir is currently \"on\n> the same branch\", and would refuse to checkout to a branch that is\n> already checkouted elsewhere.\n\nWait a second. Don't you have the HEADs and their reflogs for workdirs stored\nin the master? So you can just compare with all the other HEADs there, no?\n\nThe bookkeeping would break if you just deleted the workdir without telling\ngit. The configuration will break even more readily as renaming also breaks\nit. \n\nKind regards,\nJan\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec \n"}]}