{"thread":{"id":"5564","subject":"Notes on supporting Git operations in/on partial Working Directories","startedAt":"2006-09-14T19:05:16Z","lastAt":"2006-09-17T21:11:46Z","messageCount":13,"participants":["A Large Angry SCM","Shawn Pearce","Junio C Hamano","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"26910","messageId":"4509A7EC.9090805@gmail.com","threadId":"5564","inReplyTo":null,"subject":"Notes on supporting Git operations in/on partial Working Directories","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-14T19:05:16Z","receivedAt":"2006-09-14T19:05:16Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Notes on supporting Git operations in/on partial Working Directories\n====================================================================\n\nMotivation\n----------\nBe able to checkout only part of a tree, do some work, and commit the\nchanges. Support for partial working directories is also (almost) a\nrequirement for supporting partial repositories.\n\n\nExpectations\n------------\nAll Git commands that currently work with the index or the working\ndirectory will work with indexes or working directories that are partial\ncheckouts.\n\nLeading directories common to all objects of a partial checkout are not\npresent in the working directory.\n\nThe contents of a partial working directory can be determined on an path\nby path basis; entire directories are not required.\n\n\nImplementation Sketch\n---------------------\nThe minimum required changes[*1*][*2*][*3*] to the index file to support\npartial checkouts are:\n\n1) the addition of WD_Prefix string to hold the common path prefix of\nall objects in the working directory. For a full checkout, the WD_Prefix\nstring would be empty.\n\n2) A (new) flag for each entry in the index indicating whether or not\nthe object is in the partial checkout.\n\nThe contents of the index file still reflect the full tree but flag each\nobject (file or symlink) separately as part of the checkout or not. The\nWD_Prefix string is so that a partial checkout consisting of only\nobjects somewhere in the a/b/c/d/ tree can be found in the working\ndirectory without the a/b/c/d/ prefix to the path of the object.\n\nAll the Git commands that use the index file will need to be changed to\nsupport this but the transfer protocols do not need to change.\n\n\nNotes\n-----\n[*1*] As long as the index file structure is being changed, it may be\nworth while also including the ideas in:\n\thttp://www.gelato.unsw.edu.au/archives/git/0601/15471.html\n\thttp://www.gelato.unsw.edu.au/archives/git/0601/15483.html\n\thttp://www.gelato.unsw.edu.au/archives/git/0601/15484.html\nExcept for the \"bind\" parts since I still think that is a bad idea.\n\n[*2*] The index \"TREE\" (cache-tree) extension should also become a\nrequired part of the index.\n\n[*3*] Possibly split the index up by directory and store the parts in\nthe working directory. An index \"distributed\" in this way would have\na \"natural\" cache-tree built in and (finally) be able support empty\ndirectories.\n"},{"id":"26913","messageId":"20060914192119.GC10556@spearce.org","threadId":"5564","inReplyTo":"4509A7EC.9090805@gmail.com","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-14T19:21:19Z","receivedAt":"2006-09-14T19:21:19Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"A Large Angry SCM <gitzilla@gmail.com> wrote:\n> The contents of the index file still reflect the full tree but flag each\n> object (file or symlink) separately as part of the checkout or not. The\n> WD_Prefix string is so that a partial checkout consisting of only\n> objects somewhere in the a/b/c/d/ tree can be found in the working\n> directory without the a/b/c/d/ prefix to the path of the object.\n\nWhy not just load a partial index?\n\nIf we only want \"a/b/c/d\" subtree then only load that into the index.\nAt git-write-tree time return the new root tree by loading the tree\nof the current `HEAD` commit and walking down to a/b/c/d, updating\nthat with the tree from the index, then walking back updating each\nnode you recursed down through.  Finally output the new root tree.\n\nThe advantage is that if you have a subtree checked out you aren't\nworking with the entire massive index.\n\nBut how does this let the user checkout and work on the 10 top\nlevel directories at once and perform an atomic commit to all\nof them, but not checkout the other 100+ top level directories?\nAs I recall this was desired in the Mozilla project for example.\n \n> [*3*] Possibly split the index up by directory and store the parts in\n> the working directory. An index \"distributed\" in this way would have\n> a \"natural\" cache-tree built in and (finally) be able support empty\n> directories.\n\nPlease, no.  On a project with a large number of directories\noperations like git-write-tree would take a longer time to scan the\nindex and generate the new trees.  I unfortunately work on such\nprojects as its common for Java applications to be very deeply\nnested and large projects have a *lot* of directories.\n\n-- \nShawn.\n"},{"id":"26917","messageId":"7vu03a2po8.fsf@assigned-by-dhcp.cox.net","threadId":"5564","inReplyTo":"4509A7EC.9090805@gmail.com","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-14T19:50:31Z","receivedAt":"2006-09-14T19:50:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"A Large Angry SCM <gitzilla@gmail.com> writes:\n\n> The minimum required changes[*1*][*2*][*3*] to the index file to support\n> partial checkouts are:\n>\n> 1) the addition of WD_Prefix string to hold the common path prefix of\n> all objects in the working directory. For a full checkout, the WD_Prefix\n> string would be empty.\n>\n> 2) A (new) flag for each entry in the index indicating whether or not\n> the object is in the partial checkout.\n>\n> The contents of the index file still reflect the full tree but flag each\n> object (file or symlink) separately as part of the checkout or not. The\n> WD_Prefix string is so that a partial checkout consisting of only\n> objects somewhere in the a/b/c/d/ tree can be found in the working\n> directory without the a/b/c/d/ prefix to the path of the object.\n>\n> All the Git commands that use the index file will need to be changed to\n> support this but the transfer protocols do not need to change.\n\nWhile this may be a good start, you need a lot more than this if\nyou want to do (1) and (2):\n\nThe tree object contained by a commit is by definition a full\ntree snapshot, so if you want to do a WD_Prefix, you somehow\nneed a way to come up with the final tree that is a combination\nof what write-tree would write out from such a partial index\n(i.e. an index that describes only a subdirectory) and the rest\nof the tree from the current HEAD.  I think you can more or less\ndo this change to Porcelain using today's git core.  The\nsequence to emulate it with the today's git would be:\n\n (1) write-tree (of the WD_Prefix part of the subtree),\n (2) read-tree HEAD (to populate the index fully),\n (3) piping a massaged output from git-ls-files WD_prefix to\n     update-index --index-info, followed by read-tree\n     --prefix=WD_prefix to swap the partial tree in to\n     WD_prefix,\n (4) write-tree (to get the final result).\n\nIf you want to do per-path-inside-directory checkout (your 2),\nthis combining step would need to be even more complex.  You can\ndo that by hand (reading ls-tree and ls-files and driving\nupdate-index --index-info yourself) but it certainly would be\nmore involved.  But it's just a matter of Porcelain programming\n;-) [*1*].\n\nBut a good news is that today's git core lets you work in a\nsparsely checked out repository without any of the above\ncrap^Wcomplexity, if you drop the WD_Prefix and per\npath-inside-directory checkout \"expectations\".  Just staying\nwithin the directory you are working in, and saying \"commit .\"\nwhen you are tempted to say \"commit -a\", would be more or less\nwhat are needed.\n\nNote.  The \"git checkout\" Porcelain would want to check-out\neverythingd, so a tool to prepare such a sparsely checked out\ntree needs to be written if somebody wants to try this, since\nthe above \"good news\" is only about \"working in\" a sparsely\nchecked out tree.\n\n\n[*1*] Obviously you would also need to worry about activities\nother than making your own changes and committing.  When you are\nalways pulling from a single upstream that never rewinds the\nhead, the problem becomes simpler, but for other cases (read:\nanything that makes distributed version control more interesting\nand useful) you would need to worry about merges too.  What\nhappens when the upstream you based your changes on and the\nrepository you are pulling from today had conflicting changes\noutside of your area of interest?  Without resolving the\nconflicts, you cannot sanely claim you merged the two branches,\nand even if you wanted to resolve them yourself, an non-empty\nWD_prefix would get in your way.\n"},{"id":"26920","messageId":"4509B6AA.9040902@gmail.com","threadId":"5564","inReplyTo":"20060914192119.GC10556@spearce.org","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-14T20:08:10Z","receivedAt":"2006-09-14T20:08:10Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Shawn Pearce wrote:\n> A Large Angry SCM <gitzilla@gmail.com> wrote:\n>> The contents of the index file still reflect the full tree but flag each\n>> object (file or symlink) separately as part of the checkout or not. The\n>> WD_Prefix string is so that a partial checkout consisting of only\n>> objects somewhere in the a/b/c/d/ tree can be found in the working\n>> directory without the a/b/c/d/ prefix to the path of the object.\n> \n> Why not just load a partial index?\n> \n> If we only want \"a/b/c/d\" subtree then only load that into the index.\n> At git-write-tree time return the new root tree by loading the tree\n> of the current `HEAD` commit and walking down to a/b/c/d, updating\n> that with the tree from the index, then walking back updating each\n> node you recursed down through.  Finally output the new root tree.\n> \n> The advantage is that if you have a subtree checked out you aren't\n> working with the entire massive index.\n\nI was looking for minimal changes to the index and associated code. \nEither way works.\n\n> But how does this let the user checkout and work on the 10 top\n> level directories at once and perform an atomic commit to all\n> of them, but not checkout the other 100+ top level directories?\n> As I recall this was desired in the Mozilla project for example.\n\nThat's a partial working working directory by my definition so it would \nwork. How it's specified on the command line is TBD.\n\nIt's desired by a lot of very modular projects.\n\n>> [*3*] Possibly split the index up by directory and store the parts in\n>> the working directory. An index \"distributed\" in this way would have\n>> a \"natural\" cache-tree built in and (finally) be able support empty\n>> directories.\n> \n> Please, no.  On a project with a large number of directories\n> operations like git-write-tree would take a longer time to scan the\n> index and generate the new trees.  I unfortunately work on such\n> projects as its common for Java applications to be very deeply\n> nested and large projects have a *lot* of directories.\n\nDirectory trees without any changes might actually be less expensive to \nwork with using the split index since you could ignore all of the \nunchanged entries easily.\n"},{"id":"26923","messageId":"4509B954.60101@gmail.com","threadId":"5564","inReplyTo":"7vu03a2po8.fsf@assigned-by-dhcp.cox.net","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-14T20:19:32Z","receivedAt":"2006-09-14T20:19:32Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Junio C Hamano wrote:\n> A Large Angry SCM <gitzilla@gmail.com> writes:\n...\n> \n> While this may be a good start, you need a lot more than this if\n> you want to do (1) and (2):\n> \n> The tree object contained by a commit is by definition a full\n> tree snapshot, so if you want to do a WD_Prefix, you somehow\n> need a way to come up with the final tree that is a combination\n> of what write-tree would write out from such a partial index\n> (i.e. an index that describes only a subdirectory) and the rest\n> of the tree from the current HEAD.  I think you can more or less\n> do this change to Porcelain using today's git core.  The\n> sequence to emulate it with the today's git would be:\n\nI think you misunderstood, the index file would list all of the tree \nentries of the the checked out commit, same as the current index, but \nwould flag the entries that are actually present in the working \ndirectory. The WD_Prefix is to identify which index entries _can not_ be \npart of the working directory, and where the working directory fits in \nthe full index. That way, all the information needed by the top level \nwrite-tree is still in the index and the cache-tree extension.\n"},{"id":"26939","messageId":"7v8xkl26kb.fsf@assigned-by-dhcp.cox.net","threadId":"5564","inReplyTo":"4509B954.60101@gmail.com","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-15T02:43:16Z","receivedAt":"2006-09-15T02:43:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"A Large Angry SCM <gitzilla@gmail.com> writes:\n\n> Junio C Hamano wrote:\n>> A Large Angry SCM <gitzilla@gmail.com> writes:\n> ...\n>>\n>> While this may be a good start, you need a lot more than this if\n>> you want to do (1) and (2):\n>>\n>> The tree object contained by a commit is by definition a full\n>> tree snapshot, so if you want to do a WD_Prefix, you somehow\n>> need a way to come up with the final tree that is a combination\n>> of what write-tree would write out from such a partial index\n>> (i.e. an index that describes only a subdirectory) and the rest\n>> of the tree from the current HEAD.  I think you can more or less\n>> do this change to Porcelain using today's git core.  The\n>> sequence to emulate it with the today's git would be:\n>\n> I think you misunderstood, the index file would list all of the tree\n> entries of the the checked out commit, same as the current index, but\n> would flag the entries that are actually present in the working\n> directory. The WD_Prefix is to identify which index entries _can not_\n> be part of the working directory, and where the working directory fits\n> in the full index. That way, all the information needed by the top\n> level write-tree is still in the index and the cache-tree extension.\n\nAh indeed.  That makes it more palatable ;-).\n\nHaving said that, I do not necessarily agree that highly modular\nprojects would want to put everything in one git repository and\ntrack everything as a whole unit.\n\nThe primary audience of git, the kernel project, is reasonably\nmodular (although Andrew seems to be suffering from subsystem\nmaintainers touching overlapping areas these days and says it is\nrather unusual), and is a non-trivial size, yet it has\neverything under one umbrella.  The model makes sense in that\nproject, since the core developers need to occasionally change\nan internal API wholesale across the tree.  The people at fringe\nwho work only on limited part of the system (e.g. one particular\nfilesystem implementation), on the other hand, may not care what\nhappens in the other parts (e.g. random device drivers that\nshould not interact directly with the filesystem implementation\nin question) of the system most of the time, but they do have to\ncare if the layer closer to the core that their work depends on\nchanges (e.g. a VFS layer update changes the rule filesystems\nmust play under), so having to check out the full kernel tree\nwhile they usually work only on one part of it often cumbersome\nbut sometimes absolutely necessary so it is tolerated.\n\nEverybody is forced to work on the same codebase and merge the\nwhole tree as a unit, which might inconvenience the people\nreally at the fringe (e.g. driver writers), but being able to\nmake sure everything is in sync is a good thing to the core\ndevelopers, and that benefit outweighs the convenience of fringe\npeople (also the core people are who gets to pick the tool they\nuse ;-).\n\nIn the kernel case, out-of-tree driver people have a choice to\nbuild out of tree against just kernel headers as modlues.\nNobody (including git) gives mechanical support to enforce that\nthis version of the out-of-tree driver must be used only with\nsuch and such main tree, but build procedure and INSTALL\ndocuments of such a driver usually take care of that integration\nissues.\n\nI suspect most highly modular projects are run that way, not\njust from the version control point of view, but simply because\nof people interaction issues.  Nobody can be on top of all\npossible interface details between many modular pieces of a\ntruly huge project, so there would be clean separation of parts\nand narrow definition of how they mesh together (after all, that\nis what being highly modular is all about).  And in such a case,\nsubsystems can be (and I'd even claim they had better be)\nversion controlled more or less independently with each other,\nwith certain version dependencies, such as \"libfoo subsystem is\nused by all of our programs A, B, ..., Z, but recent libfoo 1.29\nrelease added some feature to support enhancement in version 2.4\nof program Z.  So libfoo 1.29 or later is required if you are\nbuilding the latest tip of program Z, but everybody else can\nstay at 1.28 if updating libfoo is not convenient, oh by the way\n1.30 has a thinko that broke what is used heavily only by\nprogram A, so if you are working on program A use libfoo 1.30 or\nlater, or stay at libfoo 1.28.\"  And there would be tons of tiny\ncommits between these point releases.\n\nMy point is that while there will always be _some_ version\nsynchronization requirements between subcomponents of such a\nhuge highly modular project, it is a lot looser than tracking\neach and every change in the entire tree as a whole, like git's\ncommit does.  The model of throwing all subcomponents in a\nsingle repository and trying to track everything as a whole may\nnot match the real requirement of such a project.\n\nIn other words, when somebody adds a line in a file in a tiny\ncorner of libfoo to fix a typo in the comment and makes a\ncommit, that should not have to necessarily mean the version\nnumber of the project as a whole needs to be bumped up.  It is\nmy understanding that people who house collection of related\nprojects in subversion gets this wrong, because subversion makes\nit too eacy to propagate the revision number increment up to the\nroot level when you update something in a subtree.  It may be a\ncheap operation from the storage point of view (incrementing the\nrevision number stored in a few tree nodes near the root), but\nit does not change the fact that it changes the revision of the\nwhole project and affects other parts of projects that is not\naffected by the particular change at all (it is not Subversion's\nfault, but more of a fault of people who put everything in one\nrepository).  You could manage the versions that way, but you do\nnot have to.  And if it gets in the way of things you would want\nto do, maybe you shouldn't.\n\nI think what truly huge but highly modular projects need is a\ngood support to lay-out check-outs from multiple subprojects,\neach of which is managed in its own repository but has loose\n(looser than the level of individual commits) version\ndependency.  That would need to solve three issues: (1) the\nright versions from many repositories need to be checked out in\ncorrect locations for a build, (2) after building and testing to\nmake sure they work together as a whole, these specific versions\nfrom the subcomponent repositories need to be tagged to mark a\nrelease, and (3) maybe a single large tarball that contains all\nsubprojects' checkout can be made easily.\n\nSo the issue may not be partial repository support, but support\nfor managing multiple projects.\n"},{"id":"26980","messageId":"450AEDBF.9050307@gmail.com","threadId":"5564","inReplyTo":"7v8xkl26kb.fsf@assigned-by-dhcp.cox.net","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-15T18:15:27Z","receivedAt":"2006-09-15T18:15:27Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Junio C Hamano wrote:\n...\n> \n> Having said that, I do not necessarily agree that highly modular\n> projects would want to put everything in one git repository and\n> track everything as a whole unit.\n\nAnd yet that's exactly how a lot of developers use CVS. You can argue \nthat some other way is better but when they move from CVS they're \nlooking for continuity of productivity which often means not radically \nchanging how they work. At least in the short term.\n\n> The primary audience of git, the kernel project, is reasonably\n> modular (although Andrew seems to be suffering from subsystem\n\nI no longer believe that the Linux kernel developers are the \"primary \naudience\". They are certainly an important and influential set of Git \nusers but there are also a lot of non kernel projects using Git. If not \nnow, there will soon be more non kernel Git users than kernel Git users.\n\n[Nice description of how to work with the Linux kernel code base.]\n\n[Nice description of one way a hypothetical project with dependencies on \nlibraries under active development could work.]\n\n> I think what truly huge but highly modular projects need is a\n> good support to lay-out check-outs from multiple subprojects,\n> each of which is managed in its own repository but has loose\n> (looser than the level of individual commits) version\n> dependency.  That would need to solve three issues: (1) the\n> right versions from many repositories need to be checked out in\n> correct locations for a build, (2) after building and testing to\n> make sure they work together as a whole, these specific versions\n> from the subcomponent repositories need to be tagged to mark a\n> release, and (3) maybe a single large tarball that contains all\n> subprojects' checkout can be made easily.\n >\n> So the issue may not be partial repository support, but support\n> for managing multiple projects.\n\nThere's no question that that may be better for some projects. But I \nbelieve that the project members (or owners) should decide how they use \ntheir tools.\n"},{"id":"27067","messageId":"7vvenm3h9f.fsf@assigned-by-dhcp.cox.net","threadId":"5564","inReplyTo":"450AEDBF.9050307@gmail.com","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-17T10:43:40Z","receivedAt":"2006-09-17T10:43:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"A Large Angry SCM <gitzilla@gmail.com> writes:\n\n> Junio C Hamano wrote:\n> ...\n>>\n>> Having said that, I do not necessarily agree that highly modular\n>> projects would want to put everything in one git repository and\n>> track everything as a whole unit.\n>\n> And yet that's exactly how a lot of developers use CVS. You can argue\n> that some other way is better but when they move from CVS they're\n> looking for continuity of productivity which often means not radically\n> changing how they work. At least in the short term.\n\n(Note. In the next paragraph, I used (for the want of better\n       wording) the word \"unrelated\" to mean \"indeed related,\n       but without need to be in sync at whole-tree commit\n       level, and probably insisting to be in sync at that level\n       has more disadvantages than advantages\".  Think of it to\n       refer to the relationship among more-or-less independent\n       project subcomponents in my earlier example).\n\nWell, the fact is, it does not make any difference to CVS if you\nput unrelated projects in a single \"repository\", because CVS\ndoes not even have the concept of managing a project as a whole.\nExcept perhaps that you can give the same tag to individual\nfiles and treat the set of the revs of the files that have that\nparticular tag forms a revision of a project.  And even that is\njust a kludge -- if you throw a totally unrelated project into\nsuch a \"repository\" and give files a tag that happens to be the\nsame name as the one used in another project, the tool happily\nlets you do so and checking out a revision by the tag will pull\nfiles from totally unrelated projects together.  We happen to\nuse the words \"repository\" and \"revision\" in git but what they\nmean is quite different because we are more whole-tree oriented.\nIt misses the point to compare CVS and git and say CVS allows\nplacing unrelated things in the same \"repository\".  CVS does not\neven track the whole tree state, so it does not hurt the user\nnor the tool even if you did so.  With a tool that tracks the\nwhole tree, you need a bit of thinking and planning.\n\nI do not think, by the way, we are aiming at much different\nthings.  We both know that our current tools do not support\neither mode of operation; the direction you are coming from\nwhere everything is under one roof and only parts are accessed\nwhile others are left untouched, or an organization where\nloosely-related projects from different repositories are checked\nout into a working tree hierarchy.  You alluded to \"split index\"\nin another message, and the project organization I suggested to\nkeep component projects in their own separate repositories would\nalso have separate indices in component repositories.\n\nI am certainly not opposed to the idea of making operations in\nsuch a tree (built either way) go seamlessly for the users.  A\nPorcelain that supports such mode of operation needs to be built\nor enhanced, because we do not have one.\n\nWhat I am saying is that I suspect everything-under-one-roof\napproach would incur higher damage to the core than multiple\nrepositories approach.  It is my understanding that Cogito has\nsuch a light-weight subproject support that lets you have\nseparate repositories laid out in a single working tree.\n\nFor example, users of such a Porcelain most likely would not\nworry about what is stored in .git/index and .git/remotes/\ndirectories of individual repositories that appear in such a\nsingle working tree.  The Porcelain would keep track of which\nfiles are locally modified, what components are checked out in\nwhich subdirectory, where their upstream repositories are, and\nthings like that.  It will use .git/index and .git/remotes/ of\ncomponent repositories to implement the unified tree, but the\nuse of the individual .git/ directories is its implementation\ndetail.  It is likely to do its own bookkeeping that is beyond\nwhat the current core offers, but I do not think it needs much\nadditional core support.  If such bookkeeping turns out to be\nuseful and necessary to have it in the core for whatever reason\n(either performance or interoperability across Porcelains), we\ncould certainly talk about putting that into the core.\n\nI suspect that everything-under-one-roof approach is coming from\nan observation that:\n\n - with CVS, projects that share the same cvsroot can be updated\n   with single 'cvs update' command in a directory close to the\n   root.\n\n - with git, if you use multiple repositories checked out at\n   right places in the working tree hierarchy, you need to run\n   around and say \"git checkout\" or \"git commit\" everywhere.\n\nand the latter looks very inconvenient.\n\nBut of course the latter is very inconvenient.  The current \"git\ncheckout\" nor \"git commit\" are not such subprojects-aware\nPorcelain commands.  But that does not mean you have to house\neverything in the same repository and make partial check-in to\nwork.  You will be enhancing or replacing the same \"git checkout\"\nand \"git commit\" commands to do so anyway.\n"},{"id":"27083","messageId":"450D9847.4080308@gmail.com","threadId":"5564","inReplyTo":"7vvenm3h9f.fsf@assigned-by-dhcp.cox.net","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-17T18:47:35Z","receivedAt":"2006-09-17T18:47:35Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Junio C Hamano wrote:\n> A Large Angry SCM <gitzilla@gmail.com> writes:\n> \n>> Junio C Hamano wrote:\n>> ...\n>>> Having said that, I do not necessarily agree that highly modular\n>>> projects would want to put everything in one git repository and\n>>> track everything as a whole unit.\n>> And yet that's exactly how a lot of developers use CVS. You can argue\n>> that some other way is better but when they move from CVS they're\n>> looking for continuity of productivity which often means not radically\n>> changing how they work. At least in the short term.\n> \n[...]\n\n> I suspect that everything-under-one-roof approach is coming from\n> an observation that:\n> \n>  - with CVS, projects that share the same cvsroot can be updated\n>    with single 'cvs update' command in a directory close to the\n>    root.\n> \n>  - with git, if you use multiple repositories checked out at\n>    right places in the working tree hierarchy, you need to run\n>    around and say \"git checkout\" or \"git commit\" everywhere.\n> \n> and the latter looks very inconvenient.\n> \n> But of course the latter is very inconvenient.  The current \"git\n> checkout\" nor \"git commit\" are not such subprojects-aware\n> Porcelain commands.  But that does not mean you have to house\n> everything in the same repository and make partial check-in to\n> work.  You will be enhancing or replacing the same \"git checkout\"\n> and \"git commit\" commands to do so anyway.\n\nI used CVS as an example but I've seen the \"everything-under-one-roof\" \napproach, as you put it, used in other VCS' that do work with \nchangesets. One instance, in particular, has all the source and config \nfiles in a single tree that (I'm told) would take several Gigs of \nfilesystem space to checkout fully. The codebase is modular to a great \nextent but any particular project in it usually required files from a \nlarge number of other projects. There is a lot of automation to find the \nrequired parts for builds and other actions on a project's codebase.\n\nCould this be done with multiple repositories? Yes, but we're talking \nhundreds (no exaggeration) and many of those would likely end-up \nincluding a large percentage of the other repositories the way the Git \nrepository includes the Gitk repository. It could work but their \nexisting approach already works and is likely better suited for their \ncodebase. Git can not, currently, do all the things that this \norganization wants a VCS to do, working with partial checkouts is a key one.\n\nThere is no fundamental reason Git can not support partial \ncheckouts/working directories. In fact, there is no fundamental reason \nGit can not support operations on partial (sparse?) repositories in both \nspace (working content/state, etc.) and time (history); it's just a \nmatter of record keeping[*1*]. That isn't how the Linux kernel \ndevelopers want to use their VCS but it _is_ how others want to use theirs.\n\n\nNotes:\n[*1*] I'm currently working on determining the minimum requirements \nneeded to support repositories with partial or sparse history.\n"},{"id":"27084","messageId":"eek5l4$kaa$1@sea.gmane.org","threadId":"5564","inReplyTo":"450D9847.4080308@gmail.com","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-09-17T18:55:12Z","receivedAt":"2006-09-17T18:55:12Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"A Large Angry SCM wrote:\n\n> There is no fundamental reason Git can not support partial \n> checkouts/working directories. In fact, there is no fundamental reason \n> Git can not support operations on partial (sparse?) repositories in both \n> space (working content/state, etc.) and time (history); it's just a \n> matter of record keeping[*1*]. That isn't how the Linux kernel \n> developers want to use their VCS but it _is_ how others want to use\n> theirs. \n\nThere is perhaps not much trouble with partial checkouts, but there is\nproblem with partial _commits_, at least for snapshot based SCM (as opposed\nto patchset based SCM). \n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"27085","messageId":"450DA99F.8000009@gmail.com","threadId":"5564","inReplyTo":"eek5l4$kaa$1@sea.gmane.org","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-17T20:01:35Z","receivedAt":"2006-09-17T20:01:35Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Jakub Narebski wrote:\n> A Large Angry SCM wrote:\n> \n>> There is no fundamental reason Git can not support partial \n>> checkouts/working directories. In fact, there is no fundamental reason \n>> Git can not support operations on partial (sparse?) repositories in both \n>> space (working content/state, etc.) and time (history); it's just a \n>> matter of record keeping[*1*]. That isn't how the Linux kernel \n>> developers want to use their VCS but it _is_ how others want to use\n>> theirs. \n> \n> There is perhaps not much trouble with partial checkouts, but there is\n> problem with partial _commits_, at least for snapshot based SCM (as opposed\n> to patchset based SCM). \n\nBy \"partial commit\" I take it you mean a commit with only partial \ninformation about the new (content) state? If so, the missing \ninformation about the new state can be assumed to have not changed from \nthe previous recorded state (commit).\n"},{"id":"27086","messageId":"eekb4o$5nj$1@sea.gmane.org","threadId":"5564","inReplyTo":"450DA99F.8000009@gmail.com","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-09-17T20:28:52Z","receivedAt":"2006-09-17T20:28:52Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"A Large Angry SCM wrote:\n> Jakub Narebski wrote:\n>> A Large Angry SCM wrote:\n>> \n>>> There is no fundamental reason Git can not support partial \n>>> checkouts/working directories. In fact, there is no fundamental reason \n>>> Git can not support operations on partial (sparse?) repositories in both \n>>> space (working content/state, etc.) and time (history); it's just a \n>>> matter of record keeping[*1*]. That isn't how the Linux kernel \n>>> developers want to use their VCS but it _is_ how others want to use\n>>> theirs. \n>> \n>> There is perhaps not much trouble with partial checkouts, but there is\n>> problem with partial _commits_, at least for snapshot based SCM \n>> (as opposed to patchset based SCM). \n> \n> By \"partial commit\" I take it you mean a commit with only partial \n> information about the new (content) state? If so, the missing \n> information about the new state can be assumed to have not changed from \n> the previous recorded state (commit).\n\nThat of course assumes that 1) the whole state is recorded somewhere\n(perhaps in the repository); so the partial checkout saves space only if\nrepository compress really well, 2) there are no merges outside checked out\npart.\n\nIs anybody working on \"bind\" header and subproject support?\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"27088","messageId":"450DBA12.3070700@gmail.com","threadId":"5564","inReplyTo":"eekb4o$5nj$1@sea.gmane.org","subject":"Re: Notes on supporting Git operations in/on partial Working Directories","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-17T21:11:46Z","receivedAt":"2006-09-17T21:11:46Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Jakub Narebski wrote:\n> A Large Angry SCM wrote:\n>> Jakub Narebski wrote:\n>>> A Large Angry SCM wrote:\n>>>\n>>>> There is no fundamental reason Git can not support partial \n>>>> checkouts/working directories. In fact, there is no fundamental reason \n>>>> Git can not support operations on partial (sparse?) repositories in both \n>>>> space (working content/state, etc.) and time (history); it's just a \n>>>> matter of record keeping[*1*]. That isn't how the Linux kernel \n>>>> developers want to use their VCS but it _is_ how others want to use\n>>>> theirs. \n>>> There is perhaps not much trouble with partial checkouts, but there is\n>>> problem with partial _commits_, at least for snapshot based SCM \n>>> (as opposed to patchset based SCM). \n>> By \"partial commit\" I take it you mean a commit with only partial \n>> information about the new (content) state? If so, the missing \n>> information about the new state can be assumed to have not changed from \n>> the previous recorded state (commit).\n> \n> That of course assumes that 1) the whole state is recorded somewhere\n> (perhaps in the repository); so the partial checkout saves space only if\n> repository compress really well, 2) there are no merges outside checked out\n> part.\n\n1) The TREE objects leading to the objects that are \nadded/deleted/changed objects are required. TREEs not leading to the \nadded/deleted/changed objects are not required, only their IDs. That is \nsufficient to commit the changes in a partial checkout.\n\n2) Obviously, only the part checked out can be worked on. If you want to \nmerge changes to some other part, you will need that part, and possibly \na mergebase.\n"}]}