{"thread":{"id":"7390","subject":".gitlink for Summer of Code","startedAt":"2007-03-25T12:30:38Z","lastAt":"2007-03-27T23:31:41Z","messageCount":61,"participants":["Eric Lesh","Matthieu Moy","Shawn O. Pearce","Johannes Schindelin","Junio C Hamano","Jakub Narebski","Martin Waitz","Josef Weidendorfer","David Lang","Petr Baudis","Uwe Kleine-König","Linus Torvalds","Daniel Barkalow","Steven Grimm"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"37933","messageId":"1174825838.12540.5.camel@localhost","threadId":"7390","inReplyTo":null,"subject":".gitlink for Summer of Code","fromName":"Eric Lesh","fromEmail":"eclesh@ucla.edu","sentAt":"2007-03-25T12:30:38Z","receivedAt":"2007-03-25T12:30:38Z","isPatch":false,"sender":{"key":"eclesh@ucla.edu","avatar":null},"body":"I would like to tackle .gitlink for Summer of Code. The deadline is\nabout a day away, but this is a chance to make sure you still think\n.gitlink is a good idea, and to see if there are any big problems with\nthe idea in general (before a more detailed proposal is actually submitted).\n\n.gitlink is for a lightweight checkout of a branch into a separate\ndirectory on the local filesystem. A .gitlink'ed checkout has its own\nindex+HEAD, but otherwise refers back to the main repository for\nobjects, refs, etc.\n\nJunio has said (http://www.spinics.net/lists/git/msg24964.html) he\nworks in a similar way with many work trees which are symlinked to his\nmain repo:\n: gitster git.wk0; ls -l .git/\ntotal 120\ndrwxrwsr-x 3 junio src 4096 Mar 5 16:22 ./\ndrwxrwsr-x 15 junio src 16384 Mar 5 16:23 ../\n-rw-rw-r-- 1 junio src 41 Mar 5 16:22 HEAD\nlrwxrwxrwx 1 junio src 27 Mar 3 22:53 config -> /src/git/.git/config\nlrwxrwxrwx 1 junio src 26 Mar 3 22:53 hooks -> /src/git/.git/hooks/\n-rw-rw-r-- 1 junio src 82455 Mar 5 16:22 index\nlrwxrwxrwx 1 junio src 25 Mar 3 22:53 info -> /src/git/.git/info/\ndrwxrwsr-x 3 junio src 4096 Mar 3 22:59 logs/\nlrwxrwxrwx 1 junio src 28 Mar 3 22:53 objects -> /src/git/.git/objects/\nlrwxrwxrwx 1 junio src 32 Mar 3 22:53 packed-refs -> /src/git/.git/packed-refs\nlrwxrwxrwx 1 junio src 25 Mar 3 22:53 refs -> /src/git/.git/refs/\nlrwxrwxrwx 1 junio src 28 Mar 3 22:53 remotes -> /src/git/.git/remotes/\nlrwxrwxrwx 1 junio src 29 Mar 3 22:53 rr-cache -> /src/git/.git/rr-cache/\n\nA .gitlink could simplify this, ridding the directory of symlinks and\ninstead having a .gitlink file that specifies a $GIT_DIR to which\neverything refers.\n\nMy implementation will be based on Josef Weidendorfer's \"[RFC]\nLightweight checkouts via \".gitlink\"\"\n(http://thread.gmane.org/gmane.comp.version-control.git/33755), which\ndidn't receive much objection (or comment at all). Hopefully this\ndoesn't mean you simply aren't interested.\n\nGoals:\n o Lightweight checkouts are essentially branches that reside outside of the\n main repository on the local filesystem.\n o They act as normal git checkouts (i.e. git tools know about .gitlink)\n o They can be moved around and maintain their link to the base repo\n o They can exist within other checkouts (for eventual submodule support)\n o git-clone --light-weight option to set this up\n o porcelains (i.e. cogito) don't barf\n\nIs this something that you would like to see? Any other comments?\n\nThanks,\n\nEric\n"},{"id":"37936","messageId":"vpqvegpe4e1.fsf@olympe.imag.fr","threadId":"7390","inReplyTo":"1174825838.12540.5.camel@localhost","subject":"Re: .gitlink for Summer of Code","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2007-03-25T15:20:38Z","receivedAt":"2007-03-25T15:20:38Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Eric Lesh writes:\n\n> .gitlink is for a lightweight checkout of a branch into a separate\n> directory on the local filesystem.\n\nI think it's a pitty to restrict yourself to _local_ filesystem. There\nare tons of cases where you have a fast, non-NFS, access to a machine\nand would like to host your repository there.\n\nThat said, I suppose removing this restriction moves the solution from\nthe category \"quick and efficient hack\" to something much harder.\n\n> A .gitlink'ed checkout has its own index+HEAD, but otherwise refers\n> back to the main repository for objects, refs, etc.\n\nStupid question: why .gitlink, and not .git/link or so? This file is\nnot versionned, I don't think it should be in the working tree.\n\n-- \nMatthieu\n"},{"id":"37948","messageId":"20070325203901.GB12376@spearce.org","threadId":"7390","inReplyTo":"vpqvegpe4e1.fsf@olympe.imag.fr","subject":"Re: .gitlink for Summer of Code","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-03-25T20:39:01Z","receivedAt":"2007-03-25T20:39:01Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Matthieu Moy wrote:\n> Eric Lesh writes:\n> \n> > .gitlink is for a lightweight checkout of a branch into a separate\n> > directory on the local filesystem.\n> \n> I think it's a pitty to restrict yourself to _local_ filesystem. There\n> are tons of cases where you have a fast, non-NFS, access to a machine\n> and would like to host your repository there.\n> \n> That said, I suppose removing this restriction moves the solution from\n> the category \"quick and efficient hack\" to something much harder.\n\nYes. But there's another project on the ideas list that addresses\nthat (``Lazy Clone''). It is quite a bit more difficult than the\n.gitlink idea as the implementation requires a network protocol\nclient implemented somewhere near the read_sha1_file interface.\n\nJunio and I had talked about this (I think it was on the list,\nbut maybe it was on #git) recently and considered maybe trying to\ndo it as a wrapper *above* read_sha1_file/has_sha1_file, and adjust\nthe clients that call them to invoke the wrapper instead.\n\nNothing will obviously beat having the file on a local disk; but if\nyou have a fast LAN it may be hard to beat an anonymous TCP socket\ndedicated to serving Git objects. Such a socket may even beat out\nNFS... ;-)\n \n> > A .gitlink'ed checkout has its own index+HEAD, but otherwise refers\n> > back to the main repository for objects, refs, etc.\n> \n> Stupid question: why .gitlink, and not .git/link or so? This file is\n> not versionned, I don't think it should be in the working tree.\n\nI've thought the same thing.\n\nActually, I'd almost say put it into .git/config, e.g.:\n\n\tmkdir .git\n\tcat >.git/config < wrote:\n> I would like to tackle .gitlink for Summer of Code. The deadline is\n> about a day away, but this is a chance to make sure you still think\n> .gitlink is a good idea, and to see if there are any big problems with\n> the idea in general (before a more detailed proposal is actually submitted).\n...\n> Is this something that you would like to see? Any other comments?\n\nAside from the other fork of this thread discussing the ``Lazy\nClone'' idea, I don't really have any other comments.\n\nI don't setup the same thing as Junio, but I have the same problem.\nMy day job setup has at least 5 copies of the same repository.\nI'd like an easy way to have them share the same odb, refs,\nand config.\n\nIf you choose to submit an application please see our template\n(http://git.or.cz/gitwiki/SoC2007Template) and try to talk a bit\nabout your background too.\n\n-- \nShawn.\n"},{"id":"37952","messageId":"Pine.LNX.4.63.0703252252500.4045@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"7390","inReplyTo":"20070325203901.GB12376@spearce.org","subject":"Re: .gitlink for Summer of Code","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-03-25T20:54:21Z","receivedAt":"2007-03-25T20:54:21Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 25 Mar 2007, Shawn O. Pearce wrote:\n\n> Matthieu Moy wrote:\n> > Eric Lesh writes:\n> > \n> > > .gitlink is for a lightweight checkout of a branch into a separate\n> > > directory on the local filesystem.\n> > \n> > I think it's a pitty to restrict yourself to _local_ filesystem. There\n> > are tons of cases where you have a fast, non-NFS, access to a machine\n> > and would like to host your repository there.\n> > \n> > That said, I suppose removing this restriction moves the solution from\n> > the category \"quick and efficient hack\" to something much harder.\n> \n> Yes. But there's another project on the ideas list that addresses\n> that (``Lazy Clone''). It is quite a bit more difficult than the\n> .gitlink idea as the implementation requires a network protocol\n> client implemented somewhere near the read_sha1_file interface.\n\nNot only that. You'd have to change the way read_sha1_file() is called to \nallow fetching more than one object at a time. Otherwise this will be so \nslow as to be unusable.\n\nThat's basically the reason why I changed my mind, and preferred shallow \nclones over lazy clones.\n\nCiao,\nDscho\n"},{"id":"37953","messageId":"7v3b3t3uws.fsf@assigned-by-dhcp.cox.net","threadId":"7390","inReplyTo":"20070325203901.GB12376@spearce.org","subject":"Re: .gitlink for Summer of Code","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-03-25T20:55:31Z","receivedAt":"2007-03-25T20:55:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Actually, I'd almost say put it into .git/config, e.g.:\n>\n> \tmkdir .git\n> \tcat >.git/config < \t[core]\n> \t\trepositoryversion = 0\n> \t\tfilemode = true\n> \t\tlink = /path/to/source\n> \tEOF\n>\n> as then the index and HEAD file can both be stored in .git, just\n> like with the non-gitlink case.\n\nThis is not usable at least for me as it does not allow sharing\nthe .git/config file across checkouts.\n"},{"id":"37955","messageId":"20070325210322.GF12376@spearce.org","threadId":"7390","inReplyTo":"Pine.LNX.4.63.0703252252500.4045@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .gitlink for Summer of Code","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-03-25T21:03:22Z","receivedAt":"2007-03-25T21:03:22Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin wrote:\n> On Sun, 25 Mar 2007, Shawn O. Pearce wrote:\n> > Yes. But there's another project on the ideas list that addresses\n> > that (``Lazy Clone''). It is quite a bit more difficult than the\n> > .gitlink idea as the implementation requires a network protocol\n> > client implemented somewhere near the read_sha1_file interface.\n> \n> Not only that. You'd have to change the way read_sha1_file() is called to \n> allow fetching more than one object at a time. Otherwise this will be so \n> slow as to be unusable.\n\nYes, and no. ;-)\n\nLets say we put a repository on an NFS share, and clone it using\n--shared. So it's now an alternate ODB. And read_sha1_file()\nis now doing synchronous reads, unless the client has something\ncached. Which we could just as easily cache ourselves in the\nloose object directory\n\nWe could make it faster by batching up requests. But batching\nrequests may be difficult in some contexts, as we don't know what\nelse we need until we get back that commit or tree we are trying\nto read. ;-)\n \n-- \nShawn.\n"},{"id":"37956","messageId":"20070325210504.GG12376@spearce.org","threadId":"7390","inReplyTo":"7v3b3t3uws.fsf@assigned-by-dhcp.cox.net","subject":"Re: .gitlink for Summer of Code","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-03-25T21:05:04Z","receivedAt":"2007-03-25T21:05:04Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > Actually, I'd almost say put it into .git/config, e.g.:\n> >\n> > \tmkdir .git\n> > \tcat >.git/config < > \t[core]\n> > \t\trepositoryversion = 0\n> > \t\tfilemode = true\n> > \t\tlink = /path/to/source\n> > \tEOF\n> >\n> > as then the index and HEAD file can both be stored in .git, just\n> > like with the non-gitlink case.\n> \n> This is not usable at least for me as it does not allow sharing\n> the .git/config file across checkouts.\n\nMe either.\n\nWhat I thought of after writing that was that core.link should\nalso imply reading ${core.link}/config before .git/config, so that\nthe repository config can override the master repository, but the\nmaster repository provides the bulk of the configuration.\n\n-- \nShawn.\n"},{"id":"38045","messageId":"1174929374.5662.15.camel@localhost","threadId":"7390","inReplyTo":"vpqvegpe4e1.fsf@olympe.imag.fr","subject":"Re: .gitlink for Summer of Code","fromName":"Eric Lesh","fromEmail":"eclesh@ucla.edu","sentAt":"2007-03-26T17:16:14Z","receivedAt":"2007-03-26T17:16:14Z","isPatch":false,"sender":{"key":"eclesh@ucla.edu","avatar":null},"body":"On Sun, 2007-03-25 at 17:20 +0200, Matthieu Moy wrote:\n> I think it's a pitty to restrict yourself to _local_ filesystem. There\n> are tons of cases where you have a fast, non-NFS, access to a machine\n> and would like to host your repository there.\n> \n> That said, I suppose removing this restriction moves the solution from\n> the category \"quick and efficient hack\" to something much harder.\n\nFor now, this is really meant for quick and easy access to multiple\nbranches of your own repo at the same time.\n\n\n> \n> Stupid question: why .gitlink, and not .git/link or so? This file is\n> not versionned, I don't think it should be in the working tree.\n> \n\nThere is no .git dir for these. The .gitlink refers back to the main\nrepository's .git/external/$submodule, which is a full .git dir with\nindex+HEAD and symlinks back to the .git of the main repo for everything\nelse.\n\n-Eric\n"},{"id":"38046","messageId":"vpqr6rbex7g.fsf@olympe.imag.fr","threadId":"7390","inReplyTo":"1174929374.5662.15.camel@localhost","subject":"Re: .gitlink for Summer of Code","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2007-03-26T17:22:43Z","receivedAt":"2007-03-26T17:22:43Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Eric Lesh writes:\n\n> There is no .git dir for these. The .gitlink refers back to the main\n> repository's .git/external/$submodule, which is a full .git dir with\n> index+HEAD and symlinks back to the .git of the main repo for everything\n> else.\n\nI don't see any contradiction.\n\nLight checkouts would have an almost empty .git (it still needs an\nindex, and its own head anyway) and use the .git of the main repo for\neverything else.\n\n-- \nMatthieu\n"},{"id":"38047","messageId":"eu8vrb$r15$1@sea.gmane.org","threadId":"7390","inReplyTo":"vpqvegpe4e1.fsf@olympe.imag.fr","subject":"Re: .gitlink for Summer of Code","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-03-26T17:31:18Z","receivedAt":"2007-03-26T17:31:18Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Matthieu Moy wrote:\n\n> Stupid question: why .gitlink, and not .git/link or so? This file is\n> not versionned, I don't think it should be in the working tree.\n\nIt would be special-cased, as is .git directory not versioned, the\nsame way .gitlink would be not versioned (not like .gitignore).\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"38048","messageId":"1174930688.5662.20.camel@localhost","threadId":"7390","inReplyTo":"vpqr6rbex7g.fsf@olympe.imag.fr","subject":"Re: .gitlink for Summer of Code","fromName":"Eric Lesh","fromEmail":"eclesh@ucla.edu","sentAt":"2007-03-26T17:38:08Z","receivedAt":"2007-03-26T17:38:08Z","isPatch":false,"sender":{"key":"eclesh@ucla.edu","avatar":null},"body":"On Mon, 2007-03-26 at 19:22 +0200, Matthieu Moy wrote:\n> \n> I don't see any contradiction.\n> \n> Light checkouts would have an almost empty .git (it still needs an\n> index, and its own head anyway) and use the .git of the main repo for\n> everything else.\n> \n\nJosef Weidendorfer tried to implement this before, and he concluded that\nhaving a _text file_ .git, instead of a directory, would be a good way\nto distinguish .gitlinked checkouts from normal checkouts.\n\nAs far as .git/link, that would seem to work better. As long as there\nis a sanity check to make sure that you don't manage to mix things up,\nit might be fine.\n\n-Eric\n"},{"id":"38050","messageId":"vpq3b3ruaqp.fsf@olympe.imag.fr","threadId":"7390","inReplyTo":"eu8vrb$r15$1@sea.gmane.org","subject":"Re: .gitlink for Summer of Code","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2007-03-26T18:21:18Z","receivedAt":"2007-03-26T18:21:18Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Jakub Narebski writes:\n\n> Matthieu Moy wrote:\n>\n>> Stupid question: why .gitlink, and not .git/link or so? This file is\n>> not versionned, I don't think it should be in the working tree.\n>\n> It would be special-cased, as is .git directory not versioned, the\n> same way .gitlink would be not versioned (not like .gitignore).\n\nThat's how I understand it, but why 2 special cases (.git and\n.gitlink) when you need one?\n\n-- \nMatthieu\n"},{"id":"38051","messageId":"20070326183523.GG22773@admingilde.org","threadId":"7390","inReplyTo":"1174930688.5662.20.camel@localhost","subject":"Re: .gitlink for Summer of Code","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-26T18:35:23Z","receivedAt":"2007-03-26T18:35:23Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Mon, Mar 26, 2007 at 10:38:08AM -0700, Eric Lesh wrote:\n> > Light checkouts would have an almost empty .git (it still needs an\n> > index, and its own head anyway) and use the .git of the main repo for\n> > everything else.\n> > \n> \n> Josef Weidendorfer tried to implement this before, and he concluded that\n> having a _text file_ .git, instead of a directory, would be a good way\n> to distinguish .gitlinked checkouts from normal checkouts.\n\nbut that does not allow for per-checkout HEAD and index.\nI don't see any reason for providing any sort for \"gitlink\" which\nalso uses HEAD and index from the linked location -- then you could\nuse a simple symlink, too. So having an almost empty .git directory\nand reusing parts from another .git directory makes a lot of sense to\nme.\n\n-- \nMartin Waitz\n"},{"id":"38057","messageId":"200703262133.44520.Josef.Weidendorfer@gmx.de","threadId":"7390","inReplyTo":"20070326183523.GG22773@admingilde.org","subject":"Re: .gitlink for Summer of Code","fromName":"Josef Weidendorfer","fromEmail":"josef.weidendorfer@gmx.de","sentAt":"2007-03-26T19:33:44Z","receivedAt":"2007-03-26T19:33:44Z","isPatch":false,"sender":{"key":"josef.weidendorfer@gmx.de","avatar":null},"body":"On Monday 26 March 2007, Martin Waitz wrote:\n> hoi :)\n> \n> On Mon, Mar 26, 2007 at 10:38:08AM -0700, Eric Lesh wrote:\n> > > Light checkouts would have an almost empty .git (it still needs an\n> > > index, and its own head anyway) and use the .git of the main repo for\n> > > everything else.\n> > > \n> > \n> > Josef Weidendorfer tried to implement this before, and he concluded that\n> > having a _text file_ .git, instead of a directory, would be a good way\n> > to distinguish .gitlinked checkouts from normal checkouts.\n> \n> but that does not allow for per-checkout HEAD and index.\n> I don't see any reason for providing any sort for \"gitlink\" which\n> also uses HEAD and index from the linked location -- then you could\n> use a simple symlink, too.\n\nThe idea was to make this a possible building block for submodules.\nA simple symlink does not work there when you want the checkout to\nwork even after moving the whole checkout directory around (e.g. to move the\nsubmodule around inside of the superproject).\n\n> So having an almost empty .git directory \n> and reusing parts from another .git directory makes a lot of sense to\n> me.\n\nThis would work. However, you can not clone from an almost empty .git\ndirectory with current git.\n\nThe original proposal was to have a standard .git directory for every\nlight-weight checkout inside of the base .git directory, e.g.\nin /.git/ext/.git where is some identifier for the\nlightweight checkout, either provided in the .gitlink file or\nautomatically determined.\n\nHmm... the \"almost empty .git directory\" has its merits.\nYou can override config options, and of course, the \"base\" for the\nlightweight checkout still can be a full .git dir, as would be needed\nfor submodule support. In fact, you have more freedom to choose the\npath to the base gitdir.\nI like it ;-)\n\nSo this changes the .gitlink proposal to:\n* smartly reset GIT_DIR when a core.link option is set in .git/config\n (and set GIT_WORK_DIR accordingly)\n* fake a core.name if not set (this comes from the original proposal\n to get an automatic identifier of a submodule checkout by its\n relative path to the supermodule)\n* git-checkout to allow to create a fresh light-weight checkout\n\nJosef\n"},{"id":"38058","messageId":"vpqodmfss39.fsf@olympe.imag.fr","threadId":"7390","inReplyTo":"200703262133.44520.Josef.Weidendorfer@gmx.de","subject":"Re: .gitlink for Summer of Code","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2007-03-26T19:49:30Z","receivedAt":"2007-03-26T19:49:30Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Josef Weidendorfer writes:\n\n> This would work. However, you can not clone from an almost empty .git\n> directory with current git.\n\nOf course not. If you say \"have .git/ <=> is a repository\", then your\nlightweight checkout should not have a .git/. But why should anything\nthat have a .git/ directory be a repository??\n\n> The original proposal was to have a standard .git directory for every\n> light-weight checkout inside of the base .git directory, e.g.\n> in /.git/ext/.git where is some identifier for the\n> lightweight checkout, either provided in the .gitlink file or\n> automatically determined.\n\nThat seems really weird. That implies for example:\n\n* Deleting a checkout means deleting both your local tree _and_ a part\n of the .git/ directory of the repository. Have you ever imagined\n having to do more than \"rm -fr working-tree\" even with an inferior\n VCS such as CVS?\n\n* It makes it impossible to have a checkout of a read-only location.\n For example, if one of my colleague has a repository in\n /home/otheruser/repo/, if I want to get a working tree of it, I need\n to get a complete clone of his repo to be able to do it. Assuming\n someone ever implements a \"lightweight checkout of a remote\n location\" (bzr has this for example. You can run \n \"bzr checkout --lightweigth http://whatever.com/), this would mean\n creating a directory on the server for each of the potential\n clients, not to mention the impossibility to do it over http.\n\n* You have to manage a name for each lightweight checkout. What would\n be such name? User-provided? uuidgen-like?\n\n\nI find the way bzr deals with this pretty elegant: a repository with a\nworking tree is just a working tree and a repository located in the\nsame directory. The repository stores its files (content of each\nrevisions in history, ...) in .bzr/repository/, and the working tree\nstores them (the index, pending merges, ... and pointer to the\ncorresponding branch) in .bzr/checkout/.\n\n-- \nMatthieu\n"},{"id":"38070","messageId":"20070326220302.GH22773@admingilde.org","threadId":"7390","inReplyTo":"200703262133.44520.Josef.Weidendorfer@gmx.de","subject":"Re: .gitlink for Summer of Code","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-26T22:03:08Z","receivedAt":"2007-03-26T22:03:08Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"On Mon, Mar 26, 2007 at 09:33:44PM +0200, Josef Weidendorfer wrote:\n> The idea was to make this a possible building block for submodules.\n> A simple symlink does not work there when you want the checkout to\n> work even after moving the whole checkout directory around (e.g. to move the\n> submodule around inside of the superproject).\n\nWell the submodule use case is a bit different than the lightweight\ncheckout.\nWhen you store the submodule object database inside the supermodule then\nyou only need to store the position of the submodule relative to its\nsupermodule. As you wrote this is neccessary in order to find the part\nof the object database which belongs to this one submodule.\nFinding the supermodule repository is obviously not difficult, only\nfinding the right part of it.\nBut for lightweight checkouts you need something which is closer to a\nsymlink.\n\n> > So having an almost empty .git directory \n> > and reusing parts from another .git directory makes a lot of sense to\n> > me.\n> \n> This would work. However, you can not clone from an almost empty .git\n> directory with current git.\n\nYou can't clone from a .gitlink with current git, eighter ;-).\nBut if you e.g. set git_dir according to your link then everything\nshould work quite easily.\n\n> The original proposal was to have a standard .git directory for every\n> light-weight checkout inside of the base .git directory, e.g.\n> in /.git/ext/.git where is some identifier for the\n> lightweight checkout, either provided in the .gitlink file or\n> automatically determined.\n\nWhat would you store in these per-checkout directories?\nThe index and HEAD? Anything more?\nFor submodules I currently use /.git/objects/module//\nto store the objects belonging to the submodule.\nPerhaps it makes sense to extend this to a full .git directory per\nsubmodule, I'm not yet decided on that.\nFor submodules the object store has to be different, but for normal\nlightweight checkout this should of course be shared.\n\n-- \nMartin Waitz\n"},{"id":"38077","messageId":"7vfy7rvct2.fsf@assigned-by-dhcp.cox.net","threadId":"7390","inReplyTo":"20070326220302.GH22773@admingilde.org","subject":"Re: .gitlink for Summer of Code","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-03-26T22:51:21Z","receivedAt":"2007-03-26T22:51:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Waitz writes:\n\n> For submodules I currently use /.git/objects/module//\n> to store the objects belonging to the submodule.\n\nI was not following the gitlink discussion closely, but what is\nthe motivation behind this separation of the object store?\n"},{"id":"38076","messageId":"200703270100.12920.Josef.Weidendorfer@gmx.de","threadId":"7390","inReplyTo":"20070326220302.GH22773@admingilde.org","subject":"Re: .gitlink for Summer of Code","fromName":"Josef Weidendorfer","fromEmail":"josef.weidendorfer@gmx.de","sentAt":"2007-03-26T23:00:12Z","receivedAt":"2007-03-26T23:00:12Z","isPatch":false,"sender":{"key":"josef.weidendorfer@gmx.de","avatar":null},"body":"On Tuesday 27 March 2007, Martin Waitz wrote:\n> On Mon, Mar 26, 2007 at 09:33:44PM +0200, Josef Weidendorfer wrote:\n> > The idea was to make this a possible building block for submodules.\n> > A simple symlink does not work there when you want the checkout to\n> > work even after moving the whole checkout directory around (e.g. to move the\n> > submodule around inside of the superproject).\n> \n> Well the submodule use case is a bit different than the lightweight\n> checkout.\n> When you store the submodule object database inside the supermodule then\n> you only need to store the position of the submodule relative to its\n> supermodule. As you wrote this is neccessary in order to find the part\n> of the object database which belongs to this one submodule.\n\nWhere do you store this in your module3 branch?\n\n> Finding the supermodule repository is obviously not difficult, only\n> finding the right part of it.\n> But for lightweight checkouts you need something which is closer to a\n> symlink.\n\nYes, of course.\n\n> > > So having an almost empty .git directory \n> > > and reusing parts from another .git directory makes a lot of sense to\n> > > me.\n> > \n> > This would work. However, you can not clone from an almost empty .git\n> > directory with current git.\n> \n> You can't clone from a .gitlink with current git, eighter ;-).\n> But if you e.g. set git_dir according to your link then everything\n> should work quite easily.\n\nYes.\n\n> > The original proposal was to have a standard .git directory for every\n> > light-weight checkout inside of the base .git directory, e.g.\n> > in /.git/ext/.git where is some identifier for the\n> > lightweight checkout, either provided in the .gitlink file or\n> > automatically determined.\n> \n> What would you store in these per-checkout directories?\n> The index and HEAD? Anything more?\n\nTo make it easy to implement, I thought about a standard .git layout,\nwith most directories being symlinks.\n\n> For submodules I currently use /.git/objects/module//\n> to store the objects belonging to the submodule.\n> Perhaps it makes sense to extend this to a full .git directory per\n> submodule, I'm not yet decided on that.\n\nIMHO this would be a nice property. As the submodule could exist independently\nwith its own remote heads/tags, you probably would want to at least track these,\neven if it is a submodule in your superproject.\nAnd then it makes sense to move it directly to .git/module/...\n\nThere also was a use case where one library project is used in >10\nsuperprojects. It would be nice to be able to make the submodule git dir\nbe outside of the supermodules git dir. However, this also can be done\nwith symlinks without any special support (aside from sharing the\nhead namespace).\n\nJosef\n\n> For submodules the object store has to be different, but for normal\n> lightweight checkout this should of course be shared.\n"},{"id":"38079","messageId":"200703270114.49365.Josef.Weidendorfer@gmx.de","threadId":"7390","inReplyTo":"vpqodmfss39.fsf@olympe.imag.fr","subject":"Re: .gitlink for Summer of Code","fromName":"Josef Weidendorfer","fromEmail":"josef.weidendorfer@gmx.de","sentAt":"2007-03-26T23:14:49Z","receivedAt":"2007-03-26T23:14:49Z","isPatch":false,"sender":{"key":"josef.weidendorfer@gmx.de","avatar":null},"body":"On Monday 26 March 2007, Matthieu Moy wrote:\n> Josef Weidendorfer writes:\n> > The original proposal was to have a standard .git directory for every\n> > light-weight checkout inside of the base .git directory, e.g.\n> > in /.git/ext/.git where is some identifier for the\n> > lightweight checkout, either provided in the .gitlink file or\n> > automatically determined.\n> \n> That seems really weird. That implies for example:\n> [... some good reasons to not do it this way ...]\n\nOk, you are right.\nIt is better to not touch the original repository\nfor lightweight checkouts.\n\n> * You have to manage a name for each lightweight checkout. What would\n> be such name? User-provided? uuidgen-like?\n\nSuch a name is interesting as identifier for submodules.\nIt would be the relative path of the submodule from the supermodule base;\nor user supplied.\n\nLightweight checkouts and submodules have different requirements;\nyet, the lightweight checkouts should be so flexible to be\nable to be used for submodules checkouts.\n\nJosef\n"},{"id":"38081","messageId":"20070326231637.GJ22773@admingilde.org","threadId":"7390","inReplyTo":"7vfy7rvct2.fsf@assigned-by-dhcp.cox.net","subject":"Submodule object store","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-26T23:16:38Z","receivedAt":"2007-03-26T23:16:38Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Mon, Mar 26, 2007 at 03:51:21PM -0700, Junio C Hamano wrote:\n> I was not following the gitlink discussion closely, but what is\n> the motivation behind this separation of the object store?\n\nMostly scalability.\nSome operations need to traverse all objects and this may be prohibitive\nin large repositories. So the traversal has to be split somehow to not\nrequire all objects to be loaded into RAM at the same time.\nThe most natural separation is at the submodule level.\nThe idea is to collect all the submodule references in the supermodule\nand then traverse the submodule independently with the saved references.\n\nTo make purge and fsck work we have the hard requirement that it\nmust be possible to list all objects which belong to one submodule.\nAs modules (rightfully!) don't have any project ID, we need some other\nmeans to map submodule tree entries which are stored in the supermodule to\nthe corresponding object store.\nThe most simple way is to use the location of the submodule within the\nparent to find the submodule object store. prune and fsck can then use\na path-limited commit traversal in the parent to get all relevant\nsubmodule references.\n\n\nAnd moving the submodule object store into the .git directory of the\nsupermodule has several reasons: in bare repositories it has to be\nthat way anyway and I don't want to loose the submodules history if\nthe user decides to remove the submodule from his working directory.\n\nNot the entire submodule object store has to be moved this way, only\nthat part that is referenced by the supermodule. So it can make sense\nto have a full .git repository in the submodule, together with another\nobject store (handled like alternates) in the supermodule.\n\n-- \nMartin Waitz\n"},{"id":"38080","messageId":"200703270117.59205.Josef.Weidendorfer@gmx.de","threadId":"7390","inReplyTo":"7vfy7rvct2.fsf@assigned-by-dhcp.cox.net","subject":"Re: .gitlink for Summer of Code","fromName":"Josef Weidendorfer","fromEmail":"josef.weidendorfer@gmx.de","sentAt":"2007-03-26T23:17:58Z","receivedAt":"2007-03-26T23:17:58Z","isPatch":false,"sender":{"key":"josef.weidendorfer@gmx.de","avatar":null},"body":"On Tuesday 27 March 2007, Junio C Hamano wrote:\n> Martin Waitz writes:\n> \n> > For submodules I currently use /.git/objects/module//\n> > to store the objects belonging to the submodule.\n> \n> I was not following the gitlink discussion closely, but what is\n> the motivation behind this separation of the object store?\n\nThe separation issue is about scalability of submodules, and not\ndirectly about gitlink.\n\nJosef\n"},{"id":"38088","messageId":"Pine.LNX.4.63.0703261515330.14387@qynat.qvtvafvgr.pbz","threadId":"7390","inReplyTo":"20070326233603.GL22773@admingilde.org","subject":"Re: Submodule object store","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2007-03-26T23:20:34Z","receivedAt":"2007-03-26T23:20:34Z","isPatch":false,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Tue, 27 Mar 2007, Martin Waitz wrote:\n\n> On Mon, Mar 26, 2007 at 04:28:28PM -0700, Junio C Hamano wrote:\n>> Martin Waitz writes:\n>>\n>>> To make purge and fsck work we have the hard requirement that it\n>>> must be possible to list all objects which belong to one submodule.\n>>\n>> I understand you would want to separate the ref namespace, but I\n>> still do not see why you would want to have a separate object\n>> store, laid out in a funny way. Unless you are thinking about\n>> using rsync to transfer object store, that is.\n>\n> I want to be able to list all objects which are not reachable in the\n> object store, without traversing all submodules at the same time.\n> The only way I can think of to achieve this is to have one separate\n> object store per submodule and then do the traversal per submodule.\n\nwhy do you want to optimize for the relativly rare fsck function rather then the \nmore common read functions (which would benifit from shareing object that are \nidentical between projects)?\n\nDavid Lang\n"},{"id":"38082","messageId":"7v3b3rvbae.fsf@assigned-by-dhcp.cox.net","threadId":"7390","inReplyTo":"200703270117.59205.Josef.Weidendorfer@gmx.de","subject":"Re: .gitlink for Summer of Code","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-03-26T23:24:09Z","receivedAt":"2007-03-26T23:24:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josef Weidendorfer writes:\n\n> On Tuesday 27 March 2007, Junio C Hamano wrote:\n>> Martin Waitz writes:\n>> \n>> > For submodules I currently use /.git/objects/module//\n>> > to store the objects belonging to the submodule.\n>> \n>> I was not following the gitlink discussion closely, but what is\n>> the motivation behind this separation of the object store?\n>\n> The separation issue is about scalability of submodules, and not\n> directly about gitlink.\n\nUnless you are thinking about rsync of object store, it is not\nclear what \"scalability of submodules\" has to do with having\nseparate object database.\n\nThe issue I recall from earlier discussion on scalability of\nsubmodules was about the direct placement of commit objects in\nsupermodule trees (which would force fetching in supermodule to\ndrag in all submodules even when the \"integration\" or \"build\ninfrastructure\" person does not want to have submodules pulled).\nBut I think that is an issue of the definition of connectivity.\nIt should be orthogonal to the issue of how object store is laid\nout, shouldn't it?\n"},{"id":"38083","messageId":"20070326232744.GK22773@admingilde.org","threadId":"7390","inReplyTo":"200703270100.12920.Josef.Weidendorfer@gmx.de","subject":"Re: .gitlink for Summer of Code","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-26T23:27:46Z","receivedAt":"2007-03-26T23:27:46Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Tue, Mar 27, 2007 at 01:00:12AM +0200, Josef Weidendorfer wrote:\n> On Tuesday 27 March 2007, Martin Waitz wrote:\n> > For submodules I currently use /.git/objects/module//\n> > to store the objects belonging to the submodule.\n> > Perhaps it makes sense to extend this to a full .git directory per\n> > submodule, I'm not yet decided on that.\n> \n> IMHO this would be a nice property. As the submodule could exist\n> independently with its own remote heads/tags, you probably would want\n> to at least track these, even if it is a submodule in your\n> superproject. And then it makes sense to move it directly to\n> .git/module/...\n\nI am not sure that all the other submodule heads and tags really belong\ninto the superproject. Perhaps they should simply be handled in some\nother way -- after all the submodule is a normal git repository and can\nhandle heads and tags on its own quite well.\nBut I haven't thought tag-handling for submodules through yet.\n\n> There also was a use case where one library project is used in >10\n> superprojects. It would be nice to be able to make the submodule git dir\n> be outside of the supermodules git dir. However, this also can be done\n> with symlinks without any special support (aside from sharing the\n> head namespace).\n\nOf course you can always have a normal library.git repository with all\nthe tags and stuff somewhere and just fetch from there if you need\nsome central location for it. You could even add alternate entries\npointing there to the libraries object store inside the supermodule.\n\n-- \nMartin Waitz\n"},{"id":"38084","messageId":"7vy7ljtwir.fsf@assigned-by-dhcp.cox.net","threadId":"7390","inReplyTo":"20070326231637.GJ22773@admingilde.org","subject":"Re: Submodule object store","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-03-26T23:28:28Z","receivedAt":"2007-03-26T23:28:28Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Waitz writes:\n\n> To make purge and fsck work we have the hard requirement that it\n> must be possible to list all objects which belong to one submodule.\n\nI understand you would want to separate the ref namespace, but I\nstill do not see why you would want to have a separate object\nstore, laid out in a funny way. Unless you are thinking about\nusing rsync to transfer object store, that is.\n\n> And moving the submodule object store into the .git directory of the\n> supermodule has several reasons:...\n\nOh, I am not disputing that. If you want to use a single object\nstore for both levels, I think that is a sensible thing to do.\nI just do not see the point of segregating submodule objects and\nsupermodule objects in the single object store you attach to the\nsupermodule. After all, any traversal starts from refs, so by\nsegregating the ref namespace you would limit the extent of the\ntraversal, no?\n"},{"id":"38086","messageId":"20070326233603.GL22773@admingilde.org","threadId":"7390","inReplyTo":"7vy7ljtwir.fsf@assigned-by-dhcp.cox.net","subject":"Re: Submodule object store","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-26T23:36:03Z","receivedAt":"2007-03-26T23:36:03Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"On Mon, Mar 26, 2007 at 04:28:28PM -0700, Junio C Hamano wrote:\n> Martin Waitz writes:\n> \n> > To make purge and fsck work we have the hard requirement that it\n> > must be possible to list all objects which belong to one submodule.\n> \n> I understand you would want to separate the ref namespace, but I\n> still do not see why you would want to have a separate object\n> store, laid out in a funny way. Unless you are thinking about\n> using rsync to transfer object store, that is.\n\nI want to be able to list all objects which are not reachable in the\nobject store, without traversing all submodules at the same time.\nThe only way I can think of to achieve this is to have one separate\nobject store per submodule and then do the traversal per submodule.\n\n-- \nMartin Waitz\n"},{"id":"38093","messageId":"Pine.LNX.4.63.0703261530030.14387@qynat.qvtvafvgr.pbz","threadId":"7390","inReplyTo":"20070326235527.GM22773@admingilde.org","subject":"Re: Submodule object store","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2007-03-26T23:40:15Z","receivedAt":"2007-03-26T23:40:15Z","isPatch":false,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Tue, 27 Mar 2007, Martin Waitz wrote:\n\n> hoi :)\n>\n> On Mon, Mar 26, 2007 at 03:20:34PM -0800, David Lang wrote:\n>>> I want to be able to list all objects which are not reachable in the\n>>> object store, without traversing all submodules at the same time.\n>>> The only way I can think of to achieve this is to have one separate\n>>> object store per submodule and then do the traversal per submodule.\n>>\n>> why do you want to optimize for the relativly rare fsck function rather\n>> then the more common read functions (which would benifit from shareing\n>> object that are identical between projects)?\n>\n> Because I don't know how to make it _possible_ for large repositories\n> otherwise. Consider a Linux-distribution which handles each package\n> as one submodule.\n>\n> I don't think that it's too much balanced towards fsck.\n> The separated object store also helps reduce the memory requirement for\n> large pushs/pulls.\n> Sharing objects can be achieved by alternates if you want.\n\nalternates require explicitly setting up the sharing.\n\nuseing the same object store makes this work automaticaly (think of all the \ncopies of COPYING that would end up being the same as a trivial example)\n\n> If someone comes up with a nice way to handle everything in one big\n> object store I would happily use that! :-)\n\nwhat exactly are the problems with one big object store?\n\nones that I can think of:\n\n1. when you are doing a fsck you need to walk all the trees and find out the \nlist of objects that you know about.\n\n done as a tree of binary values you can hold a LOT in memory before running \ninto swap.\n\n if it's enough larger then available ram then an option for fsck to use trees \non disk is an option.\n\n2. when creating a pack you will eventually run into pack-size limits with too \nmany objects\n\n teach the pack creators to make packs that are subsets rather then everything \n(I belive that most of the smarts are there, it just needs the upper control \nlogic to tell the existing things what to include)\n\n3. when doing a pull it takes longer to figure out what to pull to get a \nduplicate of _everything_\n\n add a way to do a 'pull projectlist' that would look at what objects are \nneeded by the project(s) requested and only try to pack up those objects\n\nwhat else is there that I'm not thinking of? so far these look like long-term \nproblems as opposed to short-term problems, and all of them have fairly simple \nfixes that can be implemented as they become an issue.\n\nDavid Lang\n"},{"id":"38089","messageId":"20070326235527.GM22773@admingilde.org","threadId":"7390","inReplyTo":"Pine.LNX.4.63.0703261515330.14387@qynat.qvtvafvgr.pbz","subject":"Re: Submodule object store","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-26T23:55:27Z","receivedAt":"2007-03-26T23:55:27Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Mon, Mar 26, 2007 at 03:20:34PM -0800, David Lang wrote:\n> >I want to be able to list all objects which are not reachable in the\n> >object store, without traversing all submodules at the same time.\n> >The only way I can think of to achieve this is to have one separate\n> >object store per submodule and then do the traversal per submodule.\n> \n> why do you want to optimize for the relativly rare fsck function rather \n> then the more common read functions (which would benifit from shareing \n> object that are identical between projects)?\n\nBecause I don't know how to make it _possible_ for large repositories\notherwise. Consider a Linux-distribution which handles each package\nas one submodule.\n\nI don't think that it's too much balanced towards fsck.\nThe separated object store also helps reduce the memory requirement for\nlarge pushs/pulls.\nSharing objects can be achieved by alternates if you want.\nIf someone comes up with a nice way to handle everything in one big\nobject store I would happily use that! :-)\n\n-- \nMartin Waitz\n"},{"id":"38096","messageId":"7vps6vttpq.fsf@assigned-by-dhcp.cox.net","threadId":"7390","inReplyTo":"20070326235527.GM22773@admingilde.org","subject":"Re: Submodule object store","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-03-27T00:29:05Z","receivedAt":"2007-03-27T00:29:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Waitz writes:\n\n> The separated object store also helps reduce the memory requirement for\n> large pushs/pulls.\n\nThat's a total bull. The size of push/pull only depends on how\nyou separate set of refs (which affects the traversal hence\naffects the set of objects to be exchanged).\n"},{"id":"38098","messageId":"200703270248.31631.jnareb@gmail.com","threadId":"7390","inReplyTo":"vpq3b3ruaqp.fsf@olympe.imag.fr","subject":"Re: .gitlink for Summer of Code","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-03-27T00:48:31Z","receivedAt":"2007-03-27T00:48:31Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Matthieu Moy wrote:\n> Jakub Narebski writes:\n> \n>> Matthieu Moy wrote:\n>>\n>>> Stupid question: why .gitlink, and not .git/link or so? This file is\n>>> not versionned, I don't think it should be in the working tree.\n>>\n>> It would be special-cased, as is .git directory not versioned, the\n>> same way .gitlink would be not versioned (not like .gitignore).\n> \n> That's how I understand it, but why 2 special cases (.git and\n> .gitlink) when you need one?\n\nWell, in the first proposal we didn't though about idea of having\nlightweight checkout .git/ directory (with .git/link, or core.link \nin .git/config) to have only parts of it, perhaps overriding main\nfiles.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"38103","messageId":"20070327034059.GJ4489@pasky.or.cz","threadId":"7390","inReplyTo":"7v3b3t3uws.fsf@assigned-by-dhcp.cox.net","subject":"Re: .gitlink for Summer of Code","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2007-03-27T03:40:59Z","receivedAt":"2007-03-27T03:40:59Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Sun, Mar 25, 2007 at 10:55:31PM CEST, Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > Actually, I'd almost say put it into .git/config, e.g.:\n> >\n> > \tmkdir .git\n> > \tcat >.git/config < > \t[core]\n> > \t\trepositoryversion = 0\n> > \t\tfilemode = true\n> > \t\tlink = /path/to/source\n> > \tEOF\n> >\n> > as then the index and HEAD file can both be stored in .git, just\n> > like with the non-gitlink case.\n> \n> This is not usable at least for me as it does not allow sharing\n> the .git/config file across checkouts.\n\nCan't you take linked .git/config and override it with stuff from local\n.git/config in that case? Don't replace, supraposition.\n\nTake somewhat contrived example of having checkout on a FAT partition\nlinking to a repository on a sane filesystem (are you permanently short\non disk space on your /home partition too, except for about two months\nright after you double your disk capacity? :). You might want to disable\ncore.fileMode there. Maybe this will not ever happen in real world and\nwe might not care. Maybe not...\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nEver try. Ever fail. No matter. // Try again. Fail again. Fail better.\n\t\t-- Samuel Beckett\n"},{"id":"38145","messageId":"20070327112549.GA12178@informatik.uni-freiburg.de","threadId":"7390","inReplyTo":"20070326233603.GL22773@admingilde.org","subject":"Re: Submodule object store","fromName":"Uwe Kleine-König","fromEmail":"ukleinek@informatik.uni-freiburg.de","sentAt":"2007-03-27T11:25:49Z","receivedAt":"2007-03-27T11:25:49Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello,\n\nMartin Waitz wrote:\n> On Mon, Mar 26, 2007 at 04:28:28PM -0700, Junio C Hamano wrote:\n> > Martin Waitz writes:\n> > \n> > > To make purge and fsck work we have the hard requirement that it\n> > > must be possible to list all objects which belong to one submodule.\n> > \n> > I understand you would want to separate the ref namespace, but I\n> > still do not see why you would want to have a separate object\n> > store, laid out in a funny way. Unless you are thinking about\n> > using rsync to transfer object store, that is.\n> \n> I want to be able to list all objects which are not reachable in the\n> object store, without traversing all submodules at the same time.\n> The only way I can think of to achieve this is to have one separate\n> object store per submodule and then do the traversal per submodule.\nI might have understood something wrongly, but to list objects that are\nnot reachable you need to traverse all trees anyhow, don't you. \n\nThen how big is the difference between a directory and an submodule?\nI'd expect it's not so big if the submodules included in different\nrevisions of the supermodule share most of their history. Of course you\nneed to exploit that. Thinking again that might be the problem?\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\n5 out of 4 people have trouble with fractions.\n"},{"id":"38148","messageId":"20070327115029.GC12178@informatik.uni-freiburg.de","threadId":"7390","inReplyTo":"20070327112549.GA12178@informatik.uni-freiburg.de","subject":"Re: Submodule object store","fromName":"Uwe Kleine-König","fromEmail":"ukleinek@informatik.uni-freiburg.de","sentAt":"2007-03-27T11:50:29Z","receivedAt":"2007-03-27T11:50:29Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hallo again,\n\nUwe Kleine-König wrote:\n> > I want to be able to list all objects which are not reachable in the\n> > object store, without traversing all submodules at the same time.\n> > The only way I can think of to achieve this is to have one separate\n> > object store per submodule and then do the traversal per submodule.\n> I might have understood something wrongly, but to list objects that are\n> not reachable you need to traverse all trees anyhow, don't you. \n> \n> Then how big is the difference between a directory and an submodule?\n> I'd expect it's not so big if the submodules included in different\n> revisions of the supermodule share most of their history. Of course you\n> need to exploit that. Thinking again that might be the problem?\nI didn't look at the code, but an other issue might be:\n\nIf you separate the odbs e.g by the pathname of the subproject, what\nhappens if I choose to move the linux kernel in my embedded Linux\nproject from /linux to /kernel/linux?\n\nOr maybe worse: If I currently track the Kernel in a tree (because of\ngit lacking submodule support) and switch to submodule. Then\nlinux/Makefile has to exist in both the supermodule's and the\nsubmodule's odb.\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=30+hours+and+4+days+in+seconds\n"},{"id":"38152","messageId":"20070327142840.GN22773@admingilde.org","threadId":"7390","inReplyTo":"7vps6vttpq.fsf@assigned-by-dhcp.cox.net","subject":"Re: Submodule object store","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-27T14:28:40Z","receivedAt":"2007-03-27T14:28:40Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Mon, Mar 26, 2007 at 05:29:05PM -0700, Junio C Hamano wrote:\n> Martin Waitz writes:\n> > The separated object store also helps reduce the memory requirement for\n> > large pushs/pulls.\n> \n> That's a total bull. The size of push/pull only depends on how\n> you separate set of refs (which affects the traversal hence\n> affects the set of objects to be exchanged).\n\nof course you are right.\n\n-- \nMartin Waitz\n"},{"id":"38153","messageId":"20070327152507.GO22773@admingilde.org","threadId":"7390","inReplyTo":"Pine.LNX.4.63.0703261530030.14387@qynat.qvtvafvgr.pbz","subject":"Re: Submodule object store","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-27T15:25:07Z","receivedAt":"2007-03-27T15:25:07Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nIts really funny that when I proposed one big object database everybody\nwanted it separated and now that I propose a separate database everybody\nwants it as one combined database.\nI read this as a sign that people really try to think critically about\nthe design, which is a good thing and will hopefully lead to a good\nand stable submodule implementation.\n\nOn Mon, Mar 26, 2007 at 03:40:15PM -0800, David Lang wrote:\n> useing the same object store makes this work automaticaly (think of all the \n> copies of COPYING that would end up being the same as a trivial example)\n\nYes, but I guess not much more than COPYING, INSTALL, some trivial\nMakefiles and empty files will be shared between subprojects.\nExcept when you have the same subproject in your tree multiple times,\nof course.\n\nYet this sharing is exactly why I started to do it that way, until Linus\nstopped me.\n\n> >If someone comes up with a nice way to handle everything in one big\n> >object store I would happily use that! :-)\n> \n> what exactly are the problems with one big object store?\n\nI think we really have to discuss this separation on several layers:\ntraversal, pack-files, and object database.\n\nFor the traversal the point of separating it into a per-module traversal\nis that only one module has to be loaded into RAM at a time.\nThis effects all operations which do a (potentially) recursive traversal:\npush, pull, fsck, prune, repack.\nHowever a separated traversal will no longer be garanteed to only list\nan object once, so this has to be handled in some way.\n\nPack files should have better access patterns if they are per-module.\nMost of the time you are only interested in one individual module and\nlocality is important here.\n\nSeparating the entire object database is a way to improve unreachability\nanalysis, as it now can be done per module.\nThe other two separations are easier to implement with a separated\nobject database, but that's not too strong an argument.\n\n\nSo if we can come up with a nice way to do unreachability analysis we\ncan indeed go on with the shared object database and tackle the\nremaining scalability issues as they arise. Those could then be added\nlater without changing the on-disk format.\n\n> ones that I can think of:\n> \n> 1. when you are doing a fsck you need to walk all the trees and find out \n> the list of objects that you know about.\n> \n> done as a tree of binary values you can hold a LOT in memory before \n> running into swap.\n\nCould you explain the algorithm you are thinking about in more detail?\n\n> if it's enough larger then available ram then an option for fsck to use \n> trees on disk is an option.\n\nThis could simplify some things.\nThere could be an on-disk index of all known objects, so that the sha1\nsums do not have to loaded into RAM all at once.\n\n-- \nMartin Waitz\n"},{"id":"38154","messageId":"20070327154614.GP22773@admingilde.org","threadId":"7390","inReplyTo":"20070327112549.GA12178@informatik.uni-freiburg.de","subject":"Re: Submodule object store","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-27T15:46:14Z","receivedAt":"2007-03-27T15:46:14Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Tue, Mar 27, 2007 at 01:25:49PM +0200, Uwe Kleine-König wrote:\n> Martin Waitz wrote:\n> > I want to be able to list all objects which are not reachable in the\n> > object store, without traversing all submodules at the same time.\n> > The only way I can think of to achieve this is to have one separate\n> > object store per submodule and then do the traversal per submodule.\n> I might have understood something wrongly, but to list objects that are\n> not reachable you need to traverse all trees anyhow, don't you. \n> \n> Then how big is the difference between a directory and an submodule?\n> I'd expect it's not so big if the submodules included in different\n> revisions of the supermodule share most of their history. Of course you\n> need to exploit that. Thinking again that might be the problem?\n\nthere is no big difference, only that the submodule may be huge by\nitself.\n\nAnd yes, every object has to be traversed anyhow, but not neccessarily by\nthe same process and at the same time.\n\n-- \nMartin Waitz\n"},{"id":"38155","messageId":"20070327155306.GQ22773@admingilde.org","threadId":"7390","inReplyTo":"20070327115029.GC12178@informatik.uni-freiburg.de","subject":"Re: Submodule object store","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-27T15:53:06Z","receivedAt":"2007-03-27T15:53:06Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Tue, Mar 27, 2007 at 01:50:29PM +0200, Uwe Kleine-König wrote:\n> If you separate the odbs e.g by the pathname of the subproject, what\n> happens if I choose to move the linux kernel in my embedded Linux\n> project from /linux to /kernel/linux?\n\nThen a new separate object database would have to be created.\nThis is the part I really don't like about separate object databases,\nbut perhaps some persistent alternates information could help here.\n\nFor any other way to separate the odb (project id, whatever), we\ncan't get a list of references into it by a path-limited traversal\nin the parent. Thus separate odbs which are not bound to a special\nlocation have some serious downsides.\n\n> Or maybe worse: If I currently track the Kernel in a tree (because of\n> git lacking submodule support) and switch to submodule. Then\n> linux/Makefile has to exist in both the supermodule's and the\n> submodule's odb.\n\nSorry, I don't understand you here.\n\n-- \nMartin Waitz\n"},{"id":"38161","messageId":"20070327164428.GR22773@admingilde.org","threadId":"7390","inReplyTo":"200703271856.09492.Josef.Weidendorfer@gmx.de","subject":"Re: Submodule object store","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-27T16:44:28Z","receivedAt":"2007-03-27T16:44:28Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Tue, Mar 27, 2007 at 06:56:09PM +0200, Josef Weidendorfer wrote:\n> On Tuesday 27 March 2007, Martin Waitz wrote:\n> > For any other way to separate the odb (project id, whatever), we\n> > can't get a list of references into it by a path-limited traversal\n> > in the parent. Thus separate odbs which are not bound to a special\n> > location have some serious downsides.\n> \n> For path-limited traversal, you still need to know all the paths\n> with super/subproject boundaries somewhere in the history.\n> Do you store this information somewhere?\n> If so, how is this different from directly storing the boundaries\n> (aside from size)?\n\nYou only need the path-limited traversal when you want to look at one\nindividual submodule. And then you start the operation in this\nsubmodule, so you know the path.\n\n-- \nMartin Waitz\n"},{"id":"38166","messageId":"Pine.LNX.4.63.0703270735550.15345@qynat.qvtvafvgr.pbz","threadId":"7390","inReplyTo":"20070327152507.GO22773@admingilde.org","subject":"Re: Submodule object store","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2007-03-27T16:53:37Z","receivedAt":"2007-03-27T16:53:37Z","isPatch":false,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Tue, 27 Mar 2007, Martin Waitz wrote:\n\n> On Mon, Mar 26, 2007 at 03:40:15PM -0800, David Lang wrote:\n>> useing the same object store makes this work automaticaly (think of all the\n>> copies of COPYING that would end up being the same as a trivial example)\n>\n> Yes, but I guess not much more than COPYING, INSTALL, some trivial\n> Makefiles and empty files will be shared between subprojects.\n> Except when you have the same subproject in your tree multiple times,\n> of course.\n\nalthough, if you end up packing multiple projects togeather you may end up \nfinding more things that diff well against each other (although it will slow \ndown the packing with more objects.\n\n> Yet this sharing is exactly why I started to do it that way, until Linus\n> stopped me.\n\nI missed that one.\n\n>>> If someone comes up with a nice way to handle everything in one big\n>>> object store I would happily use that! :-)\n>>\n>> what exactly are the problems with one big object store?\n>\n> I think we really have to discuss this separation on several layers:\n> traversal, pack-files, and object database.\n>\n> For the traversal the point of separating it into a per-module traversal\n> is that only one module has to be loaded into RAM at a time.\n> This effects all operations which do a (potentially) recursive traversal:\n> push, pull, fsck, prune, repack.\n> However a separated traversal will no longer be garanteed to only list\n> an object once, so this has to be handled in some way.\n\nan object can already appear more then once in pack files.\n\n> Pack files should have better access patterns if they are per-module.\n> Most of the time you are only interested in one individual module and\n> locality is important here.\n>\n> Separating the entire object database is a way to improve unreachability\n> analysis, as it now can be done per module.\n> The other two separations are easier to implement with a separated\n> object database, but that's not too strong an argument.\n\nif modules are really as seperate as you make them out to be then what you want \nisn't multiple modules inside one overall project (top level .git) you want \nmultiple projects and a way to link them togeather.\n\n>\n> So if we can come up with a nice way to do unreachability analysis we\n> can indeed go on with the shared object database and tackle the\n> remaining scalability issues as they arise. Those could then be added\n> later without changing the on-disk format.\n>\n>> ones that I can think of:\n>>\n>> 1. when you are doing a fsck you need to walk all the trees and find out\n>> the list of objects that you know about.\n>>\n>> done as a tree of binary values you can hold a LOT in memory before\n>> running into swap.\n>\n> Could you explain the algorithm you are thinking about in more detail?\n\nas I understand it the need is to efficiantly create a list of all the objects \nthat are reachable (so that we can then go through the objects and remove them \nif they aren't on the list).\n\nyou need these sorted to make it easy to find if something is in the list, and \nwith millions of entries you don't it to be a flat list (inserting new values \nbecomes very inefficiant) so the classic answer is to do a tree structure. you \ncan either do a tree with the object ID's in all the nodes, or you can do one \nwhere only the leaf nodes hold the object ID's and the other nodes just hold \npointers (which would then allow you to spill the leaf nodes to disk more \nefficiantly as they wouldn't need to be accessed when inserting unless the node \nitself needed to be changed. looking them up is being done more or less in alpha \norder for loose objects (and could be made to be so for objects in packs) so any \nfile I/O for lookups would be close to sequential\n\nthis sort of memory useage wouldn't be acceptable for something that happens \nfrequently, but a fsck/prune is relativly infrequent and can be run off-hours.\n\n>> if it's enough larger then available ram then an option for fsck to use\n>> trees on disk is an option.\n>\n> This could simplify some things.\n> There could be an on-disk index of all known objects, so that the sha1\n> sums do not have to loaded into RAM all at once.\n\nyou wouldn't want to trust this for a fsck/prune\n\nDavid Lang\n"},{"id":"38160","messageId":"200703271856.09492.Josef.Weidendorfer@gmx.de","threadId":"7390","inReplyTo":"20070327155306.GQ22773@admingilde.org","subject":"Re: Submodule object store","fromName":"Josef Weidendorfer","fromEmail":"josef.weidendorfer@gmx.de","sentAt":"2007-03-27T16:56:09Z","receivedAt":"2007-03-27T16:56:09Z","isPatch":false,"sender":{"key":"josef.weidendorfer@gmx.de","avatar":null},"body":"On Tuesday 27 March 2007, Martin Waitz wrote:\n> For any other way to separate the odb (project id, whatever), we\n> can't get a list of references into it by a path-limited traversal\n> in the parent. Thus separate odbs which are not bound to a special\n> location have some serious downsides.\n\nFor path-limited traversal, you still need to know all the paths\nwith super/subproject boundaries somewhere in the history.\nDo you store this information somewhere?\nIf so, how is this different from directly storing the boundaries\n(aside from size)?\n\nJosef\n"},{"id":"38163","messageId":"vpqzm5ysjuj.fsf@olympe.imag.fr","threadId":"7390","inReplyTo":"200703270114.49365.Josef.Weidendorfer@gmx.de","subject":"Re: .gitlink for Summer of Code","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2007-03-27T16:59:48Z","receivedAt":"2007-03-27T16:59:48Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Josef Weidendorfer writes:\n\n>> * You have to manage a name for each lightweight checkout. What would\n>> be such name? User-provided? uuidgen-like?\n>\n> Such a name is interesting as identifier for submodules.\n> It would be the relative path of the submodule from the supermodule base;\n> or user supplied.\n>\n> Lightweight checkouts and submodules have different requirements;\n> yet, the lightweight checkouts should be so flexible to be\n> able to be used for submodules checkouts.\n\nSure, but AAUI, submodules would use light checkouts, but light\ncheckouts are interesting by themselves, so the naming thing should be\nin submodules support.\n\n-- \nMatthieu\n"},{"id":"38168","messageId":"Pine.LNX.4.63.0703270858080.15345@qynat.qvtvafvgr.pbz","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703270952020.6730@woody.linux-foundation.org","subject":"Re: .gitlink for Summer of Code","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2007-03-27T17:00:08Z","receivedAt":"2007-03-27T17:00:08Z","isPatch":false,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Tue, 27 Mar 2007, Linus Torvalds wrote:\n\n> - walking the *global* object list is simply not possible. You need to\n> fsck every single subtree individually, and fsck the superproject on\n> its own, *without* recursing into the subprojects. And you need to be\n> able to clone the superproject and only one or two subprojects, and\n> never see it as one \"atomic\" big repository.\n\nyou can do a fsck to make sure that all needed objects are available on each \nproject individually, but a prune/gc _does_ need to go through the global object \nlist to find out what objects aren't needed anymore (otherwise, how do you know \nif the object isn't needed by another thing sharing the same object store?)\n\nDavid Lang\n"},{"id":"38165","messageId":"Pine.LNX.4.64.0703270952020.6730@woody.linux-foundation.org","threadId":"7390","inReplyTo":"200703270117.59205.Josef.Weidendorfer@gmx.de","subject":"Re: .gitlink for Summer of Code","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-27T17:04:53Z","receivedAt":"2007-03-27T17:04:53Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 27 Mar 2007, Josef Weidendorfer wrote:\n\n> On Tuesday 27 March 2007, Junio C Hamano wrote:\n> > Martin Waitz writes:\n> > \n> > > For submodules I currently use /.git/objects/module//\n> > > to store the objects belonging to the submodule.\n> > \n> > I was not following the gitlink discussion closely, but what is\n> > the motivation behind this separation of the object store?\n> \n> The separation issue is about scalability of submodules, and not\n> directly about gitlink.\n\nNOTE! It's fine to share the *object*store* for a supermodule setup.\n\nThe scalability concerns are not about the number of objects, but about \nthe operations that work on them, and specifically *traverse* the objects.\n\nSo while it's fine to share the same GIT_OBJECT_DIR for all the \nsubmodules, it's *not* ok if \"git clone\" on a supermodule will consider \nthings to be one single repository, and clone it as one huge thing, \ngenerating (and having to look up!) a ten-million object pack for a \nhundred smaller projects. THAT won't scale.\n\nBasically, a \"git-rev-list --objects HEAD\" in the super-module should only \nlist the objects in the supermodule itself, not in all the submodules. And \nthat implies that cloning a supermodule is not about cloning a single big \nrepository: it would be a matter of:\n\n - first cloning first the supermodule itself (which is often fairly \n small: just a top-level directory, with some top-level Makefiles and a \n number of directories that are submodules)\n\n - then parsing some supermodule data structure, and cloning each \n submodule individually.\n\nSimilarly for \"fetch\" (and merging too, of course - it ends up having to \nmerge each sub-project separately). \n\nThink of it this way: if you think people find it a bit annoying that you \ncurrently have to get all the history when you do clone (and why people \nhave worked on \"shallow clones\" in git), imagine just *how* frustrating it \nis if you have to get all five-hundred subprojects when you only want to \nwork on one small one!\n\nThink of something like a huge *BSD \"world\" tree, where the supermodule \ncontains *everything*. Do you really _really_ expect that every single \ndeveloper wants to clone it all? I have no idea how much that is, but I \ncan well imagine that it's several thousand subprojects, some of which are \nquite big in their own right. \n\nAlso, imagine the server side.. Anybody who thinks that the server wants \nto (or is even *able* to) do things like a fsck on the totality, or keep \nevery single object in memory, is in for a nasty surprise..\n\nSo I think that:\n\n - sharing object directories should not be a requirement, but it should \n certainly be *possible*. Quite often you might want to do it, although \n for really big superprojects it might well make sense to have \n individual object stores too.\n\n - walking the *global* object list is simply not possible. You need to \n fsck every single subtree individually, and fsck the superproject on \n its own, *without* recursing into the subprojects. And you need to be \n able to clone the superproject and only one or two subprojects, and \n never see it as one \"atomic\" big repository.\n\nI really think people should think about the *BSD kind of \"world\" setup. \nYou absolutely do _not_ want supermodules to be indivisible \"everything or \nnothign\" kind of things. You want submodules to be very much separate \nrepostories, although you *can* of course share the object store if you \nwant to (the same way git can do it between any number of totally \nunrelated repositories!)\n\n\t\tLinus\n"},{"id":"38167","messageId":"20070327172216.GA24200@informatik.uni-freiburg.de","threadId":"7390","inReplyTo":"20070327155306.GQ22773@admingilde.org","subject":"Re: Submodule object store","fromName":"Uwe Kleine-König","fromEmail":"ukleinek@informatik.uni-freiburg.de","sentAt":"2007-03-27T17:22:17Z","receivedAt":"2007-03-27T17:22:17Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hallali,\n\nMartin Waitz wrote:\n> On Tue, Mar 27, 2007 at 01:50:29PM +0200, Uwe Kleine-König wrote:\n> > If you separate the odbs e.g by the pathname of the subproject, what\n> > happens if I choose to move the linux kernel in my embedded Linux\n> > project from /linux to /kernel/linux?\n> \n> Then a new separate object database would have to be created.\n> This is the part I really don't like about separate object databases,\n> but perhaps some persistent alternates information could help here.\n> \n> For any other way to separate the odb (project id, whatever), we\n> can't get a list of references into it by a path-limited traversal\n> in the parent. Thus separate odbs which are not bound to a special\n> location have some serious downsides.\n(CVS comes to mind ...)\nThis currently convinces me that a separate odb is wrong.\n\n> > Or maybe worse: If I currently track the Kernel in a tree (because of\n> > git lacking submodule support) and switch to submodule. Then\n> > linux/Makefile has to exist in both the supermodule's and the\n> > submodule's odb.\n> \n> Sorry, I don't understand you here.\nAssume I have\n\n\tembeddedproject$ git ls-tree HEAD | grep linux\n\t040000 tree 0123456789abcdef... linux-2.6\n\nand then I commit on top of that, s.t. I get:\n\n\tembeddedproject$ git ls-tree HEAD | grep linux\n\t040000 commit 0123456789abcde0... linux-2.6\n\n(or how ever you save submodules). Then you might have to duplicate the\nobjects of linux-2.6, because they are part of both histories.\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\nprimes where sieve (p:xs) = [ x | x<-xs, x `rem` p /= 0 ]; \\\nprimes = map head (iterate sieve [2..])\n"},{"id":"38169","messageId":"20070327173512.GS22773@admingilde.org","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703270952020.6730@woody.linux-foundation.org","subject":"Re: .gitlink for Summer of Code","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-27T17:35:12Z","receivedAt":"2007-03-27T17:35:12Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Tue, Mar 27, 2007 at 10:04:53AM -0700, Linus Torvalds wrote:\n> - walking the *global* object list is simply not possible. You need to \n> fsck every single subtree individually, and fsck the superproject on \n> its own, *without* recursing into the subprojects. And you need to be \n> able to clone the superproject and only one or two subprojects, and \n> never see it as one \"atomic\" big repository.\n\nand just skip the unreachability check of fsck?\nWith this limitation a shared object store would be possible.\n\n> I really think people should think about the *BSD kind of \"world\" setup. \n> You absolutely do _not_ want supermodules to be indivisible \"everything or \n> nothign\" kind of things. You want submodules to be very much separate \n> repostories, although you *can* of course share the object store if you \n> want to (the same way git can do it between any number of totally \n> unrelated repositories!)\n\nYou already convinced me that the \"world\" use-case is worthwhile.\nAs to *can* be shared: I'd really like to have some default location for\nall objects so that it can be found automatically when you later decide\nto checkout a new submodule which has not yet been fetched.\n\n-- \nMartin Waitz\n"},{"id":"38170","messageId":"Pine.LNX.4.64.0703271338210.6485@iabervon.org","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703270952020.6730@woody.linux-foundation.org","subject":"Re: .gitlink for Summer of Code","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2007-03-27T18:09:38Z","receivedAt":"2007-03-27T18:09:38Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 27 Mar 2007, Linus Torvalds wrote:\n\n> Think of it this way: if you think people find it a bit annoying that you \n> currently have to get all the history when you do clone (and why people \n> have worked on \"shallow clones\" in git), imagine just *how* frustrating it \n> is if you have to get all five-hundred subprojects when you only want to \n> work on one small one!\n\nIs it fair to say that subproject support means that there's a use case \nwhere everybody will need shallow clones? And that it points out natural \ntriggers for shallowness?\n\nI don't see that the \"shallow clone\" mechanism is special for subprojects \n(and I don't think that a solution that depends on subprojects being what \ncauses it is a good idea), but clearly it makes sense to support: (1) no \nclone of submodules, (2) shallow clone of submodules, and (3) full clone \nof submodules.\n\nSomebody working on gcc for *BSD would presumably want to get all of gcc \nand a shallow clone of the other 1000 submodules, right? Or they'd just \nclone the submodule and ignore the superproject. At least, they'd need \nshallow clones of a bunch of the submodules, because it's not interesting \nto have the superproject otherwise.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"38171","messageId":"Pine.LNX.4.64.0703271114000.6730@woody.linux-foundation.org","threadId":"7390","inReplyTo":"Pine.LNX.4.63.0703270858080.15345@qynat.qvtvafvgr.pbz","subject":"Re: .gitlink for Summer of Code","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-27T18:15:42Z","receivedAt":"2007-03-27T18:15:42Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 27 Mar 2007, David Lang wrote:\n\n> On Tue, 27 Mar 2007, Linus Torvalds wrote:\n> \n> > - walking the *global* object list is simply not possible. You need to\n> > fsck every single subtree individually, and fsck the superproject on\n> > its own, *without* recursing into the subprojects. And you need to be\n> > able to clone the superproject and only one or two subprojects, and\n> > never see it as one \"atomic\" big repository.\n> \n> you can do a fsck to make sure that all needed objects are available on each\n> project individually, but a prune/gc _does_ need to go through the global\n\nNo it doesn't.\n\nIf you do per-project object stores, there's no need. There simply isn't \nanything to prune \"globally\". Everything is local.\n\nNow, if you share the object directory, you can't prune. But that's true \neven without any \"subproject\" support, and has nothing to do with sub- or \nsuper-projects.\n\t\n\t\tLinus\n"},{"id":"38172","messageId":"Pine.LNX.4.64.0703271115580.6730@woody.linux-foundation.org","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703271338210.6485@iabervon.org","subject":"Re: .gitlink for Summer of Code","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-27T18:19:01Z","receivedAt":"2007-03-27T18:19:01Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 27 Mar 2007, Daniel Barkalow wrote:\n> \n> Is it fair to say that subproject support means that there's a use case \n> where everybody will need shallow clones? And that it points out natural \n> triggers for shallowness?\n\nNo.\n\nI personally don't believe in shallow clones. And I *certainly* don't \nbelieve that it has anything to do with subprojects. So people may want \nshallow clones, but it's at least independent of the issue of submodules.\n\nWith subprojects, it's not that you don't want the history. It's just that \nyou don't want the history for *all* projects. Most people care about a \nvery small subset.\n\n(The exception, of course, is when the superproject simply isn't that big, \nand only has a couple of subprojects. In git, for example, the xdiff stuff \ncould be a subproject if you wanted to do it that way. But then, the \nsubproject isn't a size issue, it's purely an organizational thing, and \nthere is no argument for/against shallowness there either).\n\n\t\t\tLinus\n"},{"id":"38173","messageId":"4609643C.9050803@midwinter.com","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703271338210.6485@iabervon.org","subject":"Re: .gitlink for Summer of Code","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-03-27T18:36:44Z","receivedAt":"2007-03-27T18:36:44Z","isPatch":false,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"Daniel Barkalow wrote:\n> Somebody working on gcc for *BSD would presumably want to get all of gcc \n> and a shallow clone of the other 1000 submodules, right? Or they'd just \n> clone the submodule and ignore the superproject. At least, they'd need \n> shallow clones of a bunch of the submodules, because it's not interesting \n> to have the superproject otherwise.\n> \n\nThe obvious use case for \"I want the superproject and just one \nsubmodule\" is when the superproject has build tools, header files, or \nother pieces of data that are shared by some/all of the submodules. \nMaybe not the case in BSD per se, but having a top-level file full of \nsettings, paths to tools, etc. that gets included by the individual \nMakefiles in subdirectories isn't all that uncommon in complex \nmulti-part projects.\n\n-Steve\n"},{"id":"38174","messageId":"Pine.LNX.4.64.0703271124590.6730@woody.linux-foundation.org","threadId":"7390","inReplyTo":"20070327172216.GA24200@informatik.uni-freiburg.de","subject":"Re: Submodule object store","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-27T18:41:11Z","receivedAt":"2007-03-27T18:41:11Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 27 Mar 2007, Uwe Kleine-König wrote:\n> \n> \tembeddedproject$ git ls-tree HEAD | grep linux\n> \t040000 commit 0123456789abcde0... linux-2.6\n> \n> (or how ever you save submodules). Then you might have to duplicate the\n> objects of linux-2.6, because they are part of both histories.\n\nNo they are not. Unless you do it wrong.\n\nThe *only* object that is part of the superproject would be the tree that \n*contains* that entry itself.\n\nWe should *never* automatically follow such an entry down, *exactly* \nbecause that doesn't scale. So to actually follow that entry for something \nlike a recursive, you'd literally \"cd into linux, and start 'git diff' \nfrom commit 0123456..\"\n\nIn other words, the subproject would be its own project, and the \nsuperproject never sees it as \"part of itself\". I really think, for \nexample, that the \"git diff\" family of programs (diff-index, diff-tree, \ndiff-files) and things like \"git ls-tree\" should literally:\n\n - have a mode where they don't even recurse into subprojects, and I \n personally think that it could/should be the default!\n\n - when they recurse, they should literally (at least to begin with) do \n that kind of \"fork() ; if (child) { chdir(subproject); execve(myself) }\" \n\nThe latter is really to make sure that *even*by*mistake* we don't screw \nthings up and tie the sub/superproject together too tightly. \n\nI'm serious. I really think that the first version (which ends up being \nthe one that sets semantics) should be very careful here, so that \nsubprojects never get mixed up with the superproject.\n\nAnd I'm also serious about the \"don't recurse into subproject by default \nat all\". If I'm at the superproject, and I maintain the superproject, I \nthink the state of the subprojects themselves are a totally separate \nissue. It's quite a valid thing to do to maintain the build \ninfrastructure, and if I'm the maintainer of that, and I do \"git diff\", I \nsure as hell don't want to wait for git to do \"git diff\" on the \nsubprojects when there are 5000 of them!\n\nSure, \"git diff\" is fast (on the kernel, it takes me 0.069s on a clean \ntree), but \n\n - multiply that 0.069s by 5000 and it's not so fast any more\n\n - when you have a thousand subprojects, it's quite possible (even likely) \n that all your directories won't fit in the cache any more, and suddenly \n even a single \"git diff\" takes several seconds.\n\nReally! Try this on the Linux tree (that \"drop_caches\" thing needs root \nprivileges):\n\n\techo 3 > /proc/sys/vm/drop_caches\n\tgit diff\n\nand see it take something like 5 seconds. Now, imagine that you have a \nhundred subprojects, and they're big enough that the caches are *never* \nwarm.\n\nPeople sometimes don't seem to understand what \"scalability\" really means. \nScalability means that something that is so fast that you don't even \n*think* about it will become a major bottleneck when you do it a thousand \ntimes, and the working set has grown so big that it totally blows out \nseveral levels of caches (both CPU caches and disk caches)\n\n\t\tLinus"},{"id":"38177","messageId":"20070327194233.GA28229@informatik.uni-freiburg.de","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703271124590.6730@woody.linux-foundation.org","subject":"Re: Submodule object store","fromName":"Uwe Kleine-König","fromEmail":"ukleinek@informatik.uni-freiburg.de","sentAt":"2007-03-27T19:42:33Z","receivedAt":"2007-03-27T19:42:33Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello Linus,\n\nLinus Torvalds wrote:\n> On Tue, 27 Mar 2007, Uwe Kleine-König wrote:\n> > \n> > \tembeddedproject$ git ls-tree HEAD | grep linux\n> > \t040000 commit 0123456789abcde0... linux-2.6\n> > \n> > (or how ever you save submodules). Then you might have to duplicate the\n> > objects of linux-2.6, because they are part of both histories.\n> \n> No they are not. Unless you do it wrong.\n> \n> The *only* object that is part of the superproject would be the tree that \n> *contains* that entry itself.\nYes, I got that. I think my concern is still valid, so probably I was\njust unable to phrase it explicitly. So I retry:\n\nIn the state above (i.e. linux-2.6 being a commit) the\nsuperproject's odb doesn't necessarily needs the object\n0123456789abcde0, right. But the commit before that had linux-2.6 being\na tree. And in that state linux-2.6/Makefile has to be in the\nsuperproject's odb. So if you choose the save the objects of submodules\nin a different odb, linux-2.6/Makefile has to be in both of them.\n \nI agree with the things you said afterwards, but they don't match the\nissue I wanted to point out.\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=2004+in+roman+numerals\n"},{"id":"38179","messageId":"Pine.LNX.4.64.0703271250130.6730@woody.linux-foundation.org","threadId":"7390","inReplyTo":"20070327194233.GA28229@informatik.uni-freiburg.de","subject":"Re: Submodule object store","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-27T19:53:53Z","receivedAt":"2007-03-27T19:53:53Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 27 Mar 2007, Uwe Kleine-König wrote:\n> \n> In the state above (i.e. linux-2.6 being a commit) the\n> superproject's odb doesn't necessarily needs the object\n> 0123456789abcde0, right. But the commit before that had linux-2.6 being\n> a tree.\n\nWell, you're saying that somebody split an existing non-supermodule \nproject?\n\nIf so, the supermodule really *does* have the old tree as its state, and \nsure, there will be duplication, but it's duplication that existed in the \nactual projects themselves, not something that the superproject \nintroduced.\n\nIn other words, I don't think that's an argument for or against sharing \nthe object database. You should *always* be able to share the object \ndatabase by setting GIT_OBJECT_DIR if you want (or by using alternates). \nBut that's independent of whether you are a sub/supermodule..\n\nAfter all, if you generate two totally *separate* projects (no subproject \nat all) and they just shared some state on their own (say, git and xdiff \nboth as totally independent git repositories), they have objects that can \nbe in common. Do you want to use alternates or share an object database? \nMaybe, or maybe not. It depends on the user, not on whether it's a \nsubproject.\n\n\t\tLinus"},{"id":"38181","messageId":"Pine.LNX.4.64.0703271254190.6730@woody.linux-foundation.org","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703271250130.6730@woody.linux-foundation.org","subject":"Re: Submodule object store","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-27T19:59:22Z","receivedAt":"2007-03-27T19:59:22Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 27 Mar 2007, Linus Torvalds wrote:\n> \n> In other words, I don't think that's an argument for or against sharing \n> the object database. You should *always* be able to share the object \n> database by setting GIT_OBJECT_DIR if you want (or by using alternates). \n> But that's independent of whether you are a sub/supermodule..\n\nIn fact, I suspect that you might well have a situation where there are \nmore objects to be shared \"across\" superproject boundaries than within \nthem.\n\nFor example, say that I'm a mirror site, and I mirror two different \ndistributions, both of which use superprojects (but *different* \nsuperprojects!) to track their distro stuff.\n\nObviously, the top-level setup is likely totally different, and they \nprobably differ a bit in which subprojects they have too, but in many \ncases, those two *different* superprojects will have subprojects that \ncould often share 99% of all their objects not within the superproject, \nbut individually *across* superprojects.\n\nSo you would not want to have a object store that is tied to the \nsuperproject, but you might well want to have each superproject share the \nobject store for the subprojects that they have in common. The \"kernel\" \nsubproject in the \"ubuntu\" superproject might want to share the object \nstore for the \"linux-2.6\" subproject in the Fedora 7 superproject.\n\n(Similarly, there migh be sharing with totally *individual* projects, ie \nyou might want to make both just have an alternate that points \ndirectly to the \"official\" tree that is in neither of the two \nsuperprojects and that I maintain separately).\n\n\t\tLinus\n"},{"id":"38182","messageId":"Pine.LNX.4.64.0703271547040.6485@iabervon.org","threadId":"7390","inReplyTo":"4609643C.9050803@midwinter.com","subject":"Re: .gitlink for Summer of Code","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2007-03-27T20:02:06Z","receivedAt":"2007-03-27T20:02:06Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 27 Mar 2007, Steven Grimm wrote:\n\n> The obvious use case for \"I want the superproject and just one submodule\" is\n> when the superproject has build tools, header files, or other pieces of data\n> that are shared by some/all of the submodules. Maybe not the case in BSD per\n> se, but having a top-level file full of settings, paths to tools, etc. that\n> gets included by the individual Makefiles in subdirectories isn't all that\n> uncommon in complex multi-part projects.\n\nThis is actually the case I'm personally interested in. But in that case, \nyou want to reverse the superproject/subproject organization, because that \nway each project part can use the desired version of the common stuff, and \npeople can modify the common stuff without then testing the whole \nuniverse.\n\nI.e., at some point, you'll want to change the behavior of the build \nsystem in such a way that all of the per-part configuration information \nsets needs to be updated to work with it. If the build system is in the \nsuperproject, you need to do everything at once. If the build system is in \na subsubproject, you can make the change without affecting anything, and, \nas subprojects pull the build system change, they update the subsubproject \nentry and the configuration files as a single subproject commit. For \nsanity, you want to reach the point where all of the projects are using \nthe same subsubproject version, but that doesn't have to happen overnight, \nand you don't have a single commit which touches every subproject.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"38186","messageId":"Pine.LNX.4.63.0703271245580.15345@qynat.qvtvafvgr.pbz","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703271409070.6730@woody.linux-foundation.org","subject":"Re: .gitlink for Summer of Code","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2007-03-27T20:54:02Z","receivedAt":"2007-03-27T20:54:02Z","isPatch":false,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Tue, 27 Mar 2007, Linus Torvalds wrote:\n\n> On Tue, 27 Mar 2007, Daniel Barkalow wrote:\n>>\n>> Are you talking about submodule history, or submodule state? If they care\n>> about any state but not the corresponding history, they need to do a\n>> shallow clone of the subproject, right?\n>\n> I don't see what the confusion is about.\n>\n> Why would you want a shallow clone, and what does that have to do with\n> submodules?\n>\n> I'm saying that the *normal* case is that of the thousands of submodules,\n> you generally care about one or two (the ones you work on).\n>\n> Those modules you want full history for. The supermodule you want because\n> it contains the build infrastructure. You'd generally want full history\n> for that too.\n\nif you are working on the submodule then you are correct.\n\nhowever if you are working on the supermodule it's a different story.\n\nif I'm working on the 'ubuntu superproject' it would be nice to be able to find \nwhat is different between the 'Jan 2007' and 'April 2007' versions. one could \nhave the 2.6.19 kernel and the other would have 2.6.20. I don't care about all \nthe individual changes between these two states of the kernel, but I need to be \nable to compile either one as part of my testing. If I bisect the in the \nsuperproject to the commit that updated the kernel, then I would consider \ngetting the 'kernel subproject' history to be able to bisect the bug further (or \nI may just report it to the kernel maintainers for them to check.\n\n> There's absolutely zero reason to think shallow clones have *anything* to\n> do with this. It's a totally separate dimension. Sure, you could use\n> shallow clones *too*, but it has nothing to do with subprojects.\n\nthey are seperate, but if you need to compile the superproject you either need \nto get the full history of every subproject, or you need shallow clones (or some \nthird approach).\n\nDavid Lang\n"},{"id":"38184","messageId":"Pine.LNX.4.64.0703271602110.6485@iabervon.org","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703271115580.6730@woody.linux-foundation.org","subject":"Re: .gitlink for Summer of Code","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2007-03-27T20:54:33Z","receivedAt":"2007-03-27T20:54:33Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 27 Mar 2007, Linus Torvalds wrote:\n\n> On Tue, 27 Mar 2007, Daniel Barkalow wrote:\n> > \n> > Is it fair to say that subproject support means that there's a use case \n> > where everybody will need shallow clones? And that it points out natural \n> > triggers for shallowness?\n> \n> No.\n> \n> I personally don't believe in shallow clones. And I *certainly* don't \n> believe that it has anything to do with subprojects. So people may want \n> shallow clones, but it's at least independent of the issue of submodules.\n> \n> With subprojects, it's not that you don't want the history. It's just that \n> you don't want the history for *all* projects. Most people care about a \n> very small subset.\n\nAre you talking about submodule history, or submodule state? If they care \nabout any state but not the corresponding history, they need to do a \nshallow clone of the subproject, right?\n\nOr are you assuming that people only want to have every subproject either \nthere with full history or entirely absent?\n\nI think that one common thing would be to care about the sequence of linux \nkernel snapshots selected by openembedded in their commits, without caring \nabout the linux kernel history in between those snapshots. And they \nprobably even want to bisect the superproject (still without getting into \nkernel versions that were never in superproject commits), so they can \ntrack down what caused their PDA to stop booting, where it's not clear \nwhich program is even responsible. Maybe once the bug is down to a single \nsuperproject commit, they'd want the history for the responsible \nsubproject.\n\nIn any case, I think that the superproject object database needs to keep \ntrack of where references came from (e.g., if you pull somebody's \nsuperproject commit, and the point of that commit is to use a \ncustom-modified subproject commit, that subproject commit must come from \nthe same person, and you need to be able to fetch it correctly after the \nfact if you don't get it immediately, even if you've personally forgotten \nthe URL).\n\n> (The exception, of course, is when the superproject simply isn't that big, \n> and only has a couple of subprojects. In git, for example, the xdiff stuff \n> could be a subproject if you wanted to do it that way. But then, the \n> subproject isn't a size issue, it's purely an organizational thing, and \n> there is no argument for/against shallowness there either).\n\nOf course. And to make this use case also viable, it's probably necessary \nto be able to tell git to fetch these subprojects automatically, because \nyou'll be sad if you leave for a long plane trip with the latest git but \nnot the xdiff it uses. (Clearly an application for .gitattributes, but \nthat'd be extra fun to implement.)\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"38185","messageId":"Pine.LNX.4.64.0703271409070.6730@woody.linux-foundation.org","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703271602110.6485@iabervon.org","subject":"Re: .gitlink for Summer of Code","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-27T21:11:28Z","receivedAt":"2007-03-27T21:11:28Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 27 Mar 2007, Daniel Barkalow wrote:\n> \n> Are you talking about submodule history, or submodule state? If they care \n> about any state but not the corresponding history, they need to do a \n> shallow clone of the subproject, right?\n\nI don't see what the confusion is about.\n\nWhy would you want a shallow clone, and what does that have to do with \nsubmodules?\n\nI'm saying that the *normal* case is that of the thousands of submodules, \nyou generally care about one or two (the ones you work on).\n\nThose modules you want full history for. The supermodule you want because \nit contains the build infrastructure. You'd generally want full history \nfor that too.\n\nThere's absolutely zero reason to think shallow clones have *anything* to \ndo with this. It's a totally separate dimension. Sure, you could use \nshallow clones *too*, but it has nothing to do with subprojects.\n\n\t\tLinus\n"},{"id":"38187","messageId":"Pine.LNX.4.64.0703271420270.6730@woody.linux-foundation.org","threadId":"7390","inReplyTo":"Pine.LNX.4.64.0703271547040.6485@iabervon.org","subject":"Re: .gitlink for Summer of Code","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-27T21:27:17Z","receivedAt":"2007-03-27T21:27:17Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 27 Mar 2007, Daniel Barkalow wrote:\n> \n> This is actually the case I'm personally interested in. But in that case, \n> you want to reverse the superproject/subproject organization, because that \n> way each project part can use the desired version of the common stuff, and \n> people can modify the common stuff without then testing the whole \n> universe.\n\nThe build infrastructure is only a small part of the superproject thing.\n\nA much more interesting thing in many ways is the \"how do the pieces fit \ntogether\" question, ie the \"which library version X do I need for program \nversion Y?\"\n\nAnd that needs to be at the superproject level, obviously. The person who \nworks on the application will want to fetch the library too, but he likely \nisn't interested in all the *other* libraries that don't affect his app, \nand he likely isn't interested in things like standard libraries (which \nmay be in the superproject too, but since their versioning doesn't affect \nany normal subproject, you'd not expect application developers to have all \nof libc checked out and built, would you?).\n\nSo yes, you could have several levels: the top level for \"versioning\", the \nmiddle level for \"applications and libraries\" and some third level for \n\"build infrastructure that can be shared\". However, I've never actually \nseen any project work that way. People *always* seem to put the build \ninfrastructure at either the top level, or as one of the subprojects that \nis required for all the other subprojects.\n\nOf course, it's possible that the reason people do that is that things \nlike CVS are really really bad at the versioning stuff, and since they \naren't distributed, you cannot put the shared build infrastructure in \nmultiple projects at the same time anyway.\n\nSo with a distributed environment like git, doing the shared build \ninfrastructure as a separate sub-sub-project would work in ways it does \n*not* work in a centralized model, but I think we also want to just \nsupport the way people are used to working, and that definitely involves \nhaving the build infrastructure at or under the top level..\n\n\t\tLinus\n"},{"id":"38203","messageId":"Pine.LNX.4.63.0703271514140.15345@qynat.qvtvafvgr.pbz","threadId":"7390","inReplyTo":"euc9av$2p9$1@sea.gmane.org","subject":"Re: .gitlink for Summer of Code","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2007-03-27T23:20:54Z","receivedAt":"2007-03-27T23:20:54Z","isPatch":false,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Wed, 28 Mar 2007, Jakub Narebski wrote:\n\n>> if I'm working on the 'ubuntu superproject' it would be nice to be able to find\n>> what is different between the 'Jan 2007' and 'April 2007' versions. one could\n>> have the 2.6.19 kernel and the other would have 2.6.20. I don't care about all\n>> the individual changes between these two states of the kernel, but I need to be\n>> able to compile either one as part of my testing. If I bisect the in the\n>> superproject to the commit that updated the kernel, then I would consider\n>> getting the 'kernel subproject' history to be able to bisect the bug further (or\n>> I may just report it to the kernel maintainers for them to check.\n>\n> I'd rather call this idea _sparse_ clone (not shallow), as you have only\n> some points in the history, but they don't need to be top 'n' ones.\n\nOk I can see the difference in the definition of the two, the ideal would \nprobably be to have sparse and shallow clones be different instances of the same \nmechanism.\n\n sparse being specific points in the history, shallow being a range.\n\n allow for multiple ranges, and the ability to 'fill in the blanks' later so \nthat points can become ranges and ranges can merge.\n\nalso having the server say 'it would only be XMB more to pull everything you \ndon't have, do you want to do this?' would cause more load on the server for \neach of the partial pulls, but would encourage people to fill out partial \nrepositories instead of hitting the servers repeatedly.\n\nDavid Lang\n"},{"id":"38200","messageId":"euc9av$2p9$1@sea.gmane.org","threadId":"7390","inReplyTo":"Pine.LNX.4.63.0703271245580.15345@qynat.qvtvafvgr.pbz","subject":"Re: .gitlink for Summer of Code","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-03-27T23:31:41Z","receivedAt":"2007-03-27T23:31:41Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"David Lang wrote:\n\n> On Tue, 27 Mar 2007, Linus Torvalds wrote:\n> \n>> On Tue, 27 Mar 2007, Daniel Barkalow wrote:\n>>>\n>>> Are you talking about submodule history, or submodule state? If they care\n>>> about any state but not the corresponding history, they need to do a\n>>> shallow clone of the subproject, right?\n>>\n>> I don't see what the confusion is about.\n>>\n>> Why would you want a shallow clone, and what does that have to do with\n>> submodules?\n>>\n>> I'm saying that the *normal* case is that of the thousands of submodules,\n>> you generally care about one or two (the ones you work on).\n>>\n>> Those modules you want full history for. The supermodule you want because\n>> it contains the build infrastructure. You'd generally want full history\n>> for that too.\n> \n> if you are working on the submodule then you are correct.\n> \n> however if you are working on the supermodule it's a different story.\n> \n> if I'm working on the 'ubuntu superproject' it would be nice to be able to find \n> what is different between the 'Jan 2007' and 'April 2007' versions. one could \n> have the 2.6.19 kernel and the other would have 2.6.20. I don't care about all \n> the individual changes between these two states of the kernel, but I need to be \n> able to compile either one as part of my testing. If I bisect the in the \n> superproject to the commit that updated the kernel, then I would consider \n> getting the 'kernel subproject' history to be able to bisect the bug further (or \n> I may just report it to the kernel maintainers for them to check.\n\nI'd rather call this idea _sparse_ clone (not shallow), as you have only\nsome points in the history, but they don't need to be top 'n' ones.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"}]}