{"thread":{"id":"38958","subject":"Allowing weak references to blobs and strong references to commits","startedAt":"2015-03-31T10:07:56Z","lastAt":"2015-03-31T23:35:45Z","messageCount":10,"participants":["Mike Hommey","Philip Oakley","Junio C Hamano","Randall S. Becker","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"258737","messageId":"20150331100756.GA13377@glandium.org","threadId":"38958","inReplyTo":null,"subject":"Allowing weak references to blobs and strong references to commits","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2015-03-31T10:07:56Z","receivedAt":"2015-03-31T10:07:56Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Hi,\n\nCurrently, in git-cinnabar[1], I'm using a private namespace\n(refs/cinnabar) for various different things:\n- references to all the imported heads (which may or may not\n match remote refs),\n- the last refs used for a fetch (part of the refspec protocol for\n remote-helpers)\n- a branch containing mappings from mercurial sha1s to git objects.\n- a branch used to store all mercurial manifests.\n- a cache of some sort (used for tags only atm)\n\nSo essentially there are a bit more than twice as many refs as actually\nnecessary (and up to more than three when there is only one remote).\n\nIdeally, the mercurial manifests data would use as many refs as\nbranches, so that their parent information wouldn't have to be guessed\nfrom the corresponding git commits, but I didn't do that initially to\navoid making the number of necessary refs even bigger (that would make\nthe number three or four times as many as necessary).\n\nI won't bother you with all the whys and hows, but that ends up being\na lot of unwanted noise for users, because many commands don't limit\nthemselves to refs/heads, refs/tags and refs/remotes.\n\nOne way to reduce this noise would be for me to create fake octopus\nmerges and reduce the number of heads to one, or at least one per\ncategory. But this is cumbersome and would create a lot of useless\ncommits that would end up loose, except if they are kept forever which\nseems even worse.\n\nSo I thought, since commits are already allowed in tree objects, for\nsubmodules, why not add a bit to the mode that would tell git that\nthose commit object references are meant to always be there aka strong\nreference, as opposed to the current weak references for submodules.\nI was thinking something like 0200000, which is above S_IFMT, but I\nhaven't checked if mode is expected to be a short anywhere, maybe one of\nthe file permission flags could be abused instead (sticky bit?).\n\nI could see this used in the future to e.g. implement a fetchable reflog\n(which could be a ref to a tree with strong references to commits).\n\nThen that got me thinking that the opposite would be useful to me as\nwell: I'm currently storing mercurial manifests as git trees with\n(weak) commit references using the mercurial sha1s for files.\nUnfortunately, that doesn't allow to store the corresponding file\npermissions, so I'm going through hoops to get that. It would be simpler\nfor me if I could just declare files or symlinks with the right\npermissions and say 'the corresponding blob doesn't need to exist'.\nI'm sure other tools using git as storage would have a use for such\nweak references.\n\nWhat do you think about this? Does that seem reasonable to have in git\ncore, and if yes, how would you go about implementing it (same bit with\ndifferent meaning for blobs and commits (or would you rather that were\nonly done for commits and not for blobs)? what should I be careful\nabout, besides making sure gc and fsck don't mess up?)\n\nCheers,\n\nMike\n\n1. a git-remote-hg tool, https://github.com/glandium/git-cinnabar/\n"},{"id":"258768","messageId":"1E05987AFD4A4ABCB5515905B517C021@PhilipOakley","threadId":"38958","inReplyTo":"20150331100756.GA13377@glandium.org","subject":"Re: Allowing weak references to blobs and strong references to commits","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2015-03-31T19:55:29Z","receivedAt":"2015-03-31T19:55:29Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Mike Hommey\" \n[...]\n\n> So I thought, since commits are already allowed in tree objects, for\n> submodules, why not add a bit to the mode that would tell git that\n> those commit object references are meant to always be there aka strong\n> reference, as opposed to the current weak references for submodules.\n> I was thinking something like 0200000, which is above S_IFMT, but I\n> haven't checked if mode is expected to be a short anywhere, maybe one \n> of\n> the file permission flags could be abused instead (sticky bit?).\n>\n> I could see this used in the future to e.g. implement a fetchable \n> reflog\n> (which could be a ref to a tree with strong references to commits).\n>\n> Then that got me thinking that the opposite would be useful to me as\n> well: I'm currently storing mercurial manifests as git trees with\n> (weak) commit references using the mercurial sha1s for files.\n> Unfortunately, that doesn't allow to store the corresponding file\n> permissions, so I'm going through hoops to get that. It would be \n> simpler\n> for me if I could just declare files or symlinks with the right\n> permissions and say 'the corresponding blob doesn't need to exist'.\n> I'm sure other tools using git as storage would have a use for such\n> weak references.\n>\nThe \"weak references\" idea is something that's on my back list of \nToh-Doh's for the purpose of having a Narrow clone.\n\nHowever it's not that easy as you need to consider three areas - what's \non disk (worktree/file system), what's in the index, and what's in the \nobject store and how a coherent view is kept of all three without \nbreakage.\n\nThe 'Sparse Checkout' / 'Skip Worktree' (see `git help read-tree`) \ncovers the first two but not the third (which submodules does) [that's \nyour 'the corresponding blob doesn't need to exist' aspect from my \nperspective]\n\n\n> What do you think about this? Does that seem reasonable to have in git\n> core, and if yes, how would you go about implementing it (same bit \n> with\n> different meaning for blobs and commits (or would you rather that were\n> only done for commits and not for blobs)? what should I be careful\n> about, besides making sure gc and fsck don't mess up?)\n>\n> Cheers,\n>\n> Mike\n>\n> 1. a git-remote-hg tool, https://github.com/glandium/git-cinnabar/\n> --\nPhilip \n"},{"id":"258770","messageId":"xmqq8uecrk5g.fsf@gitster.dls.corp.google.com","threadId":"38958","inReplyTo":"20150331100756.GA13377@glandium.org","subject":"Re: Allowing weak references to blobs and strong references to commits","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-03-31T20:23:23Z","receivedAt":"2015-03-31T20:23:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mike Hommey writes:\n\n> So I thought, since commits are already allowed in tree objects, for\n> submodules, why not add a bit to the mode that would tell git that\n> those commit object references are meant to always be there aka strong\n> reference, as opposed to the current weak references for submodules.\n\nUnless you are recording the paths to these \"commits\" to be\npotentially checked out on the filesystem, do not put them in a\n\"tree\". The entries in a tree object represent \"This thing go to\nthis path in the working tree\".\n\nIt is not clear to me (and because you said \"I won't bother you with\nall the whys and hows\", I am guessing that it is OK for readers to\nbe unclear), but I think you only want to make sure \"git fetch\" and\n\"git push\" transfers these objects, the graph formed by which is\n*not* any part of the main history of the project. It is perfectly\nOK to represent these objects as a special purpose history and have\na ref point at its tip. The \"notes\" database is represented that\nway, for example. And I do not see anything wrong to use octopus\nmerges in the history if you want to represent \"here are the commit\nobjects that I care about at this point in the 'special purpose'\nhistory (not the main history)\".\n"},{"id":"258774","messageId":"006a01d06bf6$e2512540$a6f36fc0$@nexbridge.com","threadId":"38958","inReplyTo":"1E05987AFD4A4ABCB5515905B517C021@PhilipOakley","subject":"RE: Allowing weak references to blobs and strong references to commits","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2015-03-31T21:08:47Z","receivedAt":"2015-03-31T21:08:47Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On March 31, 2015 3:55 PM Philip Oakley wrote:\n> From: \"Mike Hommey\" \n> [...]\n> > So I thought, since commits are already allowed in tree objects, for\n> > submodules, why not add a bit to the mode that would tell git that\n> > those commit object references are meant to always be there aka strong\n> > reference, as opposed to the current weak references for submodules.\n> > I was thinking something like 0200000, which is above S_IFMT, but I\n> > haven't checked if mode is expected to be a short anywhere, maybe one\n> > of the file permission flags could be abused instead (sticky bit?).\n> >\n> > I could see this used in the future to e.g. implement a fetchable\n> > reflog (which could be a ref to a tree with strong references to\n> > commits).\n> >\n> > Then that got me thinking that the opposite would be useful to me as\n> > well: I'm currently storing mercurial manifests as git trees with\n> > (weak) commit references using the mercurial sha1s for files.\n> > Unfortunately, that doesn't allow to store the corresponding file\n> > permissions, so I'm going through hoops to get that. It would be\n> > simpler for me if I could just declare files or symlinks with the\n> > right permissions and say 'the corresponding blob doesn't need to\n> > exist'.\n> > I'm sure other tools using git as storage would have a use for such\n> > weak references.\n> >\n> The \"weak references\" idea is something that's on my back list of\nToh-Doh's for\n> the purpose of having a Narrow clone.\n> \n> However it's not that easy as you need to consider three areas - what's on\ndisk\n> (worktree/file system), what's in the index, and what's in the object\nstore and\n> how a coherent view is kept of all three without breakage.\n> \n> The 'Sparse Checkout' / 'Skip Worktree' (see `git help read-tree`) covers\nthe\n> first two but not the third (which submodules does) [that's your 'the\n> corresponding blob doesn't need to exist' aspect from my perspective]\n> \n> \n> > What do you think about this? Does that seem reasonable to have in git\n> > core, and if yes, how would you go about implementing it (same bit\n> > with different meaning for blobs and commits (or would you rather that\n> > were only done for commits and not for blobs)? what should I be\n> > careful about, besides making sure gc and fsck don't mess up?)\n\nI don't know whether this is relevant or not - forgiveness requested in\nadvance. It may be useful to store primarily the SHA1 for a weak object. In\na product called RMS, this was called an \"External Reference\". The file\nitself was not stored, but its signature was. It was possible to tell that\nthe commit was validly and completely on disk, only if the signature matched\n(so git status would know). If the file was missing, or had an invalid\nsignature, the working area was considered dirty (so git status would\npresumably report \"modified\"). All signatures were stored for these types of\nfiles, but the contents were not - hence \"external\". Otherwise, we stored\nall other repository attributes - except the contents, with the obvious\nrisks. This was typically used to track versions of the compilers and\nheaders being used for builds, which we did not want to store in the\nrepository, managed by a separate systems operations group, but wanted to\nknow the signatures in case we had to go back in time. From my point of\nview, I would like to be able to have /usr/include (example only) as a\nworking area where I can be 100% certain it contains what I expect it to\ncontain, but I don't really want to store the objects in a repository - and\nmay not have root anyway.\n\nCheers,\nRandall\n\n-- Brief whoami: NonStop&UNIX developer since approximately\nUNIX(421664400)/NonStop(211288444200000000)\n-- In my real life, I talk too much.\n"},{"id":"258792","messageId":"20150331223919.GA24362@glandium.org","threadId":"38958","inReplyTo":"xmqq8uecrk5g.fsf@gitster.dls.corp.google.com","subject":"Re: Allowing weak references to blobs and strong references to commits","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2015-03-31T22:39:19Z","receivedAt":"2015-03-31T22:39:19Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Mar 31, 2015 at 01:23:23PM -0700, Junio C Hamano wrote:\n> Mike Hommey writes:\n> \n> > So I thought, since commits are already allowed in tree objects, for\n> > submodules, why not add a bit to the mode that would tell git that\n> > those commit object references are meant to always be there aka strong\n> > reference, as opposed to the current weak references for submodules.\n> \n> Unless you are recording the paths to these \"commits\" to be\n> potentially checked out on the filesystem, do not put them in a\n> \"tree\". The entries in a tree object represent \"This thing go to\n> this path in the working tree\".\n> \n> It is not clear to me (and because you said \"I won't bother you with\n> all the whys and hows\", I am guessing that it is OK for readers to\n> be unclear), but I think you only want to make sure \"git fetch\" and\n> \"git push\" transfers these objects, the graph formed by which is\n> *not* any part of the main history of the project.\n\nTransfer is not the main reason I have those refs, although it's a nice\nplus. I'm using the git database as a storage for metadata I need to\nkeep track of the remote mercurial content that isn't part of the\ncontent history represented in the corresponding git history. Obviously,\nthis isn't meant to be checked out, like a notes tree. The refs are only\nthere so that a) git-cinnabar can find its data and b) git gc doesn't\nremove it.\n\n> It is perfectly OK to represent these objects as a special purpose\n> history and have a ref point at its tip. The \"notes\" database is\n> represented that way, for example.\n\nIndeed, the notes database is in a similar situation. But the fact that\nin my use-case (cloning Mozilla mercurial repositories) _thousands_ of\nrefs are involved is not really making it user-friendly. I'd rather hide\nthose away from users.\n\n> And I do not see anything wrong to use octopus merges in the history\n> if you want to represent \"here are the commit objects that I care\n> about at this point in the 'special purpose' history (not the main\n> history)\".\n\nOctopus merges are limited to 16 parents. That means to merge thousands\nof refs, I need to do that on at least 3 levels, involving hundreds of\ncommit objects, many of which would become loose quickly, or become\nextra noise in the \"metadata\" \"branches\"... and while researching this\nfurther, I realize it doesn't seem there is such a limit to the number\nof parents for octopus merges. Where did I get that from? Was there such\na limit in the past or was I high?\n\nThat being said, having names associated with those tips _is_ useful to\nme (it allows to know the corresponding mercurial sha1 and branch\nwithout having to do a note lookup for each), and a tree of commits\nwould help, here, although I could put a tree of weak refs to commits\nas the content of an octopus merge...\n\nSo I guess I could live without the strong commit refs. Weak blob refs\nwould still be useful, though.\n\nMike\n"},{"id":"258795","messageId":"xmqqiodgojqm.fsf@gitster.dls.corp.google.com","threadId":"38958","inReplyTo":"20150331223919.GA24362@glandium.org","subject":"Re: Allowing weak references to blobs and strong references to commits","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-03-31T23:00:33Z","receivedAt":"2015-03-31T23:00:33Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mike Hommey writes:\n\n> Octopus merges are limited to 16 parents.\n\nHuh?\n"},{"id":"258796","messageId":"20150331231449.GF22844@google.com","threadId":"38958","inReplyTo":"20150331223919.GA24362@glandium.org","subject":"Re: Allowing weak references to blobs and strong references to commits","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2015-03-31T23:14:49Z","receivedAt":"2015-03-31T23:14:49Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Mike Hommey wrote:\n\n> Octopus merges are limited to 16 parents.\n\nThe note about this in fast-import is out of date (e.g., see\nt/t7602-merge-octopus-many.sh and v1.6.0-rc0~194, 2008-06-27). How\nabout this patch?\n\n-- >8--\nSubject: fast-import doc: remove suggested 16-parent limit\n\nMerges with an absurd number of parents are still a bad idea because\nthey do not render well in tools like gitk, but if they are present\nin the repository being imported into git then there's no need to\navoid reproducing them faithfully.\n\nIn olden times, before v1.6.0-rc0~194 (2008-06-27), git commit-tree\nand higher-level tools built on top of it were limited to writing 16\nparents for a commit. Nowadays normal git operations are happy to\nwrite more parents when asked, so the motivation for this note in the\nfast-import documentation is gone and we can remove it.\n\nSigned-off-by: Jonathan Nieder \n---\nThanks,\nJonathan\n\n Documentation/git-fast-import.txt | 7 ++-----\n 1 file changed, 2 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex f71fb01..773584e 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -506,11 +506,8 @@ If the `from` command is\n omitted when creating a new branch, the first `merge` commit will be\n the first ancestor of the current commit, and the branch will start\n out with no files. An unlimited number of `merge` commands per\n-commit are permitted by fast-import, thereby establishing an n-way merge.\n-However Git's other tools never create commits with more than 15\n-additional ancestors (forming a 16-way merge). For this reason\n-it is suggested that frontends do not use more than 15 `merge`\n-commands per commit; 16, if starting a new, empty branch.\n+commit are permitted by fast-import and other git commands, thereby\n+establishing an n-way merge.\n \n Here `` is any of the commit specification expressions\n also accepted by `from` (see above).\n-- \n2.2.0.rc0.207.ga3a616c\n"},{"id":"258797","messageId":"20150331231807.GG22844@google.com","threadId":"38958","inReplyTo":"20150331231449.GF22844@google.com","subject":"Re: Allowing weak references to blobs and strong references to commits","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2015-03-31T23:18:07Z","receivedAt":"2015-03-31T23:18:07Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n\n> How\n> about this patch?\n\nI think I botched the wording. Here's a second try.\n\n-- >8 --\nSubject: fast-import doc: remove suggested 16-parent limit\n\nMerges with an absurd number of parents are still a bad idea because\nthey do not render well in tools like gitk, but if they are present\nin the repository being imported into git then there's no need to\navoid reproducing them faithfully.\n\nIn olden times, before v1.6.0-rc0~194 (2008-06-27), git commit-tree\nand higher-level tools built on top of it were limited to writing 16\nparents for a commit. Nowadays normal git operations are happy to\nwrite more parents when asked, so the motivation for this note in the\nfast-import documentation is gone and we can remove it.\n\nSigned-off-by: Jonathan Nieder \n---\n Documentation/git-fast-import.txt | 4 ----\n 1 file changed, 4 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex f71fb01..690fed3 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -507,10 +507,6 @@ omitted when creating a new branch, the first `merge` commit will be\n the first ancestor of the current commit, and the branch will start\n out with no files. An unlimited number of `merge` commands per\n commit are permitted by fast-import, thereby establishing an n-way merge.\n-However Git's other tools never create commits with more than 15\n-additional ancestors (forming a 16-way merge). For this reason\n-it is suggested that frontends do not use more than 15 `merge`\n-commands per commit; 16, if starting a new, empty branch.\n \n Here `` is any of the commit specification expressions\n also accepted by `from` (see above).\n-- \n"},{"id":"258799","messageId":"CAPc5daWEitgc_69Cqp0daaMpFwEwoVAi62fgeDoyJKz8wBaBrg@mail.gmail.com","threadId":"38958","inReplyTo":"20150331231449.GF22844@google.com","subject":"Re: Allowing weak references to blobs and strong references to commits","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-03-31T23:25:35Z","receivedAt":"2015-03-31T23:25:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Tue, Mar 31, 2015 at 4:14 PM, Jonathan Nieder wrote:\n> Mike Hommey wrote:\n>\n>> Octopus merges are limited to 16 parents.\n>\n> The note about this in fast-import is out of date (e.g., see\n> t/t7602-merge-octopus-many.sh and v1.6.0-rc0~194, 2008-06-27). How\n> about this patch?\n\nAhh, I thought we eradicated all mentions of that ancient limit back then.\n\nThanks for catching.\n"},{"id":"258802","messageId":"20150331233545.GA16511@glandium.org","threadId":"38958","inReplyTo":"20150331231449.GF22844@google.com","subject":"Re: Allowing weak references to blobs and strong references to commits","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2015-03-31T23:35:45Z","receivedAt":"2015-03-31T23:35:45Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Mar 31, 2015 at 04:14:49PM -0700, Jonathan Nieder wrote:\n> Mike Hommey wrote:\n> \n> > Octopus merges are limited to 16 parents.\n> \n> The note about this in fast-import is out of date (e.g., see\n> t/t7602-merge-octopus-many.sh and v1.6.0-rc0~194, 2008-06-27). How\n> about this patch?\n\nAha! I wasn't stoned! Thanks for fixing this :)\n\nMike\n"}]}