{"thread":{"id":"35781","subject":"Creating own hierarchies under $GITDIR/refs ?","startedAt":"2014-02-02T10:37:39Z","lastAt":"2014-02-02T23:44:56Z","messageCount":11,"participants":["David Kastrup","Duy Nguyen","John Keeping","Jeff King","Jed Brown","Andreas Schwab"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"234032","messageId":"87a9e92424.fsf@fencepost.gnu.org","threadId":"35781","inReplyTo":null,"subject":"Creating own hierarchies under $GITDIR/refs ?","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-02T10:37:39Z","receivedAt":"2014-02-02T10:37:39Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"\nHi,\n\nin the context of an ongoing discussion on the Emacs developer list of\nconverting the Bzr repository of Emacs, one question (with different\napproaches) is where to put the information regarding preexisting Bazaar\nrevision numbers and bug tracker ids: those are not present in the\ncurrent Git mirror.\n\nPutting them in the commit messages would require a full history\nrewrite, and if some are missed in the process, this cannot be fixed\nafterwards.\n\nSo I mused: refs/heads contains branches, refs/tags contains tags. The\nrespective information would likely easily enough be stored in refs/bzr\nand refs/bugs and in that manner would not pollute the \"ordinary\" tag\nand branch spaces, rendering \"git tag\" and/or \"git branch\" output mostly\nunusable. I tested creating such a directory and entries and indeed\nreferences like bzr/39005 then worked.\n\nHowever, cloning from the repository did not copy those directories and\nreferences, so without modification, this scheme would not work for\ncloned repositories.\n\nAre there some measures one can take/configure in the parent repository\nsuch that (named or all) additional directories inside of $GITDIR/refs\nwould get cloned along with the rest?\n\nIt would definitely open viable options for dealing with mirrors and/or\nrepository migrations in general.\n\n-- \nDavid Kastrup\n"},{"id":"234034","messageId":"CACsJy8CdKRQ_au3QqVoUdedvPpkPh_2vodKJwLZ7VrrwRJSDXQ@mail.gmail.com","threadId":"35781","inReplyTo":"87a9e92424.fsf@fencepost.gnu.org","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-02T11:00:37Z","receivedAt":"2014-02-02T11:00:37Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Feb 2, 2014 at 5:37 PM, David Kastrup wrote:\n> in the context of an ongoing discussion on the Emacs developer list of\n> converting the Bzr repository of Emacs, one question (with different\n> approaches) is where to put the information regarding preexisting Bazaar\n> revision numbers and bug tracker ids: those are not present in the\n> current Git mirror.\n>\n> Putting them in the commit messages would require a full history\n> rewrite, and if some are missed in the process, this cannot be fixed\n> afterwards.\n\nWhat do you need them for? Perhaps putting everything in a file, maybe\nsorted by SHA-1, would suffice? It should not be too hard to write a\nscript to map bug tracker id to a commit id. The file is for past\ncommits only. New commits can contain these info in their messages.\n-- \nDuy\n"},{"id":"234119","messageId":"m2mwi9rd1p.fsf@linux-m68k.org","threadId":"35781","inReplyTo":"87a9e92424.fsf@fencepost.gnu.org","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2014-02-02T11:04:18Z","receivedAt":"2014-02-02T11:04:18Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"David Kastrup writes:\n\n> Are there some measures one can take/configure in the parent repository\n> such that (named or all) additional directories inside of $GITDIR/refs\n> would get cloned along with the rest?\n\n$ git config --add remote.orgin.fetch '+refs/notes/*:refs/notes/*'\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"234036","messageId":"8761ox2240.fsf@fencepost.gnu.org","threadId":"35781","inReplyTo":"CACsJy8CdKRQ_au3QqVoUdedvPpkPh_2vodKJwLZ7VrrwRJSDXQ@mail.gmail.com","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-02T11:19:43Z","receivedAt":"2014-02-02T11:19:43Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Duy Nguyen writes:\n\n> On Sun, Feb 2, 2014 at 5:37 PM, David Kastrup wrote:\n>> in the context of an ongoing discussion on the Emacs developer list of\n>> converting the Bzr repository of Emacs, one question (with different\n>> approaches) is where to put the information regarding preexisting Bazaar\n>> revision numbers and bug tracker ids: those are not present in the\n>> current Git mirror.\n>>\n>> Putting them in the commit messages would require a full history\n>> rewrite, and if some are missed in the process, this cannot be fixed\n>> afterwards.\n>\n> What do you need them for?\n\nResolving references typically found in commit messages. Also\nestablishing correlation to bug issue numbers.\n\n> Perhaps putting everything in a file, maybe sorted by SHA-1, would\n> suffice? It should not be too hard to write a script to map bug\n> tracker id to a commit id.\n\nWe are not talking about \"it should not be too hard\". We are talking\nabout \"obvious and reliable enough to render a complete history rewrite\npointless\".\n\n> The file is for past commits only.\n\n> New commits can contain these info in their messages.\n\nIf it's not forgotten. Experience shows that things like issue numbers\nhave a tendency to be omitted, and then they stay missing.\n\nAt any rate, this is exactly the kind of stuff that tags are useful for,\nexcept that using them for all that would render the \"tag space\"\novercrowded.\n\nRest assured that the \"standard\" answers have been beat to death in the\nEmacs developer list thread several times over.\n\nSo I'm more interested in getting actual answers dealing with the\nquestion I have asked rather than suggestions for questions that would\nbe easier to answer.\n\nSince Git has a working facility for references that is catered to do\nexactly this kind of mapping and already _does_, it seems like a\nconvenient path to explore.\n\nIt apparently even already works with --decorate:\n\ncommit c92b1fb3ad8514f08fc4cec531211717955a5c29 (tag: release/2.19.1-1, origin/release/unstable, tag: refs/bzr/r15000)\nAuthor: Phil Holmes \nDate: Sun Jan 19 15:01:48 2014 +0000\n\n Release: update news.\n\n-- \nDavid Kastrup\n"},{"id":"234039","messageId":"20140202113141.GB29976@serenity.lan","threadId":"35781","inReplyTo":"8761ox2240.fsf@fencepost.gnu.org","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2014-02-02T11:31:41Z","receivedAt":"2014-02-02T11:31:41Z","isPatch":false,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Feb 02, 2014 at 12:19:43PM +0100, David Kastrup wrote:\n> Duy Nguyen writes:\n> \n> > The file is for past commits only.\n> \n> > New commits can contain these info in their messages.\n> \n> If it's not forgotten. Experience shows that things like issue numbers\n> have a tendency to be omitted, and then they stay missing.\n> \n> At any rate, this is exactly the kind of stuff that tags are useful for,\n> except that using them for all that would render the \"tag space\"\n> overcrowded.\n\nActually, I would say this is exactly the sort of thing notes are for.\n\ngit.git uses them to map commits back to mailing list discussions:\n\n git fetch git://github.com/gitster/git +refs/notes/amlog:refs/notes/amlog &&\n git log --notes=amlog\n\nSee also notes.displayRef in git-config(1).\n\nNotes aren't fetch by default, but it's not hard for those interested to\nadd a remote.*.fetch line to their config.\n"},{"id":"234041","messageId":"87wqhdzqo3.fsf@fencepost.gnu.org","threadId":"35781","inReplyTo":"20140202113141.GB29976@serenity.lan","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-02T11:42:52Z","receivedAt":"2014-02-02T11:42:52Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"John Keeping writes:\n\n> On Sun, Feb 02, 2014 at 12:19:43PM +0100, David Kastrup wrote:\n>> Duy Nguyen writes:\n>> \n>> > The file is for past commits only.\n>> \n>> > New commits can contain these info in their messages.\n>> \n>> If it's not forgotten. Experience shows that things like issue numbers\n>> have a tendency to be omitted, and then they stay missing.\n>> \n>> At any rate, this is exactly the kind of stuff that tags are useful for,\n>> except that using them for all that would render the \"tag space\"\n>> overcrowded.\n>\n> Actually, I would say this is exactly the sort of thing notes are for.\n>\n> git.git uses them to map commits back to mailing list discussions:\n\nBut that's the wrong direction. What is needed in the Emacs case is\nmapping the Bazaar reference numbers (and bug numbers) to commits.\n\nWhile it is true that the history rewriting approach would not deliver\nthis either (short of git log --grep with suitable patterns), I was\nlooking for something less of a crutch here.\n\n> Notes aren't fetch by default, but it's not hard for those interested\n> to add a remote.*.fetch line to their config.\n\nIf we are talking about measures everybody has to actively take before\ngetting access to functionality, this does not cross the convenience\nthreshold making it a solution preferred over others. But it's probably\nfeasible to configure a fetch line doing this that will get cloned when\nfirst cloning a repository. That's not too hot for people with existing\nrepositories, but since we are talking about a migration from Bazaar\nanyway, Git users currently are so by choice and so might be more\nwilling to update their configuration if it helps with avoiding a fully\nnew clone.\n\n-- \nDavid Kastrup\n"},{"id":"234042","messageId":"CACsJy8A44EoxEb3b=ZZ-jgLLtn1ttr0P8gGwWJ2dNYCWi3MRUw@mail.gmail.com","threadId":"35781","inReplyTo":"8761ox2240.fsf@fencepost.gnu.org","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-02T12:00:12Z","receivedAt":"2014-02-02T12:00:12Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Feb 2, 2014 at 6:19 PM, David Kastrup wrote:\n> Since Git has a working facility for references that is catered to do\n> exactly this kind of mapping and already _does_, it seems like a\n> convenient path to explore.\n\nIt will not scale. If you make those refs available for\ncloning/fetching, all of them will be advertised first thing when git\nstarts negotiate. Imagine thousands of refs (and keep increasing) sent\nto the receiver at the beginning of every connection. Something like\n\"reverse git-notes\" may transfer more efficiently. Or we need to\nimprove git protocol to handle massive refs better, something that's\nbeen discussed for a while without any outcome.\n-- \nDuy\n"},{"id":"234044","messageId":"87sis1zpf3.fsf@fencepost.gnu.org","threadId":"35781","inReplyTo":"CACsJy8A44EoxEb3b=ZZ-jgLLtn1ttr0P8gGwWJ2dNYCWi3MRUw@mail.gmail.com","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-02T12:09:52Z","receivedAt":"2014-02-02T12:09:52Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Duy Nguyen writes:\n\n> On Sun, Feb 2, 2014 at 6:19 PM, David Kastrup wrote:\n>> Since Git has a working facility for references that is catered to do\n>> exactly this kind of mapping and already _does_, it seems like a\n>> convenient path to explore.\n>\n> It will not scale. If you make those refs available for\n> cloning/fetching, all of them will be advertised first thing when git\n> starts negotiate. Imagine thousands of refs (and keep increasing) sent\n> to the receiver at the beginning of every connection.\n\nIn current LilyPond repository:\ngit tag|wc\n 969 969 15161\n\nIn current Emacs mirror:\ngit tag|wc\n 1202 1202 15729\n\nIn current Git repository:\ngit tag|wc\n 498 498 4820\n\n> Something like \"reverse git-notes\" may transfer more efficiently. Or\n> we need to improve git protocol to handle massive refs better,\n> something that's been discussed for a while without any outcome.\n\nI think that even disregarding special use of references, _existing_\npractice would already appear to warrant being able to deal with\nthousands of refs in a reasonable manner.\n\nIt's a reasonable expectation to have a tag per (potentially\nintermediate) release or release candidate. For any project publishing\nreproducible daily snapshots, the threshold of 1000 will get reached\nwithin few years.\n\nOf course, it is relevant information to know that right _now_\nreferences will not scale. But that does not seem like a defensible\nlong-term perspective.\n\n-- \nDavid Kastrup\n"},{"id":"234046","messageId":"20140202122432.GC29976@serenity.lan","threadId":"35781","inReplyTo":"87wqhdzqo3.fsf@fencepost.gnu.org","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2014-02-02T12:24:32Z","receivedAt":"2014-02-02T12:24:32Z","isPatch":false,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Feb 02, 2014 at 12:42:52PM +0100, David Kastrup wrote:\n> John Keeping writes:\n> \n> > On Sun, Feb 02, 2014 at 12:19:43PM +0100, David Kastrup wrote:\n> >> Duy Nguyen writes:\n> >> \n> >> > The file is for past commits only.\n> >> \n> >> > New commits can contain these info in their messages.\n> >> \n> >> If it's not forgotten. Experience shows that things like issue numbers\n> >> have a tendency to be omitted, and then they stay missing.\n> >> \n> >> At any rate, this is exactly the kind of stuff that tags are useful for,\n> >> except that using them for all that would render the \"tag space\"\n> >> overcrowded.\n> >\n> > Actually, I would say this is exactly the sort of thing notes are for.\n> >\n> > git.git uses them to map commits back to mailing list discussions:\n> \n> But that's the wrong direction. What is needed in the Emacs case is\n> mapping the Bazaar reference numbers (and bug numbers) to commits.\n\nAh, OK. I hadn't quite read carefully enough.\n\nI actually wonder if you could do this with notes and git-grep; for\nexample:\n\n git grep -l keeping.me.uk refs/notes/amlog |\n sed -e 's/.*://' -e 's!/!!g'\n\nThat should be relatively efficient since you're only looking at the\ncurrent notes tree.\n\n> While it is true that the history rewriting approach would not deliver\n> this either (short of git log --grep with suitable patterns), I was\n> looking for something less of a crutch here.\n> \n> > Notes aren't fetch by default, but it's not hard for those interested\n> > to add a remote.*.fetch line to their config.\n> \n> If we are talking about measures everybody has to actively take before\n> getting access to functionality, this does not cross the convenience\n> threshold making it a solution preferred over others. But it's probably\n> feasible to configure a fetch line doing this that will get cloned when\n> first cloning a repository.\n\nI'm assuming you'll need some form of tool (at least a script) to\nmanipulate this feature; it wouldn't be too hard for that to set this up\nthe first time it's run.\n"},{"id":"234066","messageId":"20140202232652.GD16196@sigill.intra.peff.net","threadId":"35781","inReplyTo":"87a9e92424.fsf@fencepost.gnu.org","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-02-02T23:26:52Z","receivedAt":"2014-02-02T23:26:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Feb 02, 2014 at 11:37:39AM +0100, David Kastrup wrote:\n\n> So I mused: refs/heads contains branches, refs/tags contains tags. The\n> respective information would likely easily enough be stored in refs/bzr\n> and refs/bugs and in that manner would not pollute the \"ordinary\" tag\n> and branch spaces, rendering \"git tag\" and/or \"git branch\" output mostly\n> unusable. I tested creating such a directory and entries and indeed\n> references like bzr/39005 then worked.\n\nYes. The names \"refs/tags\" and \"refs/heads\" are special by convention,\nand there is no reason you cannot have other hierarchies (and indeed, we\nalready have \"refs/notes\" and \"refs/remotes\" as common hierarchies).\n\n> However, cloning from the repository did not copy those directories and\n> references, so without modification, this scheme would not work for\n> cloned repositories.\n\nCorrect. Anyone who wants them will have to ask for them manually, like:\n\n git config --add remote.origin.fetch '+refs/bzr/*:refs/bzr/*'\n\nafter which any \"git fetch\" will retrieve them.\n\n> Are there some measures one can take/configure in the parent repository\n> such that (named or all) additional directories inside of $GITDIR/refs\n> would get cloned along with the rest?\n\nNo. It is up to the client to decide which parts of the ref namespace\nthey want to fetch. The server only advertises what it has, and the\nclient selects from that.\n\n\nOthers mentioned that refs were never really intended to scale to\none-per-commit. We serve some repositories with tens of thousands of\nrefs from GitHub, and it does work. On the backend, we even have some\nrepos in the hundreds of thousands (but these are not client facing).\nMost of the pain points (like O(n^2) loops) have been ironed out, but\nthe two big ones are still:\n\n - server ref advertisement lists _all_ refs at the start of the\n conversation. So, e.g.,\n\n git fetch git://github.com/Homebrew/homebrew.git\n\n sends 2MB of advertisement just so a client can find out \"nope,\n nothing to fetch\".\n\n - the packed-refs storage is rather monolithic. Reading a value from\n it currently requires parsing the whole file. Likewise, deleting a\n ref requires rewriting the whole file.\n\nSo what you are proposing will work, but do note that there is a cost.\n\n-Peff\n"},{"id":"234068","messageId":"878utt84g7.fsf@jedbrown.org","threadId":"35781","inReplyTo":"20140202122432.GC29976@serenity.lan","subject":"Re: Creating own hierarchies under $GITDIR/refs ?","fromName":"Jed Brown","fromEmail":"jed@59a2.org","sentAt":"2014-02-02T23:44:56Z","receivedAt":"2014-02-02T23:44:56Z","isPatch":false,"sender":{"key":"jed@59a2.org","avatar":"https://gravatar.com/avatar/1391d04d82555f9058a9fdf5eead233e909a48e40480db31fc554e7afeb301da?d=mp&s=160"},"body":"John Keeping writes:\n> I actually wonder if you could do this with notes and git-grep; for\n> example:\n>\n> git grep -l keeping.me.uk refs/notes/amlog |\n> sed -e 's/.*://' -e 's!/!!g'\n>\n> That should be relatively efficient since you're only looking at the\n> current notes tree.\n\nI added notes handling to gitifyhg and would search it similar to this.\nSince gitifyhg is two-way, I could not modify the commits. Later, when\nwe converted several repositories (up to 50k commits/80 MB), I appended\n\n Hg-commit: $Hg_commit_hash\n\nto all the commit messages. This way it shows up on the web interface,\nusers don't have to obtain the notes specially, and \"git log --grep\"\nworks naturally. I think it's worth considering this simple solution;\nexisting Git users won't mind recloning once.\n"}]}