{"thread":{"id":"31619","subject":"Quickly searching for a note","startedAt":"2012-09-21T14:41:04Z","lastAt":"2012-09-25T16:19:53Z","messageCount":19,"participants":["Joshua Jensen","Andreas Schwab","Junio C Hamano","Johannes Sixt","Jeff King","Michael J Gruber"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"199660","messageId":"505C7C80.3000700@workspacewhiz.com","threadId":"31619","inReplyTo":null,"subject":"Quickly searching for a note","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2012-09-21T14:41:04Z","receivedAt":"2012-09-21T14:41:04Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":"Background: To tie Perforce changelists to Git commits, I add a note to \na commit with the form \"P4@123456\". Later, I use the note to sync down \nthe closest Perforce changelist matching the Git commit.\n\nI search for these notes by getting a list of revisions:\n\n git rev-list --max-count=1000\n\nI iterate those revisions and run git show and grep on each:\n\n git show -s --format=%N%n%s --show-notes=p4notes COMMIT\n\nFor short runs, this isn't so bad. For longer runs of commits (I just \nwalked through approximately 100), it takes a long time. Running 'git \nshow' is costing me about 7/10 of second, presumably because I am on \nWindows.\n\nIs there a faster way to do this?\n\nThanks.\n\nJosh\n"},{"id":"199661","messageId":"m2d31fbedd.fsf@igel.home","threadId":"31619","inReplyTo":"505C7C80.3000700@workspacewhiz.com","subject":"Re: Quickly searching for a note","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2012-09-21T15:10:54Z","receivedAt":"2012-09-21T15:10:54Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Joshua Jensen writes:\n\n> Background: To tie Perforce changelists to Git commits, I add a note to a\n> commit with the form \"P4@123456\". Later, I use the note to sync down the\n> closest Perforce changelist matching the Git commit.\n>\n> I search for these notes by getting a list of revisions:\n>\n> git rev-list --max-count=1000\n>\n> I iterate those revisions and run git show and grep on each:\n>\n> git show -s --format=%N%n%s --show-notes=p4notes COMMIT\n\nHow about \"git grep P4@123456 notes/p4notes\"?\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"199673","messageId":"7vy5k370n7.fsf@alter.siamese.dyndns.org","threadId":"31619","inReplyTo":"505C7C80.3000700@workspacewhiz.com","subject":"Re: Quickly searching for a note","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-21T17:21:00Z","receivedAt":"2012-09-21T17:21:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joshua Jensen writes:\n\n> Background: To tie Perforce changelists to Git commits, I add a note\n> to a commit with the form \"P4@123456\". Later, I use the note to sync\n> down the closest Perforce changelist matching the Git commit.\n>\n> I search for these notes by getting a list of revisions:\n>\n> git rev-list --max-count=1000\n>\n> I iterate those revisions and run git show and grep on each:\n>\n> git show -s --format=%N%n%s --show-notes=p4notes COMMIT\n>\n> For short runs, this isn't so bad. For longer runs of commits (I just\n> walked through approximately 100), it takes a long time. Running 'git\n> show' is costing me about 7/10 of second, presumably because I am on\n> Windows.\n\nIs there any particular reason you do that as two separate steps?\nIt would feel more natural, at least to me, to do something along\nthe lines of\n\n\tgit log --show-notes=p4notes -1000\n"},{"id":"199681","messageId":"505CB21E.4040607@workspacewhiz.com","threadId":"31619","inReplyTo":"7vy5k370n7.fsf@alter.siamese.dyndns.org","subject":"Re: Quickly searching for a note","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2012-09-21T18:29:50Z","receivedAt":"2012-09-21T18:29:50Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":"----- Original Message -----\nFrom: Junio C Hamano\nDate: 9/21/2012 11:21 AM\n> Joshua Jensen writes:\n>\n>> Background: To tie Perforce changelists to Git commits, I add a note\n>> to a commit with the form \"P4@123456\". Later, I use the note to sync\n>> down the closest Perforce changelist matching the Git commit.\n>>\n>> I search for these notes by getting a list of revisions:\n>>\n>> git rev-list --max-count=1000\n>>\n>> I iterate those revisions and run git show and grep on each:\n>>\n>> git show -s --format=%N%n%s --show-notes=p4notes COMMIT\n>>\n>> For short runs, this isn't so bad. For longer runs of commits (I just\n>> walked through approximately 100), it takes a long time. Running 'git\n>> show' is costing me about 7/10 of second, presumably because I am on\n>> Windows.\n> Is there any particular reason you do that as two separate steps?\n> It would feel more natural, at least to me, to do something along\n> the lines of\n>\n> \tgit log --show-notes=p4notes -1000\n>\n>\nThanks for the reply.\n\nI did not make clear above that I want to stop looking when I find the \nfirst commit that has the note.\n\nIn the case of 'git log --show-notes=p4notes -1000', Git will process \nand hand me the log output for 1,000 commits. It is rare I need to walk \nthat deep. We saw 300 commits deep once on a long-lived branch that \nhadn't been merged in yet, but I'd be surprised to see 1,000.\n\nStill, it shows an arbitrary choice. Really, I want to say to Git: Walk \nup the history as far as you need to go from HEAD and return to me the \nfirst commit containing the text \"P4@\".\n\nAny other thoughts?\n\n-Josh\n"},{"id":"199682","messageId":"505CB337.706@workspacewhiz.com","threadId":"31619","inReplyTo":"m2d31fbedd.fsf@igel.home","subject":"Re: Quickly searching for a note","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2012-09-21T18:34:31Z","receivedAt":"2012-09-21T18:34:31Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":"----- Original Message -----\nFrom: Andreas Schwab\nDate: 9/21/2012 9:10 AM\n> Joshua Jensen writes:\n>\n>> Background: To tie Perforce changelists to Git commits, I add a note to a\n>> commit with the form \"P4@123456\". Later, I use the note to sync down the\n>> closest Perforce changelist matching the Git commit.\n>>\n>> I search for these notes by getting a list of revisions:\n>>\n>> git rev-list --max-count=1000\n>>\n>> I iterate those revisions and run git show and grep on each:\n>>\n>> git show -s --format=%N%n%s --show-notes=p4notes COMMIT\n> How about \"git grep P4@123456 notes/p4notes\"?\n>\n> Andreas.\n>\nThanks for the reply.\n\nI should have labeled the format above as \"P4@#######\". The numeric \npart will change. The \"P4@\" will not.\n\nSo, I run \"git grep P4@ notes/p4notes\". I get a bunch of responses. I \nneed the closest commit to HEAD that contains the P4@ text.\n\nAny ideas?\n\n-Josh\n"},{"id":"199691","messageId":"7vtxur3zxi.fsf@alter.siamese.dyndns.org","threadId":"31619","inReplyTo":"505CB21E.4040607@workspacewhiz.com","subject":"Re: Quickly searching for a note","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-21T20:04:41Z","receivedAt":"2012-09-21T20:04:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joshua Jensen writes:\n\n>> Is there any particular reason you do that as two separate steps?\n>> It would feel more natural, at least to me, to do something along\n>> the lines of\n>>\n>> \tgit log --show-notes=p4notes -1000\n>>\n>>\n> Thanks for the reply.\n>\n> I did not make clear above that I want to stop looking when I find the\n> first commit that has the note.\n>\n> In the case of 'git log --show-notes=p4notes -1000', Git will process\n> and hand me the log output for 1,000 commits. It is rare I need to\n> walk that deep.\n\nI simply matched it with your initial \"rev-list --max-count=1000\".\nThe \"log\" command pages and you can hit 'q' once you saw enough (in\nother words, you do not have to say -1000).\n"},{"id":"199694","messageId":"505CCD2A.8020003@workspacewhiz.com","threadId":"31619","inReplyTo":"7vtxur3zxi.fsf@alter.siamese.dyndns.org","subject":"Re: Quickly searching for a note","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2012-09-21T20:25:14Z","receivedAt":"2012-09-21T20:25:14Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":"----- Original Message -----\nFrom: Junio C Hamano\nDate: 9/21/2012 2:04 PM\n> Joshua Jensen writes:\n>\n>>> Is there any particular reason you do that as two separate steps?\n>>> It would feel more natural, at least to me, to do something along\n>>> the lines of\n>>>\n>>> \tgit log --show-notes=p4notes -1000\n>>>\n>>>\n>> Thanks for the reply.\n>>\n>> I did not make clear above that I want to stop looking when I find the\n>> first commit that has the note.\n>>\n>> In the case of 'git log --show-notes=p4notes -1000', Git will process\n>> and hand me the log output for 1,000 commits. It is rare I need to\n>> walk that deep.\n> I simply matched it with your initial \"rev-list --max-count=1000\".\n> The \"log\" command pages and you can hit 'q' once you saw enough (in\n> other words, you do not have to say -1000).\n>\nThis is run via script without user intervention. Presumably, Git will \ndo 1,000 commits of work when it may only need to do 1 or 5 or 10?\n\n-Josh\n"},{"id":"199697","messageId":"505CD2FA.80200@kdbg.org","threadId":"31619","inReplyTo":"505CCD2A.8020003@workspacewhiz.com","subject":"Re: Quickly searching for a note","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2012-09-21T20:50:02Z","receivedAt":"2012-09-21T20:50:02Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 21.09.2012 22:25, schrieb Joshua Jensen:\n> ----- Original Message -----\n> From: Junio C Hamano\n> Date: 9/21/2012 2:04 PM\n>> Joshua Jensen writes:\n>>\n>>>> Is there any particular reason you do that as two separate steps?\n>>>> It would feel more natural, at least to me, to do something along\n>>>> the lines of\n>>>>\n>>>> git log --show-notes=p4notes -1000\n>>>>\n>>>>\n>>> Thanks for the reply.\n>>>\n>>> I did not make clear above that I want to stop looking when I find the\n>>> first commit that has the note.\n>>>\n>>> In the case of 'git log --show-notes=p4notes -1000', Git will process\n>>> and hand me the log output for 1,000 commits. It is rare I need to\n>>> walk that deep.\n>> I simply matched it with your initial \"rev-list --max-count=1000\".\n>> The \"log\" command pages and you can hit 'q' once you saw enough (in\n>> other words, you do not have to say -1000).\n>>\n> This is run via script without user intervention. Presumably, Git will\n> do 1,000 commits of work when it may only need to do 1 or 5 or 10?\n\nThe trick is to pipe 'git log' output into another process that reads no\nmore than it needs and exits. Then 'git log' dies from SIGPIPE before it\nprocessed all 1000 commits because its down-stream has gone away.\n\nFor example:\n\n git log --show-notes=p4notes -1000 |\n sed -n -e '/^commit /h' -e '/P4@/{H;g;p;q}'\n\n(The pipeline keeps track of the most recent 'commit' line, and when it\nfinds the 'P4@' it prints the most recent 'commit' line followed by the\n'P4@' line.)\n\n-- Hannes\n"},{"id":"199703","messageId":"505CD7D0.2000505@workspacewhiz.com","threadId":"31619","inReplyTo":"505CD2FA.80200@kdbg.org","subject":"Re: Quickly searching for a note","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2012-09-21T21:10:40Z","receivedAt":"2012-09-21T21:10:40Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":"----- Original Message -----\nFrom: Johannes Sixt\nDate: 9/21/2012 2:50 PM\n> The trick is to pipe 'git log' output into another process that reads no\n> more than it needs and exits. Then 'git log' dies from SIGPIPE before it\n> processed all 1000 commits because its down-stream has gone away.\n>\n> For example:\n>\n> git log --show-notes=p4notes -1000 |\n> sed -n -e '/^commit /h' -e '/P4@/{H;g;p;q}'\n>\n> (The pipeline keeps track of the most recent 'commit' line, and when it\n> finds the 'P4@' it prints the most recent 'commit' line followed by the\n> 'P4@' line.)\n>\nGot it. I'll try that out now.\n\n-Josh\n"},{"id":"199712","messageId":"20120921233723.GA29433@sigill.intra.peff.net","threadId":"31619","inReplyTo":"505CD7D0.2000505@workspacewhiz.com","subject":"Re: Quickly searching for a note","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-09-21T23:37:24Z","receivedAt":"2012-09-21T23:37:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 21, 2012 at 03:10:40PM -0600, Joshua Jensen wrote:\n\n> ----- Original Message -----\n> From: Johannes Sixt\n> Date: 9/21/2012 2:50 PM\n> >The trick is to pipe 'git log' output into another process that reads no\n> >more than it needs and exits. Then 'git log' dies from SIGPIPE before it\n> >processed all 1000 commits because its down-stream has gone away.\n> >\n> >For example:\n> >\n> > git log --show-notes=p4notes -1000 |\n> > sed -n -e '/^commit /h' -e '/P4@/{H;g;p;q}'\n> >\n> >(The pipeline keeps track of the most recent 'commit' line, and when it\n> >finds the 'P4@' it prints the most recent 'commit' line followed by the\n> >'P4@' line.)\n> >\n> Got it. I'll try that out now.\n\nI think people have provided sane techniques for doing this with a\npipeline. But there is really no reason not to have --grep-notes, just\nas we have --grep. It's simply that nobody has implemented it yet (and\nnobody is working on it as far as I know). It would actually be a fairly\nsimple feature to add if somebody wanted to get their feet wet with git.\n\n-Peff\n"},{"id":"199714","messageId":"7v7grn3pfo.fsf@alter.siamese.dyndns.org","threadId":"31619","inReplyTo":"20120921233723.GA29433@sigill.intra.peff.net","subject":"Re: Quickly searching for a note","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-21T23:51:23Z","receivedAt":"2012-09-21T23:51:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I think people have provided sane techniques for doing this with a\n> pipeline. But there is really no reason not to have --grep-notes, just\n> as we have --grep. It's simply that nobody has implemented it yet (and\n> nobody is working on it as far as I know). It would actually be a fairly\n> simple feature to add if somebody wanted to get their feet wet with git.\n\nI agree that the implementation will be simple once you figure out\nwhat the sensible semantics and external interfaces are. The latter\nis not that simple and certainly not something for newbies to solve\non their own. That is why I didn't mention it.\n\nBut now you brought it up, here are a few thinking-points as a\nstarter:\n\n - Wouldn't it be more intuitive to just let the normal \"--grep\" to\n also hit what \"--show-notes\" would add to the output? Does it\n really add value to the end user experience to add a separate\n \"--grep-notes=P4[0-9]*\" option, even though it would give you\n more flexibility?\n\n Not having thought things through thorouly, I still answer this\n question both ways myself and what the right user experience\n should look like.\n\n - Do we want to be limited to one notes tree? Would it make sense\n to show notes from the usual commit notes but use different notes\n tree for the sole purpose of restricting visibility? If we\n wanted to allow that for people who want flexibility, but still\n want to use only one and the same by default, what should the\n command line options look like?\n\n - Would it be common to say \"I want commits with _any_ notes from\n this notes tree\"? Having to say \"--grep-notes=.\" for such a\n simple task, if it is common, feels a bit clunky.\n"},{"id":"199724","messageId":"7v1uhu4q4f.fsf@alter.siamese.dyndns.org","threadId":"31619","inReplyTo":"505C7C80.3000700@workspacewhiz.com","subject":"Re: Quickly searching for a note","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-22T04:51:12Z","receivedAt":"2012-09-22T04:51:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joshua Jensen writes:\n\n> Background: To tie Perforce changelists to Git commits, I add a note\n> to a commit with the form \"P4@123456\". Later, I use the note to sync\n> down the closest Perforce changelist matching the Git commit.\n\nI noticed that nobody brought this up, but probably it should not be\nleft unsaid, so...\n\nFor annotating commits with additional pieces of data, notes is a\nreasonable mechanism, but the user should be aware that it is\nheavily geared towards one-way mapping. When you have a commit and\nwant to know something about it, it will give you the associated\ninformation reasonably efficiently.\n\nBut it is not a good mechanism for retrieval if the primary way you\nuse the stored information is to go from the associated information\nto find the commit that has that note attached to it. Your usage\npattern that triggered this thread may fall into that category.\n\nIt may still be a reasonable mechanism to use notes to exchange the\ninformation across repositories, but if your application relies\nheavily on mapping the information in the opposite way, you may want\nto maintain a local cache of the reverse mapping in a more efficient\nfashion. For example, every time your notes tree is updated, you\ncan loop over \"git notes list\" output and register the contents of\nthe blob object that annotates each commit as the key and the commit\nobject name as the value to a repository-local sqlite database or\nsomething (and depending on the nature of the frequent query, have\nefficient index on the key).\n\nHaving mentioned an external database as the most generic approach,\nI suspect that one important way to use notes is to associate\ncommits with some other (presumably unique) ID to interface with the\nexternal world. For example, I maintain \"amlog\" notes to record the\noriginal message-ID for each commit that resulted from \"git am\".\nThe primary use of this is to find the message-ID for a commit that\nwas made some time ago and later found to be questionable, so that I\ncan find the relevant discussion thread, but the information could\nbe used to see if a given message I see in the mail archive has been\nalready applied, and this needs a fast reverse mapping.\n\nIt actually is fairly trivial to maintain both forward and reverse\nmapping for this kind of use case. For example, your gateway that\nsyncs from Perforce may currently be doing something like this at\nthe end of it:\n\n git notes --ref p4notes add -m \"P4@$p4_change_id\" HEAD\n\nto give a quick mapping the commit object name of the resulting\ncommit (in HEAD) to \"P4@123456\".\n\nThis is stored as a mapping from the object name of HEAD to the\nobject name of a blob whose contents is \"P4@123456\" You can see it\nin action with\n\n $ git notes --ref p4notes list HEAD\n\nthat gives the blob object name that stores the note for the HEAD.\n\nNow, there is _no_ reason why you cannot attach notes to these blob\nobjects. For example, your \"Perforce to Git\" gateway can end with\nsomething like this instead:\n\n HEAD=$(git rev-parse --verify HEAD)\n git notes --ref p4notes add -m \"P4@$p4_change_id\" $HEAD\n noteblob=$(git notes --ref p4notes list $HEAD)\n git notes --ref p4notes add -m \"$HEAD\" $noteblob\n\nThen when you want to map P4@123456 to Git commit, you could\n\n $ noteblob=$(echo P4@123456 | git hash-object --stdin)\n $ git notes --ref p4notes show $noteblob\n\nto see the commit object name that is associated with that notes.\nOf course, the same notes tree holds the forward mapping as before,\nso \n\n $ git notes --ref p4notes show HEAD\n\nwill give you the \"P4@123456\".\n\nWe may want to support such a reverse mapping natively so that\n\"notes rewrite\" logic maintains the mapping in both direction.\n\nI've CC'ed people who may want to be involved in further design\nwork.\n"},{"id":"199740","messageId":"505DE30B.2000805@drmicha.warpmail.net","threadId":"31619","inReplyTo":"7v7grn3pfo.fsf@alter.siamese.dyndns.org","subject":"Re: Quickly searching for a note","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2012-09-22T16:10:51Z","receivedAt":"2012-09-22T16:10:51Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Junio C Hamano venit, vidit, dixit 22.09.2012 01:51:\n> Jeff King writes:\n> \n>> I think people have provided sane techniques for doing this with a\n>> pipeline. But there is really no reason not to have --grep-notes, just\n>> as we have --grep. It's simply that nobody has implemented it yet (and\n>> nobody is working on it as far as I know). It would actually be a fairly\n>> simple feature to add if somebody wanted to get their feet wet with git.\n> \n> I agree that the implementation will be simple once you figure out\n> what the sensible semantics and external interfaces are. The latter\n> is not that simple and certainly not something for newbies to solve\n> on their own. That is why I didn't mention it.\n> \n> But now you brought it up, here are a few thinking-points as a\n> starter:\n> \n> - Wouldn't it be more intuitive to just let the normal \"--grep\" to\n> also hit what \"--show-notes\" would add to the output? Does it\n> really add value to the end user experience to add a separate\n> \"--grep-notes=P4[0-9]*\" option, even though it would give you\n> more flexibility?\n> \n> Not having thought things through thorouly, I still answer this\n> question both ways myself and what the right user experience\n> should look like.\n> \n> - Do we want to be limited to one notes tree? Would it make sense\n> to show notes from the usual commit notes but use different notes\n> tree for the sole purpose of restricting visibility? If we\n> wanted to allow that for people who want flexibility, but still\n> want to use only one and the same by default, what should the\n> command line options look like?\n> \n> - Would it be common to say \"I want commits with _any_ notes from\n> this notes tree\"? Having to say \"--grep-notes=.\" for such a\n> simple task, if it is common, feels a bit clunky.\n> \n\nOn my mental scratch pad (yeah, that's where the bald spots are) I have\nthe following more general idea to enhance the revision parser:\n\n--limit-run=