{"thread":{"id":"41899","subject":"What is an efficient way to get all blobs / trees that have notes attached?","startedAt":"2016-04-01T10:51:52Z","lastAt":"2016-04-04T17:33:38Z","messageCount":5,"participants":["Sebastian Schuberth","Johan Herland"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"282445","messageId":"ndljs8$vj3$1@ger.gmane.org","threadId":"41899","inReplyTo":null,"subject":"What is an efficient way to get all blobs / trees that have notes attached?","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2016-04-01T10:51:52Z","receivedAt":"2016-04-01T10:51:52Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"Hi,\n\nI'm curious whether there's a more efficient way to get a list of blobs \n/ trees (and their names) that have notes attached than doing this:\n\n1) Get all notes refs I'm interested in (git-for-each-ref).\n\n2) For each notes ref, get the list of notes (git-notes list) and store \nthem in a hash table that maps object hashes to notes.\n\n3) Recursively list all blobs / trees (git-ls-tree) and look whether an \nobject's hash is conatined in our table to get its notes.\n\nIn particular 3) could be expensive for repos with a lot of files as \nwe're looking at all of them just to see whether they have notes attached.\n\nRegards,\nSebastian\n"},{"id":"282455","messageId":"CALKQrgdytYJtMTBHXbcRQ_iT5rWakZCxxqRW1rpLsPiSejud-Q@mail.gmail.com","threadId":"41899","inReplyTo":"ndljs8$vj3$1@ger.gmane.org","subject":"Re: What is an efficient way to get all blobs / trees that have notes attached?","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2016-04-01T12:16:33Z","receivedAt":"2016-04-01T12:16:33Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Fri, Apr 1, 2016 at 12:51 PM, Sebastian Schuberth\n wrote:\n> Hi,\n>\n> I'm curious whether there's a more efficient way to get a list of blobs /\n> trees (and their names) that have notes attached than doing this:\n>\n> 1) Get all notes refs I'm interested in (git-for-each-ref).\n>\n> 2) For each notes ref, get the list of notes (git-notes list) and store them\n> in a hash table that maps object hashes to notes.\n>\n> 3) Recursively list all blobs / trees (git-ls-tree) and look whether an\n> object's hash is conatined in our table to get its notes.\n>\n> In particular 3) could be expensive for repos with a lot of files as we're\n> looking at all of them just to see whether they have notes attached.\n\nIn (3), why would you need to search through _all_ blobs/trees? Would\nit not be cheaper to simply query the object type of each annotated\nobject from (2)? I.e. something like:\n\nfor notes_ref in $(git for-each-ref refs/notes | cut -c 49-)\ndo\n echo \"--- $notes_ref ---\"\n for annotated_obj in $(git notes --ref=$notes_ref list | cut -c 41-)\n do\n type=$(git cat-file -t \"$annotated_obj\")\n if test \"$type\" != \"commit\"\n then\n echo \"$annotated_obj: $type\"\n fi\n done\ndone\n\nCan probably be made even faster by using the --batch option to cat-file...\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"282453","messageId":"CALKQrgdONBZOgw5--x35Gq2+2fze=mm1t_162UgC0W12JGg6Hw@mail.gmail.com","threadId":"41899","inReplyTo":"CALKQrgdytYJtMTBHXbcRQ_iT5rWakZCxxqRW1rpLsPiSejud-Q@mail.gmail.com","subject":"Re: What is an efficient way to get all blobs / trees that have notes attached?","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2016-04-01T12:23:32Z","receivedAt":"2016-04-01T12:23:32Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Fri, Apr 1, 2016 at 2:16 PM, Johan Herland wrote:\n> for notes_ref in $(git for-each-ref refs/notes | cut -c 49-)\n> do\n> echo \"--- $notes_ref ---\"\n> for annotated_obj in $(git notes --ref=$notes_ref list | cut -c 41-)\n> do\n> type=$(git cat-file -t \"$annotated_obj\")\n> if test \"$type\" != \"commit\"\n> then\n> echo \"$annotated_obj: $type\"\n> fi\n> done\n> done\n>\n> Can probably be made even faster by using the --batch option to cat-file...\n\nFor example:\n\nfor notes_ref in $(git for-each-ref refs/notes | cut -c 49-)\ndo\n echo \"--- $notes_ref ---\"\n git notes --ref=$notes_ref list | cut -c 42- | git cat-file\n--batch-check=\"%(objecttype) %(objectname)\" | grep\n'^\\(\\(blob\\)\\|\\(tree\\)\\) '\ndone\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"282604","messageId":"CAHGBnuNQYrg70h+mjzDpYyKhHiR6tWx9j989KLO3bW-jC7XmZw@mail.gmail.com","threadId":"41899","inReplyTo":"CALKQrgdytYJtMTBHXbcRQ_iT5rWakZCxxqRW1rpLsPiSejud-Q@mail.gmail.com","subject":"Re: What is an efficient way to get all blobs / trees that have notes attached?","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2016-04-04T07:46:40Z","receivedAt":"2016-04-04T07:46:40Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On Fri, Apr 1, 2016 at 2:16 PM, Johan Herland wrote:\n\n>> 3) Recursively list all blobs / trees (git-ls-tree) and look whether an\n>> object's hash is conatined in our table to get its notes.\n>>\n>> In particular 3) could be expensive for repos with a lot of files as we're\n>> looking at all of them just to see whether they have notes attached.\n>\n> In (3), why would you need to search through _all_ blobs/trees? Would\n> it not be cheaper to simply query the object type of each annotated\n> object from (2)? I.e. something like:\n>\n> for notes_ref in $(git for-each-ref refs/notes | cut -c 49-)\n> do\n> echo \"--- $notes_ref ---\"\n> for annotated_obj in $(git notes --ref=$notes_ref list | cut -c 41-)\n> do\n> type=$(git cat-file -t \"$annotated_obj\")\n> if test \"$type\" != \"commit\"\n> then\n> echo \"$annotated_obj: $type\"\n> fi\n> done\n> done\n\nThanks for the idea. The problem is that I do want to list the notes\nby path of the object they belong to. As a blob could potentially\nbelong to more than one path (copies of files in the repo), I do not\nsee another way of getting that information other than iterating over\nall blobs and checking what path(s) they belong to.\n\n-- \nSebastian Schuberth\n"},{"id":"282627","messageId":"CALKQrgctNZU79rreN1RWEd+KOMNMmMgTce_wV4iCKyNw+TwmWg@mail.gmail.com","threadId":"41899","inReplyTo":"CAHGBnuNQYrg70h+mjzDpYyKhHiR6tWx9j989KLO3bW-jC7XmZw@mail.gmail.com","subject":"Re: What is an efficient way to get all blobs / trees that have notes attached?","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2016-04-04T17:33:38Z","receivedAt":"2016-04-04T17:33:38Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Mon, Apr 4, 2016 at 9:46 AM, Sebastian Schuberth\n wrote:\n> On Fri, Apr 1, 2016 at 2:16 PM, Johan Herland wrote:\n>>> 3) Recursively list all blobs / trees (git-ls-tree) and look whether an\n>>> object's hash is conatined in our table to get its notes.\n>>>\n>>> In particular 3) could be expensive for repos with a lot of files as we're\n>>> looking at all of them just to see whether they have notes attached.\n>>\n>> In (3), why would you need to search through _all_ blobs/trees? Would\n>> it not be cheaper to simply query the object type of each annotated\n>> object from (2)? I.e. something like:\n>>\n>> for notes_ref in $(git for-each-ref refs/notes | cut -c 49-)\n>> do\n>> echo \"--- $notes_ref ---\"\n>> for annotated_obj in $(git notes --ref=$notes_ref list | cut -c 41-)\n>> do\n>> type=$(git cat-file -t \"$annotated_obj\")\n>> if test \"$type\" != \"commit\"\n>> then\n>> echo \"$annotated_obj: $type\"\n>> fi\n>> done\n>> done\n>\n> Thanks for the idea. The problem is that I do want to list the notes\n> by path of the object they belong to. As a blob could potentially\n> belong to more than one path (copies of files in the repo), I do not\n> see another way of getting that information other than iterating over\n> all blobs and checking what path(s) they belong to.\n\nTrue; fundamentally what you want is a blob/tree ID -> path(s) mapping,\nwhich is an independent problem, unrelated to to the initial notes lookup.\n\nI don't know of a solution faster than the brute-force search you already\nsketched. If this lookup is important to your use case, you could consider\nbuilding/caching the required mapping when the notes are added in the\nfirst place, but I don't know if that is possible in your scenario...\n\n\n...Johan\n\n> --\n> Sebastian Schuberth\n\n-- \nJohan Herland, \nwww.herland.net\n"}]}