{"thread":{"id":"38052","subject":"\"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","startedAt":"2014-11-26T00:42:42Z","lastAt":"2014-11-26T12:30:57Z","messageCount":10,"participants":["Mike Hommey","Jeff King","Johan Herland"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"252556","messageId":"20141126004242.GA13915@glandium.org","threadId":"38052","inReplyTo":null,"subject":"\"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-11-26T00:42:42Z","receivedAt":"2014-11-26T00:42:42Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Hi,\n\nI have a note tree with a bit more than 200k notes.\n\n$ time git notes --ref foo show $sha1 > /dev/null\nreal    0m0.147s\nuser    0m0.136s\nsys     0m0.008s\n\nThat's a lot of time, especially when you have a script that does that\non a fair amount of sha1s.\n\nNow, the interesting thing is this:\n\n$ time git ls-tree -r refs/notes/foo $sha1 ${sha1:0:2}/${sha1:2:38} ${sha1:0:2}/${sha1:2:2}/${sha1:4:36} > /dev/null\nreal    0m0.006s\nuser    0m0.008s\nsys     0m0.000s\n\n$ time git cat-file blob $objsha1 > /dev/null\nreal    0m0.002s\nuser    0m0.000s\nsys     0m0.000s\n\nAnd even better:\n\n$ wc -l /tmp/note\n39 /tmp/note\n$ time git ls-tree refs/notes/foo $(awk '{print $1, substr($1,1,2) \"/\" substr($1,3), substr($1,1,2) \"/\" substr($1,3,2) \"/\" substr($1,5)}' /tmp/note) | awk '{print $3}' | git cat-file --batch > /dev/null\nreal    0m0.035s\nuser    0m0.028s\nsys     0m0.004s\n\nReading 39 notes with ls-tree and cat-file --batch takes less time than\nusing git notes show for only one of them...\n\n(and reading all 39 of them with git notes show takes 5.5s)\n\nMike\n"},{"id":"252558","messageId":"20141126010051.GA29830@peff.net","threadId":"38052","inReplyTo":"20141126004242.GA13915@glandium.org","subject":"Re: \"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-11-26T01:00:51Z","receivedAt":"2014-11-26T01:00:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 26, 2014 at 09:42:42AM +0900, Mike Hommey wrote:\n\n> I have a note tree with a bit more than 200k notes.\n>\n> $ time git notes --ref foo show $sha1 > /dev/null\n> real    0m0.147s\n> user    0m0.136s\n> sys     0m0.008s\n> \n> That's a lot of time, especially when you have a script that does that\n> on a fair amount of sha1s.\n\nIIRC, the notes code populates an in-memory data structure, which gives\nfaster per-commit lookup at the cost of some setup time. Obviously for a\nsingle lookup, that's going to be a bad tradeoff (but it does make sense\nfor \"git log --notes\"). I don't know offhand how difficult it would be\nto tune the data structure differently (or avoid it altogether) if we\nknow ahead of time we are only going to do a small number of lookups.\nBut Johan (cc'd) might.\n\n-Peff\n"},{"id":"252560","messageId":"20141126012448.GA11183@peff.net","threadId":"38052","inReplyTo":"20141126010051.GA29830@peff.net","subject":"Re: \"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-11-26T01:24:49Z","receivedAt":"2014-11-26T01:24:49Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 25, 2014 at 08:00:51PM -0500, Jeff King wrote:\n\n> On Wed, Nov 26, 2014 at 09:42:42AM +0900, Mike Hommey wrote:\n> \n> > I have a note tree with a bit more than 200k notes.\n> >\n> > $ time git notes --ref foo show $sha1 > /dev/null\n> > real    0m0.147s\n> > user    0m0.136s\n> > sys     0m0.008s\n> > \n> > That's a lot of time, especially when you have a script that does that\n> > on a fair amount of sha1s.\n> \n> IIRC, the notes code populates an in-memory data structure, which gives\n> faster per-commit lookup at the cost of some setup time. Obviously for a\n> single lookup, that's going to be a bad tradeoff (but it does make sense\n> for \"git log --notes\"). I don't know offhand how difficult it would be\n> to tune the data structure differently (or avoid it altogether) if we\n> know ahead of time we are only going to do a small number of lookups.\n> But Johan (cc'd) might.\n\nOne other question: how were your notes created?\n\nI tried to replicate your setup by creating one note per commit in\nlinux.git (over 400k notes total). I did it with one big mktree,\ncreating a single top-level notes tree. Doing a single \"git notes show\"\nlookup on the tree was something like 800ms.\n\nHowever, this is not what trees created by git-notes look like. It\nshards the object sha1s into subtrees (1a/2b/{36}), and I think does so\ndynamically in a way that keeps each individual tree size low. The\nin-memory data structure then only \"faults in\" tree objects as they are\nneeded. So a single lookup should only hit a small part of the total\ntree.\n\nDoing a single \"git notes edit HEAD\" in my case caused the notes code to\nwrite the result using its sharding algorithm. Subsequent \"git notes\nshow\" invocations were only 14ms.\n\nDid you use something besides git-notes to create the tree? From your\nexamples, it looks like you were accounting for the sharding during\nlookup, so maybe this is leading in the wrong direction (but if so, I\ncould not reproduce your times at all even with a much larger case).\n\n-Peff\n"},{"id":"252563","messageId":"20141126013456.GA13622@peff.net","threadId":"38052","inReplyTo":"20141126012448.GA11183@peff.net","subject":"Re: \"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-11-26T01:34:57Z","receivedAt":"2014-11-26T01:34:57Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 25, 2014 at 08:24:48PM -0500, Jeff King wrote:\n\n> However, this is not what trees created by git-notes look like. It\n> shards the object sha1s into subtrees (1a/2b/{36}), and I think does so\n> dynamically in a way that keeps each individual tree size low. The\n> in-memory data structure then only \"faults in\" tree objects as they are\n> needed. So a single lookup should only hit a small part of the total\n> tree.\n> \n> Doing a single \"git notes edit HEAD\" in my case caused the notes code to\n> write the result using its sharding algorithm. Subsequent \"git notes\n> show\" invocations were only 14ms.\n> \n> Did you use something besides git-notes to create the tree? From your\n> examples, it looks like you were accounting for the sharding during\n> lookup, so maybe this is leading in the wrong direction (but if so, I\n> could not reproduce your times at all even with a much larger case).\n\nHmph. Having just written all that, I looked at your example again, and\nyou are running \"git ls-tree -r\", which would read the whole tree\nanyway. So \"git notes\" should be _faster_ for a single lookup.\n\nSomething weird is definitely going on. Can you use \"strace\" or \"perf\"\nto get a sense of where the time is going? Has your repository been\npacked recently?\n\n-Peff\n"},{"id":"252565","messageId":"20141126022553.GA16199@glandium.org","threadId":"38052","inReplyTo":"20141126012448.GA11183@peff.net","subject":"Re: \"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-11-26T02:25:53Z","receivedAt":"2014-11-26T02:25:53Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Nov 25, 2014 at 08:24:49PM -0500, Jeff King wrote:\n> On Tue, Nov 25, 2014 at 08:00:51PM -0500, Jeff King wrote:\n> \n> > On Wed, Nov 26, 2014 at 09:42:42AM +0900, Mike Hommey wrote:\n> > \n> > > I have a note tree with a bit more than 200k notes.\n> > >\n> > > $ time git notes --ref foo show $sha1 > /dev/null\n> > > real    0m0.147s\n> > > user    0m0.136s\n> > > sys     0m0.008s\n> > > \n> > > That's a lot of time, especially when you have a script that does that\n> > > on a fair amount of sha1s.\n> > \n> > IIRC, the notes code populates an in-memory data structure, which gives\n> > faster per-commit lookup at the cost of some setup time. Obviously for a\n> > single lookup, that's going to be a bad tradeoff (but it does make sense\n> > for \"git log --notes\"). I don't know offhand how difficult it would be\n> > to tune the data structure differently (or avoid it altogether) if we\n> > know ahead of time we are only going to do a small number of lookups.\n> > But Johan (cc'd) might.\n> \n> One other question: how were your notes created?\n> \n> I tried to replicate your setup by creating one note per commit in\n> linux.git (over 400k notes total). I did it with one big mktree,\n> creating a single top-level notes tree. Doing a single \"git notes show\"\n> lookup on the tree was something like 800ms.\n> \n> However, this is not what trees created by git-notes look like. It\n> shards the object sha1s into subtrees (1a/2b/{36}), and I think does so\n> dynamically in a way that keeps each individual tree size low. The\n> in-memory data structure then only \"faults in\" tree objects as they are\n> needed. So a single lookup should only hit a small part of the total\n> tree.\n> \n> Doing a single \"git notes edit HEAD\" in my case caused the notes code to\n> write the result using its sharding algorithm. Subsequent \"git notes\n> show\" invocations were only 14ms.\n> \n> Did you use something besides git-notes to create the tree? From your\n> examples, it looks like you were accounting for the sharding during\n> lookup, so maybe this is leading in the wrong direction (but if so, I\n> could not reproduce your times at all even with a much larger case).\n\nSo... this is interesting. I happen to have recreated the notes tree\n\"manually\", and now each git notes show takes under 10ms.\n\nNow, looking at the notes tree reflog, I see that at some point, some\nnotes were added at the top-level of the tree, without being nested,\nwhich is strange.\n\nAnd it looks like it's related to how I've been adding them, through\ngit-fast-import. I was using notemodify commands, and was using the\nfilemodify command to load the previous notes tree instead of using the\nfrom command because I don't care about keeping the notes history.\nSo fast-import was actually filling the notes tree as if it were\nstarting over with whatever new notes were added with notemodify (which,\nin a case where there were many, it filled with one level of\nindirection)\n\nI'm not sure this is a case worth fixing in fast-import. I can easily\nwork around it.\n\nMike\n"},{"id":"252566","messageId":"20141126023039.GA19401@glandium.org","threadId":"38052","inReplyTo":"20141126013456.GA13622@peff.net","subject":"Re: \"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-11-26T02:30:39Z","receivedAt":"2014-11-26T02:30:39Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Nov 25, 2014 at 08:34:57PM -0500, Jeff King wrote:\n> On Tue, Nov 25, 2014 at 08:24:48PM -0500, Jeff King wrote:\n> \n> > However, this is not what trees created by git-notes look like. It\n> > shards the object sha1s into subtrees (1a/2b/{36}), and I think does so\n> > dynamically in a way that keeps each individual tree size low. The\n> > in-memory data structure then only \"faults in\" tree objects as they are\n> > needed. So a single lookup should only hit a small part of the total\n> > tree.\n> > \n> > Doing a single \"git notes edit HEAD\" in my case caused the notes code to\n> > write the result using its sharding algorithm. Subsequent \"git notes\n> > show\" invocations were only 14ms.\n> > \n> > Did you use something besides git-notes to create the tree? From your\n> > examples, it looks like you were accounting for the sharding during\n> > lookup, so maybe this is leading in the wrong direction (but if so, I\n> > could not reproduce your times at all even with a much larger case).\n> \n> Hmph. Having just written all that, I looked at your example again, and\n> you are running \"git ls-tree -r\", which would read the whole tree\n> anyway. So \"git notes\" should be _faster_ for a single lookup.\n\nThe -r actually doesn't matter, since what's being listed is a blob, not\na tree, so there is no recursion.\n\nMike\n"},{"id":"252568","messageId":"20141126044642.GA15252@peff.net","threadId":"38052","inReplyTo":"20141126022553.GA16199@glandium.org","subject":"Re: \"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-11-26T04:46:42Z","receivedAt":"2014-11-26T04:46:42Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 26, 2014 at 11:25:53AM +0900, Mike Hommey wrote:\n\n> Now, looking at the notes tree reflog, I see that at some point, some\n> notes were added at the top-level of the tree, without being nested,\n> which is strange.\n\nThat's somewhat expected. The fanout is dynamic based on the number of\nnotes, so early on I think some notes may be found at the top of the tree.\n\n> And it looks like it's related to how I've been adding them, through\n> git-fast-import. I was using notemodify commands, and was using the\n> filemodify command to load the previous notes tree instead of using the\n> from command because I don't care about keeping the notes history.\n> So fast-import was actually filling the notes tree as if it were\n> starting over with whatever new notes were added with notemodify (which,\n> in a case where there were many, it filled with one level of\n> indirection)\n\nAh, that sort of makes sense. This confused the code to adjust the\nfanout, because we track \"number of notes\" independently of \"number of\nfiles\" (even though they are really the same thing in a notes tree).\n\n> I'm not sure this is a case worth fixing in fast-import. I can easily\n> work around it.\n\nYeah. Probably fast-import could be smarter here, but I think ultimately\nit makes sense to stick to using the note commands. I think what you\nwant is a version of \"from\" that takes an existing tree (and number of\nnotes!) from a commit, but does not add it as a parent. AFAIK,\nfast-import doesn't have a way to do that.\n\nProbably the simplest thing is to build it with history via fast-import,\nand then just truncate the history at the end with:\n\n  commit=$(echo \"final notes tree\" | git commit-tree refs/notes/foo^{tree})\n  git update-ref refs/notes/foo $commit\n\n-Peff\n"},{"id":"252569","messageId":"20141126044935.GB15252@peff.net","threadId":"38052","inReplyTo":"20141126023039.GA19401@glandium.org","subject":"Re: \"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-11-26T04:49:35Z","receivedAt":"2014-11-26T04:49:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 26, 2014 at 11:30:39AM +0900, Mike Hommey wrote:\n\n> > Hmph. Having just written all that, I looked at your example again, and\n> > you are running \"git ls-tree -r\", which would read the whole tree\n> > anyway. So \"git notes\" should be _faster_ for a single lookup.\n> \n> The -r actually doesn't matter, since what's being listed is a blob, not\n> a tree, so there is no recursion.\n\nAh, right. I should have looked more carefully. I took the \"-r\" and the\npatterns to mean \"recursively list the tree, and I will grep for these\nelements\". But you were actually generating a set of pathspecs, which\ngit could then use to limit some of the walk.\n\n-Peff\n"},{"id":"252578","messageId":"CALKQrgftttSpuw8kc+jC6E5RBet39wHKy3670Z5iG=KQSmrCAw@mail.gmail.com","threadId":"38052","inReplyTo":"20141126044642.GA15252@peff.net","subject":"Re: \"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2014-11-26T11:46:20Z","receivedAt":"2014-11-26T11:46:20Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"(First of all, thanks to both for great investigation and analysis)\n\nOn Wed, Nov 26, 2014 at 5:46 AM, Jeff King <peff@peff.net> wrote:\n> On Wed, Nov 26, 2014 at 11:25:53AM +0900, Mike Hommey wrote:\n>\n>> Now, looking at the notes tree reflog, I see that at some point, some\n>> notes were added at the top-level of the tree, without being nested,\n>> which is strange.\n>\n> That's somewhat expected. The fanout is dynamic based on the number of\n> notes, so early on I think some notes may be found at the top of the tree.\n>\n>> And it looks like it's related to how I've been adding them, through\n>> git-fast-import. I was using notemodify commands, and was using the\n>> filemodify command to load the previous notes tree instead of using the\n>> from command because I don't care about keeping the notes history.\n\nI'd very much like to see this fast-import stream (or a script\ngenerating it). I'm assuming that it roughly follows along these lines:\n\n - Start a new commit from a clean slate (no 'from')\n\n - Do a single filemodify to \"load\" the previous notes tree.\n   Exactly what does this filemodify command look like? I'm\n   guessing you're using the root tree object from the previous\n   notes tree as <dataref>, and an empty <path>, i.e.:\n\n     M 040000 $previous_notes_tree_root_sha1 \\n\n\n - Do a series of notemodify commands for the notes being added in\n   this commit.\n\n - End of commit.\n\nBrowsing the fast-import code, I believe I can confirm some of the\nbehavior you're seeing. Your commit starts from a clean slate\n(with branch->num_notes == 0). The initial filemodify command loads\nthe previous notes tree, but does not adjust branch->num_notes.\nThe subsequent notemodify commands each increment branch->num_notes.\nThen, when wrapping up the commit, branch->num_notes is used to\ncalculate the new fanout (the fast-import code does a simple\ndivide-by-256 heuristic). If you happened to have added more than\n256 notes in this commit, you get a 2/38 fanout, otherwise you get a\nflat notes tree. This is obviously incorrect, because the notes that\nwere \"preloaded\" by the filemodify command are not being accounted in\nthe fanout calculation. Thus, you end up with a seriously suboptimal\n(notes) tree structure, which kills your lookup times.\n\nHowever, at the start of note_change_n() (which handles the \"notemodify\"\ncommands), there is some logic to deal with this. The initial comment\nthere suggests that it was written for the case when the commit uses\n\"from refs/notes/foo\" to add notes to an existing notes tree, but I don't\nreally see why it should not also apply to the case where the notemodify\ncommands are preceded by a filemodify to \"preload\" the notes tree.\n\nFurthermore, It is not obvious why this logic (that counts the number of\nexiting notes before we add the first note) does not ALREADY trigger in\nyour case (both b->num_notes and *old_fanout should be == 0). However, I\nwonder if the \"counting mode\" is in fact invoked, but that there is some\nnon-obvious interplay with how \"struct tree_entry\"s are prepared by your\nuse of filemodify:\n\nWhen using 'from ...' to preload b->branch_tree, the code in parse_from()\nprepares a b->branch_tree tree_entry looking like this:\n\n  .versions[1].sha1 = $some_sha1\n  .tree == NULL\n\nFollowing this, parse_new_commit() goes on to call load_branch(b), which\nAFAICS, populates the .tree:\n\n  .versions[1].sha1 = $some_sha1\n  .tree == <tree structure loaded from $some_sha1>\n\nHowever, when you use the filemodify command to replace the root tree\nobject, we end up calling\n\n  tree_content_replace(&b->branch_tree, sha1, mode, NULL);\n\nnear the end of file_change_m(), which causes the b->branch_tree tree_entry\nto look like this:\n\n  .versions[1].sha1 = $some_sha1\n  .tree == NULL\n\n(i.e. same as parse_from()), BUT there is never a follow-up call to\nload_branch(b).\n\nSo, at the time we handle the first notemodify command, the\nb->branch_tree.tree entry is either non-NULL (in the case of 'from ...'),\nor NULL (in the case of your \"preloading\" filemodify). And when we start\nthe \"counting mode\" of change_note_fanout(), the for loop in\ndo_change_note_fanout() depends on the .tree entry being already loaded.\n\nA possible fix would be to ensure .tree is loaded before starting the for\nloop in do_change_note_fanout(). But it's been too long since I thoroughly\ngrokked the fast-import code to say if simply inserting a\n\n  load_tree(&b->branch_tree);\n\nbefore the for loop is Safe(tm) and Correct(tm)...\n\n>> So fast-import was actually filling the notes tree as if it were\n>> starting over with whatever new notes were added with notemodify (which,\n>> in a case where there were many, it filled with one level of\n>> indirection)\n>\n> Ah, that sort of makes sense. This confused the code to adjust the\n> fanout, because we track \"number of notes\" independently of \"number of\n> files\" (even though they are really the same thing in a notes tree).\n\nNot always. There is no way to summarily prevent someone from adding\nobjects at arbitrary locations in a notes tree. There \"non-notes\" must be\n(and are) handled by the notes code (albeit less than gracefully).\n\n>> I'm not sure this is a case worth fixing in fast-import. I can easily\n>> work around it.\n>\n> Yeah. Probably fast-import could be smarter here, but I think ultimately\n> it makes sense to stick to using the note commands. I think what you\n> want is a version of \"from\" that takes an existing tree (and number of\n> notes!) from a commit, but does not add it as a parent. AFAIK,\n> fast-import doesn't have a way to do that.\n>\n> Probably the simplest thing is to build it with history via fast-import,\n> and then just truncate the history at the end with:\n>\n>   commit=$(echo \"final notes tree\" | git commit-tree refs/notes/foo^{tree})\n>   git update-ref refs/notes/foo $commit\n\nI agree that this is probably the best workaround for now.\n\n\n...Johan\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"252579","messageId":"20141126123057.GA14275@glandium.org","threadId":"38052","inReplyTo":"CALKQrgftttSpuw8kc+jC6E5RBet39wHKy3670Z5iG=KQSmrCAw@mail.gmail.com","subject":"Re: \"git notes show\" is orders of magnitude slower than doing it manually with ls-tree and cat-file","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2014-11-26T12:30:57Z","receivedAt":"2014-11-26T12:30:57Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Wed, Nov 26, 2014 at 12:46:20PM +0100, Johan Herland wrote:\n> (First of all, thanks to both for great investigation and analysis)\n> \n> On Wed, Nov 26, 2014 at 5:46 AM, Jeff King <peff@peff.net> wrote:\n> > On Wed, Nov 26, 2014 at 11:25:53AM +0900, Mike Hommey wrote:\n> >\n> >> Now, looking at the notes tree reflog, I see that at some point, some\n> >> notes were added at the top-level of the tree, without being nested,\n> >> which is strange.\n> >\n> > That's somewhat expected. The fanout is dynamic based on the number of\n> > notes, so early on I think some notes may be found at the top of the tree.\n> >\n> >> And it looks like it's related to how I've been adding them, through\n> >> git-fast-import. I was using notemodify commands, and was using the\n> >> filemodify command to load the previous notes tree instead of using the\n> >> from command because I don't care about keeping the notes history.\n> \n> I'd very much like to see this fast-import stream (or a script\n> generating it). I'm assuming that it roughly follows along these lines:\n> \n>  - Start a new commit from a clean slate (no 'from')\n> \n>  - Do a single filemodify to \"load\" the previous notes tree.\n>    Exactly what does this filemodify command look like? I'm\n>    guessing you're using the root tree object from the previous\n>    notes tree as <dataref>, and an empty <path>, i.e.:\n> \n>      M 040000 $previous_notes_tree_root_sha1 \\n\n> \n>  - Do a series of notemodify commands for the notes being added in\n>    this commit.\n> \n>  - End of commit.\n\nThat's exactly the scenario.\n\n<snip>\n> >   commit=$(echo \"final notes tree\" | git commit-tree refs/notes/foo^{tree})\n> >   git update-ref refs/notes/foo $commit\n> \n> I agree that this is probably the best workaround for now.\n\nIndeed, that's about what I had in mind when I said I could easily work\naround (except I use the ls command in fast-import), and what I\nimplemented.\n\nMike\n"}]}