git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 4/4] unpack-trees: cheaper index update when walking by cache-tree

From
Elijah Newren <newren@gmail.com>
Date
Aug 10, 2018, 18:39 UTC
Message-ID
<CABPp-BGU6QnUwQgkhwx6vLBc3ozoEScQ4DaZd-9ZZfQhXfxPww@mail.gmail.com>
In-Reply-To
<CACsJy8DF5XLf-RF3SwTpRynYALJUPO_VTK=fpx1oabwB80ZpPw@mail.gmail.com>
On Fri, Aug 10, 2018 at 9:39 AM Duy Nguyen <pclouds@gmail.com> wrote:
>
> On Wed, Aug 8, 2018 at 8:46 PM Elijah Newren <newren@gmail.com> wrote:
> > > @@ -701,6 +702,24 @@ static int traverse_by_cache_tree(int pos, int nr_entries, int nr_names,
Show 12 quoted lines
> > If we're going to go this route, I think we should first check that
> > o->fn is one of those known safe functions.  And if we're going that
> > route, the comments I bring up on patch 2 about possibly avoiding
> > call_unpack_fn() altogether might even obviate this patch while
> > speeding things up more.
>
> Yes I do need to check o->fn. I might have to think more about
> avoiding call_unpack_fn(). Even if we avoid it though, we still go
> through add_index_entry() and suffer the same checks every time unless
> we do somethine like this (but then of course it's safer because
> you're doing it in a specific x-way merge, not generic code like
> this).

Why do we still need to go through add_index_entry()? I thought that the whole point was that you already checked that at the current path, the trees being unpacked were all equal and matched both the index and the cache_tree. If so, why is there any need for an update at all? (Did I read your all_trees_same_as_cache_tree() function wrong, and you don't actually know these all match in some important way?)

Show 24 quoted lines
> > > @@ -1561,6 +1581,13 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
> > >                 if (!ret) {
> > >                         if (!o->result.cache_tree)
> > >                                 o->result.cache_tree = cache_tree();
> > > +                       /*
> > > +                        * TODO: Walk o.src_index->cache_tree, quickly check
> > > +                        * if o->result.cache has the exact same content for
> > > +                        * any valid cache-tree in o.src_index, then we can
> > > +                        * just copy the cache-tree over instead of hashing a
> > > +                        * new tree object.
> > > +                        */
> >
> > Interesting.  I really don't know how cache_tree works...but if we
> > avoided calling call_unpack_fn, and thus left the original index entry
> > in place instead of replacing it with an equal one, would that as a
> > side effect speed up the cache_tree_valid/cache_tree_update calls for
> > us?  Or is there still work here?
>
> Naah. Notice that we don't care at all about the source's cache-tree
> when we update o->result one (and we never ever do anything about
> o->result's cache-tree during the merge). Whether you invalidate or
> not, o->result's cache-tree is always empty and you still have to
> recreate all cache-tree in o->result. You essentially play full cost
> of "git write-tree" here if I'm not mistaken.

Oh...perhaps that answers my question above. So we have to call add_index_entry() for the side effect of populating the new cache_tree?

Previous: Duy NguyenNext: Duy Nguyen
Message 35 of 121 in “[RFC] Speeding up checkout (and merge, rebase, etc)”
  1. 0/3 [RFC] Speeding up checkout (and merge, rebase, etc)Ben Peart, Jul 18, 2018
  2. 1/3 add unbounded Multi-Producer-Multi-Consumer queueBen Peart, Jul 18, 2018
  3. Stefan BellerJul 18, 2018
  4. Junio C HamanoJul 19, 2018
  5. 2/3 add performance tracing around traverse_trees() in unpack_trees()Ben Peart, Jul 18, 2018
  6. 3/3 Add initial parallel version of unpack_trees()Ben Peart, Jul 18, 2018
  7. Junio C HamanoJul 18, 2018
  8. Stefan BellerJul 18, 2018
  9. Jeff KingJul 18, 2018
  10. Ben PeartJul 23, 2018
  11. Duy NguyenJul 23, 2018
  12. Ben PeartJul 23, 2018
  13. Jeff KingJul 24, 2018
  14. Duy NguyenJul 24, 2018
  15. Ben PeartJul 25, 2018
  16. Duy NguyenJul 26, 2018
  17. Duy NguyenJul 26, 2018
  18. Junio C HamanoJul 26, 2018
  19. Duy NguyenJul 27, 2018
  20. Ben PeartJul 27, 2018
  21. Duy NguyenJul 27, 2018
  22. Junio C HamanoJul 27, 2018
  23. Duy NguyenJul 27, 2018
  24. Duy NguyenJul 29, 2018
  25. 0/4 Speed up unpack_trees()Nguyễn Thái Ngọc Duy, Jul 29, 2018
  26. 1/4 unpack-trees.c: add performance tracingNguyễn Thái Ngọc Duy, Jul 29, 2018
  27. Ben PeartJul 30, 2018
  28. 2/4 unpack-trees: optimize walking same trees with cache-treeNguyễn Thái Ngọc Duy, Jul 29, 2018
  29. Ben PeartJul 30, 2018
  30. 3/4 unpack-trees: reduce malloc in cache-tree walkNguyễn Thái Ngọc Duy, Jul 29, 2018
  31. Ben PeartJul 30, 2018
  32. 4/4 unpack-trees: cheaper index update when walking by cache-treeNguyễn Thái Ngọc Duy, Jul 29, 2018
  33. Elijah NewrenAug 8, 2018
  34. Duy NguyenAug 10, 2018
  35. Elijah NewrenAug 10, 2018
  36. Duy NguyenAug 10, 2018
  37. Elijah NewrenAug 10, 2018
  38. Duy NguyenAug 10, 2018
  39. Ben PeartJul 30, 2018
  40. Duy NguyenJul 31, 2018
  41. Ben PeartJul 31, 2018
  42. Ben PeartJul 31, 2018
  43. Duy NguyenAug 1, 2018
  44. Ben PeartAug 8, 2018
  45. Ben PeartAug 9, 2018
  46. Duy NguyenAug 10, 2018
  47. Duy NguyenAug 10, 2018
  48. Ben PeartJul 30, 2018
  49. 0/4 Speed up unpack_trees()Nguyễn Thái Ngọc Duy, Aug 4, 2018
  50. 1/4 unpack-trees: add performance tracingNguyễn Thái Ngọc Duy, Aug 4, 2018
  51. 2/4 unpack-trees: optimize walking same trees with cache-treeNguyễn Thái Ngọc Duy, Aug 4, 2018
  52. Elijah NewrenAug 8, 2018
  53. Duy NguyenAug 10, 2018
  54. Elijah NewrenAug 10, 2018
  55. 3/4 unpack-trees: reduce malloc in cache-tree walkNguyễn Thái Ngọc Duy, Aug 4, 2018
  56. Elijah NewrenAug 8, 2018
  57. 4/4 unpack-trees: cheaper index update when walking by cache-treeNguyễn Thái Ngọc Duy, Aug 4, 2018
  58. Junio C HamanoAug 6, 2018
  59. Duy NguyenAug 6, 2018
  60. Junio C HamanoAug 6, 2018
  61. Ben PeartAug 8, 2018
  62. Junio C HamanoAug 8, 2018
  63. Junio C HamanoAug 8, 2018
  64. Junio C HamanoAug 8, 2018
  65. Duy NguyenAug 10, 2018
  66. 0/5 Speed up unpack_trees()Nguyễn Thái Ngọc Duy, Aug 12, 2018
  67. 3/5 unpack-trees: optimize walking same trees with cache-treeNguyễn Thái Ngọc Duy, Aug 12, 2018
  68. Ben PeartAug 13, 2018
  69. Duy NguyenAug 15, 2018
  70. 1/5 trace.h: support nested performance tracingNguyễn Thái Ngọc Duy, Aug 12, 2018
  71. Ben PeartAug 13, 2018
  72. 2/5 unpack-trees: add performance tracingNguyễn Thái Ngọc Duy, Aug 12, 2018
  73. Thomas AdamAug 12, 2018
  74. Junio C HamanoAug 13, 2018
  75. Ben PeartAug 13, 2018
  76. Jeff KingAug 13, 2018
  77. Stefan BellerAug 13, 2018
  78. Ben PeartAug 13, 2018
  79. Duy NguyenAug 13, 2018
  80. Jeff KingAug 13, 2018
  81. Junio C HamanoAug 13, 2018
  82. Jeff HostetlerAug 14, 2018
  83. Duy NguyenAug 14, 2018
  84. Stefan BellerAug 14, 2018
  85. Duy NguyenAug 14, 2018
  86. Jeff KingAug 14, 2018
  87. Junio C HamanoAug 14, 2018
  88. Duy NguyenAug 15, 2018
  89. Junio C HamanoAug 15, 2018
  90. Jeff HostetlerAug 14, 2018
  91. 4/5 unpack-trees: reduce malloc in cache-tree walkNguyễn Thái Ngọc Duy, Aug 12, 2018
  92. 5/5 unpack-trees: reuse (still valid) cache-tree from src_indexNguyễn Thái Ngọc Duy, Aug 12, 2018
  93. Elijah NewrenAug 13, 2018
  94. Duy NguyenAug 13, 2018
  95. Ben PeartAug 13, 2018
  96. Duy NguyenAug 13, 2018
  97. Ben PeartAug 13, 2018
  98. Junio C HamanoAug 13, 2018
  99. Ben PeartAug 14, 2018
  100. 0/7 Speed up unpack_trees()Nguyễn Thái Ngọc Duy, Aug 18, 2018
  101. 1/7 trace.h: support nested performance tracingNguyễn Thái Ngọc Duy, Aug 18, 2018
  102. 2/7 unpack-trees: add performance tracingNguyễn Thái Ngọc Duy, Aug 18, 2018
  103. 3/7 unpack-trees: optimize walking same trees with cache-treeNguyễn Thái Ngọc Duy, Aug 18, 2018
  104. Ben PeartAug 20, 2018
  105. 5/7 unpack-trees: reuse (still valid) cache-tree from src_indexNguyễn Thái Ngọc Duy, Aug 18, 2018
  106. 6/7 unpack-trees: add missing cache invalidationNguyễn Thái Ngọc Duy, Aug 18, 2018
  107. 4/7 unpack-trees: reduce malloc in cache-tree walkNguyễn Thái Ngọc Duy, Aug 18, 2018
  108. 7/7 cache-tree: verify valid cache-tree in the test suiteNguyễn Thái Ngọc Duy, Aug 18, 2018
  109. Elijah NewrenAug 18, 2018
  110. Elijah NewrenAug 18, 2018
  111. Duy NguyenAug 19, 2018
  112. Document update for nd/unpack-trees-with-cache-treeNguyễn Thái Ngọc Duy, Aug 25, 2018
  113. Martin ÅgrenAug 25, 2018
  114. Document update for nd/unpack-trees-with-cache-treeNguyễn Thái Ngọc Duy, Aug 25, 2018
  115. Ben PeartJul 27, 2018
  116. Duy NguyenJul 26, 2018
  117. Junio C HamanoJul 24, 2018
  118. Duy NguyenJul 24, 2018
  119. Jeff KingJul 24, 2018
  120. Ben PeartJul 25, 2018
  121. Jeff KingJul 24, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.