git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 0/2] Optimization batch 6: make full use of exact renames

From
Elijah Newren <newren@gmail.com>
Date
Feb 4, 2021, 00:05 UTC
Message-ID
<CABPp-BHMDi2i=9etq-L=0H=OO7Mbo=RFwMWTNebNB3H9Vbm3+A@mail.gmail.com>
In-Reply-To
<YBszm/s9na3ixUsO@coredump.intra.peff.net>
On Wed, Feb 3, 2021 at 3:37 PM Jeff King <peff@peff.net> wrote:
Show 24 quoted lines
>
> On Wed, Feb 03, 2021 at 03:06:26PM -0800, Elijah Newren wrote:
>
> > >    In an early attempt, I tried to retire rename_src[j], once
> > >    rename_dst[i] has been found to be a "good enough" match for it,
> > >    from the pool of rename src candidates to find a good match for
> > >    rename_dst[k] for i < k, but naive implementation of it would not
> > >    work well for obvious reasons---rename_src[j] may match a lot
> > >    better with rename_dst[k] than rename_dst[i] but we do not know
> > >    that until we try to estimate similarity with rename_dst[k].
> >
> > You may really like the next two series I submit.  I have a smarter
> > way to find a "good enough" match (comparing to exactly one other file
> > and often finding sufficient similarity), and one that'll make
> > intuitive sense to users.
>
> Here's a really old thread with an approach that may or may not be
> similar to what you're thinking of:
>
>   https://lore.kernel.org/git/596909b30710220240g665054d8hc40bc5d2234ba9e1@mail.gmail.com/
>
> Though maybe start with this summary message:
>
>   https://lore.kernel.org/git/596909b30710220309l1a28e646r9fd47f967dc32574@mail.gmail.com/

Interesting thread; thanks for the link. It's not remotely similar to what I have done, but a brief glance through it reminds me of the ideas at https://github.com/gitgitgadget/git/issues/519.

Show 10 quoted lines
> I remember experimenting some with it at the time, but never making
> things faster. It's entirely possible (likely, even) that I was simply
> not doing it well enough.
>
> It's also been long enough since I looked at the rename code that I'm
> not sure how different it is in practice. It still has a quadratic
> matrix in the end. We basically do a similar strategy of
> rolling-hash-fingerprint-and-see-where-things-collide, but I think we
> may end up with more work during the quadratic part (again, it's been
> a while, so I may just be totally off-base).

I'm not sure if I should spoil the surprise for the patchsets I haven't yet submitted but... when I get done, for the testcases I have looked at, rename detection is no longer the slowest part of merging/rebasing/cherry-picking -- not even when there are tens of thousands of renames on one side of history. And I didn't achieve that by making other parts of the code slower.

If someone can come up with a real-world testcase where rename detection is still really slow in a merge/rebase/cherry-pick with my implemented optimizations, I've got at least one more optimization that I hadn't bothered implementing because everything was fast enough for me already. And the idea you linked above and the ideas in the gitgitgadget issue linked above all have more possibilities that are complementary to what I have done that might help further.

> I've also wondered if something similar might be helpful for delta
> compression (which is now done with an O(m*n) sliding window).
Previous: Jeff KingNext: Elijah Newren via GitGitGadget
Message 21 of 25 in “Optimization batch 6: make full use of exact renames”
  1. 0/2 Optimization batch 6: make full use of exact renamesElijah Newren via GitGitGadget, Feb 3, 2021
  2. 1/2 diffcore-rename: no point trying to find a match better than exactElijah Newren via GitGitGadget, Feb 3, 2021
  3. Derrick StoleeFeb 3, 2021
  4. Elijah NewrenFeb 3, 2021
  5. Junio C HamanoFeb 3, 2021
  6. Elijah NewrenFeb 3, 2021
  7. 2/2 diffcore-rename: filter rename_src list when possibleElijah Newren via GitGitGadget, Feb 3, 2021
  8. Junio C HamanoFeb 3, 2021
  9. Elijah NewrenFeb 3, 2021
  10. 0/2 Optimization batch 6: make full use of exact renamesElijah Newren via GitGitGadget, Feb 3, 2021
  11. 1/2 diffcore-rename: no point trying to find a match better than exactElijah Newren via GitGitGadget, Feb 3, 2021
  12. 2/2 diffcore-rename: filter rename_src list when possibleElijah Newren via GitGitGadget, Feb 3, 2021
  13. Junio C HamanoFeb 13, 2021
  14. Elijah NewrenFeb 13, 2021
  15. Junio C HamanoFeb 13, 2021
  16. Elijah NewrenFeb 13, 2021
  17. Junio C HamanoFeb 3, 2021
  18. Elijah NewrenFeb 3, 2021
  19. Junio C HamanoFeb 3, 2021
  20. Jeff KingFeb 3, 2021
  21. Elijah NewrenFeb 4, 2021
  22. 0/2 Optimization batch 6: make full use of exact renamesElijah Newren via GitGitGadget, Feb 14, 2021
  23. 1/2 diffcore-rename: no point trying to find a match better than exactElijah Newren via GitGitGadget, Feb 14, 2021
  24. 2/2 diffcore-rename: filter rename_src list when possibleElijah Newren via GitGitGadget, Feb 14, 2021
  25. Elijah NewrenFeb 3, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.