git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH 3/3] git-rebase.sh: make git-rebase--interactive the default

From
Johannes Schindelin <johannes.schindelin@gmx.de>
Date
Jun 21, 2018, 10:57 UTC
Message-ID
<nycvar.QRO.7.76.6.1806211242440.11870@tvgsbejvaqbjf.bet>
In-Reply-To
<CABPp-BEnJ4q9WGi4BgikpLJ9Aty5-3MR4VozjBk7ie4rGMc-ng@mail.gmail.com>
Hi Elijah,
On Wed, 20 Jun 2018, Elijah Newren wrote:
Show 13 quoted lines
> On Sun, Jun 17, 2018 at 2:44 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> 
> > I was really referring to speed. But I have to admit that I do not have
> > any current numbers.
> >
> > Another issue just hit me, though: rebase --am does not need to look at as
> > many Git objects as rebase --merge or rebase -i. Therefore, GVFS users
> > will still want to use --am wherever possible, to avoid "hydrating"
> > many objects during their rebase.
> 
> What is it that makes rebase --am need fewer Git objects than rebase
> --merge or rebase -i?

Multiple things. The most obvious thing: --cherry-pick. When you call `git rebase --am`, it does not try to exclude the patches by looking at `rev-list --cherry-pick --right-only <upstream>..<head>`

This really bit us in GVFS Git because we have several thousand developers working on the same code base, and you probably read the numbers elsewhere how many commits are introduced while a developer goes on so much as a week-long vacation.

Next, rename detection.

I think that bit us even harder because it tries to look at all the files that have been changed in upstream in the meantime, which can be *a lot*. And if you know that moving files outside of your cozy little sparse checkout is unlikely (or moving from the outside into your checkout), then all this hydration is pretty wasteful. That's why we had to switch off rename detection in GVFS Git.

Show 9 quoted lines
> My guess at what objects are needed by each type:
> 
> At a high level, rebase --am for each commit will need to compare the
> commit to its parent to generate a diff (which thus involves walking
> over the objects in both the commit and its parent, though it should
> be able to skip over subtrees that are equal), and then will need to
> look at all the objects in the target commit on which it needs to
> apply the patch (in order to properly fill the index for a starting
> point, and used later when creating a new commit).

No, in --am mode, it does not even need to look at all the objects in the target commits. Only those objects that correspond to the files that are touched by the diff.

In a massive code base, such as Windows', this makes a huge difference. Even comparing the number of files touched by the patches that are to be rebased to the number of files that were touched in upstream since you rebased last is ridiculous. And a three-way merge needs to consider that latter set of files.

> If the application of the diff fails, it falls back to a three-way
> merge, though the three-way merge shouldn't need any additional objects.

The three-way merge needs to reconcile the diff between branch point and your changes to the diff between branch point and upstream's changes. The latter can be a lot bigger than the former, in which case --am's complexity (O(former)) is much nicer than --merge's (O(former u latter)).

Show 44 quoted lines
> So, to summarize, rebase--am needs objects from the commit being
> rebased, its parent, and the target commit onto which it is applying,
> though it can short circuit some objects when the commit and its parent
> have matching subtree(s).
> 
> rebase -i, if I understand correctly, does a three-way merge between
> the commit, its parent, and the target commit.  Thus, we again walk
> over objects in those three commits; I think unpack_trees() does not
> take advantage of matching trees to avoid descending into subtrees,
> but if so that's an optimization that we may be able to implement
> (though it would require diving into unpack_trees() code, which is
> never easy...).
> 
> (Side notes: (1) rebase --merge is basically the same as rebase -i
> here; it's path to reaching the recursive merge machinery is a bit
> different but the resulting arguments are the same; (2) a real merge
> between branches would require more objects because it would have to
> do some revision walking to find a merge base, and a real merge base
> is likely to differ more than just the parent commit.  But finding
> merge bases isn't relevant to rebase -m or rebase -i)
> 
> Is there something else I'm missing that fundamentally makes rebase -i
> need more objects?
> 
> > As to speed: that might be harder. But then, the performance might already
> > be good enough. I do not have numbers (nor the time to generate them) to
> > back up my hunch that --am is substantially faster than --merge.
> 
> I too have a hunch that --am is faster than --merge, on big enough
> repos or repos with enough renames.  I can partially back it up with
> an indirect number: at [1], it was reported that cherry-picks could be
> sped up by a factor of 20-30 on some repos with lots of renames.  I
> believe there are other performance improvements possible too, for the
> --merge or -i cases.
> 
> I'm also curious now whether your comment on hydrating objects might
> uncover additional areas where performance improvements could be made
> for non-am-based rebases of large-enough repos.
> 
> Elijah
> 
> [1] https://public-inbox.org/git/CABPp-BH4LLzeJjE5cvwWQJ8xTj3m9oC-41Tu8BM8c7R0gQTjWw@mail.gmail.com/
> (see also Peter's last reply in that thread, and compare to his first
> post)

I am too unfamiliar with the rename detection to understand the details, but you are familiar with it, so I trust your judgement!

Ciao, Dscho

Previous: Elijah NewrenNext: Elijah Newren
Message 120 of 130 in “RFC: rebase inconsistency in 2.18 -- course of action?”
  1. Elijah NewrenJun 7, 2018
  2. t3401: add directory rename testcases for rebase and amElijah Newren, Jun 7, 2018
  3. Elijah NewrenJun 25, 2018
  4. apply: fix grammar error in commentElijah Newren, Jun 7, 2018
  5. t5407: fix test to cover intended argumentsElijah Newren, Jun 7, 2018
  6. git-rebase--merge: modernize "git-$cmd" to "git $cmd"Elijah Newren, Jun 7, 2018
  7. Elijah NewrenJun 7, 2018
  8. git-rebase--merge: modernize "git-$cmd" to "git $cmd"Elijah Newren, Jun 27, 2018
  9. Johannes SchindelinJul 12, 2018
  10. Elijah NewrenJul 13, 2018
  11. Johannes SchindelinJul 14, 2018
  12. 1/2 t3422: new testcases for checking when incompatible options passedElijah Newren, Jun 7, 2018
  13. 2/2 git-rebase: error out when incompatible options passedElijah Newren, Jun 7, 2018
  14. Phillip WoodJun 10, 2018
  15. Elijah NewrenJun 11, 2018
  16. Phillip WoodJun 11, 2018
  17. Elijah NewrenJun 11, 2018
  18. Phillip WoodJun 12, 2018
  19. 0/7 Document/fix/warn about rebase incompatibilities and inconsistencesElijah Newren, Jun 17, 2018
  20. 1/7 git-rebase.txt: document incompatible optionsElijah Newren, Jun 17, 2018
  21. Phillip WoodJun 17, 2018
  22. Elijah NewrenJun 20, 2018
  23. Eric SunshineJun 17, 2018
  24. Elijah NewrenJun 20, 2018
  25. 2/7 git-rebase.sh: update help messages a bitElijah Newren, Jun 17, 2018
  26. 3/7 t3422: new testcases for checking when incompatible options passedElijah Newren, Jun 17, 2018
  27. 4/7 git-rebase: error out when incompatible options passedElijah Newren, Jun 17, 2018
  28. 7/7 git-rebase: make --allow-empty-message the defaultElijah Newren, Jun 17, 2018
  29. Phillip WoodJun 17, 2018
  30. Elijah NewrenJun 20, 2018
  31. 6/7 git-rebase.txt: address confusion between --no-ff vs --force-rebaseElijah Newren, Jun 17, 2018
  32. 5/7 git-rebase.txt: document behavioral inconsistencies between modesElijah Newren, Jun 17, 2018
  33. Phillip WoodJun 17, 2018
  34. 0/7 Document/fix/warn about rebase incompatibilities and inconsistenciesElijah Newren, Jun 21, 2018
  35. 7/7 git-rebase: make --allow-empty-message the defaultElijah Newren, Jun 21, 2018
  36. 3/7 t3422: new testcases for checking when incompatible options passedElijah Newren, Jun 21, 2018
  37. Junio C HamanoJun 21, 2018
  38. Elijah NewrenJun 21, 2018
  39. Eric SunshineJun 21, 2018
  40. 2/7 git-rebase.sh: update help messages a bitElijah Newren, Jun 21, 2018
  41. Junio C HamanoJun 21, 2018
  42. Elijah NewrenJun 21, 2018
  43. 5/7 git-rebase.txt: document behavioral inconsistencies between modesElijah Newren, Jun 21, 2018
  44. Junio C HamanoJun 21, 2018
  45. Elijah NewrenJun 21, 2018
  46. 1/7 git-rebase.txt: document incompatible optionsElijah Newren, Jun 21, 2018
  47. Junio C HamanoJun 21, 2018
  48. Elijah NewrenJun 21, 2018
  49. SZEDER GáborJun 22, 2018
  50. 4/7 git-rebase: error out when incompatible options passedElijah Newren, Jun 21, 2018
  51. Junio C HamanoJun 21, 2018
  52. Elijah NewrenJun 21, 2018
  53. 6/7 git-rebase.txt: address confusion between --no-ff vs --force-rebaseElijah Newren, Jun 21, 2018
  54. Junio C HamanoJun 21, 2018
  55. Elijah NewrenJun 21, 2018
  56. 0/9 Document/fix/warn about rebase incompatibilities and inconsistenciesElijah Newren, Jun 25, 2018
  57. 3/9 t3422: new testcases for checking when incompatible options passedElijah Newren, Jun 25, 2018
  58. Junio C HamanoJun 26, 2018
  59. Elijah NewrenJun 27, 2018
  60. 2/9 git-rebase.sh: update help messages a bitElijah Newren, Jun 25, 2018
  61. 5/9 git-rebase.txt: address confusion between --no-ff vs --force-rebaseElijah Newren, Jun 25, 2018
  62. 9/9 git-rebase: make --allow-empty-message the defaultElijah Newren, Jun 25, 2018
  63. 8/9 t3401: add directory rename testcases for rebase and amElijah Newren, Jun 25, 2018
  64. 7/9 git-rebase.txt: document behavioral differences between modesElijah Newren, Jun 25, 2018
  65. 1/9 git-rebase.txt: document incompatible optionsElijah Newren, Jun 25, 2018
  66. 6/9 directory-rename-detection.txt: technical docs on abilities and limitationsElijah Newren, Jun 25, 2018
  67. 4/9 git-rebase: error out when incompatible options passedElijah Newren, Jun 25, 2018
  68. 0/9 Document/fix/warn about rebase incompatibilities and inconsistenciesElijah Newren, Jun 27, 2018
  69. 2/9 git-rebase.sh: update help messages a bitElijah Newren, Jun 27, 2018
  70. 1/9 git-rebase.txt: document incompatible optionsElijah Newren, Jun 27, 2018
  71. 7/9 git-rebase.txt: document behavioral differences between modesElijah Newren, Jun 27, 2018
  72. 9/9 git-rebase: make --allow-empty-message the defaultElijah Newren, Jun 27, 2018
  73. 2.19.0 regression: leaving editor with empty commit message doesn't stop rebase [was: Re: [RFC PATCH v5 9/9] git-rebase: make --allow-empty-message the default]SZEDER Gábor, Sep 12, 2018
  74. Elijah NewrenSep 12, 2018
  75. sequencer: fix --allow-empty-message behavior, make it smarterElijah Newren, Sep 12, 2018
  76. Junio C HamanoSep 13, 2018
  77. 6/9 directory-rename-detection.txt: technical docs on abilities and limitationsElijah Newren, Jun 27, 2018
  78. 8/9 t3401: add directory rename testcases for rebase and amElijah Newren, Jun 27, 2018
  79. 4/9 git-rebase: error out when incompatible options passedElijah Newren, Jun 27, 2018
  80. 5/9 git-rebase.txt: address confusion between --no-ff vs --force-rebaseElijah Newren, Jun 27, 2018
  81. 3/9 t3422: new testcases for checking when incompatible options passedElijah Newren, Jun 27, 2018
  82. Phillip WoodJun 29, 2018
  83. git-rebase.sh: handle keep-empty like all other optionsElijah Newren, Jun 7, 2018
  84. Phillip WoodJun 10, 2018
  85. Elijah NewrenJun 11, 2018
  86. Phillip WoodJun 11, 2018
  87. Elijah NewrenJun 11, 2018
  88. Phillip WoodJun 12, 2018
  89. 1/2 t3418: add testcase showing problems with rebase -i and strategy optionsElijah Newren, Jun 7, 2018
  90. 2/2 Fix use of strategy options with interactive rebasesElijah Newren, Jun 7, 2018
  91. 0/2 Fix use of strategy options with interactive rebasesElijah Newren, Jun 27, 2018
  92. 1/2 t3418: add testcase showing problems with rebase -i and strategy optionsElijah Newren, Jun 27, 2018
  93. Eric SunshineJun 27, 2018
  94. Elijah NewrenJun 27, 2018
  95. Junio C HamanoJun 27, 2018
  96. Elijah NewrenJun 27, 2018
  97. Johannes SixtJun 27, 2018
  98. Eric SunshineJun 27, 2018
  99. SZEDER GáborJun 27, 2018
  100. Junio C HamanoJun 27, 2018
  101. Junio C HamanoJun 27, 2018
  102. 2/2 Fix use of strategy options with interactive rebasesElijah Newren, Jun 27, 2018
  103. 0/2 Fix use of strategy options with interactive rebasesElijah Newren, Jun 27, 2018
  104. 2/2 Fix use of strategy options with interactive rebasesElijah Newren, Jun 27, 2018
  105. Johannes SchindelinJul 12, 2018
  106. Elijah NewrenJul 13, 2018
  107. 1/2 t3418: add testcase showing problems with rebase -i and strategy optionsElijah Newren, Jun 27, 2018
  108. SZEDER GáborJun 27, 2018
  109. SZEDER GáborJul 11, 2018
  110. Elijah NewrenJul 11, 2018
  111. 0/3 Send more rebases through interactive machineryElijah Newren, Jun 7, 2018
  112. 1/3 git-rebase, sequencer: add a --quiet option for the interactive machineryElijah Newren, Jun 7, 2018
  113. Johannes SchindelinJun 9, 2018
  114. Elijah NewrenJun 9, 2018
  115. 3/3 git-rebase.sh: make git-rebase--interactive the defaultElijah Newren, Jun 7, 2018
  116. Johannes SchindelinJun 9, 2018
  117. Elijah NewrenJun 10, 2018
  118. Johannes SchindelinJun 17, 2018
  119. Elijah NewrenJun 20, 2018
  120. Johannes SchindelinJun 21, 2018
  121. Elijah NewrenJun 21, 2018
  122. Johannes SchindelinJan 21, 2019
  123. Elijah NewrenJan 21, 2019
  124. Johannes SchindelinJan 22, 2019
  125. 2/3 rebase: Implement --merge via git-rebase--interactiveElijah Newren, Jun 7, 2018
  126. Johannes SchindelinJun 9, 2018
  127. Elijah NewrenJun 10, 2018
  128. Phillip WoodJun 11, 2018
  129. Junio C HamanoJun 11, 2018
  130. Elijah NewrenJun 11, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.