git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 1/5] test-list-objects: List a subset of object ids

From
Derrick Stolee <stolee@gmail.com>
Date
Oct 5, 2017, 12:39 UTC
Message-ID
<6222f65a-c518-74a4-9df4-89bf997425cb@gmail.com>
In-Reply-To
<20171005100001.do5m4zhexcsltvw7@sigill.intra.peff.net>
On 10/5/2017 6:00 AM, Jeff King wrote:
Show 40 quoted lines
> On Thu, Oct 05, 2017 at 06:48:10PM +0900, Junio C Hamano wrote:
>
>> Jeff King <peff@peff.net> writes:
>>
>>> This is weirdly specific. Can we accomplish the same thing with existing
>>> tools?
>>>
>>> E.g., could:
>>>
>>>    git cat-file --batch-all-objects --batch-check='%(objectname)' |
>>>    shuffle |
>>>    head -n 100
>>>
>>> do the same thing?
>>>
>>> I know that "shuffle" isn't available everywhere, but I'd much rather
>>> see us fill in portability gaps in a general way, rather than
>>> introducing one-shot C code that needs to be maintained (and you
>>> wouldn't _think_ that t/helper programs need much maintenance, but try
>>> perusing "git log t/helper" output; they have to adapt to the same
>>> tree-wide changes as the rest of the code).
>> I was thinking about this a bit more, and came to the conclusion
>> that "sort -R" and "shuf" are wrong tools to use.  We would want to
>> measure with something close to real world workload.  for example,
>> letting
>>
>> 	git rev-list --all --objects
>>
>> produce the listof objects in traversal order (i.e. this is very
>> similar to the order in which "git log -p" needs to access the
>> objects) and chomping at the number of sample objects you need in
>> your test would give you such a list.
> Actually, I'd just as soon see timings for "git log --format=%h" or "git
> log --raw", as opposed to patches 1 and 2.
>
> You won't see a 90% speedup there, but you will see the actual
> improvement that real-world users are going to experience, which is way
> more important, IMHO.
>
> -Peff
Thanks for thinking hard about this.

For some real-user context: Some engineers using Git for the Windows repo were seeing extremely slow commands, such as 'fetch' or 'commit', and when we took a trace we saw most of the time spinning in this abbreviation code. Our workaround so far has been to set core.abbrev=40.

I'll run some perf numbers for these commands you recommend, and also see if I can replicate some of the pain points that triggered this change using the Linux repo.

Thanks, -Stolee

Previous: Junio C HamanoNext: Jeff King
Message 13 of 47 in “Improve abbreviation disambiguation”
  1. 0/5 Improve abbreviation disambiguationDerrick Stolee, Sep 25, 2017
  2. 2/5 p0008-abbrev.sh: Test find_unique_abbrev() perfDerrick Stolee, Sep 25, 2017
  3. Junio C HamanoSep 26, 2017
  4. Jeff KingOct 5, 2017
  5. Jeff KingOct 5, 2017
  6. 5/5 sha1_name: Minimize OID comparisons during disambiguationDerrick Stolee, Sep 25, 2017
  7. 1/5 test-list-objects: List a subset of object idsDerrick Stolee, Sep 25, 2017
  8. Junio C HamanoSep 26, 2017
  9. Jeff KingOct 5, 2017
  10. Junio C HamanoOct 5, 2017
  11. Jeff KingOct 5, 2017
  12. Junio C HamanoOct 5, 2017
  13. Derrick StoleeOct 5, 2017
  14. Jeff KingOct 6, 2017
  15. Derrick StoleeOct 7, 2017
  16. Jeff KingOct 7, 2017
  17. Junio C HamanoOct 8, 2017
  18. 4/5 sha1_name: Parse less while finding common prefixDerrick Stolee, Sep 25, 2017
  19. Stefan BellerSep 25, 2017
  20. Derrick StoleeOct 2, 2017
  21. 3/5 sha1_name: Unroll len loop in find_unique_abbrev_rDerrick Stolee, Sep 25, 2017
  22. 1/5 test-list-objects: List a subset of object idsDerrick Stolee, Oct 2, 2017
  23. Junio C HamanoOct 3, 2017
  24. 2/5 p0008-abbrev.sh: Test find_unique_abbrev() perfDerrick Stolee, Oct 2, 2017
  25. 0/5 Improve abbreviation disambituationDerrick Stolee, Oct 2, 2017
  26. Jeff KingOct 5, 2017
  27. 4/5 sha1_name: Parse less while finding common prefixDerrick Stolee, Oct 2, 2017
  28. Junio C HamanoOct 4, 2017
  29. 3/5 sha1_name: Unroll len loop in find_unique_abbrev_rDerrick Stolee, Oct 2, 2017
  30. Junio C HamanoOct 3, 2017
  31. Derrick StoleeOct 3, 2017
  32. Junio C HamanoOct 4, 2017
  33. Derrick StoleeOct 4, 2017
  34. Junio C HamanoOct 4, 2017
  35. Derrick StoleeOct 4, 2017
  36. Junio C HamanoOct 5, 2017
  37. Jeff KingOct 5, 2017
  38. Junio C HamanoOct 5, 2017
  39. 5/5 sha1_name: Minimize OID comparisons during disambiguationDerrick Stolee, Oct 2, 2017
  40. Stefan BellerOct 3, 2017
  41. Derrick StoleeOct 3, 2017
  42. Jeff KingOct 5, 2017
  43. cleanup: fix possible overflow errors in binary searchDerrick Stolee, Oct 6, 2017
  44. Jeff KingOct 6, 2017
  45. Derrick StoleeOct 6, 2017
  46. cleanup: fix possible overflow errors in binary searchDerrick Stolee, Oct 8, 2017
  47. Jeff KingOct 9, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.