git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 7/7] index-pack: make quantum of work smaller

From
Junio C Hamano <gitster@pobox.com>
Date
Aug 24, 2020, 21:19 UTC
Message-ID
<xmqqk0xn3fo8.fsf@gitster.c.googlers.com>
In-Reply-To
<3e69f41f19535fa2c04fae6adc78bcc4f052d523.1598296530.git.jonathantanmy@google.com>
Jonathan Tan <jonathantanmy@google.com> writes:
Show 9 quoted lines
> Currently, when index-pack resolves deltas, it does not split up delta
> trees into threads: each delta base root (an object that is not a
> REF_DELTA or OFS_DELTA) can go into its own thread, but all deltas on
> that root (direct or indirect) are processed in the same thread.
>
> This is a problem when a repository contains a large text file (thus,
> delta-able) that is modified many times - delta resolution time during
> fetching is dominated by processing the deltas corresponding to that
> text file.

We favor wide and shallow delta trees for both reduced storage footprint and lower runtime overhead, so optimizing the index-pack for that use case makes a lot of sense.

Previous: Jonathan TanNext: Jonathan Tan
Message 11 of 28 in “Better threaded delta resolution in index-pack (another try)”
  1. 0/7 Better threaded delta resolution in index-pack (another try)Jonathan Tan, Aug 24, 2020
  2. 1/7 Documentation: deltaBaseCacheLimit is per-threadJonathan Tan, Aug 24, 2020
  3. fetch-pack: in partial clone, pass --promisorJonathan Tan, Aug 24, 2020
  4. Jonathan TanAug 24, 2020
  5. 2/7 index-pack: remove redundant parameterJonathan Tan, Aug 24, 2020
  6. Junio C HamanoAug 24, 2020
  7. 3/7 index-pack: unify threaded and unthreaded codeJonathan Tan, Aug 24, 2020
  8. Junio C HamanoAug 24, 2020
  9. 5/7 index-pack: calculate {ref,ofs}_{first,last} earlyJonathan Tan, Aug 24, 2020
  10. 7/7 index-pack: make quantum of work smallerJonathan Tan, Aug 24, 2020
  11. Junio C HamanoAug 24, 2020
  12. 6/7 index-pack: make resolve_delta() assume base dataJonathan Tan, Aug 24, 2020
  13. 4/7 index-pack: remove redundant child fieldJonathan Tan, Aug 24, 2020
  14. Junio C HamanoAug 24, 2020
  15. fixup! index-pack: make quantum of work smallerJonathan Tan, Aug 24, 2020
  16. Jeff KingAug 24, 2020
  17. Jonathan TanAug 25, 2020
  18. Jeff KingAug 25, 2020
  19. Jeff KingAug 25, 2020
  20. 0/7 Better threaded delta resolution in index-pack (another try)Jonathan Tan, Sep 8, 2020
  21. 2/7 index-pack: remove redundant parameterJonathan Tan, Sep 8, 2020
  22. 3/7 index-pack: unify threaded and unthreaded codeJonathan Tan, Sep 8, 2020
  23. 7/7 index-pack: make quantum of work smallerJonathan Tan, Sep 8, 2020
  24. 6/7 index-pack: make resolve_delta() assume base dataJonathan Tan, Sep 8, 2020
  25. 5/7 index-pack: calculate {ref,ofs}_{first,last} earlyJonathan Tan, Sep 8, 2020
  26. 4/7 index-pack: remove redundant child fieldJonathan Tan, Sep 8, 2020
  27. 1/7 Documentation: deltaBaseCacheLimit is per-threadJonathan Tan, Sep 8, 2020
  28. Junio C HamanoSep 8, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.