git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 12/15] [GSOC] cat-file: reuse ref-filter logic

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Jul 2, 2021, 13:36 UTC
Message-ID
<877di8al8n.fsf@evledraar.gmail.com>
In-Reply-To
<e04b970ccb0cad8c0b651ab11f5f52063bd84606.1625155693.git.gitgitgadget@gmail.com>
On Thu, Jul 01 2021, ZheNing Hu via GitGitGadget wrote:
Show 37 quoted lines
> From: ZheNing Hu <adlternative@gmail.com>
>
> In order to let cat-file use ref-filter logic, let's do the
> following:
>
> 1. Change the type of member `format` in struct `batch_options`
> to `ref_format`, we will pass it to ref-filter later.
> 2. Let `batch_objects()` add atoms to format, and use
> `verify_ref_format()` to check atoms.
> 3. Use `format_ref_array_item()` in `batch_object_write()` to
> get the formatted data corresponding to the object. If the
> return value of `format_ref_array_item()` is equals to zero,
> use `batch_write()` to print object data; else if the return
> value is less than zero, use `die()` to print the error message
> and exit; else if return value is greater than zero, only print
> the error message, but don't exit.
> 4. Use free_ref_array_item_value() to free ref_array_item's
> value.
>
> Most of the atoms in `for-each-ref --format` are now supported,
> such as `%(tree)`, `%(parent)`, `%(author)`, `%(tagger)`, `%(if)`,
> `%(then)`, `%(else)`, `%(end)`. But these atoms will be rejected:
> `%(refname)`, `%(symref)`, `%(upstream)`, `%(push)`, `%(worktreepath)`,
> `%(flag)`, `%(HEAD)`, because these atoms are unique to those objects
> that pointed to by a ref, "for-each-ref"'s family can naturally use
> these atoms, but not all objects are pointed to be a ref, so "cat-file"
> will not be able to use them.
>
> The performance for `git cat-file --batch-all-objects
> --batch-check` on the Git repository itself with performance
> testing tool `hyperfine` changes from 669.4 ms ±  31.1 ms to
> 1.134 s ±  0.063 s.
>
> The performance for `git cat-file --batch-all-objects --batch
>>/dev/null` on the Git repository itself with performance testing
> tool `time` change from "27.37s user 0.29s system 98% cpu 28.089
> total" to "33.69s user 1.54s system 87% cpu 40.258 total".

This new feature is really nice, but that's a really bad performance regression. A lot of software in the wild relies on "cat-file --batch" to be *the* performant interface to git for mass-extrction of object data.

That's in increase of ~70% and ~20%, respectively. Have you dug into (e.g. with a profiler) where we're now spending all this time?

Previous: ZheNing Hu via GitGitGadgetNext: Christian Couder
Message 22 of 39 in “[GSOC] cat-file: reuse ref-filter logic”
  1. 00/15 [GSOC] cat-file: reuse ref-filter logicZheNing Hu via GitGitGadget, Jul 1, 2021
  2. 01/15 [GSOC] ref-filter: add obj-type check in grab contentsZheNing Hu via GitGitGadget, Jul 1, 2021
  3. 03/15 [GSOC] ref-filter: --format=%(raw) re-support --perlZheNing Hu via GitGitGadget, Jul 1, 2021
  4. Ævar Arnfjörð BjarmasonJul 2, 2021
  5. ZheNing HuJul 3, 2021
  6. 02/15 [GSOC] ref-filter: add %(raw) atomZheNing Hu via GitGitGadget, Jul 1, 2021
  7. Ævar Arnfjörð BjarmasonJul 2, 2021
  8. ZheNing HuJul 3, 2021
  9. 05/15 [GSOC] ref-filter: add %(rest) atomZheNing Hu via GitGitGadget, Jul 1, 2021
  10. 04/15 [GSOC] ref-filter: use non-const ref_format in *_atom_parser()ZheNing Hu via GitGitGadget, Jul 1, 2021
  11. 06/15 [GSOC] ref-filter: pass get_object() return value to their callersZheNing Hu via GitGitGadget, Jul 1, 2021
  12. 08/15 [GSOC] ref-filter: add cat_file_mode in struct ref_formatZheNing Hu via GitGitGadget, Jul 1, 2021
  13. Ævar Arnfjörð BjarmasonJul 2, 2021
  14. Eric SunshineJul 2, 2021
  15. Christian CouderJul 2, 2021
  16. ZheNing HuJul 3, 2021
  17. Ævar Arnfjörð BjarmasonJul 5, 2021
  18. 07/15 [GSOC] ref-filter: introduce free_ref_array_item_value() functionZheNing Hu via GitGitGadget, Jul 1, 2021
  19. 09/15 [GSOC] ref-filter: modify the error message and value in get_objectZheNing Hu via GitGitGadget, Jul 1, 2021
  20. 11/15 [GSOC] cat-file: change batch_objects parameter nameZheNing Hu via GitGitGadget, Jul 1, 2021
  21. 12/15 [GSOC] cat-file: reuse ref-filter logicZheNing Hu via GitGitGadget, Jul 1, 2021
  22. Ævar Arnfjörð BjarmasonJul 2, 2021
  23. Christian CouderJul 2, 2021
  24. ZheNing HuJul 3, 2021
  25. ZheNing HuJul 3, 2021
  26. Ævar Arnfjörð BjarmasonJul 3, 2021
  27. ZheNing HuJul 4, 2021
  28. Ævar Arnfjörð BjarmasonJul 5, 2021
  29. 10/15 [GSOC] cat-file: add has_object_file() checkZheNing Hu via GitGitGadget, Jul 1, 2021
  30. Ævar Arnfjörð BjarmasonJul 2, 2021
  31. ZheNing HuJul 3, 2021
  32. 13/15 [GSOC] cat-file: reuse err buf in batch_object_write()ZheNing Hu via GitGitGadget, Jul 1, 2021
  33. 14/15 [GSOC] cat-file: re-implement --textconv, --filters optionsZheNing Hu via GitGitGadget, Jul 1, 2021
  34. Junio C HamanoJul 1, 2021
  35. Junio C HamanoJul 1, 2021
  36. ZheNing HuJul 2, 2021
  37. Junio C HamanoJul 2, 2021
  38. ZheNing HuJul 3, 2021
  39. 15/15 [GSOC] ref-filter: remove grab_oid() functionZheNing Hu via GitGitGadget, Jul 1, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.