git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 2/8] maintenance: add loose-objects task

From
Derrick Stolee <stolee@gmail.com>
Date
Sep 24, 2020, 13:45 UTC
Message-ID
<446e805a-6b83-a6b7-18a3-f99ac85a6fe1@gmail.com>
In-Reply-To
<20200922230937.906065-1-jonathantanmy@google.com>
On 9/22/2020 7:09 PM, Jonathan Tan wrote:
Show 31 quoted lines
>> Create a 'loose-objects' task for the 'git maintenance run' command.
>> This helps clean up loose objects without disrupting concurrent Git
>> commands using the following sequence of events:
>>
>> 1. Run 'git prune-packed' to delete any loose objects that exist
>>    in a pack-file. Concurrent commands will prefer the packed
>>    version of the object to the loose version. (Of course, there
>>    are exceptions for commands that specifically care about the
>>    location of an object. These are rare for a user to run on
>>    purpose, and we hope a user that has selected background
>>    maintenance will not be trying to do foreground maintenance.)
>>
>> 2. Run 'git pack-objects' on a batch of loose objects. These
>>    objects are grouped by scanning the loose object directories in
>>    lexicographic order until listing all loose objects -or-
>>    reaching 50,000 objects. This is more than enough if the loose
>>    objects are created only by a user doing normal development.
>>    We noticed users with _millions_ of loose objects because VFS
>>    for Git downloads blobs on-demand when a file read operation
>>    requires populating a virtual file. 
> 
> [snip]
> 
>> This has potential of
>>    happening in partial clones if someone runs 'git grep' or
>>    otherwise evades the batch-download feature for requesting
>>    promisor objects.
> 
> This part is not strictly true, as even when Git lazy-fetches one
> object, it fetches it in the form of a packfile - so maybe remove this
> sentence.

This is a good point. I just did some testing and we do store these single-object downloads as pack-files. My misunderstanding is due to my own bias and experience with the GVFS protocol.

I have also heard that "git fetch" might explode some small pack-files into loose objects, and I guess I expected the same here. However, that is not the case for partial clone. I'll remove this.

Show 6 quoted lines
> This is nevertheless a good feature to have - loose objects may not be
> created during lazy fetches, but they definitely are created during
> normal operation (e.g. commits). Git, as a whole, prefers packfiles over
> loose objects, and just packing the loose objects themselves instead of
> running repack (which goes through all reachable objects) is definitely
> better for large repositories.

Thanks, -Stolee

Previous: Jonathan TanNext: Derrick Stolee via GitGitGadget
Message 38 of 66 in “Maintenance II: prefetch, loose-objects, incremental-repack tasks”
  1. 0/9 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Aug 6, 2020
  2. 8/9 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Aug 6, 2020
  3. Son Luong NgocAug 6, 2020
  4. Derrick StoleeAug 6, 2020
  5. 6/9 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Aug 6, 2020
  6. 9/9 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Aug 6, 2020
  7. 7/9 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Aug 6, 2020
  8. 5/9 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Aug 6, 2020
  9. 4/9 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Aug 6, 2020
  10. 3/9 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Aug 6, 2020
  11. Emily ShafferAug 12, 2020
  12. Derrick StoleeAug 14, 2020
  13. 2/9 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Aug 6, 2020
  14. Emily ShafferAug 12, 2020
  15. Derrick StoleeAug 14, 2020
  16. 1/9 fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano via GitGitGadget, Aug 6, 2020
  17. Emily ShafferAug 12, 2020
  18. Junio C HamanoAug 13, 2020
  19. Jonathan NiederAug 13, 2020
  20. fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano, Aug 13, 2020
  21. Derrick StoleeAug 14, 2020
  22. Junio C HamanoAug 14, 2020
  23. 0/9 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Aug 18, 2020
  24. 1/9 fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano via GitGitGadget, Aug 18, 2020
  25. 3/9 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Aug 18, 2020
  26. 5/9 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Aug 18, 2020
  27. 2/9 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Aug 18, 2020
  28. 8/9 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Aug 18, 2020
  29. 9/9 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Aug 18, 2020
  30. 7/9 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Aug 18, 2020
  31. 6/9 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Aug 18, 2020
  32. 4/9 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Aug 18, 2020
  33. 0/8 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Aug 25, 2020
  34. 1/8 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Aug 25, 2020
  35. Jonathan TanSep 22, 2020
  36. 2/8 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Aug 25, 2020
  37. Jonathan TanSep 22, 2020
  38. Derrick StoleeSep 24, 2020
  39. 3/8 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Aug 25, 2020
  40. Jonathan TanSep 22, 2020
  41. Derrick StoleeSep 24, 2020
  42. 4/8 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Aug 25, 2020
  43. Jonathan TanSep 22, 2020
  44. Derrick StoleeSep 24, 2020
  45. 5/8 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Aug 25, 2020
  46. 6/8 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Aug 25, 2020
  47. Jonathan TanSep 22, 2020
  48. Derrick StoleeSep 24, 2020
  49. Jonathan TanSep 24, 2020
  50. 8/8 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Aug 25, 2020
  51. Jonathan TanSep 22, 2020
  52. 7/8 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Aug 25, 2020
  53. Junio C HamanoAug 25, 2020
  54. Son Luong NgocAug 26, 2020
  55. Derrick StoleeAug 26, 2020
  56. 0/8 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Sep 25, 2020
  57. 1/8 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Sep 25, 2020
  58. 3/8 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Sep 25, 2020
  59. Junio C HamanoSep 25, 2020
  60. Derrick StoleeSep 25, 2020
  61. 4/8 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Sep 25, 2020
  62. 2/8 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Sep 25, 2020
  63. 5/8 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Sep 25, 2020
  64. 7/8 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Sep 25, 2020
  65. 8/8 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Sep 25, 2020
  66. 6/8 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Sep 25, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.