git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/9] pack-objects: add `--print-filtered` to print omitted objects

From
Christian Couder <christian.couder@gmail.com>
Date
Jun 21, 2023, 11:11 UTC
Message-ID
<CAP8UFD1cAMg-54YTmW6N9sbi3KKs2OmPRz7zshWDGkStgiZnUQ@mail.gmail.com>
In-Reply-To
<ZJLWgfCNoqE6xNqq@nand.local>
On Wed, Jun 21, 2023 at 12:52 PM Taylor Blau <me@ttaylorr.com> wrote:
>
> On Thu, Jun 15, 2023 at 03:50:17PM -0700, Junio C Hamano wrote:
Show 22 quoted lines
> > Makes sense.  It is a bit sad that we have to accumulate everything
> > until the end at which time we have to dump the accumulated in bulk,
> > but that is a current limitation of list-objects-filter API and not
> > within the scope of this change.  We may in the longer term want to
> > see if we can make the collection of filtered-out objects streamable
> > by replacing the .omits object array with a callback function, or do
> > something along that line.
>
> Hmm. I think it is possible to use something like `git pack-objects`'s
> `--stdin-packs` mode to accomplish this without needing to keep track of
> the set of discarded objects (i.e. those which don't match the filter).
>
> IIUC, the set of objects which don't match the filter is the same as the
> set of all objects in packs beforehand, differenced with the set of
> objects that shows up in the pack containing objects which *do* match
> the filter.
>
> If you mark all of the "before" packs with `-` in the input to
> `--stdin-packs`, and then pass along the pack containing the filtered
> set without `-` (to indicate that the resulting pack should not contain
> any objects which appear in that pack), I think you would end up with
> the set of non-matching objects.

I agree that it can be done like this, but I am not sure it's very efficient to do it like this. When we create the pack with filtered out objects, we know the set of objects we filtered out, so it doesn't seem efficient to make `git pack-objects --stdin-packs` read more packfiles or their indexes than necessary and compute that set of objects again.

Now I haven't checked if there is a real performance difference for large packfiles, and perhaps `git pack-objects --stdin-packs` is very efficient. But I hope that going the way I implemented it and perhaps using some optimization ideas that Junio suggested above, will make it easier to improve performance in the future.

Previous: Taylor BlauNext: Taylor Blau
Message 8 of 161 in “Repack objects into separate packfiles based on a filter”
  1. 0/9 Repack objects into separate packfiles based on a filterChristian Couder, Jun 14, 2023
  2. 1/9 pack-objects: allow `--filter` without `--stdout`Christian Couder, Jun 14, 2023
  3. Taylor BlauJun 21, 2023
  4. Christian CouderJul 5, 2023
  5. 2/9 pack-objects: add `--print-filtered` to print omitted objectsChristian Couder, Jun 14, 2023
  6. Junio C HamanoJun 15, 2023
  7. Taylor BlauJun 21, 2023
  8. Christian CouderJun 21, 2023
  9. Taylor BlauJun 21, 2023
  10. 3/9 t/helper: add 'find-pack' test-toolChristian Couder, Jun 14, 2023
  11. Junio C HamanoJun 15, 2023
  12. Christian CouderJun 21, 2023
  13. Taylor BlauJun 21, 2023
  14. 5/9 repack: refactor finishing pack-objects commandChristian Couder, Jun 14, 2023
  15. Junio C HamanoJun 16, 2023
  16. Taylor BlauJun 21, 2023
  17. Christian CouderJun 21, 2023
  18. Taylor BlauJun 21, 2023
  19. 4/9 repack: refactor piping an oid to a commandChristian Couder, Jun 14, 2023
  20. Junio C HamanoJun 15, 2023
  21. Taylor BlauJun 21, 2023
  22. Christian CouderJun 21, 2023
  23. 6/9 repack: add `--filter=<filter-spec>` optionChristian Couder, Jun 14, 2023
  24. Junio C HamanoJun 16, 2023
  25. Taylor BlauJun 21, 2023
  26. Christian CouderJun 21, 2023
  27. Taylor BlauJun 22, 2023
  28. Christian CouderJun 21, 2023
  29. Junio C HamanoJun 21, 2023
  30. Christian CouderJun 22, 2023
  31. Junio C HamanoJun 22, 2023
  32. Taylor BlauJun 21, 2023
  33. Christian CouderJul 5, 2023
  34. 8/9 repack: implement `--filter-to` for storing filtered out objectsChristian Couder, Jun 14, 2023
  35. Junio C HamanoJun 16, 2023
  36. Taylor BlauJun 21, 2023
  37. Christian CouderJun 21, 2023
  38. Taylor BlauJun 21, 2023
  39. Junio C HamanoJun 21, 2023
  40. Christian CouderJul 5, 2023
  41. 7/9 gc: add `gc.repackFilter` config optionChristian Couder, Jun 14, 2023
  42. 9/9 gc: add `gc.repackFilterTo` config optionChristian Couder, Jun 14, 2023
  43. Junio C HamanoJun 16, 2023
  44. Junio C HamanoJun 14, 2023
  45. Junio C HamanoJun 16, 2023
  46. 0/8 Repack objects into separate packfiles based on a filterChristian Couder, Jul 5, 2023
  47. 1/8 pack-objects: allow `--filter` without `--stdout`Christian Couder, Jul 5, 2023
  48. 2/8 t/helper: add 'find-pack' test-toolChristian Couder, Jul 5, 2023
  49. 3/8 repack: refactor finishing pack-objects commandChristian Couder, Jul 5, 2023
  50. 4/8 repack: refactor finding pack prefixChristian Couder, Jul 5, 2023
  51. 5/8 repack: add `--filter=<filter-spec>` optionChristian Couder, Jul 5, 2023
  52. Junio C HamanoJul 5, 2023
  53. Christian CouderJul 24, 2023
  54. Junio C HamanoJul 24, 2023
  55. Christian CouderJul 25, 2023
  56. Junio C HamanoJul 25, 2023
  57. Junio C HamanoJul 25, 2023
  58. Christian CouderAug 8, 2023
  59. Taylor BlauAug 9, 2023
  60. Junio C HamanoAug 9, 2023
  61. Junio C HamanoAug 9, 2023
  62. Jeff KingAug 10, 2023
  63. Junio C HamanoJul 5, 2023
  64. Christian CouderJul 24, 2023
  65. 6/8 gc: add `gc.repackFilter` config optionChristian Couder, Jul 5, 2023
  66. 8/8 gc: add `gc.repackFilterTo` config optionChristian Couder, Jul 5, 2023
  67. 7/8 repack: implement `--filter-to` for storing filtered out objectsChristian Couder, Jul 5, 2023
  68. Junio C HamanoJul 5, 2023
  69. Christian CouderJul 24, 2023
  70. Junio C HamanoJul 24, 2023
  71. Robert CoupJul 25, 2023
  72. Junio C HamanoJul 25, 2023
  73. Christian CouderJul 25, 2023
  74. 0/8 Repack objects into separate packfiles based on a filterChristian Couder, Jul 24, 2023
  75. 1/8 pack-objects: allow `--filter` without `--stdout`Christian Couder, Jul 24, 2023
  76. Taylor BlauJul 25, 2023
  77. Junio C HamanoJul 25, 2023
  78. 3/8 repack: refactor finishing pack-objects commandChristian Couder, Jul 24, 2023
  79. Taylor BlauJul 25, 2023
  80. 4/8 repack: refactor finding pack prefixChristian Couder, Jul 24, 2023
  81. Taylor BlauJul 25, 2023
  82. Christian CouderAug 8, 2023
  83. 2/8 t/helper: add 'find-pack' test-toolChristian Couder, Jul 24, 2023
  84. Taylor BlauJul 25, 2023
  85. Christian CouderAug 8, 2023
  86. 5/8 repack: add `--filter=<filter-spec>` optionChristian Couder, Jul 24, 2023
  87. Taylor BlauJul 25, 2023
  88. Christian CouderAug 8, 2023
  89. Taylor BlauAug 9, 2023
  90. 6/8 gc: add `gc.repackFilter` config optionChristian Couder, Jul 24, 2023
  91. Taylor BlauJul 25, 2023
  92. Christian CouderAug 8, 2023
  93. Taylor BlauAug 9, 2023
  94. 8/8 gc: add `gc.repackFilterTo` config optionChristian Couder, Jul 24, 2023
  95. 7/8 repack: implement `--filter-to` for storing filtered out objectsChristian Couder, Jul 24, 2023
  96. Taylor BlauJul 25, 2023
  97. 0/8 Repack objects into separate packfiles based on a filterChristian Couder, Aug 8, 2023
  98. 7/8 repack: implement `--filter-to` for storing filtered out objectsChristian Couder, Aug 8, 2023
  99. 5/8 repack: add `--filter=<filter-spec>` optionChristian Couder, Aug 8, 2023
  100. Taylor BlauAug 9, 2023
  101. 4/8 repack: refactor finding pack prefixChristian Couder, Aug 8, 2023
  102. Taylor BlauAug 9, 2023
  103. 3/8 repack: refactor finishing pack-objects commandChristian Couder, Aug 8, 2023
  104. 6/8 gc: add `gc.repackFilter` config optionChristian Couder, Aug 8, 2023
  105. 8/8 gc: add `gc.repackFilterTo` config optionChristian Couder, Aug 8, 2023
  106. 1/8 pack-objects: allow `--filter` without `--stdout`Christian Couder, Aug 8, 2023
  107. 2/8 t/helper: add 'find-pack' test-toolChristian Couder, Aug 8, 2023
  108. Taylor BlauAug 9, 2023
  109. Taylor BlauAug 9, 2023
  110. Junio C HamanoAug 9, 2023
  111. Christian CouderAug 12, 2023
  112. 0/8 Repack objects into separate packfiles based on a filterChristian Couder, Aug 12, 2023
  113. 1/8 pack-objects: allow `--filter` without `--stdout`Christian Couder, Aug 12, 2023
  114. 2/8 t/helper: add 'find-pack' test-toolChristian Couder, Aug 12, 2023
  115. 3/8 repack: refactor finishing pack-objects commandChristian Couder, Aug 12, 2023
  116. 4/8 repack: refactor finding pack prefixChristian Couder, Aug 12, 2023
  117. 5/8 repack: add `--filter=<filter-spec>` optionChristian Couder, Aug 12, 2023
  118. 6/8 gc: add `gc.repackFilter` config optionChristian Couder, Aug 12, 2023
  119. 7/8 repack: implement `--filter-to` for storing filtered out objectsChristian Couder, Aug 12, 2023
  120. 8/8 gc: add `gc.repackFilterTo` config optionChristian Couder, Aug 12, 2023
  121. Junio C HamanoAug 15, 2023
  122. Taylor BlauAug 15, 2023
  123. Junio C HamanoAug 15, 2023
  124. Taylor BlauAug 15, 2023
  125. Junio C HamanoAug 15, 2023
  126. Taylor BlauAug 16, 2023
  127. Junio C HamanoAug 16, 2023
  128. Christian CouderSep 11, 2023
  129. 0/9 Repack objects into separate packfiles based on a filterChristian Couder, Sep 11, 2023
  130. 4/9 repack: refactor finding pack prefixChristian Couder, Sep 11, 2023
  131. 5/9 pack-bitmap-write: rebuild using new bitmap when remappingChristian Couder, Sep 11, 2023
  132. 2/9 t/helper: add 'find-pack' test-toolChristian Couder, Sep 11, 2023
  133. 6/9 repack: add `--filter=<filter-spec>` optionChristian Couder, Sep 11, 2023
  134. 1/9 pack-objects: allow `--filter` without `--stdout`Christian Couder, Sep 11, 2023
  135. 8/9 repack: implement `--filter-to` for storing filtered out objectsChristian Couder, Sep 11, 2023
  136. 7/9 gc: add `gc.repackFilter` config optionChristian Couder, Sep 11, 2023
  137. 3/9 repack: refactor finishing pack-objects commandChristian Couder, Sep 11, 2023
  138. 9/9 gc: add `gc.repackFilterTo` config optionChristian Couder, Sep 11, 2023
  139. 0/9 Repack objects into separate packfiles based on a filterChristian Couder, Sep 25, 2023
  140. 1/9 pack-objects: allow `--filter` without `--stdout`Christian Couder, Sep 25, 2023
  141. 2/9 t/helper: add 'find-pack' test-toolChristian Couder, Sep 25, 2023
  142. 3/9 repack: refactor finishing pack-objects commandChristian Couder, Sep 25, 2023
  143. 4/9 repack: refactor finding pack prefixChristian Couder, Sep 25, 2023
  144. 5/9 pack-bitmap-write: rebuild using new bitmap when remappingChristian Couder, Sep 25, 2023
  145. 7/9 gc: add `gc.repackFilter` config optionChristian Couder, Sep 25, 2023
  146. 6/9 repack: add `--filter=<filter-spec>` optionChristian Couder, Sep 25, 2023
  147. 8/9 repack: implement `--filter-to` for storing filtered out objectsChristian Couder, Sep 25, 2023
  148. 9/9 gc: add `gc.repackFilterTo` config optionChristian Couder, Sep 25, 2023
  149. Junio C HamanoSep 25, 2023
  150. Taylor BlauSep 25, 2023
  151. 0/9 Repack objects into separate packfiles based on a filterChristian Couder, Oct 2, 2023
  152. 1/9 pack-objects: allow `--filter` without `--stdout`Christian Couder, Oct 2, 2023
  153. 2/9 t/helper: add 'find-pack' test-toolChristian Couder, Oct 2, 2023
  154. 3/9 repack: refactor finishing pack-objects commandChristian Couder, Oct 2, 2023
  155. 4/9 repack: refactor finding pack prefixChristian Couder, Oct 2, 2023
  156. 5/9 pack-bitmap-write: rebuild using new bitmap when remappingChristian Couder, Oct 2, 2023
  157. 7/9 gc: add `gc.repackFilter` config optionChristian Couder, Oct 2, 2023
  158. 9/9 gc: add `gc.repackFilterTo` config optionChristian Couder, Oct 2, 2023
  159. 8/9 repack: implement `--filter-to` for storing filtered out objectsChristian Couder, Oct 2, 2023
  160. 6/9 repack: add `--filter=<filter-spec>` optionChristian Couder, Oct 2, 2023
  161. Taylor BlauOct 2, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.