git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Question: .idx without .pack causes performance issues?

From
Doug Kelly <dougk.ff7@gmail.com>
Date
Aug 7, 2015, 21:36 UTC
Message-ID
<CAEtYS8SGnFFHM5BFzAo+Z2BzUGbp47AibA3v6qm_uEboRmfaNQ@mail.gmail.com>
In-Reply-To
<xmqqk2tb6dxc.fsf@gitster.dls.corp.google.com>
On Mon, Aug 3, 2015 at 8:27 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 25 quoted lines
> Doug Kelly <dougk.ff7@gmail.com> writes:
>
>> Here's a change to prune.c that at least addresses the issue by removing
>> .idx files without an associated pack, but it's by no means pretty.  If anyone
>> has any feedback before I turn this into a formal patch, it's more than welcome!
>
> I'd hesitate to see removal of a file (for that matter, a creation
> too) inside a "while (de = readdir)" loop.  As the original function
> is about temporary files, and the new thing is not about temporary
> files at all, I'd further prefer that we do not do it in the same
> loop.
>
> I am wondering if we can add a new mode to report_pack_garbage() in
> sha1_file.c to allow it to remove stale and lone ".idx".  Most of
> the time we are accessing packs read-only, and I do not want the
> function to unconditionally remove lone ".idx", but perhaps we
> can teach "prune" to set a custom report_garbage() routine and
> react to a call to its custom report_garbage()?
>
> Perhaps that custom report_garbage() can make a list of ".idx"
> files, iterate over it to pick the lone one without ".pack" and
> remove them.  Or the custom report_garbage() can make a list of lone
> ".idx" files, if you tweak the interface to report_garbage() to
> contain th seen_bits value, avoiding the need to check the existence
> of ".pack" for the second time.

Yeah, I didn't think this was the cleanest solution, and I wasn't even thinking about removing while inside the readdir loop, but I can see how that might be a very bad idea. In any case, thanks for the suggestions... I'll be completely blunt in saying I'm far less than well-versed in the Git internals. Looking at the implementation of report_pack_garbage(), it does look like seen_bits already has this logic, and indeed, git count-objects -v reports the files as garbage.

So, I think you're right: prune would need to set report_garbage appropriately, then call count-objects to clean that up. If we wanted it to *only* care for lone idx files, we would have to string match on the message (seems fragile), but perhaps a more observant approach would be to add a custom flag to prune to clean *all* garbage in the repository, as passed to report_garbage? Probably wouldn't want to be enabled by default, but only on invocation or with careful consideration and setting an appropriate config flag.

Thoughts?
Previous: Junio C HamanoNext: Junio C Hamano
Message 8 of 34 in “Question: .idx without .pack causes performance issues?”
  1. Doug KellyJul 21, 2015
  2. Junio C HamanoJul 21, 2015
  3. Junio C HamanoJul 21, 2015
  4. Junio C HamanoJul 21, 2015
  5. Doug KellyJul 21, 2015
  6. Doug KellyAug 3, 2015
  7. Junio C HamanoAug 4, 2015
  8. Doug KellyAug 7, 2015
  9. Junio C HamanoAug 7, 2015
  10. 1/2 prepare_packed_git(): refactor garbage reporting in pack directoryDoug Kelly, Aug 13, 2015
  11. 2/2 gc: Remove garbage .idx files from pack dirDoug Kelly, Aug 13, 2015
  12. Junio C HamanoAug 17, 2015
  13. Junio C HamanoAug 17, 2015
  14. Eric SunshineAug 13, 2015
  15. Junio C HamanoAug 17, 2015
  16. Junio C HamanoOct 28, 2015
  17. Doug KellyOct 28, 2015
  18. 1/3 prepare_packed_git(): refactor garbage reporting in pack directoryDoug Kelly, Nov 4, 2015
  19. 2/3 t5304: Add test for cleaning pack garbageDoug Kelly, Nov 4, 2015
  20. 3/3 gc: Remove garbage .idx files from pack dirDoug Kelly, Nov 4, 2015
  21. Doug KellyNov 4, 2015
  22. Junio C HamanoNov 4, 2015
  23. Doug KellyNov 4, 2015
  24. Jeff KingNov 4, 2015
  25. Doug KellyNov 4, 2015
  26. Jeff KingNov 4, 2015
  27. Jeff KingDec 30, 2015
  28. Doug KellyJan 13, 2016
  29. Junio C HamanoJan 13, 2016
  30. Doug KellyJan 13, 2016
  31. Jeff KingJan 13, 2016
  32. Jeff KingNov 4, 2015
  33. Doug KellyJul 21, 2015
  34. Fwd: Question: .idx without .pack causes performance issues?Thomas Berg, Nov 11, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.