git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Question: .idx without .pack causes performance issues?

From
Junio C Hamano <gitster@pobox.com>
Date
Jul 21, 2015, 18:57 UTC
Message-ID
<xmqq4mkxwd77.fsf@gitster.dls.corp.google.com>
In-Reply-To
<CAEtYS8QWCg5_DtrJw-e+c50vcG0OpciR6LWon-3GgyngGn+0pQ@mail.gmail.com>
Doug Kelly <dougk.ff7@gmail.com> writes:
Show 14 quoted lines
> I just wanted to relay an issue we've seen before at my day job (and
> it just recently cropped up again).  When moving users from Git for
> Windows 1.8.3 to 1.9.5, we found a few users started having operations
> take an excruciatingly long amount of time.  At some point, we traced
> the issue to a number of .pack files had been deleted (possibly
> garbage collected?) -- but their associated .idx files were still
> present.  Upon removing the "orphaned" idx files, we found performance
> returned to normal.  Otherwise, git fsck reported no issues with the
> repositories.
>
> Other users have noted that using git gc would sometimes correct the
> issue for them, but not always.
>
> Anyway, has anyone else experienced this performance degradation?

I wouldn't be surprised if such a configuration to have leftover ".idx" files that lack ".pack" affected performance, but I think you really have to work on getting into such a situation (unless your operating system is very cooperative and tries hard to corrupt your repository, that is ;-), so I wouldn't be surprised if you were the first one to report this.

We open the ".idx" file and try to keep as many of them in-core, without opening corresponding ".pack" until the data is needed.

When we need an object, we learn from an ".idx" file that a particular pack ought to have a copy of it, and then attempt to open the corresponding ".pack" file. If this fails, we do protect ourselves from strange repositories with only ".idx" files by not using that ".idx" and try to see if the sought-after object exists elsewhere (and if there isn't we say "no such object", which is also a correct thing to do).

I however do not think that we mark the in-core structure that corresponds to an open ".idx" file in any way when such a failure happens. If we really cared enough, we could do so, saying "we know there is .idx file, but do not bother looking at it again, as we know the corresponding .pack is missing", and that would speed things up a bit, essentially bringing us back to a sane situation without any ".idx" without corresponding ".pack".

I do not think it is worth the effort, though. It would be more fruitful to find out how you end up with ".idx exists but not corresponding .pack" and if that is some systemic failure, see if there is a way to prevent that from happening in the first place.

Also, I think it may not be a bad idea to teach "gc" to remove stale ".idx" files that do not have corresponding ".pack" as garbage.

Previous: Doug KellyNext: Junio C Hamano
Message 2 of 34 in “Question: .idx without .pack causes performance issues?”
  1. Doug KellyJul 21, 2015
  2. Junio C HamanoJul 21, 2015
  3. Junio C HamanoJul 21, 2015
  4. Junio C HamanoJul 21, 2015
  5. Doug KellyJul 21, 2015
  6. Doug KellyAug 3, 2015
  7. Junio C HamanoAug 4, 2015
  8. Doug KellyAug 7, 2015
  9. Junio C HamanoAug 7, 2015
  10. 1/2 prepare_packed_git(): refactor garbage reporting in pack directoryDoug Kelly, Aug 13, 2015
  11. 2/2 gc: Remove garbage .idx files from pack dirDoug Kelly, Aug 13, 2015
  12. Junio C HamanoAug 17, 2015
  13. Junio C HamanoAug 17, 2015
  14. Eric SunshineAug 13, 2015
  15. Junio C HamanoAug 17, 2015
  16. Junio C HamanoOct 28, 2015
  17. Doug KellyOct 28, 2015
  18. 1/3 prepare_packed_git(): refactor garbage reporting in pack directoryDoug Kelly, Nov 4, 2015
  19. 2/3 t5304: Add test for cleaning pack garbageDoug Kelly, Nov 4, 2015
  20. 3/3 gc: Remove garbage .idx files from pack dirDoug Kelly, Nov 4, 2015
  21. Doug KellyNov 4, 2015
  22. Junio C HamanoNov 4, 2015
  23. Doug KellyNov 4, 2015
  24. Jeff KingNov 4, 2015
  25. Doug KellyNov 4, 2015
  26. Jeff KingNov 4, 2015
  27. Jeff KingDec 30, 2015
  28. Doug KellyJan 13, 2016
  29. Junio C HamanoJan 13, 2016
  30. Doug KellyJan 13, 2016
  31. Jeff KingJan 13, 2016
  32. Jeff KingNov 4, 2015
  33. Doug KellyJul 21, 2015
  34. Fwd: Question: .idx without .pack causes performance issues?Thomas Berg, Nov 11, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.