git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Fwd: Question: .idx without .pack causes performance issues?

From
TBThomas Berg <merlin66b@gmail.com>
Date
Nov 11, 2015, 14:58 UTC
Message-ID
<CABYiQpmYP=x-Urbwd0e_aa=iAMM4wP2bvdXwDN0=htEr5iOZAw@mail.gmail.com>
In-Reply-To
<CABYiQpn7r2Vcf=S5RaWHBN85eBYGPV_e02+BY=4L98qfUzDT1Q@mail.gmail.com>
Hi all,
(re-sending because my first e-mail was rejected due to html formatting)

While debugging a git fetch performance problem on Windows I came across this thread. The problem in our case was also caused by orphaned .idx files.

On Tue, Jul 21, 2015 at 9:15 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 19 quoted lines
>
> Junio C Hamano <gitster@pobox.com> writes:
>
> > I however do not think that we mark the in-core structure that
> > corresponds to an open ".idx" file in any way when such a failure
> > happens.  If we really cared enough, we could do so, saying "we know
> > there is .idx file, but do not bother looking at it again, as we
> > know the corresponding .pack is missing", and that would speed things
> > up a bit, essentially bringing us back to a sane situation without
> > any ".idx" without corresponding ".pack".
> >
> > I do not think it is worth the effort, though.  It would be more
> > fruitful to find out how you end up with ".idx exists but not
> > corresponding .pack" and if that is some systemic failure, see if
> > there is a way to prevent that from happening in the first place.
>
> While I still think that it is more important to prevent such a
> situation from occurring in the first place, ignoring .idx that lack
> corresponding .pack should be fairly simple, perhaps like this.

I have observed the following: if garbage collection is triggered during a git fetch, I always get messages like this:

$ git fetch origin
Show 13 quoted lines
> Auto packing the repository for optimum performance. You may also
> run "git gc" manually. See "git help gc" for more information.
> Counting objects: 396468, done.
> Delta compression using up to 12 threads.
> Compressing objects: 100% (98683/98683), done.
> Writing objects: 100% (396468/396468), done.
> Total 396468 (delta 289422), reused 395212 (delta 288289)
> Unlink of file '.git/objects/pack/pack-343b6cfdf58171f53c235b900a75d09bd9219e06.pack' failed. Should I try again? (y/n) n
> Unlink of file '.git/objects/pack/pack-343b6cfdf58171f53c235b900a75d09bd9219e06.idx' failed. Should I try again? (y/n) n
> Unlink of file '.git/objects/pack/pack-63a6cb5e2a9f72eea72b02ac74a167e1d71d417f.idx' failed. Should I try again? (y/n) n
> Unlink of file '.git/objects/pack/pack-9b616a2501bb9c13acecf3e981c39868dd2f5ff7.pack' failed. Should I try again? (y/n) n
> Unlink of file '.git/objects/pack/pack-9b616a2501bb9c13acecf3e981c39868dd2f5ff7.idx' failed. Should I try again? (y/n) n
> Checking connectivity: 396468, done.

Windows has the property that if a file is open it can't be deleted. If so, it could be that git fetch needs to close the files first. I can't remember observing this problem when running git gc by itself.

In the repos where we have problems I observed both unnecessary .pack files and .idx files, but way more .idx files. Maybe, over time, unnecessary pack files have been cleaned up but not .idx files?

If so, this would explain how we get into this situation. I have been testing this with very old git versions on Windows (1.7.4 and 1.8.4), sorry if these problems are already fixed in later versions.

- Thomas
Previous: Doug Kelly
Message 34 of 34 in “Question: .idx without .pack causes performance issues?”
  1. Doug KellyJul 21, 2015
  2. Junio C HamanoJul 21, 2015
  3. Junio C HamanoJul 21, 2015
  4. Junio C HamanoJul 21, 2015
  5. Doug KellyJul 21, 2015
  6. Doug KellyAug 3, 2015
  7. Junio C HamanoAug 4, 2015
  8. Doug KellyAug 7, 2015
  9. Junio C HamanoAug 7, 2015
  10. 1/2 prepare_packed_git(): refactor garbage reporting in pack directoryDoug Kelly, Aug 13, 2015
  11. 2/2 gc: Remove garbage .idx files from pack dirDoug Kelly, Aug 13, 2015
  12. Junio C HamanoAug 17, 2015
  13. Junio C HamanoAug 17, 2015
  14. Eric SunshineAug 13, 2015
  15. Junio C HamanoAug 17, 2015
  16. Junio C HamanoOct 28, 2015
  17. Doug KellyOct 28, 2015
  18. 1/3 prepare_packed_git(): refactor garbage reporting in pack directoryDoug Kelly, Nov 4, 2015
  19. 2/3 t5304: Add test for cleaning pack garbageDoug Kelly, Nov 4, 2015
  20. 3/3 gc: Remove garbage .idx files from pack dirDoug Kelly, Nov 4, 2015
  21. Doug KellyNov 4, 2015
  22. Junio C HamanoNov 4, 2015
  23. Doug KellyNov 4, 2015
  24. Jeff KingNov 4, 2015
  25. Doug KellyNov 4, 2015
  26. Jeff KingNov 4, 2015
  27. Jeff KingDec 30, 2015
  28. Doug KellyJan 13, 2016
  29. Junio C HamanoJan 13, 2016
  30. Doug KellyJan 13, 2016
  31. Jeff KingJan 13, 2016
  32. Jeff KingNov 4, 2015
  33. Doug KellyJul 21, 2015
  34. Fwd: Question: .idx without .pack causes performance issues?Thomas Berg, Nov 11, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.