git_checkattr() is inefficient when repeated [Re: [PATCH 00/11] Micro-optimizing lookup_object()]

Junio C Hamano wrote:

Cool stuff!

While looking at the performance of it, I noticed something odd about packing: stracing the command you gave for your timings

  strace -o pack.trace \
    ./git-pack-objects --count-only --keep-true-parents --honor-pack-keep \
    --non-empty --all --reflog --no-reuse-delta --delta-base-offset \
    --stdout  </dev/null >/dev/null

yields the fairly crazy

  $ grep -c 'open.*attrib' pack.trace 
  4398

including runs such as (with a line of context for clarity)

  munmap(0x7f9cd39f7000, 4096)            = 0
  open("compat/.gitattributes", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("compat/.gitattributes", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("compat/.gitattributes", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("compat/.gitattributes", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("compat/.gitattributes", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("compat/.gitattributes", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("compat/.gitattributes", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("compat/.gitattributes", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("t/.gitattributes", O_RDONLY)      = 3
  fstat(3, {st_mode=S_IFREG|0644, st_size=36, ...}) = 0

So calling git_checkattr in a loop is quite inefficient. Indeed there's a good optimization opportunity: compiled for 4-way hashing I have (best of 3)

  6.76user 0.23system 0:07.02elapsed 99%CPU (0avgtext+0avgdata 479792maxresident)

but making no_try_delta() in pack-objects.c a dummy 'return 0' gives

  6.45user 0.13system 0:06.61elapsed 99%CPU (0avgtext+0avgdata 478256maxresident)

Which would be a 4.5% speedup. Obviously that won't quite be attainable since we want the attributes mechanism to work, but we still shouldn't have to open 4398 .gitattributes files when there are only 8 .gitattributes plus one .git/info/attributes.