git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 0/4] commit-graph: handle file descriptor exhaustion

From
Taylor Blau <me@ttaylorr.com>
Date
Apr 23, 2020, 21:40 UTC
Message-ID
<cover.1587677671.git.me@ttaylorr.com>
Hi,

This is another series to handle an issue we started seeing at GitHub on repositories that have too many commit-graph layers.

This issue evaded me for several days, but the gist is that having too many descriptors open caused us to fail 'git_mkstemp_mode', which in turn left some values in 'ctx->commit_graph_filenames_after' uninitialized. This in turn caused a general protection fault when trying to free these uninitialized entries. Frustratingly, it causes stray 'commit-graph-chain.lock' files to stick around even after the process has gone away, causing all subsequent commit-graph write operations to fail.

The first two patches are semi-related cleanups and other preparation. The third patch fixes the main issue described here. Peff contributed the fourth patch which teaches the commit-graph machinery to be more willing to close file descriptors.

These patches are very much owed to the help that Stolee and Peff provided while writing them and for their willingness to let me bounce some ideas off of them.

Thanks in advance for your review.
Jeff King (1):
  commit-graph: close descriptors after mmap
Taylor Blau (3):
  commit-graph.c: don't use discarded graph_name in error
  t/test-lib.sh: make ULIMIT_FILE_DESCRIPTORS available to tests
  commit-graph.c: gracefully handle file descriptor exhaustion
 commit-graph.c                | 21 ++++++++-------------
 commit-graph.h                |  3 +--
 fuzz-commit-graph.c           |  5 ++---
 t/t1400-update-ref.sh         |  9 ---------
 t/t5324-split-commit-graph.sh | 13 +++++++++++++
 t/test-lib.sh                 |  9 +++++++++
 6 files changed, 33 insertions(+), 27 deletions(-)

-- 2.26.0.113.ge9739cdccc

Next: Taylor Blau
Message 1 of 14 in “commit-graph: handle file descriptor exhaustion”
  1. 0/4 commit-graph: handle file descriptor exhaustionTaylor Blau, Apr 23, 2020
  2. 1/4 commit-graph.c: don't use discarded graph_name in errorTaylor Blau, Apr 23, 2020
  3. 2/4 t/test-lib.sh: make ULIMIT_FILE_DESCRIPTORS available to testsTaylor Blau, Apr 23, 2020
  4. 4/4 commit-graph: close descriptors after mmapTaylor Blau, Apr 23, 2020
  5. Junio C HamanoApr 23, 2020
  6. Jeff KingApr 24, 2020
  7. Derrick StoleeApr 24, 2020
  8. Taylor BlauApr 24, 2020
  9. Junio C HamanoApr 24, 2020
  10. Derrick StoleeApr 27, 2020
  11. 3/4 commit-graph.c: gracefully handle file descriptor exhaustionTaylor Blau, Apr 23, 2020
  12. t5324-split-commit-graph.sh flaky due to assumptions about ulimit behaviorÆvar Arnfjörð Bjarmason, Jun 24, 2021
  13. Jeff KingJun 24, 2021
  14. Taylor BlauApr 23, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.