git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCHv2 1/5] update-ref: test handling large transactions properly

From
Jeff King <peff@peff.net>
Date
Jan 22, 2015, 13:07 UTC
Message-ID
<20150122130746.GD19681@peff.net>
In-Reply-To
<54C0D6F9.1000808@alum.mit.edu>
On Thu, Jan 22, 2015 at 11:54:49AM +0100, Michael Haggerty wrote:
Show 23 quoted lines
> > +run_with_limited_open_files () {
> > +	(ulimit -n 32 && "$@")
> > +}
> 
> Regarding the choice of "32", I wonder what is the worst-case number of
> open file descriptors that are needed *before* counting the ones that
> are currently wasted on open loose-reference locks. On Linux it seems to
> be only 4 with my setup:
> 
>     $ (ulimit -n 3 && git update-ref --stdin </dev/null)
>     bash: /dev/null: Too many open files
>     $ (ulimit -n 4 && git update-ref --stdin </dev/null)
>     $
> 
> This number might depend a little bit on details of the repository, like
> whether config files import other config files. But as long as the
> "background" number of fds required is at least a few less than 32, then
> your number should be safe.
> 
> Does anybody know of a platform where file descriptors are eaten up
> gluttonously by, for example, each shared library that is in use or
> something? That's the only think I can think of that could potentially
> make your choice of 32 problematic.

It's not just choice of platform. There could be inherited descriptors in the environment (e.g., the test suite is being run by a shell that keeps a pipe to CI server open, or something). And the test suite itself uses several extra descriptors for hiding and showing output.

I think this is the sort of thing that we have to determine with a mix of guessing and empiricism. 4 is almost certainly too low. 32 looks "pretty big" in practice but not so big that it will make the test slow. I think our best bet is probably to ship it and see if anybody reports problems while the patch cooks. Then we can bump the number (or find a new approach) as necessary.

-Peff
Previous: Michael HaggertyNext: Stefan Beller
Message 4 of 45 in “[PATCHv2 0/5] Fix bug in large transactions”
  1. Stefan BellerJan 22, 2015
  2. 1/5 update-ref: test handling large transactions properlyStefan Beller, Jan 22, 2015
  3. Michael HaggertyJan 22, 2015
  4. Jeff KingJan 22, 2015
  5. 2/5 t7004: rename ULIMIT test prerequisite to ULIMIT_STACK_SIZEStefan Beller, Jan 22, 2015
  6. 3/5 refs.c: remove lock_fd from struct ref_lockStefan Beller, Jan 22, 2015
  7. 4/5 refs.c: have a write_sha1_to_lock_file wrapperStefan Beller, Jan 22, 2015
  8. 5/5 refs.c: enable large transactionsStefan Beller, Jan 22, 2015
  9. Michael HaggertyJan 22, 2015
  10. Jeff KingJan 22, 2015
  11. Michael HaggertyJan 22, 2015
  12. Stefan BellerJan 22, 2015
  13. 0/5 So you dislike the sequence of system calls?Stefan Beller, Jan 22, 2015
  14. 1/5 fixup for "refs.c: enable large transactions"Stefan Beller, Jan 22, 2015
  15. 2/5 refs.c: remove unlock_ref from write_ref_sha1Stefan Beller, Jan 22, 2015
  16. 3/5 refs.c: move static functions to close and commit refsStefan Beller, Jan 22, 2015
  17. 4/5 refs.c: remove committing the ref from write_ref_sha1Stefan Beller, Jan 22, 2015
  18. 5/5 refs.c: write values to lock files early for committingStefan Beller, Jan 22, 2015
  19. Ramsay JonesJan 22, 2015
  20. Stefan BellerJan 22, 2015
  21. Ramsay JonesJan 22, 2015
  22. Ramsay JonesJan 22, 2015
  23. Stefan BellerJan 22, 2015
  24. Ramsay JonesJan 22, 2015
  25. Michael HaggertyJan 22, 2015
  26. 0/6 Fix bug in large transactionsStefan Beller, Jan 23, 2015
  27. 1/6 update-ref: test handling large transactions properlyStefan Beller, Jan 23, 2015
  28. 2/6 t7004: rename ULIMIT test prerequisite to ULIMIT_STACK_SIZEStefan Beller, Jan 23, 2015
  29. 3/6 refs.c: remove lock_fd from struct ref_lockStefan Beller, Jan 23, 2015
  30. 4/6 refs.c: move static functions to close and commit refsStefan Beller, Jan 23, 2015
  31. 5/6 refs.c: remove unlock_ref and commit_ref from write_ref_sha1Stefan Beller, Jan 23, 2015
  32. Junio C HamanoJan 23, 2015
  33. Stefan BellerJan 24, 2015
  34. Junio C HamanoJan 24, 2015
  35. Stefan BellerJan 24, 2015
  36. Junio C HamanoJan 24, 2015
  37. 6/6 refs.c: enable large transactionsStefan Beller, Jan 23, 2015
  38. Junio C HamanoJan 24, 2015
  39. Stefan BellerJan 24, 2015
  40. Junio C HamanoJan 24, 2015
  41. Stefan BellerJan 26, 2015
  42. refs.c: clean up write_ref_sha1 returnsStefan Beller, Jan 26, 2015
  43. Junio C HamanoJan 27, 2015
  44. Stefan BellerJan 28, 2015
  45. Junio C HamanoJan 27, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.