git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 1/2] sha1_file: introduce close_one_pack() to close packs on fd pressure

From
Brandon Casey <drafnel@gmail.com>
Date
Aug 1, 2013, 20:37 UTC
Message-ID
<CA+sFfMdJgQaEBx_FsYPz1rC3--jknnb4Zwr+FOaL+9gNe4xwyw@mail.gmail.com>
In-Reply-To
<7vy58lqiwd.fsf@alter.siamese.dyndns.org>
On Thu, Aug 1, 2013 at 1:02 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 24 quoted lines
> Brandon Casey <drafnel@gmail.com> writes:
>
>> On Thu, Aug 1, 2013 at 11:39 AM, Junio C Hamano <gitster@pobox.com> wrote:
>>> That makes me feel somewhat uneasy.  Yes, you can open/mmap/close
>>> and hold onto the contents of a file still mapped in-core, and it
>>> may not count as "open filedescriptor", but do OSes allow infinite
>>> such mmapped regions to us?  We do keep track of number of open
>>> windows, but is there a way for us to learn how close we are to the
>>> limit?
>>
>> Not that I know of, but xmmap() does already try to unmap existing
>> windows when mmap() fails, and then retries the mmap.  It calls
>> release_pack_memory() which calls unuse_one_window().  mmap returns
>> ENOMEM when either there is no available memory or if the limit of
>> mmap mappings has been exceeded.
>
> OK, so if there were such an OS limit, the unuse_one_window() will
> hopefully reduce the number of open windows and as a side effect we
> may go below that limit.
>
> What I was worried about was if there was a limit on the number of
> files we have windows into (i.e. having one window each in N files,
> with fds all closed, somehow costs more than having N window in one
> file with the fd closed).

Ah, yeah, I've never heard of that type of limit and I do not know if there is one.

If there is such a limit, like you said unuse_one_window() will _hopefully_ release enough windows to reduce the number of packs we have windows into, but it is certainly not guaranteed.

Show 6 quoted lines
> We currently have knobs for total number
> of windows and number of open fds consumed for packs, and the latter
> indirectly controls the number of active packfiles we have windows
> into. Your proposed change will essentially make the number of
> active packfiles unlimited by any of our knobs, and that was where
> my uneasiness was coming from.

Yes and no. The limit on the number of open fds used for packs only indirectly controls the number of active packfiles we have windows into for the packs that are larger than packedGitWindowSize. For pack files smaller than packedGitWindowSize, the number was unlimited too since we close the file descriptor if the whole pack fits within one window.

-Brandon
Previous: Junio C HamanoNext: Brandon Casey
Message 20 of 23 in “sha1_file: introduce close_one_pack() to close packs on fd pressure”
  1. sha1_file: introduce close_one_pack() to close packs on fd pressureBrandon Casey, Jul 30, 2013
  2. Eric SunshineJul 30, 2013
  3. Junio C HamanoJul 30, 2013
  4. Jeff KingJul 30, 2013
  5. Brandon CaseyJul 30, 2013
  6. 1/2 sha1_file: introduce close_one_pack() to close packs on fd pressureBrandon Casey, Jul 31, 2013
  7. 2/2 Don't close pack fd when free'ing pack windowsBrandon Casey, Jul 31, 2013
  8. Antoine PelisseJul 31, 2013
  9. Fredrik GustafssonJul 31, 2013
  10. Brandon CaseyJul 31, 2013
  11. Fredrik GustafssonJul 31, 2013
  12. Brandon CaseyJul 31, 2013
  13. Thomas RastJul 31, 2013
  14. Junio C HamanoAug 1, 2013
  15. Brandon CaseyAug 1, 2013
  16. Junio C HamanoAug 1, 2013
  17. Brandon CaseyAug 1, 2013
  18. Brandon CaseyAug 1, 2013
  19. Junio C HamanoAug 1, 2013
  20. Brandon CaseyAug 1, 2013
  21. sha1_file: introduce close_one_pack() to close packs on fd pressureBrandon Casey, Aug 2, 2013
  22. Junio C HamanoAug 2, 2013
  23. Brandon CaseyAug 2, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.