git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v5 4/4] convert: Stream from fd to required clean filter instead of mmap

From
Jeff King <peff@peff.net>
Date
Aug 26, 2014, 17:54 UTC
Message-ID
<20140826175428.GA17546@peff.net>
In-Reply-To
<E23693B7-0D9D-477D-A303-4A68433EAB79@zib.de>
On Mon, Aug 25, 2014 at 06:55:51PM +0200, Steffen Prohaska wrote:
Show 9 quoted lines
> It could be handled that way, but we would be back to the original problem
> that 32-bit git fails for large files.  The convert code path currently
> assumes that all data is available in a single buffer at some point to apply
> crlf and ident filters.
> 
> If the initial filter, which is assumed to reduce the file size, fails, we
> could seek to 0 and read the entire file.  But git would then fail for large
> files with out-of-memory.  We would not gain anything for the use case that
> I describe in the commit message's first paragraph.

Ah. So the real problem is that we cannot handle _other_ conversions for large files, and we must try to intercept the data before it gets to them. So this is really just helping "reduction" filters. Even if our streaming filter succeeds, it does not help the situation if it did not reduce the large file to a smaller one.

It would be nice in the long run to let the other filters stream, too, but that is not a problem we need to solve immediately. Your patch is a strict improvement.

Thanks for the explanation; your approach makes a lot more sense to me now.

-Peff
Previous: Junio C Hamano
Message 15 of 15 in “Stream fd to clean filter; GIT_MMAP_LIMIT, GIT_ALLOC_LIMIT with git_parse_ulong()”
  1. 0/4 Stream fd to clean filter; GIT_MMAP_LIMIT, GIT_ALLOC_LIMIT with git_parse_ulong()Steffen Prohaska, Aug 24, 2014
  2. 1/4 convert: Refactor would_convert_to_git() to single arg 'path'Steffen Prohaska, Aug 24, 2014
  3. Junio C HamanoAug 25, 2014
  4. 2/4 Change GIT_ALLOC_LIMIT check to use git_parse_ulong()Steffen Prohaska, Aug 24, 2014
  5. Jeff KingAug 25, 2014
  6. Steffen ProhaskaAug 25, 2014
  7. Jeff KingAug 25, 2014
  8. 3/4 Introduce GIT_MMAP_LIMIT to allow testing expected mmap sizeSteffen Prohaska, Aug 24, 2014
  9. 4/4 convert: Stream from fd to required clean filter instead of mmapSteffen Prohaska, Aug 24, 2014
  10. Jeff KingAug 25, 2014
  11. Steffen ProhaskaAug 25, 2014
  12. Junio C HamanoAug 25, 2014
  13. Jeff KingAug 26, 2014
  14. Junio C HamanoAug 26, 2014
  15. Jeff KingAug 26, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.