Re: [PATCH] don't use mmap() to hash files

From: Thomas Rast <trast@student.ethz.ch>
Date: Feb 14, 2010, 11:14 UTC
Message-ID: <201002141214.15025.trast@student.ethz.ch>
In-Reply-To: <7v8wawy0ee.fsf@alter.siamese.dyndns.org>

On Sunday 14 February 2010 04:14:01 Junio C Hamano wrote:

Show 11 quoted lines

> Later, we might want to split the codepath again to:
> 
>  (0) see if it is huge or small if we are not reading from pipe;
> 
>  (1) if we do not know the size or if it is moderately tiny, keep doing
>      what your code does;
> 
>  (2) if we know we are reading something huge with known size, then have a
>      loop to read-a-bit-compress-and-write-it-out-while-hashing, and
>      finally rename the loose resulting object to the final name.  Or we
>      may even want to do that into a new pack on its own.

There's a slight problem with that code (I tried to finish last night's attempt but got stuck on this):

The create_tmpfile() and move_temp_to_file() duo goes to some lengths to ensure that the file is created in the same directory that we want it to end up in. However, in the block-based scheme, you cannot know which directory this will be before you have already written the entire output.

So again I guess there are a few possible solutions:

* Try the cross-directory rename anyway, but if it doesn't work,
  copy&unlink.  This of course means that you may write the same
  object over network twice.

* Declare that keeping the memory usage near what it is today (the
  full output buffer plus a constant) is okay.

* Give up and stick with Dmitry's patch :-)

-- 
Thomas Rast
trast@{inf,student}.ethz.ch