git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] read-cache: call verify_hdr() in a background thread

From
JHJeff Hostetler <git@jeffhostetler.com>
Date
Mar 24, 2017, 15:46 UTC
Message-ID
<1743d048-8d6b-c14e-161b-351e6d79fb95@jeffhostetler.com>
In-Reply-To
<20170324153617.smmzju5mygqt5vux@sigill.intra.peff.net>
On 3/24/2017 11:36 AM, Jeff King wrote:
Show 23 quoted lines
> On Fri, Mar 24, 2017 at 01:27:51PM +0000, git@jeffhostetler.com wrote:
>
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Teash do_read_index() in read-cache.c to call verify_hdr()
>> in a background thread while the forground thread parses
>> the index and builds the_index.
>>
>> This is a performance optimization to reduce the overall
>> time required to get the index into memory.
>>
>> Testing on Windows (using the OpenSSL SHA1 routine) showed
>> that parsing the index and computing the SHA1 take almost
>> equal time, so this patch effectively reduces the startup
>> time by 1/2.
>
> Have you considered just skipping the sha1 check on read (possibly with
> an option)?
>
> Its purpose is to detect disk bit-rot. Checking it for every single
> operation may be a bit excessive, especially because it gets rewritten a
> lot. Nobody really cared until now because they don't have index files
> that are hundreds of megabytes.

Yes, we have done that in a version of the client customized for the virtual file system effort and have been using it internally without incident for quite a while.

And we can get a patch for that instead (or include it with this one) if there's interest.

I tried to limit my proposal here to increasing performance while preserving the existing behavior.

Jeff
Previous: Jeff KingNext: Jonathan Nieder
Message 4 of 7 in “read-cache: call verify_hdr() in a background thread”
  1. read-cache: call verify_hdr() in a background threadgit@jeffhostetler.com, Mar 24, 2017
  2. read-cache: call verify_hdr() in a background threadgit@jeffhostetler.com, Mar 24, 2017
  3. Jeff KingMar 24, 2017
  4. Jeff HostetlerMar 24, 2017
  5. Jonathan NiederMar 24, 2017
  6. Jeff HostetlerMar 24, 2017
  7. Jonathan NiederMar 24, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.