git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 6/6] Automatically switch to crc32 checksum for index when it's too large

From
Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date
Feb 6, 2012, 08:54 UTC
Message-ID
<CACsJy8Dv9fUzL3COZKVw_KR6aF20kHaw8M4CdBXJDA9H3fbxLw@mail.gmail.com>
In-Reply-To
<E799595D-61B3-4978-BCE1-BA6A33034B55@apple.com>
2012/2/6 Dave Zarzycki <zarzycki@apple.com>:
> Which crc32 polynomial is being used? crc32c (a.k.a. Castagnoli)? It would be great if this were the same polynomial that Intel implements in hardware via SSE4.2.
It's zlib's crc32.
Show 7 quoted lines
> On Feb 5, 2012, at 9:48 PM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
>
>> An experiment with -O3 is done on Intel D510@1.66GHz. At around 250k
>> entries, index reading time exceeds 0.5s. Switching to crc32 brings it
>> back lower than 0.2s.
>>
>> On 4M files index, reading time with SHA-1 takes ~8.4, crc32 2.8s.
-- 
Duy
Previous: Dave ZarzyckiNext: Dave Zarzycki
Message 17 of 20 in “read-cache: use sha1file for sha1 calculation”
  1. 1/6 read-cache: use sha1file for sha1 calculationNguyễn Thái Ngọc Duy, Feb 6, 2012
  2. 2/6 csum-file: make sha1 calculation optionalNguyễn Thái Ngọc Duy, Feb 6, 2012
  3. 3/6 Stop producing index version 2Nguyễn Thái Ngọc Duy, Feb 6, 2012
  4. Junio C HamanoFeb 6, 2012
  5. Shawn PearceFeb 7, 2012
  6. Nguyen Thai Ngoc DuyFeb 7, 2012
  7. Nguyen Thai Ngoc DuyFeb 7, 2012
  8. Junio C HamanoFeb 7, 2012
  9. Thomas RastFeb 7, 2012
  10. 4/6 Introduce index version 4 with global flagsNguyễn Thái Ngọc Duy, Feb 6, 2012
  11. 5/6 Allow to use crc32 as a lighter checksum on indexNguyễn Thái Ngọc Duy, Feb 6, 2012
  12. Shawn PearceFeb 7, 2012
  13. Dave ZarzyckiFeb 7, 2012
  14. Dave ZarzyckiFeb 7, 2012
  15. 6/6 Automatically switch to crc32 checksum for index when it's too largeNguyễn Thái Ngọc Duy, Feb 6, 2012
  16. Dave ZarzyckiFeb 6, 2012
  17. Nguyen Thai Ngoc DuyFeb 6, 2012
  18. Dave ZarzyckiFeb 6, 2012
  19. Junio C HamanoFeb 6, 2012
  20. Nguyen Thai Ngoc DuyFeb 6, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.