git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/9] Prefix-compress on-disk index entries

From
Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date
Apr 4, 2012, 12:34 UTC
Message-ID
<CACsJy8A+cJtzKdqJSWbmjT1LgP10LB69-NHfOv8S6BusGcMeFw@mail.gmail.com>
In-Reply-To
<1333493596-14202-1-git-send-email-gitster@pobox.com>
On Wed, Apr 4, 2012 at 5:53 AM, Junio C Hamano <gitster@pobox.com> wrote:
> This is still rough,
but nice cleanups
Show 19 quoted lines
> but with this patch I am getting:
>
>    $ ls -l .git/index*
>    -rw-r----- 1 jch eng 25586488 2012-04-03 15:27 .git/index
>    -rw-r----- 1 jch eng 14654328 2012-04-03 15:38 .git/index-4
>
> in a clone of WebKit repository that has 183175 paths.
>
> With hot-cache with no local modification:
>
>    $ time sh -c 'GIT_INDEX_FILE=.git/index-4 git diff'
>    real  0m0.469s
>    user  0m0.130s
>    sys   0m0.330s
>
>    $ time sh -c 'git diff'
>    real  0m0.677s
>    user  0m0.290s
>    sys   0m0.370s

I wonder what causes user time drop from .29s to .13s here. I think the main patch should increase computation, even only slightly, not less. Or is it noise?

> The updated format is not documented yet, as I didn't intend (and I still
> am not committed) to declare a change along this line the official "v4"
> format; I was merely being curious to see how much improvements we can get
> from a trivial approach like this.

Anything else you have in mind for v4? Any chance we can adopt crc32 instead of sha-1? We could divide the index into many smaller parts for checksum, for example one crc32 every 100 entries, and one (or sha-1) for each extension. It should not complicate the code too much.

-- 
Duy
Previous: Junio C HamanoNext: Junio C Hamano
Message 15 of 27 in “Prefix-compress on-disk index entries”
  1. 0/9 Prefix-compress on-disk index entriesJunio C Hamano, Apr 3, 2012
  2. 1/9 varint: make it available outside the context of packJunio C Hamano, Apr 3, 2012
  3. 2/9 cache.h: hide on-disk index detailsJunio C Hamano, Apr 3, 2012
  4. 3/9 read-cache.c: allow unaligned mapping of the index fileJunio C Hamano, Apr 3, 2012
  5. 4/9 read-cache.c: make create_from_disk() report number of bytes it consumedJunio C Hamano, Apr 3, 2012
  6. 5/9 read-cache.c: report the header version we do not understandJunio C Hamano, Apr 3, 2012
  7. 6/9 read-cache.c: move code to copy ondisk to incore cache to a helper functionJunio C Hamano, Apr 3, 2012
  8. 7/9 read-cache.c: move code to copy incore to ondisk cache to a helper functionJunio C Hamano, Apr 3, 2012
  9. 8/9 read-cache.c: read prefix-compressed names in index on-disk version v4Junio C Hamano, Apr 3, 2012
  10. 9/9 read-cache.c: write index v4 formatJunio C Hamano, Apr 3, 2012
  11. David BarrApr 4, 2012
  12. Junio C HamanoApr 4, 2012
  13. Junio C HamanoApr 4, 2012
  14. 2/2 update-index: upgrade/downgrade on-disk index versionJunio C Hamano, Apr 4, 2012
  15. Nguyen Thai Ngoc DuyApr 4, 2012
  16. Junio C HamanoApr 4, 2012
  17. David BarrApr 6, 2012
  18. Nguyen Thai Ngoc DuyMay 2, 2012
  19. David BarrMay 2, 2012
  20. 1/2 unpack-trees: preserve the index file version of originalJunio C Hamano, Apr 27, 2012
  21. 2/2 index-v4: document the entry formatJunio C Hamano, Apr 27, 2012
  22. Thomas RastApr 30, 2012
  23. Junio C HamanoMay 1, 2012
  24. Thomas RastMay 1, 2012
  25. Shawn PearceMay 2, 2012
  26. Junio C HamanoMay 2, 2012
  27. Shawn PearceMay 2, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.