git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: GSoC - Designing a faster index format

From
ESelton sky <eltonsky9404@gmail.com>
Date
Mar 22, 2012, 20:32 UTC
Message-ID
<CAKTdtZmYc=xz4zCPQiuSTUvdmbLRKXNWNL3N6_4Bj0gujYmRvw@mail.gmail.com>
In-Reply-To
<CAKTdtZkGP3KbMGf88yW7zcCjemUyEy_4CVNkLD0SV=Lm7=Kveg@mail.gmail.com>
Got a few questions:
1. index is used for building next commit, so it should only include
files created/modified/deleted. But I see it has all entries for
current working dir. why?
2. From read_index_from() I see the whole index is read into mem, and
write one by one (entry/ext) back to disk. This makes sense. But why
we have to compute Sha1 for all entries, especially unchanged entries?
3. how does git track updated files? Does it compare the ts between
working dir and index ? Or they are recorded somewhere?
4. When does git insert to cache tree? and when it retrieve from it?
Some early thoughts for the tree format:

We can use B tree like format. Keep the header in the beginning of the file as is, but add file length (4bytes) and the pointer to extensions (8bytes) into header. Entry list follows the header. The entry starts with number of children offsets (1 byte) followed by list of offsets (4 bytes each). We can limit the number for balance. Other fields leave as is. Extensions can locate in between entries.

Use Sha1 , rather than the path, as the key for each entry node. This beats the case like 1000 files in a dir which breaks the balance of the tree, as Thomas mentioned. If a file is updated, the old Sha1 can be found in object dir. This also gives flexibility. We may use splay tree, in order to move updated nodes close to the root. The downside is full path has to be stored in entry.

Regards, Elton

On Wed, Mar 21, 2012 at 11:01 PM, elton sky <eltonsky9404@gmail.com> wrote:
Show 30 quoted lines
> Hi Nguyen, Thomas
>
> Thanks for the points &clues. Processing them...
>
> -Elton
>
> On Wed, Mar 21, 2012 at 10:25 PM, Thomas Rast <trast@student.ethz.ch> wrote:
>> elton sky <eltonsky9404@gmail.com> writes:
>>
>>> I got questions like: how each operations affect index? how cache tree
>>> data and index is stored?
>>> Maybe you can point me how I should catch up quickly. I went through
>>> the article "git-for-computer-scientists", that quite makes sense.
>>
>> In addition to what Nguyen Thai Ngoc Duy said, check out the
>> (sub)threads
>>
>>  http://thread.gmane.org/gmane.comp.version-control.git/190016/focus=190132
>>  [origins of the GSoC project idea]
>>
>>  http://thread.gmane.org/gmane.comp.version-control.git/192014/focus=192025
>>  [perspectives of core developers in reply to the idea]
>>
>>  http://thread.gmane.org/gmane.comp.version-control.git/186244/focus=186282
>>  http://thread.gmane.org/gmane.comp.version-control.git/186357
>>  [the last few discussions about cache-tree]
>>
>> --
>> Thomas Rast
>> trast@{inf,student}.ethz.ch
Previous: elton skyNext: Jakub Narebski
Message 5 of 33 in “GSoC - Designing a faster index format”
  1. elton skyMar 20, 2012
  2. Nguyen Thai Ngoc DuyMar 21, 2012
  3. Thomas RastMar 21, 2012
  4. elton skyMar 21, 2012
  5. elton skyMar 22, 2012
  6. Jakub NarebskiMar 23, 2012
  7. Nguyen Thai Ngoc DuyMar 23, 2012
  8. elton skyMar 23, 2012
  9. Nguyen Thai Ngoc DuyMar 23, 2012
  10. Nguyen Thai Ngoc DuyMar 24, 2012
  11. elton skyMar 26, 2012
  12. elton skyMar 26, 2012
  13. Thomas RastMar 26, 2012
  14. Nguyen Thai Ngoc DuyMar 26, 2012
  15. Shawn PearceMar 26, 2012
  16. elton skyMar 27, 2012
  17. David BarrMar 27, 2012
  18. Nguyen Thai Ngoc DuyMar 27, 2012
  19. Jeff KingMar 29, 2012
  20. Nguyen Thai Ngoc DuyMar 27, 2012
  21. Nguyen Thai Ngoc DuyMar 26, 2012
  22. elton skyMar 27, 2012
  23. Nguyen Thai Ngoc DuyMar 27, 2012
  24. elton skyApr 2, 2012
  25. Nguyen Thai Ngoc DuyApr 2, 2012
  26. Shawn PearceApr 2, 2012
  27. Nguyen Thai Ngoc DuyApr 2, 2012
  28. elton skyApr 4, 2012
  29. Nguyen Thai Ngoc DuyApr 4, 2012
  30. elton skyApr 4, 2012
  31. elton skyApr 6, 2012
  32. elton skyApr 6, 2012
  33. elton skyApr 7, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.