git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 00/24] Index-v5

From
Thomas Gummerer <t.gummerer@gmail.com>
Date
Aug 31, 2013, 05:23 UTC
Message-ID
<87vc2mo2lt.fsf@gmail.com>
In-Reply-To
<xmqq61uuo4dl.fsf@gitster.dls.corp.google.com>
Junio C Hamano <gitster@pobox.com> writes:
Show 24 quoted lines
> Duy Nguyen <pclouds@gmail.com> writes:
>
>> On Mon, Aug 19, 2013 at 2:41 AM, Thomas Gummerer <t.gummerer@gmail.com> wrote:
>>
>> I'm done reviewing this version (I neglected the extension writing
>> patches because after spending hours on the main write patch I don't
>> want to look at them anymore :p). Now that rc period is over, with a
>> partial write proof-of-concept, I think it's enough to call Junio's
>> attention on the series, see if we have any chance of merging it. The
>> partial write POC is needed to make sure we don't overlook anything,
>> just support update-index is enough.
>
> I've been following the review comment threads after looking at the
> patches myself when they were posted. I was hoping to see some API
> improvement over the current "we (have to) have everything available
> in-core in a flat array" model, which gives a lot of convenience and
> IO overhead at the same time, that would make me say "yes, this
> operation, that we need to do very often, will certainly be helped
> by this new API, and in order to support that style of API better,
> the current file format is inadequate and we do need to go to the
> proposed tree like on-disk format" for at least one, but
> unfortunately I haven't found any (yet).
>
> So...

I think the issue is a bit different. The current API, with some small additions (e.g. read_index_filtered()) works well as in-memory format, even for partial reading/writing. I will try to write a POC for partial writing to show that the current in-memory format works for this too. As Duy wrote in the other email, some API changes will be necessary to allow that, but not a big API change moving from a flat array to a tree based format.

I think it comes down to "this operation will be helped by partial loading/writing and we need this small API changes (read_index_filtered() for now, more to follow) and the index format change to be able to do that".

Does that make sense, with at least Duy's comments in the review addressed and a POC for partial writing?

Previous: Duy Nguyen
Message 55 of 55 in “Index-v5”
  1. 00/24 Index-v5Thomas Gummerer, Aug 18, 2013
  2. 01/24 t2104: Don't fail for index versions other than [23]Thomas Gummerer, Aug 18, 2013
  3. 02/24 read-cache: use fixed width integer typesThomas Gummerer, Aug 18, 2013
  4. Eric SunshineAug 18, 2013
  5. Junio C HamanoAug 20, 2013
  6. Thomas GummererAug 21, 2013
  7. 03/24 read-cache: split index file version specific functionalityThomas Gummerer, Aug 18, 2013
  8. 04/24 read-cache: clear version in discard_index()Thomas Gummerer, Aug 18, 2013
  9. Junio C HamanoAug 20, 2013
  10. Thomas GummererAug 21, 2013
  11. 05/24 read-cache: move index v2 specific functions to their own fileThomas Gummerer, Aug 18, 2013
  12. 06/24 read-cache: Don't compare uid, gid and ino on cygwinThomas Gummerer, Aug 18, 2013
  13. Ramsay JonesAug 18, 2013
  14. Thomas GummererAug 20, 2013
  15. 07/24 read-cache: Re-read index if index file changedThomas Gummerer, Aug 18, 2013
  16. 08/24 add documentation for the index apiThomas Gummerer, Aug 18, 2013
  17. Eric SunshineAug 18, 2013
  18. 09/24 read-cache: add index reading apiThomas Gummerer, Aug 18, 2013
  19. 10/24 make sure partially read index is not changedThomas Gummerer, Aug 18, 2013
  20. Eric SunshineAug 18, 2013
  21. Thomas GummererAug 20, 2013
  22. 11/24 grep.c: use index apiThomas Gummerer, Aug 18, 2013
  23. 12/24 ls-files.c: use index apiThomas Gummerer, Aug 18, 2013
  24. 13/24 documentation: add documentation of the index-v5 file formatThomas Gummerer, Aug 18, 2013
  25. 14/24 read-cache: make in-memory format aware of stat_crcThomas Gummerer, Aug 18, 2013
  26. 15/24 read-cache: read index-v5Thomas Gummerer, Aug 18, 2013
  27. Eric SunshineAug 19, 2013
  28. Duy NguyenAug 20, 2013
  29. Thomas GummererAug 20, 2013
  30. Duy NguyenAug 21, 2013
  31. Duy NguyenAug 20, 2013
  32. Thomas GummererAug 20, 2013
  33. Duy NguyenAug 23, 2013
  34. 16/24 read-cache: read resolve-undo dataThomas Gummerer, Aug 18, 2013
  35. Eric SunshineAug 19, 2013
  36. 17/24 read-cache: read cache-tree in index-v5Thomas Gummerer, Aug 18, 2013
  37. Duy NguyenAug 24, 2013
  38. Thomas GummererNov 25, 2013
  39. 18/24 read-cache: write index-v5Thomas Gummerer, Aug 18, 2013
  40. Duy NguyenAug 24, 2013
  41. Thomas GummererNov 25, 2013
  42. Duy NguyenAug 24, 2013
  43. Duy NguyenAug 24, 2013
  44. 19/24 read-cache: write index-v5 cache-tree dataThomas Gummerer, Aug 18, 2013
  45. 20/24 read-cache: write resolve-undo data for index-v5Thomas Gummerer, Aug 18, 2013
  46. 21/24 update-index.c: rewrite index when index-version is givenThomas Gummerer, Aug 18, 2013
  47. 22/24 p0003-index.sh: add perf test for the index formatsThomas Gummerer, Aug 18, 2013
  48. 23/24 introduce GIT_INDEX_VERSION environment variableThomas Gummerer, Aug 18, 2013
  49. Duy NguyenAug 21, 2013
  50. Thomas GummererAug 21, 2013
  51. 24/24 test-lib: allow setting the index format versionThomas Gummerer, Aug 18, 2013
  52. Duy NguyenAug 24, 2013
  53. Junio C HamanoAug 25, 2013
  54. Duy NguyenAug 25, 2013
  55. Thomas GummererAug 31, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.