git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos

From
Jonathan Nieder <jrnieder@gmail.com>
Date
Jan 17, 2018, 18:16 UTC
Message-ID
<20180117181618.GA112319@aiede.svl.corp.google.com>
In-Reply-To
<CACsJy8A9AheuBK-mKy9d9m5yfSFFZ9N8=wg8FZD_C6mHHTdqXg@mail.gmail.com>
Hi,
Duy Nguyen wrote:
> On Wed, Jan 17, 2018 at 4:42 AM, Brandon Williams <bmwill@google.com> wrote:
Show 12 quoted lines
>>                                  IIUC Split index is an index extension
>> that can be enabled to limit the size of the index file that is written
>> when making changes to the index.  It breaks the index into two pieces,
>> index (which contains only changes) and sharedindex.XXXXX (which
>> contains unchanged information) where 'XXXXX' is a value found in the
>> index file.  If we don't do anything fancy then these two files live
>> next to one another in a repository's git directory at $GIT_DIR/index
>> and $GIT_DIR/sharedindex.XXXXX.  This seems to work all well and fine
>> except that this isn't always the case and the read_index_from function
>> takes this into account by enabling a caller to specify a path to where
>> the index file is located.  We can do this by specifying the index file
>> we want to use by setting GIT_INDEX_FILE.
[...]
Show 17 quoted lines
>> In this case if i were to specify a location of an
>> index file in my home directory '~/index' and be using the split index
>> feature then the corresponding sharedindex file would live in my
>> repository's git directory '~/project/.git/sharedindex.XXXXX'.  So the
>> sharedindex file is always located relative to the project's git
>> directory and not the index file itself, which is kind of confusing.
>> Maybe a better design would be to have the sharedindex file located
>> relative to the index file.
>
> That adds more problems. Now when you move the index file around you
> have to move the shared index file too (think about atomic rename
> which we use in plenty of places, we can't achieve that by moving two
> files). A new dependency to $GIT_DIR is not that confusing to me, the
> index file is useless anyway if you don't have access to
> $GIT_DIR/objects. There was always the option to _not_ split the index
> when $GIT_INDEX_FILE is specified, I think I did consider that but I
> dropped it because we'd lose the performance gain by splitting.
Can you elaborate a little more on this?

At first glance, it seems simpler to say "paths in index extensions named in the index file are relative to the location of the index file" and to make moving the index file also require moving the shared index file, exactly as you say. So at least from a "principle of least surprise" perspective I would be tempted to go that way.

It's true that we rely on atomic rename in plenty of places, but only within a directory. (Filesystem boundaries, NFS, etc mean that atomic renames across directories are a lost cause.)

Fortunately index files (including temp index files used by scripts) tend to only be in $GIT_DIR, for exactly that reason. So I am wondering if switching to index-file-relative semantics would be an invasive move and what the pros and cons of such a move are.

Thanks, Jonathan

Previous: Brandon WilliamsNext: Duy Nguyen
Message 45 of 95 in “fixes for split index mode”
  1. 0/3 fixes for split index modeThomas Gummerer, Dec 10, 2017
  2. 1/3 repository: fix repo_read_index with submodulesThomas Gummerer, Dec 10, 2017
  3. Brandon WilliamsDec 11, 2017
  4. Thomas GummererDec 11, 2017
  5. 2/3 prune: fix pruning with multiple worktrees and split indexThomas Gummerer, Dec 10, 2017
  6. Brandon WilliamsDec 11, 2017
  7. Thomas GummererDec 11, 2017
  8. 3/3 travis: run tests with GIT_TEST_SPLIT_INDEXThomas Gummerer, Dec 10, 2017
  9. Eric SunshineDec 10, 2017
  10. SZEDER GáborDec 11, 2017
  11. Thomas GummererDec 11, 2017
  12. Lars SchneiderDec 12, 2017
  13. Junio C HamanoDec 12, 2017
  14. Thomas GummererDec 12, 2017
  15. Junio C HamanoDec 12, 2017
  16. Thomas GummererDec 13, 2017
  17. Lars SchneiderDec 13, 2017
  18. Junio C HamanoDec 13, 2017
  19. Lars SchneiderDec 13, 2017
  20. Thomas GummererDec 13, 2017
  21. 0/3 fixes for split index modeThomas Gummerer, Dec 17, 2017
  22. 2/3 prune: fix pruning with multiple worktrees and split indexThomas Gummerer, Dec 17, 2017
  23. Brandon WilliamsDec 18, 2017
  24. Thomas GummererJan 3, 2018
  25. Junio C HamanoJan 4, 2018
  26. 1/3 repository: fix repo_read_index with submodulesThomas Gummerer, Dec 17, 2017
  27. Brandon WilliamsDec 18, 2017
  28. Thomas GummererDec 18, 2017
  29. Brandon WilliamsDec 18, 2017
  30. 3/3 travis: run tests with GIT_TEST_SPLIT_INDEXThomas Gummerer, Dec 17, 2017
  31. Lars SchneiderDec 18, 2017
  32. Thomas GummererJan 4, 2018
  33. Lars SchneiderJan 5, 2018
  34. Thomas GummererJan 7, 2018
  35. 0/3 fixes for split index modeThomas Gummerer, Jan 7, 2018
  36. 1/3 read-cache: fix reading the shared index for other reposThomas Gummerer, Jan 7, 2018
  37. Duy NguyenJan 8, 2018
  38. Thomas GummererJan 8, 2018
  39. Thomas GummererJan 13, 2018
  40. Brandon WilliamsJan 8, 2018
  41. Duy NguyenJan 9, 2018
  42. Brandon WilliamsJan 16, 2018
  43. Duy NguyenJan 17, 2018
  44. Brandon WilliamsJan 17, 2018
  45. Jonathan NiederJan 17, 2018
  46. Duy NguyenJan 18, 2018
  47. Junio C HamanoJan 19, 2018
  48. Thomas GummererJan 20, 2018
  49. Junio C HamanoJan 22, 2018
  50. Thomas GummererJan 27, 2018
  51. Junio C HamanoFeb 7, 2018
  52. 2/3 split-index: don't write cache tree with null oid entriesThomas Gummerer, Jan 7, 2018
  53. 3/3 travis: run tests with GIT_TEST_SPLIT_INDEXThomas Gummerer, Jan 7, 2018
  54. 4/3 read-cache: don't try to write index if we can't write shared indexThomas Gummerer, Jan 13, 2018
  55. Duy NguyenJan 14, 2018
  56. 1/3 read-cache.c: change type of "temp" in write_shared_index()Nguyễn Thái Ngọc Duy, Jan 14, 2018
  57. 2/3 read-cache.c: move tempfile creation/cleanup out of write_shared_indexNguyễn Thái Ngọc Duy, Jan 14, 2018
  58. 3/3 read-cache: don't write index twice if we can't write shared indexNguyễn Thái Ngọc Duy, Jan 14, 2018
  59. SZEDER GáborJan 18, 2018
  60. Duy NguyenJan 18, 2018
  61. Jeff KingJan 18, 2018
  62. Duy NguyenJan 18, 2018
  63. Duy NguyenJan 18, 2018
  64. Jeff KingJan 18, 2018
  65. SZEDER GáborJan 18, 2018
  66. Duy NguyenJan 19, 2018
  67. 0/5 Travis CI: don't run the test suite as root in the 32 bit Linux buildSZEDER Gábor, Jan 22, 2018
  68. 2/5 travis-ci: use 'set -e' in the 32 bit Linux build jobSZEDER Gábor, Jan 22, 2018
  69. Jeff KingJan 23, 2018
  70. Jeff KingJan 23, 2018
  71. SZEDER GáborJan 24, 2018
  72. Jeff KingJan 24, 2018
  73. 5/5 travis-ci: don't fail if user already exists on 32 bit Linux build jobSZEDER Gábor, Jan 22, 2018
  74. Jeff KingJan 23, 2018
  75. Duy NguyenJan 24, 2018
  76. SZEDER GáborJan 24, 2018
  77. 1/5 travis-ci: use 'set -x' for the commands under 'su' in the 32 bit Linux buildSZEDER Gábor, Jan 22, 2018
  78. 3/5 travis-ci: don't repeat the path of the cache directorySZEDER Gábor, Jan 22, 2018
  79. Jeff KingJan 23, 2018
  80. SZEDER GáborJan 24, 2018
  81. 4/5 travis-ci: don't run the test suite as root in the 32 bit Linux buildSZEDER Gábor, Jan 22, 2018
  82. Jeff KingJan 23, 2018
  83. SZEDER GáborJan 24, 2018
  84. Jeff KingJan 24, 2018
  85. Jeff KingJan 24, 2018
  86. Jeff KingJan 24, 2018
  87. SZEDER GáborJan 22, 2018
  88. Eric SunshineJan 22, 2018
  89. SZEDER GáborJan 22, 2018
  90. Duy NguyenJan 24, 2018
  91. Lars SchneiderJan 26, 2018
  92. Thomas GummererJan 14, 2018
  93. Thomas GummererJan 18, 2018
  94. Junio C HamanoJan 19, 2018
  95. Thomas GummererJan 19, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.