git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v1 16/19] read-cache: unlink old sharedindex files

From
Christian Couder <christian.couder@gmail.com>
Date
Oct 27, 2016, 12:14 UTC
Message-ID
<CAP8UFD0uiQ5RFYPqGizUqPb-QoTHjrJJmmj4ZXLyBnpJcoyGKA@mail.gmail.com>
In-Reply-To
<CACsJy8BPe085Qu4GpJ3MLTPeZM_7pbVrX31enQrGQn0oyjSrdg@mail.gmail.com>
On Thu, Oct 27, 2016 at 12:25 PM, Duy Nguyen <pclouds@gmail.com> wrote:
Show 28 quoted lines
> On Tue, Oct 25, 2016 at 5:43 PM, Duy Nguyen <pclouds@gmail.com> wrote:
>>>  static int write_shared_index(struct index_state *istate,
>>> @@ -2211,8 +2269,11 @@ static int write_shared_index(struct index_state *istate,
>>>         }
>>>         ret = rename_tempfile(&temporary_sharedindex,
>>>                               git_path("sharedindex.%s", sha1_to_hex(si->base->sha1)));
>>> -       if (!ret)
>>> +       if (!ret) {
>>>                 hashcpy(si->base_sha1, si->base->sha1);
>>> +               clean_shared_index_files(sha1_to_hex(si->base->sha1));
>>
>> This operation is technically garbage collection and should belong to
>> "git gc --auto", which is already called automatically in a few
>> places. Is it not called often enough that we need to do the cleaning
>> up right after a new shared index is created?
>
> Christian, if we assume to go with Junio's suggestion to disable
> split-index on temporary files, the only files left we have to take
> care of are index and index.lock. I believe pruning here in this code
> will have an advantage over in "git gc --auto" because when this is
> executed, we know we're holding index.lock, so nobody else is updating
> the index, it's race-free. All we need to do is peek in $GIT_DIR/index
> to see what shared index file it requires and keep it alive too, the
> remaining of shared index files can be deleted safely. We don't even
> need to fall back to mtime.
>
> git-gc just can't match this because while it's running, somebody else
> may be updating $GIT_DIR/index. Handling races would be a lot harder.

Yeah, I agree that if we disable split-index on temporary files and on commands like "git read-tree --index-output=<file>" then it is the right thing to do it in write_shared_index(). (In fact when I wrote the previous RFC series I didn't think about those special cases, and that was the reason why I did it like this. So I just need to go back to the implementation that was in the previous RFC series.)

I am just still wondering if disabling split-index on temporary files could not have a bad performance impact for some use cases, but I guess we could always come back to problem again if that happens.

Thanks, Christian.

Previous: Duy NguyenNext: Junio C Hamano
Message 47 of 66 in “Add configuration options for split-index”
  1. 00/19 Add configuration options for split-indexChristian Couder, Oct 23, 2016
  2. 02/19 config: add git_config_get_split_index()Christian Couder, Oct 23, 2016
  3. 05/19 update-index: warn in case of split-index incoherencyChristian Couder, Oct 23, 2016
  4. Duy NguyenOct 25, 2016
  5. Christian CouderOct 29, 2016
  6. Junio C HamanoNov 1, 2016
  7. Christian CouderNov 1, 2016
  8. Junio C HamanoNov 2, 2016
  9. 04/19 read-cache: add and then use tweak_split_index()Christian Couder, Oct 23, 2016
  10. 01/19 split-index: s/eith/with/ typo fixChristian Couder, Oct 23, 2016
  11. 03/19 split-index: add {add,remove}_split_index() functionsChristian Couder, Oct 23, 2016
  12. Duy NguyenOct 25, 2016
  13. Christian CouderOct 29, 2016
  14. Duy NguyenNov 7, 2016
  15. Duy NguyenNov 9, 2016
  16. Christian CouderNov 9, 2016
  17. 07/19 Documentation/config: add information for core.splitIndexChristian Couder, Oct 23, 2016
  18. 09/19 config: add git_config_get_max_percent_split_change()Christian Couder, Oct 23, 2016
  19. Duy NguyenOct 25, 2016
  20. Christian CouderOct 29, 2016
  21. Junio C HamanoNov 1, 2016
  22. Christian CouderNov 5, 2016
  23. 13/19 sha1_file: make check_and_freshen_file() non staticChristian Couder, Oct 23, 2016
  24. 12/19 Documentation/config: add splitIndex.maxPercentChangeChristian Couder, Oct 23, 2016
  25. Junio C HamanoNov 1, 2016
  26. Christian CouderNov 5, 2016
  27. Junio C HamanoNov 6, 2016
  28. 11/19 t1700: add tests for splitIndex.maxPercentChangeChristian Couder, Oct 23, 2016
  29. Junio C HamanoNov 1, 2016
  30. 10/19 read-cache: regenerate shared index if necessaryChristian Couder, Oct 23, 2016
  31. Ramsay JonesOct 23, 2016
  32. Christian CouderOct 29, 2016
  33. Duy NguyenOct 25, 2016
  34. Christian CouderOct 29, 2016
  35. 15/19 config: add git_config_get_date_string() from gc.cChristian Couder, Oct 23, 2016
  36. Junio C HamanoNov 1, 2016
  37. Christian CouderNov 23, 2016
  38. Junio C HamanoNov 23, 2016
  39. Christian CouderNov 28, 2016
  40. Junio C HamanoNov 28, 2016
  41. 17/19 t1700: test shared index file expirationChristian Couder, Oct 23, 2016
  42. 18/19 Documentation/config: add splitIndex.sharedIndexExpireChristian Couder, Oct 23, 2016
  43. 19/19 Documentation/git-update-index: explain splitIndex.*Christian Couder, Oct 23, 2016
  44. 16/19 read-cache: unlink old sharedindex filesChristian Couder, Oct 23, 2016
  45. Duy NguyenOct 25, 2016
  46. Duy NguyenOct 27, 2016
  47. Christian CouderOct 27, 2016
  48. Junio C HamanoOct 27, 2016
  49. Duy NguyenOct 29, 2016
  50. 08/19 Documentation/git-update-index: talk about core.splitIndex config varChristian Couder, Oct 23, 2016
  51. 14/19 read-cache: touch shared index files when usedChristian Couder, Oct 23, 2016
  52. Duy NguyenOct 25, 2016
  53. Junio C HamanoNov 1, 2016
  54. 06/19 t1700: add tests for core.splitIndexChristian Couder, Oct 23, 2016
  55. Junio C HamanoOct 24, 2016
  56. Duy NguyenOct 25, 2016
  57. Junio C HamanoOct 25, 2016
  58. Duy NguyenOct 26, 2016
  59. Junio C HamanoOct 26, 2016
  60. Duy NguyenOct 25, 2016
  61. Christian CouderNov 3, 2016
  62. Duy NguyenNov 7, 2016
  63. Christian CouderNov 18, 2016
  64. Duy NguyenNov 22, 2016
  65. Christian CouderNov 22, 2016
  66. Duy NguyenNov 22, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.