{"thread":{"id":"55049","subject":"[PATCH] doc/reftable: document how to handle windows","startedAt":"2021-01-26T05:00:45Z","lastAt":"2021-02-23T16:58:19Z","messageCount":7,"participants":["Han-Wen Nienhuys via GitGitGadget","Han-Wen Nienhuys","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"415266","messageId":"pull.951.git.git.1611589125365.gitgitgadget@gmail.com","threadId":"55049","inReplyTo":null,"subject":"[PATCH] doc/reftable: document how to handle windows","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-25T15:38:45Z","receivedAt":"2021-01-26T05:00:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nOn Windows we can't delete or overwrite files opened by other processes. Here we\nsketch how to handle this situation.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n doc/reftable: document how to handle windows\n \n On Windows we can't delete or overwrite files opened by other processes.\n Here we sketch how to handle this situation.\n \n Signed-off-by: Han-Wen Nienhuys hanwen@google.com\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-951%2Fhanwen%2Fwindows-doc-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-951/hanwen/windows-doc-v1\nPull-Request: https://github.com/git/git/pull/951\n\n Documentation/technical/reftable.txt | 38 +++++++++++++++++++++-------\n 1 file changed, 29 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/technical/reftable.txt b/Documentation/technical/reftable.txt\nindex 8095ab2590c..d8be27d88c1 100644\n--- a/Documentation/technical/reftable.txt\n+++ b/Documentation/technical/reftable.txt\n@@ -876,13 +876,13 @@ A collection of reftable files are stored in the `$GIT_DIR/reftable/`\n directory:\n \n ....\n-00000001-00000001.log\n-00000002-00000002.ref\n-00000003-00000003.ref\n+00000001-00000001-RANDOM1.log\n+00000002-00000002-RANDOM2.ref\n+00000003-00000003-RANDOM3.ref\n ....\n \n where reftable files are named by a unique name such as produced by the\n-function `${min_update_index}-${max_update_index}.ref`.\n+function `${min_update_index}-${max_update_index}-${random}.ref`.\n \n Log-only files use the `.log` extension, while ref-only and mixed ref\n and log files use `.ref`. extension.\n@@ -893,9 +893,9 @@ current files, one per line, in order, from oldest (base) to newest\n \n ....\n $ cat .git/reftable/tables.list\n-00000001-00000001.log\n-00000002-00000002.ref\n-00000003-00000003.ref\n+00000001-00000001-RANDOM1.log\n+00000002-00000002-RANDOM2.ref\n+00000003-00000003-RANDOM3.ref\n ....\n \n Readers must read `$GIT_DIR/reftable/tables.list` to determine which\n@@ -906,6 +906,10 @@ Reftable files not listed in `tables.list` may be new (and about to be\n added to the stack by the active writer), or ancient and ready to be\n pruned.\n \n+The random suffix added to table filenames ensures that we never attempt to\n+overwrite an existing table, which is necessary for this scheme to work on\n+Windows\n+\n Backward compatibility\n ^^^^^^^^^^^^^^^^^^^^^^\n \n@@ -940,7 +944,7 @@ new reftable and atomically appending it to the stack:\n 3. Select `update_index` to be most recent file's\n `max_update_index + 1`.\n 4. Prepare temp reftable `tmp_XXXXXX`, including log entries.\n-5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}.ref`.\n+5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}-${random}.ref`.\n 6. Copy `tables.list` to `tables.list.lock`, appending file from (5).\n 7. Rename `tables.list.lock` to `tables.list`.\n \n@@ -993,7 +997,7 @@ prevents other processes from trying to compact these files.\n should always be the case, assuming that other processes are adhering to\n the locking protocol.\n 7. Rename `${min_update_index}-${max_update_index}_XXXXXX` to\n-`${min_update_index}-${max_update_index}.ref`.\n+`${min_update_index}-${max_update_index}-${random}.ref`.\n 8. Write the new stack to `tables.list.lock`, replacing `B` and `C`\n with the file from (4).\n 9. Rename `tables.list.lock` to `tables.list`.\n@@ -1005,6 +1009,22 @@ This strategy permits compactions to proceed independently of updates.\n Each reftable (compacted or not) is uniquely identified by its name, so\n open reftables can be cached by their name.\n \n+Windows\n+^^^^^^^\n+\n+On windows, and other systems that do not allow deleting or renaming to open\n+files, compaction may succeed, but other readers may prevent obsolete tables\n+from being deleted.\n+\n+On these platforms, the following strategy can be followed: on closing a\n+reftable stack, reload `tables.list`, and delete any tables no longer mentioned\n+in `tables.list`.\n+\n+Irregular program exit may still leave about unused files. In this case, a\n+cleanup operation can read `tables.list`, note its modification timestamp, and\n+delete any unreferenced `*.ref` files that are older.\n+\n+\n Alternatives considered\n ~~~~~~~~~~~~~~~~~~~~~~~\n \n\nbase-commit: 66e871b6647ffea61a77a0f82c7ef3415f1ee79c\n-- \ngitgitgadget\n"},{"id":"415278","messageId":"CAFQ2z_PCh2RfWALhAUXm01Xq0o+ibuEGJ2p9sCtvTASQ0FLUag@mail.gmail.com","threadId":"55049","inReplyTo":"xmqq35yo459k.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] doc/reftable: document how to handle windows","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-01-26T11:38:38Z","receivedAt":"2021-01-26T11:53:33Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Jan 26, 2021 at 6:49 AM Junio C Hamano wrote:\n>\n> \"Han-Wen Nienhuys via GitGitGadget\" writes:\n>\n> > $ cat .git/reftable/tables.list\n> > -00000001-00000001.log\n> > -00000002-00000002.ref\n> > -00000003-00000003.ref\n> > +00000001-00000001-RANDOM1.log\n> > +00000002-00000002-RANDOM2.ref\n> > +00000003-00000003-RANDOM3.ref\n> > ....\n> > @@ -940,7 +944,7 @@ new reftable and atomically appending it to the stack:\n> > 3. Select `update_index` to be most recent file's\n> > `max_update_index + 1`.\n> > 4. Prepare temp reftable `tmp_XXXXXX`, including log entries.\n> > -5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}.ref`.\n> > +5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}-${random}.ref`.\n> > 6. Copy `tables.list` to `tables.list.lock`, appending file from (5).\n> > 7. Rename `tables.list.lock` to `tables.list`.\n>\n> Is this because we have been assuming that in step 5. we can\n> \"overwrite\" (i.e. take over the name, implicitly unlinking the\n> existing one) the existing 0000001-00000001.ref with the newly\n> prepared one, which is not doable on Windows?\n\nNo, the protocol for adding a table to the end of the stack is\nimpervious to problems on Windows, as everything happens under lock,\nso there is no possibility of collisions.\n\n> We must prepare for two \"randoms\" colliding and retrying the\n> renaming step anyway, so would it make more sense to instead\n> use a non-random suffix (i.e. try \"-0.ref\" first, and when it\n> fails, readdir for 0000001-00000001-*.ref to find the latest\n> suffix and increment it)?\n\nThis is a lot of complexity, and both transactions and compactions can\nalways fail because they fail to get the lock, or because the data to\nbe written is out of date. So callers need to be prepared for a retry\nanyway.\n\n> > @@ -993,7 +997,7 @@ prevents other processes from trying to compact these files.\n> > should always be the case, assuming that other processes are adhering to\n> > the locking protocol.\n> > 7. Rename `${min_update_index}-${max_update_index}_XXXXXX` to\n> > -`${min_update_index}-${max_update_index}.ref`.\n> > +`${min_update_index}-${max_update_index}-${random}.ref`.\n> > 8. Write the new stack to `tables.list.lock`, replacing `B` and `C`\n> > with the file from (4).\n>\n> Likewise.\n\nThis case is different. Consider the following situation\n\n1-1.ref:\n main=abc123 @ timestamp 1\n master=abc123 @ timestamp 1\n2-2.ref: bla=456def @ timestamp 2\n3-3.ref:\n bla delete @ timestamp 3\n master delete @timestamp 3\n\nThe result of compacting this together would be a table containing\n\n main = abc123 @ timestamp 1\n\nbut in the previous naming convention, we'd name the resulting table\n\"1-1.ref\", which conflicts with the table in our starting situation.\n\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"415331","messageId":"CAFQ2z_PN8K6sKq=Rdw=maVhd67GhCtxWgGSUb5KhZ85EYV6jOw@mail.gmail.com","threadId":"55049","inReplyTo":"xmqqtur338bg.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] doc/reftable: document how to handle windows","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-01-26T18:11:50Z","receivedAt":"2021-01-26T22:08:56Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Jan 26, 2021 at 6:40 PM Junio C Hamano wrote:\n>\n> Han-Wen Nienhuys writes:\n>\n> >> Is this because we have been assuming that in step 5. we can\n> >> \"overwrite\" (i.e. take over the name, implicitly unlinking the\n> >> existing one) the existing 0000001-00000001.ref with the newly\n> >> prepared one, which is not doable on Windows?\n> >\n> > No, the protocol for adding a table to the end of the stack is\n> > impervious to problems on Windows, as everything happens under lock,\n> > so there is no possibility of collisions.\n> >\n> >> We must prepare for two \"randoms\" colliding and retrying the\n> >> renaming step anyway, so would it make more sense to instead\n> >> use a non-random suffix (i.e. try \"-0.ref\" first, and when it we\n> >> fails, readdir for 0000001-00000001-*.ref to find the latest\n> >> suffix and increment it)?\n> >\n> > This is a lot of complexity, and both transactions and compactions can\n> > always fail because they fail to get the lock, or because the data to\n> > be written is out of date. So callers need to be prepared for a retry\n> > anyway.\n>\n> Sorry, are we saying the same thing and reaching different\n> conclusions?\n>\n> My question was, under the assumption that the callers need to be\n> prepared for a retry anyway,\n>\n> (1) would it be possible to use \"seq\" (or \"take max from existing\n> and add one\") as the random number generator for the ${random}\n> part of your document, and\n>\n> (2) if the answer to the previous question is yes, would it result\n> in a system that is easier for Git developers, who observe what\n> happens inside the .git directory, to understand the behaviour\n> of the system, as they can immediately see that 1-1-47 is newer\n> than 1-1-22 instead of 1-1-$random1 and 1-1-$random2 that\n> cannot be compared?\n\nThe first two parts of the file name (${min}-${max}) already provide\nvisibility into what is going on, and the file system timestamp\nalready indicates which file is newer. I picked a random name as\nsuffix, as it gets the job done and is simple.\n\nWe could do what you suggest, but it adds semantics to the filenames\nthat aren't really there: currently, tables.list is a list of\nfilenames, and no part of the code parses back the file names. If we'd\ndo what you suggest, we have more ways in which the system can break\nsubtly, and needs to handle error conditions if the names are\nmalformed. This is the complexity I was alluding to in my previous\nmessage.\n\nWe could stipulate that a compaction must always increase the logical\ntimestamp, ie. in the scenario I sketched, the compacted table should\nbe written with a max-timestamp of 3, even though it contains no\nentries at timestamp 3. This avoids the error condition, but it's\nalso surprising because it is actually inconsistent with how the\nformat is described. But maybe we could update the description of the\nformat.\n\nOr, we could rename to ${min}-${max}-0 and if that fails try\n${min}-${max}-1, and if that fails ${min}-${max}-2 etc. I think that\nis somewhat nicer than parsing back a counter from the existing\nfilenames, but it could have the effect that 1-1-0 could be newer than\n1-1-2.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"415342","messageId":"xmqqtur338bg.fsf@gitster.c.googlers.com","threadId":"55049","inReplyTo":"CAFQ2z_PCh2RfWALhAUXm01Xq0o+ibuEGJ2p9sCtvTASQ0FLUag@mail.gmail.com","subject":"Re: [PATCH] doc/reftable: document how to handle windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-26T17:40:51Z","receivedAt":"2021-01-26T22:19:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Han-Wen Nienhuys writes:\n\n>> Is this because we have been assuming that in step 5. we can\n>> \"overwrite\" (i.e. take over the name, implicitly unlinking the\n>> existing one) the existing 0000001-00000001.ref with the newly\n>> prepared one, which is not doable on Windows?\n>\n> No, the protocol for adding a table to the end of the stack is\n> impervious to problems on Windows, as everything happens under lock,\n> so there is no possibility of collisions.\n>\n>> We must prepare for two \"randoms\" colliding and retrying the\n>> renaming step anyway, so would it make more sense to instead\n>> use a non-random suffix (i.e. try \"-0.ref\" first, and when it\n>> fails, readdir for 0000001-00000001-*.ref to find the latest\n>> suffix and increment it)?\n>\n> This is a lot of complexity, and both transactions and compactions can\n> always fail because they fail to get the lock, or because the data to\n> be written is out of date. So callers need to be prepared for a retry\n> anyway.\n\nSorry, are we saying the same thing and reaching different\nconclusions? \n\nMy question was, under the assumption that the callers need to be\nprepared for a retry anyway,\n\n (1) would it be possible to use \"seq\" (or \"take max from existing\n and add one\") as the random number generator for the ${random}\n part of your document, and\n\n (2) if the answer to the previous question is yes, would it result\n in a system that is easier for Git developers, who observe what\n happens inside the .git directory, to understand the behaviour\n of the system, as they can immediately see that 1-1-47 is newer\n than 1-1-22 instead of 1-1-$random1 and 1-1-$random2 that\n cannot be compared?\n"},{"id":"415350","messageId":"xmqqk0rz1mpy.fsf@gitster.c.googlers.com","threadId":"55049","inReplyTo":"CAFQ2z_PN8K6sKq=Rdw=maVhd67GhCtxWgGSUb5KhZ85EYV6jOw@mail.gmail.com","subject":"Re: [PATCH] doc/reftable: document how to handle windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-26T20:12:41Z","receivedAt":"2021-01-26T22:19:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Han-Wen Nienhuys writes:\n\n> The first two parts of the file name (${min}-${max}) already provide\n> visibility into what is going on, and the file system timestamp\n> already indicates which file is newer. I picked a random name as\n> suffix, as it gets the job done and is simple.\n\nOK, as long as two paths of the same ${min}-${max} part would not\nconfuse people, I am perfectly fine.\n\n> Or, we could rename to ${min}-${max}-0 and if that fails try\n> ${min}-${max}-1, and if that fails ${min}-${max}-2 etc. I think that\n> is somewhat nicer than parsing back a counter from the existing\n> filenames, but it could have the effect that 1-1-0 could be newer than\n> 1-1-2.\n\nI agree that such an approach that can get fooled by an existing gap\nwould not achieve anything over the ${random} approach.\n\nThanks.\n\n"},{"id":"415361","messageId":"xmqq35yo459k.fsf@gitster.c.googlers.com","threadId":"55049","inReplyTo":"pull.951.git.git.1611589125365.gitgitgadget@gmail.com","subject":"Re: [PATCH] doc/reftable: document how to handle windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-26T05:49:11Z","receivedAt":"2021-01-26T22:19:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> $ cat .git/reftable/tables.list\n> -00000001-00000001.log\n> -00000002-00000002.ref\n> -00000003-00000003.ref\n> +00000001-00000001-RANDOM1.log\n> +00000002-00000002-RANDOM2.ref\n> +00000003-00000003-RANDOM3.ref\n> ....\n> @@ -940,7 +944,7 @@ new reftable and atomically appending it to the stack:\n> 3. Select `update_index` to be most recent file's\n> `max_update_index + 1`.\n> 4. Prepare temp reftable `tmp_XXXXXX`, including log entries.\n> -5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}.ref`.\n> +5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}-${random}.ref`.\n> 6. Copy `tables.list` to `tables.list.lock`, appending file from (5).\n> 7. Rename `tables.list.lock` to `tables.list`.\n\nIs this because we have been assuming that in step 5. we can\n\"overwrite\" (i.e. take over the name, implicitly unlinking the\nexisting one) the existing 0000001-00000001.ref with the newly\nprepared one, which is not doable on Windows?\n\nWe must prepare for two \"randoms\" colliding and retrying the\nrenaming step anyway, so would it make more sense to instead\nuse a non-random suffix (i.e. try \"-0.ref\" first, and when it\nfails, readdir for 0000001-00000001-*.ref to find the latest\nsuffix and increment it)?\n\n> @@ -993,7 +997,7 @@ prevents other processes from trying to compact these files.\n> should always be the case, assuming that other processes are adhering to\n> the locking protocol.\n> 7. Rename `${min_update_index}-${max_update_index}_XXXXXX` to\n> -`${min_update_index}-${max_update_index}.ref`.\n> +`${min_update_index}-${max_update_index}-${random}.ref`.\n> 8. Write the new stack to `tables.list.lock`, replacing `B` and `C`\n> with the file from (4).\n\nLikewise.\n"},{"id":"417567","messageId":"pull.951.v2.git.git.1614099444126.gitgitgadget@gmail.com","threadId":"55049","inReplyTo":"pull.951.git.git.1611589125365.gitgitgadget@gmail.com","subject":"[PATCH v2] doc/reftable: document how to handle windows","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T16:57:23Z","receivedAt":"2021-02-23T16:58:19Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nOn Windows we can't delete or overwrite files opened by other processes. Here we\nsketch how to handle this situation.\n\nWe propose to use a random element in the filename. It's possible to design an\nalternate solution based on counters, but that would assign semantics to the\nfilenames that complicates implementation.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n doc/reftable: document how to handle windows\n \n On Windows we can't delete or overwrite files opened by other processes.\n Here we sketch how to handle this situation.\n \n Signed-off-by: Han-Wen Nienhuys hanwen@google.com\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-951%2Fhanwen%2Fwindows-doc-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-951/hanwen/windows-doc-v2\nPull-Request: https://github.com/git/git/pull/951\n\nRange-diff vs v1:\n\n 1: a952bc478f86 ! 1: e3854f2cc106 doc/reftable: document how to handle windows\n @@ Commit message\n On Windows we can't delete or overwrite files opened by other processes. Here we\n sketch how to handle this situation.\n \n + We propose to use a random element in the filename. It's possible to design an\n + alternate solution based on counters, but that would assign semantics to the\n + filenames that complicates implementation.\n +\n Signed-off-by: Han-Wen Nienhuys \n \n ## Documentation/technical/reftable.txt ##\n -@@ Documentation/technical/reftable.txt: A collection of reftable files are stored in the `$GIT_DIR/reftable/`\n - directory:\n +@@ Documentation/technical/reftable.txt: A repository must set its `$GIT_DIR/config` to configure reftable:\n + Layout\n + ^^^^^^\n \n - ....\n +-A collection of reftable files are stored in the `$GIT_DIR/reftable/`\n +-directory:\n +-\n +-....\n -00000001-00000001.log\n -00000002-00000002.ref\n -00000003-00000003.ref\n -+00000001-00000001-RANDOM1.log\n -+00000002-00000002-RANDOM2.ref\n -+00000003-00000003-RANDOM3.ref\n - ....\n - \n - where reftable files are named by a unique name such as produced by the\n +-....\n +-\n +-where reftable files are named by a unique name such as produced by the\n -function `${min_update_index}-${max_update_index}.ref`.\n -+function `${min_update_index}-${max_update_index}-${random}.ref`.\n ++A collection of reftable files are stored in the `$GIT_DIR/reftable/` directory.\n ++Their names should have a random element, such that each filename is globally\n ++unique; this helps avoid spurious failures on Windows, where open files cannot\n ++be removed or overwritten. It suggested to use\n ++`${min_update_index}-${max_update_index}-${random}.ref` as a naming convention.\n \n Log-only files use the `.log` extension, while ref-only and mixed ref\n and log files use `.ref`. extension.\n @@ Documentation/technical/reftable.txt: current files, one per line, in order, fro\n ....\n \n Readers must read `$GIT_DIR/reftable/tables.list` to determine which\n -@@ Documentation/technical/reftable.txt: Reftable files not listed in `tables.list` may be new (and about to be\n - added to the stack by the active writer), or ancient and ready to be\n - pruned.\n - \n -+The random suffix added to table filenames ensures that we never attempt to\n -+overwrite an existing table, which is necessary for this scheme to work on\n -+Windows\n -+\n - Backward compatibility\n - ^^^^^^^^^^^^^^^^^^^^^^\n - \n @@ Documentation/technical/reftable.txt: new reftable and atomically appending it to the stack:\n 3. Select `update_index` to be most recent file's\n `max_update_index + 1`.\n\n\n Documentation/technical/reftable.txt | 42 +++++++++++++++++-----------\n 1 file changed, 26 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/technical/reftable.txt b/Documentation/technical/reftable.txt\nindex 8095ab2590c8..3ef169af27d8 100644\n--- a/Documentation/technical/reftable.txt\n+++ b/Documentation/technical/reftable.txt\n@@ -872,17 +872,11 @@ A repository must set its `$GIT_DIR/config` to configure reftable:\n Layout\n ^^^^^^\n \n-A collection of reftable files are stored in the `$GIT_DIR/reftable/`\n-directory:\n-\n-....\n-00000001-00000001.log\n-00000002-00000002.ref\n-00000003-00000003.ref\n-....\n-\n-where reftable files are named by a unique name such as produced by the\n-function `${min_update_index}-${max_update_index}.ref`.\n+A collection of reftable files are stored in the `$GIT_DIR/reftable/` directory.\n+Their names should have a random element, such that each filename is globally\n+unique; this helps avoid spurious failures on Windows, where open files cannot\n+be removed or overwritten. It suggested to use\n+`${min_update_index}-${max_update_index}-${random}.ref` as a naming convention.\n \n Log-only files use the `.log` extension, while ref-only and mixed ref\n and log files use `.ref`. extension.\n@@ -893,9 +887,9 @@ current files, one per line, in order, from oldest (base) to newest\n \n ....\n $ cat .git/reftable/tables.list\n-00000001-00000001.log\n-00000002-00000002.ref\n-00000003-00000003.ref\n+00000001-00000001-RANDOM1.log\n+00000002-00000002-RANDOM2.ref\n+00000003-00000003-RANDOM3.ref\n ....\n \n Readers must read `$GIT_DIR/reftable/tables.list` to determine which\n@@ -940,7 +934,7 @@ new reftable and atomically appending it to the stack:\n 3. Select `update_index` to be most recent file's\n `max_update_index + 1`.\n 4. Prepare temp reftable `tmp_XXXXXX`, including log entries.\n-5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}.ref`.\n+5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}-${random}.ref`.\n 6. Copy `tables.list` to `tables.list.lock`, appending file from (5).\n 7. Rename `tables.list.lock` to `tables.list`.\n \n@@ -993,7 +987,7 @@ prevents other processes from trying to compact these files.\n should always be the case, assuming that other processes are adhering to\n the locking protocol.\n 7. Rename `${min_update_index}-${max_update_index}_XXXXXX` to\n-`${min_update_index}-${max_update_index}.ref`.\n+`${min_update_index}-${max_update_index}-${random}.ref`.\n 8. Write the new stack to `tables.list.lock`, replacing `B` and `C`\n with the file from (4).\n 9. Rename `tables.list.lock` to `tables.list`.\n@@ -1005,6 +999,22 @@ This strategy permits compactions to proceed independently of updates.\n Each reftable (compacted or not) is uniquely identified by its name, so\n open reftables can be cached by their name.\n \n+Windows\n+^^^^^^^\n+\n+On windows, and other systems that do not allow deleting or renaming to open\n+files, compaction may succeed, but other readers may prevent obsolete tables\n+from being deleted.\n+\n+On these platforms, the following strategy can be followed: on closing a\n+reftable stack, reload `tables.list`, and delete any tables no longer mentioned\n+in `tables.list`.\n+\n+Irregular program exit may still leave about unused files. In this case, a\n+cleanup operation can read `tables.list`, note its modification timestamp, and\n+delete any unreferenced `*.ref` files that are older.\n+\n+\n Alternatives considered\n ~~~~~~~~~~~~~~~~~~~~~~~\n \n\nbase-commit: 66e871b6647ffea61a77a0f82c7ef3415f1ee79c\n-- \ngitgitgadget\n"}]}