# [PATCH] credential-store: use timeout when locking file 7 messages from 2020-10-30 to 2020-11-26. Participants: Simão Afonso, Junio C Hamano, Jeff King. Thread: https://gitlist.dev/t/54539 ## Simão Afonso, 2020-10-30 18:07 Subject: [PATCH] credential-store: use timeout when locking file Message-ID: <20201030180718.4i7txqkgye7r6pkb@safonso-t430> URL: https://gitlist.dev/e/20201030180718.4i7txqkgye7r6pkb%40safonso-t430 ``` When holding the lock for rewriting the credential file, use a timeout to avoid race conditions when the credentials file needs to be updated in parallel. An example would be doing `fetch --all` on a repository with several remotes that need credentials, using parallel fetching. The timeout is hardcoded to 1 second, since this is just to solve a race condition. This was reported here: https://lore.kernel.org/git/20201029192020.mcri76ylbdure2o7@safonso-t430/ --- builtin/credential-store.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/builtin/credential-store.c b/builtin/credential-store.c index 5331ab151..acff4abae 100644 --- a/builtin/credential-store.c +++ b/builtin/credential-store.c @@ -58,8 +58,9 @@ static void print_line(struct strbuf *buf) static void rewrite_credential_file(const char *fn, struct credential *c, struct strbuf *extra) { - if (hold_lock_file_for_update(&credential_lock, fn, 0) < 0) - die_errno("unable to get credential storage lock"); + static long timeout = 1000; + if (hold_lock_file_for_update_timeout(&credential_lock, fn, 0, timeout) < 0) + die_errno("unable to get credential storage lock in %ld ms", timeout); if (extra) print_line(extra); parse_credential_file(fn, c, NULL, print_line); -- 2.29.1 ``` ## Junio C Hamano, 2020-10-30 19:49 Subject: Re: [PATCH] credential-store: use timeout when locking file Message-ID: URL: https://gitlist.dev/e/xmqq361v334r.fsf%40gitster.c.googlers.com In-Reply-To: <20201030180718.4i7txqkgye7r6pkb@safonso-t430> ``` Simão Afonso writes: > When holding the lock for rewriting the credential file, use a timeout > to avoid race conditions when the credentials file needs to be updated > in parallel. > > An example would be doing `fetch --all` on a repository with several > remotes that need credentials, using parallel fetching. OK. > The timeout is hardcoded to 1 second, since this is just to solve a race > condition. It is unclear what this sentence wants to explain. How does "this is to solve a race" justifies the choice of "1 second" (as opposed to say 10 seconds or 0.5 second)? Or is this trying to justify why there is no configurability? If so, why is it OK to hardcode it if it is done to solve a race? Are we assuming certain maximum rate of operation that is "reasonable"? > This was reported here: > https://lore.kernel.org/git/20201029192020.mcri76ylbdure2o7@safonso-t430/ > --- Missing sign-off. cf. https://git-scm.com/docs/SubmittingPatches.html#sign-off > builtin/credential-store.c | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/builtin/credential-store.c b/builtin/credential-store.c > index 5331ab151..acff4abae 100644 > --- a/builtin/credential-store.c > +++ b/builtin/credential-store.c > @@ -58,8 +58,9 @@ static void print_line(struct strbuf *buf) > static void rewrite_credential_file(const char *fn, struct credential *c, > struct strbuf *extra) > { > - if (hold_lock_file_for_update(&credential_lock, fn, 0) < 0) > - die_errno("unable to get credential storage lock"); > + static long timeout = 1000; Why "static"? It would make your code easier to follow if you limit use of "static" to only cases where you want to take advantage of the fact that the value previously left by the earlier call is seen by the next call, and/or the address of the variable must be valid even after the control returns to the caller. I would understand if this were "const long timeout = 1000". If this were an identifier with longer lifespan, I would have suggested to include some scale in the variable name (e.g. timeout_ms to clarify that it is counted in milliseconds), but it is just for this short function, so let's say "timeout" is just fine. > + if (hold_lock_file_for_update_timeout(&credential_lock, fn, 0, timeout) < 0) > + die_errno("unable to get credential storage lock in %ld ms", timeout); > if (extra) > print_line(extra); > parse_credential_file(fn, c, NULL, print_line); Thanks. ``` ## Simão Afonso, 2020-11-24 19:32 Subject: [PATCH v2] crendential-store: use timeout when locking file Message-ID: <20201124193208.74fe4mticgkbxou3@safonso-t430> URL: https://gitlist.dev/e/20201124193208.74fe4mticgkbxou3%40safonso-t430 In-Reply-To: ``` When holding the lock for rewriting the credential file, use a timeout to avoid race conditions when the credentials file needs to be updated in parallel. An example would be doing `fetch --all` on a repository with several remotes that need credentials, using parallel fetching. The timeout can be configured using "credentialStore.fileTimeout", defaulting to 1 second. Signed-off-by: Simão Afonso --- Thanks for the review. I got stuck with work and only got around to tweak this now. I added a configurable timeout with the old value as default. I think that txt file is the only documentation that requires update for a new configuration value. Documentation/config/credential.txt | 6 ++++++ builtin/credential-store.c | 8 ++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Documentation/config/credential.txt b/Documentation/config/credential.txt index 9d01641c2..132e04b47 100644 --- a/Documentation/config/credential.txt +++ b/Documentation/config/credential.txt @@ -28,3 +28,9 @@ credential..*:: credentialCache.ignoreSIGHUP:: Tell git-credential-cache--daemon to ignore SIGHUP, instead of quitting. + +credentialStore.fileTimeout:: + The length of time, in milliseconds, for git-credential-store to retry + when trying to lock the credentials file. Value 0 means not to retry at + all; -1 means to try indefinitely. Default is 1000 (i.e., retry for + 1s). diff --git a/builtin/credential-store.c b/builtin/credential-store.c index 5331ab151..82284176e 100644 --- a/builtin/credential-store.c +++ b/builtin/credential-store.c @@ -1,4 +1,5 @@ #include "builtin.h" +#include "config.h" #include "lockfile.h" #include "credential.h" #include "string-list.h" @@ -58,8 +59,11 @@ static void print_line(struct strbuf *buf) static void rewrite_credential_file(const char *fn, struct credential *c, struct strbuf *extra) { - if (hold_lock_file_for_update(&credential_lock, fn, 0) < 0) - die_errno("unable to get credential storage lock"); + int timeout_ms = 1000; + git_config_get_int("credentialstore.filetimeout", &timeout_ms); + + if (hold_lock_file_for_update_timeout(&credential_lock, fn, 0, timeout_ms) < 0) + die_errno("unable to get credential storage lock in %d ms", timeout_ms); if (extra) print_line(extra); parse_credential_file(fn, c, NULL, print_line); -- 2.29.2 ``` ## Junio C Hamano, 2020-11-24 22:08 Subject: Re: [PATCH v2] crendential-store: use timeout when locking file Message-ID: URL: https://gitlist.dev/e/xmqq4kleif26.fsf%40gitster.c.googlers.com In-Reply-To: <20201124193208.74fe4mticgkbxou3@safonso-t430> ``` Simão Afonso writes: > +credentialStore.fileTimeout:: > + The length of time, in milliseconds, for git-credential-store to retry > + when trying to lock the credentials file. Value 0 means not to retry at > + all; -1 means to try indefinitely. Default is 1000 (i.e., retry for > + 1s). I do not remember what was said in the first round of the review, but I wonder if this is the best name for users. I think it is good enough, but do ".lockTimeout" or ".lockTimeoutMS" make it even easier to grok, perhaps? > diff --git a/builtin/credential-store.c b/builtin/credential-store.c > index 5331ab151..82284176e 100644 > --- a/builtin/credential-store.c > +++ b/builtin/credential-store.c > @@ -1,4 +1,5 @@ > #include "builtin.h" > +#include "config.h" > #include "lockfile.h" > #include "credential.h" > #include "string-list.h" > @@ -58,8 +59,11 @@ static void print_line(struct strbuf *buf) > static void rewrite_credential_file(const char *fn, struct credential *c, > struct strbuf *extra) > { > - if (hold_lock_file_for_update(&credential_lock, fn, 0) < 0) > - die_errno("unable to get credential storage lock"); > + int timeout_ms = 1000; > + git_config_get_int("credentialstore.filetimeout", &timeout_ms); Please have a blank line before the first statement. > + > + if (hold_lock_file_for_update_timeout(&credential_lock, fn, 0, timeout_ms) < 0) > + die_errno("unable to get credential storage lock in %d ms", timeout_ms); Should this be die_errno()? Looking at lock_file_timeout(), I am not sure if the value of errno is valid in all codepaths that return failure. In any case, the message should be markd with _() for translation. Other than that, it looks good. Thanks. > if (extra) > print_line(extra); > parse_credential_file(fn, c, NULL, print_line); ``` ## Jeff King, 2020-11-25 09:37 Subject: Re: [PATCH v2] crendential-store: use timeout when locking file Message-ID: URL: https://gitlist.dev/e/X74lw%2BfyurzqVbg1%40coredump.intra.peff.net In-Reply-To: ``` On Tue, Nov 24, 2020 at 02:08:01PM -0800, Junio C Hamano wrote: > Simão Afonso writes: > > > +credentialStore.fileTimeout:: > > + The length of time, in milliseconds, for git-credential-store to retry > > + when trying to lock the credentials file. Value 0 means not to retry at > > + all; -1 means to try indefinitely. Default is 1000 (i.e., retry for > > + 1s). > > I do not remember what was said in the first round of the review, > but I wonder if this is the best name for users. I think it is good > enough, but do ".lockTimeout" or ".lockTimeoutMS" make it even > easier to grok, perhaps? Yeah, I think those are a bit more obvious. > > + > > + if (hold_lock_file_for_update_timeout(&credential_lock, fn, 0, timeout_ms) < 0) > > + die_errno("unable to get credential storage lock in %d ms", timeout_ms); > > Should this be die_errno()? Looking at lock_file_timeout(), I am > not sure if the value of errno is valid in all codepaths that return > failure. I think it's the right thing here. Inside hold_lock_file_for_update_timeout(), we'd pass errno to unable_to_lock_die(), etc. So if there is a code path in lock_file_timeout() that isn't setting errno properly, we should probably be fixing that. Another option would be to just pass LOCK_DIE_ON_ERROR here, but I think for this use I prefer the smaller "unable to lock" to the big "another git process may have crashed" advice message we'd give in that case. -Peff ``` ## Simão Afonso, 2020-11-25 18:31 Subject: [PATCH v3] crendential-store: use timeout when locking file Message-ID: <20201125183123.47praa6hcgrgrib4@safonso-t430> URL: https://gitlist.dev/e/20201125183123.47praa6hcgrgrib4%40safonso-t430 In-Reply-To: ``` When holding the lock for rewriting the credential file, use a timeout to avoid race conditions when the credentials file needs to be updated in parallel. An example would be doing `fetch --all` on a repository with several remotes that need credentials, using parallel fetching. The timeout can be configured using "credentialStore.lockTimeoutMS", defaulting to 1 second. Signed-off-by: Simão Afonso --- lockTimeoutMS sounds like the consensual name, implemented. Documentation/config/credential.txt | 6 ++++++ builtin/credential-store.c | 8 ++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Documentation/config/credential.txt b/Documentation/config/credential.txt index 9d01641c2..512f31876 100644 --- a/Documentation/config/credential.txt +++ b/Documentation/config/credential.txt @@ -28,3 +28,9 @@ credential..*:: credentialCache.ignoreSIGHUP:: Tell git-credential-cache--daemon to ignore SIGHUP, instead of quitting. + +credentialStore.lockTimeoutMS:: + The length of time, in milliseconds, for git-credential-store to retry + when trying to lock the credentials file. Value 0 means not to retry at + all; -1 means to try indefinitely. Default is 1000 (i.e., retry for + 1s). diff --git a/builtin/credential-store.c b/builtin/credential-store.c index 5331ab151..ae3c1ba75 100644 --- a/builtin/credential-store.c +++ b/builtin/credential-store.c @@ -1,4 +1,5 @@ #include "builtin.h" +#include "config.h" #include "lockfile.h" #include "credential.h" #include "string-list.h" @@ -58,8 +59,11 @@ static void print_line(struct strbuf *buf) static void rewrite_credential_file(const char *fn, struct credential *c, struct strbuf *extra) { - if (hold_lock_file_for_update(&credential_lock, fn, 0) < 0) - die_errno("unable to get credential storage lock"); + int timeout_ms = 1000; + + git_config_get_int("credentialstore.locktimeoutms", &timeout_ms); + if (hold_lock_file_for_update_timeout(&credential_lock, fn, 0, timeout_ms) < 0) + die_errno(_("unable to get credential storage lock in %d ms"), timeout_ms); if (extra) print_line(extra); parse_credential_file(fn, c, NULL, print_line); -- 2.29.2 ``` ## Jeff King, 2020-11-26 07:37 Subject: Re: [PATCH v3] crendential-store: use timeout when locking file Message-ID: URL: https://gitlist.dev/e/X79bUrreRkpiiorU%40coredump.intra.peff.net In-Reply-To: <20201125183123.47praa6hcgrgrib4@safonso-t430> ``` On Wed, Nov 25, 2020 at 06:31:23PM +0000, Simão Afonso wrote: > When holding the lock for rewriting the credential file, use a timeout > to avoid race conditions when the credentials file needs to be updated > in parallel. > > An example would be doing `fetch --all` on a repository with several > remotes that need credentials, using parallel fetching. > > The timeout can be configured using "credentialStore.lockTimeoutMS", > defaulting to 1 second. > > Signed-off-by: Simão Afonso > --- > > lockTimeoutMS sounds like the consensual name, implemented. Thanks, this version looks good to me. -Peff ```