{"thread":{"id":"54190","subject":"[PATCH] push: make `--force-with-lease[=<ref>]` safer","startedAt":"2020-09-04T18:53:09Z","lastAt":"2020-10-03T12:11:03Z","messageCount":120,"participants":["Srinidhi Kaushik","Phillip Wood","Junio C Hamano","Johannes Schindelin","SZEDER Gábor","Philip Oakley"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"405055","messageId":"20200904185147.77439-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":null,"subject":"[PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-04T18:51:47Z","receivedAt":"2020-09-04T18:53:09Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"The `--force-with-lease` option in `git-push`, makes sure that\nrefs on remote aren't clobbered by unexpected changes when the\n\"<expect>\" ref value is explicitly specified.\n\nFor other cases (i.e., `--force-with-lease[=<ref>]`) where the tip\nof the remote tracking branch is populated as the \"<expect>\" value,\nthere is a possibility of allowing unwanted overwrites on the remote\nside when some tools that implicitly fetch remote-tracking refs in\nthe background are used with the repository. If a remote-tracking ref\nwas updated when a rewrite is happening locally and if those changes\nare pushed by omitting the \"<expect>\" value in `--force-with-lease`,\nany new changes from the updated tip will be lost locally and will\nbe overwritten on the remote.\n\nThis problem can be addressed by checking the `reflog` of the branch\nthat is being pushed and verify if there in a entry with the remote\ntracking ref. By running this check, we can ensure that refs being\nare fetched in the background while a \"lease\" is being held are not\noverlooked before a push, and any new changes can be acknowledged\nand (if necessary) integrated locally.\n\nThe new check will cause `git-push` to fail if it detects the presence\nof any updated refs that we do not have locally and reject the push\nstating `implicit fetch` as the reason.\n\nAn experimental configuration setting: `push.rejectImplicitFetch`\nwhich defaults to `true` (when `features.experimental` is enabled)\nhas been added, to allow `git-push` to reject a push if the check\nfails.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n\nHello,\nI picked this up from #leftoverbits over at GitHub [1] from the open\nissues list. This idea [2], for a safer `--force-with-lease` was\noriginally proposed by Johannes on the mailing list.\n\n[1]: https://github.com/gitgitgadget/git/issues/640\n[2]: https://lore.kernel.org/git/nycvar.QRO.7.76.6.1808272306271.73@tvgsbejvaqbjf.bet/\n\nThanks.\n\n Documentation/config/feature.txt |  3 +\n Documentation/config/push.txt    | 14 +++++\n Documentation/git-push.txt       |  6 ++\n builtin/send-pack.c              |  5 ++\n remote.c                         | 96 +++++++++++++++++++++++++++++---\n remote.h                         |  4 +-\n send-pack.c                      |  1 +\n t/t5533-push-cas.sh              | 86 ++++++++++++++++++++++++++++\n transport-helper.c               |  5 ++\n transport.c                      |  5 ++\n 10 files changed, 217 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config/feature.txt b/Documentation/config/feature.txt\nindex c0cbf2bb1c..f93e9fd898 100644\n--- a/Documentation/config/feature.txt\n+++ b/Documentation/config/feature.txt\n@@ -18,6 +18,9 @@ skipping more commits at a time, reducing the number of round trips.\n * `protocol.version=2` speeds up fetches from repositories with many refs by\n allowing the client to specify which refs to list before the server lists\n them.\n++\n+* `push.rejectImplicitFetch=true` runs additional checks for linkgit:git-push[1]\n+`--force-with-lease` to mitigate implicit updates of remote-tracking refs.\n\n feature.manyFiles::\n \tEnable config options that optimize for repos with many files in the\ndiff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\nindex f5e5b38c68..1a7184034d 100644\n--- a/Documentation/config/push.txt\n+++ b/Documentation/config/push.txt\n@@ -114,3 +114,17 @@ push.recurseSubmodules::\n \tspecifying '--recurse-submodules=check|on-demand|no'.\n \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n \tset (in which case a 'true' value means 'on-demand').\n+\n+push.rejectImplicitFetch::\n+\tIf set to `true`, runs additional checks for the `--force-with-lease`\n+\toption when used with linkgit:git-push[1] if the expected value for\n+\tthe remote ref is unspecified (`--force-with-lease[=<ref>]`), and\n+\tinstead asked depend on the current value of the remote-tracking ref.\n+\tThe check ensures that the commit at the tip of the remote-tracking\n+\tbranch -- which may have been implicitly updated by tools that fetch\n+\tremote refs by running linkgit:git-fetch[1] in the background -- has\n+\tbeen integrated locally, when holding the \"lease\". If the new changes\n+\tfrom such remote-tracking refs have not been updated locally before\n+\tpushing, linkgit:git-push[1] will fail indicating the reject reason\n+\tas `implicit fetch`. Enabling `feature.experimental` makes this option\n+\tdefault to `true`.\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..2176a743f3 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, setting the (experimental) `push.rejectImplicitFetch` option\n+to `true` will ensure changes from remote-tracking refs that are updated in the\n+background using linkgit:git-fetch[1] are accounted for (either by integrating\n+them locally, or explicitly specifying an overwrite), by rejecting to update\n+such refs.\n\n -f::\n --force::\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 2b9610f121..6500a8267a 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n\n+\t\tcase REF_STATUS_REJECT_IMPLICIT_FETCH:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"implicit fetch\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\ndiff --git a/remote.c b/remote.c\nindex c5ed74f91c..ee2dedd15b 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -49,6 +49,8 @@ static const char *pushremote_name;\n static struct rewrites rewrites;\n static struct rewrites rewrites_push;\n\n+static struct object_id cas_reflog_check_oid;\n+\n static int valid_remote(const struct remote *remote)\n {\n \treturn (!!remote->url) || (!!remote->foreign_vcs);\n@@ -1446,6 +1448,22 @@ int match_push_refs(struct ref *src, struct ref **dst,\n \treturn 0;\n }\n\n+/*\n+ * Consider `push.rejectImplicitFetch` to be set to true if experimental\n+ * features are enabled; use user-defined value if set explicitly.\n+ */\n+int reject_implicit_fetch()\n+{\n+\tint conf = 0;\n+\tif (!git_config_get_bool(\"push.rejectImplicitFetch\", &conf))\n+\t\treturn conf;\n+\n+\tif (!git_config_get_bool(\"feature.experimental\", &conf))\n+\t\treturn conf;\n+\n+\treturn conf;\n+}\n+\n void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t\t     int force_update)\n {\n@@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t * If the remote ref has moved and is now different\n \t\t * from what we expect, reject any push.\n \t\t *\n-\t\t * It also is an error if the user told us to check\n-\t\t * with the remote-tracking branch to find the value\n-\t\t * to expect, but we did not have such a tracking\n-\t\t * branch.\n+\t\t * It also is an error if the user told us to check with the\n+\t\t * remote-tracking branch to find the value to expect, but we\n+\t\t * did not have such a tracking branch, or we have one that\n+\t\t * has new changes.\n \t\t */\n \t\tif (ref->expect_old_sha1) {\n \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n+\t\t\telse if (reject_implicit_fetch() && ref->implicit_fetch)\n+\t\t\t\treject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;\n \t\t\telse\n-\t\t\t\t/* If the ref isn't stale then force the update. */\n+\t\t\t\t/*\n+\t\t\t\t * If the ref isn't stale, or there was no\n+\t\t\t\t * implicit fetch, force the update.\n+\t\t\t\t */\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n\n@@ -2272,23 +2295,67 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \treturn 0;\n }\n\n+static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,\n+\t\t\t     const char *ident, timestamp_t timestamp, int tz,\n+\t\t\t     const char *message, void *cb_data)\n+{\n+\treturn oideq(noid, &cas_reflog_check_oid);\n+}\n+\n+/*\n+ * Iterate through the reflog of a local branch and check if the tip of the\n+ * remote-tracking branch is reachable from one of the entries.\n+ */\n+static int remote_ref_in_reflog(const struct object_id *r_oid,\n+\t\t\t\tconst struct object_id *l_oid,\n+\t\t\t\tconst char *local_ref_name)\n+{\n+\tint ret = 0;\n+\tcas_reflog_check_oid = *r_oid;\n+\n+\tstruct commit *r_commit, *l_commit;\n+\tl_commit = lookup_commit_reference(the_repository, l_oid);\n+\tr_commit = lookup_commit_reference(the_repository, r_oid);\n+\n+\t/*\n+\t * If the remote-tracking ref is an ancestor of the local ref (a merge,\n+\t * for instance) there is no need to iterate through the reflog entries\n+\t * to ensure reachability; it can be skipped to return early instead.\n+\t */\n+\tret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;\n+\tif (ret)\n+\t\tgoto skip;\n+\n+\tret = for_each_reflog_ent_reverse(local_ref_name,\n+\t\t\t\t\t  oid_in_reflog_ent,\n+\t\t\t\t\t  NULL);\n+skip:\n+\treturn ret;\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n \t\t      struct ref *ref)\n {\n-\tint i;\n+\tint i, do_reflog_check = 0;\n+\tstruct object_id oid;\n+\tstruct ref *local_ref = get_local_ref(ref->name);\n\n \t/* Find an explicit --<option>=<name>[:<value>] entry */\n \tfor (i = 0; i < cas->nr; i++) {\n \t\tstruct push_cas *entry = &cas->entry[i];\n \t\tif (!refname_match(entry->refname, ref->name))\n \t\t\tcontinue;\n+\n \t\tref->expect_old_sha1 = 1;\n \t\tif (!entry->use_tracking)\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n \t\t\toidclr(&ref->old_oid_expect);\n-\t\treturn;\n+\t\telse\n+\t\t\tdo_reflog_check = 1;\n+\n+\t\tgoto reflog_check;\n \t}\n\n \t/* Are we using \"--<option>\" to cover all? */\n@@ -2298,6 +2365,21 @@ static void apply_cas(struct push_cas_option *cas,\n \tref->expect_old_sha1 = 1;\n \tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n \t\toidclr(&ref->old_oid_expect);\n+\telse\n+\t\tdo_reflog_check = 1;\n+\n+reflog_check:\n+\t/*\n+\t * For cases where \"--force-with-lease[=<refname>]\" i.e., when the\n+\t * \"<expect>\" value is unspecified, run additional checks to verify\n+\t * if the tip of the remote-tracking branch (if implicitly updated\n+\t * when a \"lease\" is being held) is reachable from at least one entry\n+\t * in the reflog of the local branch that is being pushed, ensuring\n+\t * new changes (if any) have been integrated locally.\n+\t */\n+\tif (do_reflog_check && local_ref && !read_ref(local_ref->name, &oid))\n+\t\tref->implicit_fetch = !remote_ref_in_reflog(&ref->old_oid, &oid,\n+\t\t\t\t\t\t\t    local_ref->name);\n }\n\n void apply_push_cas(struct push_cas_option *cas,\ndiff --git a/remote.h b/remote.h\nindex 5e3ea5a26d..f859fa5fed 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -104,7 +104,8 @@ struct ref {\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\timplicit_fetch:1;\n\n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -133,6 +134,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_FETCH_FIRST,\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n+\t\tREF_STATUS_REJECT_IMPLICIT_FETCH,\n \t\tREF_STATUS_REJECT_SHALLOW,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\ndiff --git a/send-pack.c b/send-pack.c\nindex 632f1580ca..fe7f14add4 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_IMPLICIT_FETCH:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..840b2a95f9 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -13,6 +13,41 @@ setup_srcdst_basic () {\n \t)\n }\n\n+setup_implicit_fetch () {\n+\trm -fr src dup dst &&\n+\tgit init --bare dst &&\n+\tgit clone --no-local dst src &&\n+\tgit clone --no-local dst dup\n+\t(\n+\t\tcd src &&\n+\t\ttest_commit A &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit fetch &&\n+\t\tgit merge origin/master &&\n+\t\ttest_commit B &&\n+\t\tgit switch -c branch master~1 &&\n+\t\ttest_commit C &&\n+\t\ttest_commit D &&\n+\t\tgit push --all\n+\t) &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch master &&\n+\t\tgit fetch --all &&\n+\t\tgit branch branch --track origin/branch &&\n+\t\tgit merge origin/master\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit switch master &&\n+\t\ttest_commit E &&\n+\t\tgit push origin master:master\n+\t)\n+}\n+\n test_expect_success setup '\n \t# create template repository\n \ttest_commit A &&\n@@ -256,4 +291,55 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n\n+test_expect_success 'implicit updates to remote-tracking refs with `push.rejectImplicitFetch` set (protected, all refs)' '\n+\tsetup_implicit_fetch &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch master &&\n+\t\ttest_commit G &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit fetch --all &&\n+\t\tgit config --local feature.experimental true &&\n+\t\ttest_must_fail git push --force-with-lease --all 2>err &&\n+\t\tgrep \"implicit fetch\" err\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit config --local feature.experimental false &&\n+\t\tgit push --force-with-lease --all 2>err &&\n+\t\tgrep \"forced update\" err\n+\t)\n+'\n+\n+test_expect_success 'implicit updates to remote-tracking refs with `push.rejectImplicitFetch` set (protected, specific ref)' '\n+\tsetup_implicit_fetch &&\n+\tgit ls-remote dst refs/heads/master >actual &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit F &&\n+\t\tgit switch master &&\n+\t\ttest_commit G &&\n+\t\tgit fetch  &&\n+\t\tgit config --local push.rejectImplicitFetch true &&\n+\t\ttest_must_fail git push --force-with-lease=master --all 2>err &&\n+\t\tgrep \"implicit fetch\" err\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >expect &&\n+\ttest_cmp expect actual &&\n+\t(\n+\t\tcd src &&\n+\t\tgit push --force --force-with-lease --all 2>err &&\n+\t\tgrep \"forced update\" err\n+\t)\n+'\n+\n test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c52c99d829..75b4c1b758 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"ignored fetch\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_IMPLICIT_FETCH;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -896,6 +900,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tswitch (ref->status) {\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n+\t\tcase REF_STATUS_REJECT_IMPLICIT_FETCH:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\ndiff --git a/transport.c b/transport.c\nindex 43e24bf1e5..588575498f 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -567,6 +567,10 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"stale info\", porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_IMPLICIT_FETCH:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"implicit fetch\", porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -1101,6 +1105,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_IMPLICIT_FETCH) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n\n \t\tstrbuf_reset(&buf);\n--\n2.28.0\n"},{"id":"405123","messageId":"624d9e35-29b8-4012-a3d6-e9b00a9e4485@gmail.com","threadId":"54190","inReplyTo":"20200904185147.77439-1-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-09-07T15:23:35Z","receivedAt":"2020-09-07T15:24:42Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Srinidhi\n\nThanks for working on this, making --force-with-lease safer would be a \nvaluable contribution\n\nOn 04/09/2020 19:51, Srinidhi Kaushik wrote:\n> The `--force-with-lease` option in `git-push`, makes sure that\n> refs on remote aren't clobbered by unexpected changes when the\n> \"<expect>\" ref value is explicitly specified.\n\nI think it would help to write out \n`--force-with-lease[=<refname>[:<expect>]]` so readers know what \n\"<expect>\" is referring to\n\n> For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip\n> of the remote tracking branch is populated as the \"<expect>\" value,\n> there is a possibility of allowing unwanted overwrites on the remote\n> side when some tools that implicitly fetch remote-tracking refs in\n> the background are used with the repository. If a remote-tracking ref\n> was updated when a rewrite is happening locally and if those changes\n> are pushed by omitting the \"<expect>\" value in `--force-with-lease`,\n> any new changes from the updated tip will be lost locally and will\n> be overwritten on the remote.\n> \n> This problem can be addressed by checking the `reflog` of the branch\n> that is being pushed and verify if there in a entry with the remote\n> tracking ref. By running this check, we can ensure that refs being\n> are fetched in the background while a \"lease\" is being held are not\n> overlooked before a push, and any new changes can be acknowledged\n> and (if necessary) integrated locally.\n\nAn addition safety measure would be to check the reflog of the local \ncommit and the tip of the remote tracking branch dates overlap. \nOtherwise if there is an implicit fetch of a remote head that has been \nrewound we still push the local branch when we shouldn't.\n\n> The new check will cause `git-push` to fail if it detects the presence\n> of any updated refs that we do not have locally and reject the push\n> stating `implicit fetch` as the reason.\n\n'implicit fetch' is a rather terse message - can we say something along \nthe lines of \"the remote has been updated since the last merge/push\"?\n\n> An experimental configuration setting: `push.rejectImplicitFetch`\n> which defaults to `true` (when `features.experimental` is enabled)\n> has been added, to allow `git-push` to reject a push if the check\n> fails.\n\nMaking this available with features.experimental initially is probably a \ngood idea, I hope it will become the default if in future versions.\n\n> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n> ---\n> \n> Hello,\n> I picked this up from #leftoverbits over at GitHub [1] from the open\n> issues list. This idea [2], for a safer `--force-with-lease` was\n> originally proposed by Johannes on the mailing list.\n> \n> [1]: https://github.com/gitgitgadget/git/issues/640\n> [2]: https://lore.kernel.org/git/nycvar.QRO.7.76.6.1808272306271.73@tvgsbejvaqbjf.bet/\n> \n> Thanks.\n> \n>   Documentation/config/feature.txt |  3 +\n>   Documentation/config/push.txt    | 14 +++++\n>   Documentation/git-push.txt       |  6 ++\n>   builtin/send-pack.c              |  5 ++\n>   remote.c                         | 96 +++++++++++++++++++++++++++++---\n>   remote.h                         |  4 +-\n>   send-pack.c                      |  1 +\n>   t/t5533-push-cas.sh              | 86 ++++++++++++++++++++++++++++\n>   transport-helper.c               |  5 ++\n>   transport.c                      |  5 ++\n>   10 files changed, 217 insertions(+), 8 deletions(-)\n> \n> diff --git a/Documentation/config/feature.txt b/Documentation/config/feature.txt\n> index c0cbf2bb1c..f93e9fd898 100644\n> --- a/Documentation/config/feature.txt\n> +++ b/Documentation/config/feature.txt\n> @@ -18,6 +18,9 @@ skipping more commits at a time, reducing the number of round trips.\n>   * `protocol.version=2` speeds up fetches from repositories with many refs by\n>   allowing the client to specify which refs to list before the server lists\n>   them.\n> ++\n> +* `push.rejectImplicitFetch=true` runs additional checks for linkgit:git-push[1]\n> +`--force-with-lease` to mitigate implicit updates of remote-tracking refs.\n> \n>   feature.manyFiles::\n>   \tEnable config options that optimize for repos with many files in the\n> diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\n> index f5e5b38c68..1a7184034d 100644\n> --- a/Documentation/config/push.txt\n> +++ b/Documentation/config/push.txt\n> @@ -114,3 +114,17 @@ push.recurseSubmodules::\n>   \tspecifying '--recurse-submodules=check|on-demand|no'.\n>   \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n>   \tset (in which case a 'true' value means 'on-demand').\n> +\n> +push.rejectImplicitFetch::\n> +\tIf set to `true`, runs additional checks for the `--force-with-lease`\n> +\toption when used with linkgit:git-push[1] if the expected value for\n> +\tthe remote ref is unspecified (`--force-with-lease[=<ref>]`), and\n> +\tinstead asked depend on the current value of the remote-tracking ref.\n> +\tThe check ensures that the commit at the tip of the remote-tracking\n> +\tbranch -- which may have been implicitly updated by tools that fetch\n> +\tremote refs by running linkgit:git-fetch[1] in the background -- has\n> +\tbeen integrated locally, when holding the \"lease\". If the new changes\n> +\tfrom such remote-tracking refs have not been updated locally before\n> +\tpushing, linkgit:git-push[1] will fail indicating the reject reason\n> +\tas `implicit fetch`. Enabling `feature.experimental` makes this option\n> +\tdefault to `true`.\n> diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\n> index 3b8053447e..2176a743f3 100644\n> --- a/Documentation/git-push.txt\n> +++ b/Documentation/git-push.txt\n> @@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally\n>   force push changes to `master` if the remote version is still at\n>   `base`, regardless of what your local `remotes/origin/master` has been\n>   updated to in the background.\n> ++\n> +Alternatively, setting the (experimental) `push.rejectImplicitFetch` option\n> +to `true` will ensure changes from remote-tracking refs that are updated in the\n> +background using linkgit:git-fetch[1] are accounted for (either by integrating\n> +them locally, or explicitly specifying an overwrite), by rejecting to update\n> +such refs.\n> \n>   -f::\n>   --force::\n> diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n> index 2b9610f121..6500a8267a 100644\n> --- a/builtin/send-pack.c\n> +++ b/builtin/send-pack.c\n> @@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)\n>   \t\t\tmsg = \"stale info\";\n>   \t\t\tbreak;\n> \n> +\t\tcase REF_STATUS_REJECT_IMPLICIT_FETCH:\n> +\t\t\tres = \"error\";\n> +\t\t\tmsg = \"implicit fetch\";\n> +\t\t\tbreak;\n> +\n>   \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n>   \t\t\tres = \"error\";\n>   \t\t\tmsg = \"already exists\";\n> diff --git a/remote.c b/remote.c\n> index c5ed74f91c..ee2dedd15b 100644\n> --- a/remote.c\n> +++ b/remote.c\n> @@ -49,6 +49,8 @@ static const char *pushremote_name;\n>   static struct rewrites rewrites;\n>   static struct rewrites rewrites_push;\n> \n> +static struct object_id cas_reflog_check_oid;\n> +\n\nrather than using a global variable I think it would be better just to \npass this value around using the cb_data argument of the reflog callback \nfunction\n\n>   static int valid_remote(const struct remote *remote)\n>   {\n>   \treturn (!!remote->url) || (!!remote->foreign_vcs);\n> @@ -1446,6 +1448,22 @@ int match_push_refs(struct ref *src, struct ref **dst,\n>   \treturn 0;\n>   }\n> \n> +/*\n> + * Consider `push.rejectImplicitFetch` to be set to true if experimental\n> + * features are enabled; use user-defined value if set explicitly.\n> + */\n> +int reject_implicit_fetch()\n> +{\n> +\tint conf = 0;\n> +\tif (!git_config_get_bool(\"push.rejectImplicitFetch\", &conf))\n> +\t\treturn conf;\n> +\n> +\tif (!git_config_get_bool(\"feature.experimental\", &conf))\n> +\t\treturn conf;\n> +\n> +\treturn conf;\n> +}\n> +\n>   void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n>   \t\t\t     int force_update)\n>   {\n> @@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n>   \t\t * If the remote ref has moved and is now different\n>   \t\t * from what we expect, reject any push.\n>   \t\t *\n> -\t\t * It also is an error if the user told us to check\n> -\t\t * with the remote-tracking branch to find the value\n> -\t\t * to expect, but we did not have such a tracking\n> -\t\t * branch.\n> +\t\t * It also is an error if the user told us to check with the\n> +\t\t * remote-tracking branch to find the value to expect, but we\n> +\t\t * did not have such a tracking branch, or we have one that\n> +\t\t * has new changes.\n>   \t\t */\n>   \t\tif (ref->expect_old_sha1) {\n>   \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n>   \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n> +\t\t\telse if (reject_implicit_fetch() && ref->implicit_fetch)\n> +\t\t\t\treject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;\n>   \t\t\telse\n> -\t\t\t\t/* If the ref isn't stale then force the update. */\n> +\t\t\t\t/*\n> +\t\t\t\t * If the ref isn't stale, or there was no\n> +\t\t\t\t * implicit fetch, force the update.\n> +\t\t\t\t */\n>   \t\t\t\tforce_ref_update = 1;\n>   \t\t}\n> \n> @@ -2272,23 +2295,67 @@ static int remote_tracking(struct remote *remote, const char *refname,\n>   \treturn 0;\n>   }\n> \n> +static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,\n> +\t\t\t     const char *ident, timestamp_t timestamp, int tz,\n> +\t\t\t     const char *message, void *cb_data)\n> +{\n\nusing the callback data we would have something like\n\nstruct oid *remote_head = cb_data;\nreturn oideq(noid, remote_head);\n\n> +\treturn oideq(noid, &cas_reflog_check_oid);\n> +}\n> +\n> +/*\n> + * Iterate through the reflog of a local branch and check if the tip of the\n> + * remote-tracking branch is reachable from one of the entries.\n> + */\n> +static int remote_ref_in_reflog(const struct object_id *r_oid,\n> +\t\t\t\tconst struct object_id *l_oid,\n> +\t\t\t\tconst char *local_ref_name)\n> +{\n> +\tint ret = 0;\n> +\tcas_reflog_check_oid = *r_oid;\n> +\n> +\tstruct commit *r_commit, *l_commit;\n\nOur coding style is to declare all variables before any statements, so \nthis should come above `cas_reflog_check_oid = *r_oid` but that line \nwants to go away anyway.\n\n> +\tl_commit = lookup_commit_reference(the_repository, l_oid);\n> +\tr_commit = lookup_commit_reference(the_repository, r_oid);\n> +\n> +\t/*\n> +\t * If the remote-tracking ref is an ancestor of the local ref (a merge,\n> +\t * for instance) there is no need to iterate through the reflog entries\n> +\t * to ensure reachability; it can be skipped to return early instead.\n> +\t */\n> +\tret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;\n> +\tif (ret)\n> +\t\tgoto skip;\n\nRather than using a goto it would perhaps be better to do\n\nif (!ret)\n\tret = for_each_reflog_...\n\n> +\n> +\tret = for_each_reflog_ent_reverse(local_ref_name,\n> +\t\t\t\t\t  oid_in_reflog_ent,\n> +\t\t\t\t\t  NULL);\n\nusing the callback data we'd pass r_oid rather than NULL as the last \nargument\n\n> +skip:\n> +\treturn ret;\n> +}\n> +\n>   static void apply_cas(struct push_cas_option *cas,\n>   \t\t      struct remote *remote,\n>   \t\t      struct ref *ref)\n>   {\n> -\tint i;\n> +\tint i, do_reflog_check = 0;\n> +\tstruct object_id oid;\n> +\tstruct ref *local_ref = get_local_ref(ref->name);\n> \n>   \t/* Find an explicit --<option>=<name>[:<value>] entry */\n>   \tfor (i = 0; i < cas->nr; i++) {\n>   \t\tstruct push_cas *entry = &cas->entry[i];\n>   \t\tif (!refname_match(entry->refname, ref->name))\n>   \t\t\tcontinue;\n> +\n>   \t\tref->expect_old_sha1 = 1;\n>   \t\tif (!entry->use_tracking)\n>   \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n>   \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n>   \t\t\toidclr(&ref->old_oid_expect);\n> -\t\treturn;\n> +\t\telse\n> +\t\t\tdo_reflog_check = 1;\n> +\n> +\t\tgoto reflog_check;\n\nI'm not too keen in jumping here, can't we just check `do_reflog_check` \nbelow?\n\nBest Wishes\n\nPhillip\n\n>   \t}\n> \n>   \t/* Are we using \"--<option>\" to cover all? */\n> @@ -2298,6 +2365,21 @@ static void apply_cas(struct push_cas_option *cas,\n>   \tref->expect_old_sha1 = 1;\n>   \tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n>   \t\toidclr(&ref->old_oid_expect);\n> +\telse\n> +\t\tdo_reflog_check = 1;\n> +\n> +reflog_check:\n> +\t/*\n> +\t * For cases where \"--force-with-lease[=<refname>]\" i.e., when the\n> +\t * \"<expect>\" value is unspecified, run additional checks to verify\n> +\t * if the tip of the remote-tracking branch (if implicitly updated\n> +\t * when a \"lease\" is being held) is reachable from at least one entry\n> +\t * in the reflog of the local branch that is being pushed, ensuring\n> +\t * new changes (if any) have been integrated locally.\n> +\t */\n> +\tif (do_reflog_check && local_ref && !read_ref(local_ref->name, &oid))\n> +\t\tref->implicit_fetch = !remote_ref_in_reflog(&ref->old_oid, &oid,\n> +\t\t\t\t\t\t\t    local_ref->name);\n>   }\n> \n>   void apply_push_cas(struct push_cas_option *cas,\n> diff --git a/remote.h b/remote.h\n> index 5e3ea5a26d..f859fa5fed 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -104,7 +104,8 @@ struct ref {\n>   \t\tforced_update:1,\n>   \t\texpect_old_sha1:1,\n>   \t\texact_oid:1,\n> -\t\tdeletion:1;\n> +\t\tdeletion:1,\n> +\t\timplicit_fetch:1;\n> \n>   \tenum {\n>   \t\tREF_NOT_MATCHED = 0, /* initial value */\n> @@ -133,6 +134,7 @@ struct ref {\n>   \t\tREF_STATUS_REJECT_FETCH_FIRST,\n>   \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n>   \t\tREF_STATUS_REJECT_STALE,\n> +\t\tREF_STATUS_REJECT_IMPLICIT_FETCH,\n>   \t\tREF_STATUS_REJECT_SHALLOW,\n>   \t\tREF_STATUS_UPTODATE,\n>   \t\tREF_STATUS_REMOTE_REJECT,\n> diff --git a/send-pack.c b/send-pack.c\n> index 632f1580ca..fe7f14add4 100644\n> --- a/send-pack.c\n> +++ b/send-pack.c\n> @@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n>   \tcase REF_STATUS_REJECT_FETCH_FIRST:\n>   \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n>   \tcase REF_STATUS_REJECT_STALE:\n> +\tcase REF_STATUS_REJECT_IMPLICIT_FETCH:\n>   \tcase REF_STATUS_REJECT_NODELETE:\n>   \t\treturn CHECK_REF_STATUS_REJECTED;\n>   \tcase REF_STATUS_UPTODATE:\n> diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\n> index 0b0eb1d025..840b2a95f9 100755\n> --- a/t/t5533-push-cas.sh\n> +++ b/t/t5533-push-cas.sh\n> @@ -13,6 +13,41 @@ setup_srcdst_basic () {\n>   \t)\n>   }\n> \n> +setup_implicit_fetch () {\n> +\trm -fr src dup dst &&\n> +\tgit init --bare dst &&\n> +\tgit clone --no-local dst src &&\n> +\tgit clone --no-local dst dup\n> +\t(\n> +\t\tcd src &&\n> +\t\ttest_commit A &&\n> +\t\tgit push\n> +\t) &&\n> +\t(\n> +\t\tcd dup &&\n> +\t\tgit fetch &&\n> +\t\tgit merge origin/master &&\n> +\t\ttest_commit B &&\n> +\t\tgit switch -c branch master~1 &&\n> +\t\ttest_commit C &&\n> +\t\ttest_commit D &&\n> +\t\tgit push --all\n> +\t) &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch master &&\n> +\t\tgit fetch --all &&\n> +\t\tgit branch branch --track origin/branch &&\n> +\t\tgit merge origin/master\n> +\t) &&\n> +\t(\n> +\t\tcd dup &&\n> +\t\tgit switch master &&\n> +\t\ttest_commit E &&\n> +\t\tgit push origin master:master\n> +\t)\n> +}\n> +\n>   test_expect_success setup '\n>   \t# create template repository\n>   \ttest_commit A &&\n> @@ -256,4 +291,55 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n>   \t)\n>   '\n> \n> +test_expect_success 'implicit updates to remote-tracking refs with `push.rejectImplicitFetch` set (protected, all refs)' '\n> +\tsetup_implicit_fetch &&\n> +\ttest_when_finished \"rm -fr dst src dup\" &&\n> +\tgit ls-remote dst refs/heads/master >expect.master &&\n> +\tgit ls-remote dst refs/heads/master >expect.branch &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch master &&\n> +\t\ttest_commit G &&\n> +\t\tgit switch branch &&\n> +\t\ttest_commit H &&\n> +\t\tgit fetch --all &&\n> +\t\tgit config --local feature.experimental true &&\n> +\t\ttest_must_fail git push --force-with-lease --all 2>err &&\n> +\t\tgrep \"implicit fetch\" err\n> +\t) &&\n> +\tgit ls-remote dst refs/heads/master >actual.master &&\n> +\tgit ls-remote dst refs/heads/master >actual.branch &&\n> +\ttest_cmp expect.master actual.master &&\n> +\ttest_cmp expect.branch actual.branch &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit config --local feature.experimental false &&\n> +\t\tgit push --force-with-lease --all 2>err &&\n> +\t\tgrep \"forced update\" err\n> +\t)\n> +'\n> +\n> +test_expect_success 'implicit updates to remote-tracking refs with `push.rejectImplicitFetch` set (protected, specific ref)' '\n> +\tsetup_implicit_fetch &&\n> +\tgit ls-remote dst refs/heads/master >actual &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch branch &&\n> +\t\ttest_commit F &&\n> +\t\tgit switch master &&\n> +\t\ttest_commit G &&\n> +\t\tgit fetch  &&\n> +\t\tgit config --local push.rejectImplicitFetch true &&\n> +\t\ttest_must_fail git push --force-with-lease=master --all 2>err &&\n> +\t\tgrep \"implicit fetch\" err\n> +\t) &&\n> +\tgit ls-remote dst refs/heads/master >expect &&\n> +\ttest_cmp expect actual &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit push --force --force-with-lease --all 2>err &&\n> +\t\tgrep \"forced update\" err\n> +\t)\n> +'\n> +\n>   test_done\n> diff --git a/transport-helper.c b/transport-helper.c\n> index c52c99d829..75b4c1b758 100644\n> --- a/transport-helper.c\n> +++ b/transport-helper.c\n> @@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,\n>   \t\t\tstatus = REF_STATUS_REJECT_STALE;\n>   \t\t\tFREE_AND_NULL(msg);\n>   \t\t}\n> +\t\telse if (!strcmp(msg, \"ignored fetch\")) {\n> +\t\t\tstatus = REF_STATUS_REJECT_IMPLICIT_FETCH;\n> +\t\t\tFREE_AND_NULL(msg);\n> +\t\t}\n>   \t\telse if (!strcmp(msg, \"forced update\")) {\n>   \t\t\tforced = 1;\n>   \t\t\tFREE_AND_NULL(msg);\n> @@ -896,6 +900,7 @@ static int push_refs_with_push(struct transport *transport,\n>   \t\tswitch (ref->status) {\n>   \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n>   \t\tcase REF_STATUS_REJECT_STALE:\n> +\t\tcase REF_STATUS_REJECT_IMPLICIT_FETCH:\n>   \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n>   \t\t\tif (atomic) {\n>   \t\t\t\treject_atomic_push(remote_refs, mirror);\n> diff --git a/transport.c b/transport.c\n> index 43e24bf1e5..588575498f 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -567,6 +567,10 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,\n>   \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n>   \t\t\t\t \"stale info\", porcelain, summary_width);\n>   \t\tbreak;\n> +\tcase REF_STATUS_REJECT_IMPLICIT_FETCH:\n> +\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n> +\t\t\t\t \"implicit fetch\", porcelain, summary_width);\n> +\t\tbreak;\n>   \tcase REF_STATUS_REJECT_SHALLOW:\n>   \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n>   \t\t\t\t \"new shallow roots not allowed\",\n> @@ -1101,6 +1105,7 @@ static int run_pre_push_hook(struct transport *transport,\n>   \t\tif (!r->peer_ref) continue;\n>   \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n>   \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n> +\t\tif (r->status == REF_STATUS_REJECT_IMPLICIT_FETCH) continue;\n>   \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n> \n>   \t\tstrbuf_reset(&buf);\n> --\n> 2.28.0\n> \n"},{"id":"405127","messageId":"xmqqmu211s43.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200904185147.77439-1-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-07T16:14:36Z","receivedAt":"2020-09-07T16:17:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> The `--force-with-lease` option in `git-push`, makes sure that\n> refs on remote aren't clobbered by unexpected changes when the\n> \"<expect>\" ref value is explicitly specified.\n>\n> For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip\n> of the remote tracking branch is populated as the \"<expect>\" value,\n> there is a possibility of allowing unwanted overwrites on the remote\n> side when some tools that implicitly fetch remote-tracking refs in\n> the background are used with the repository. If a remote-tracking ref\n> was updated when a rewrite is happening locally and if those changes\n> are pushed by omitting the \"<expect>\" value in `--force-with-lease`,\n> any new changes from the updated tip will be lost locally and will\n> be overwritten on the remote.\n\nHmph, I am not sure if we are on the same page as the problem with\nthe form of force-with-lease without <expect>.\n\nIn this sequence of end-user operation\n\n    $ git checkout --detach origin/target\n    ... edit working tree files ...\n    $ git commit --amend -a\n    $ git push origin +HEAD:target\n\nthe user wanted to fix the topmost commit at the 'target' branch at\nthe origin repository, and force-update it.\n\nThe --force-with-lease is a way to make sure that the only commit\nbeing lost by the force-update is the commit the user wanted to\namend.  If other users pushed to the 'target' branch in the\nmeantime, the forced push at the last step will lose it.\n\n    $ git checkout --detach origin/target\n    $ TO_LOSE=$(git rev-parse HEAD)\n    ... edit working tree files ...\n    $ git commit --amend -a\n    $ git push origin --force-with-lease=target:$TO_LOSE HEAD:target\n\nSo we say \"I knew, when I started working on the replacement, I\nstarted at the commit $TO_LOSE; please stop my forced push if the\ntip of 'target' was moved by somebody else, away from $TO_LOSE\".\n\nThe force-with-lease without the exact <expect> object name, i.e.\n\n    $ git push origin --force-with-lease=target HEAD:target\n\nwould break if 'origin/target' was updated anytime between the time\nwhen the first \"git checkout --detach\" step finishes and the time\nthe last \"git push\" is run, because 'origin/target' would be\ndifferent from $TO_LOSE and things that were pushed to 'origin/target'\nby others in the meantime will be lost, in addition to $TO_LOSE commit\nthat the user is willing to discard and replace.\n\n> This problem can be addressed by checking the `reflog` of the branch\n> that is being pushed and verify if there in a entry with the remote\n> tracking ref.\n\nSorry, but it is unclear how reflog would help.\n\nBefore the \"git checkout\" step in the example, there would have been\na \"git fetch\" from the origin that brought the remote-tracking\nbranch 'origin/target' to the current state with a reflog entry for\nit.  If an automated background process makes another fetch while\nthe user is editing files in the working tree, such a fetch may also\nadd another reflog entry for that action.\n\nUnless you make a snapshot of the reflog state immediately after you\ndo \"git checkout\" in the example, you wouldn't know if there were\nunexpected updates to the remote-tracking branch even if you check\nthe reflog.\n\nBesides, you do not control the parenthood relationship between the\ncommits _other_ people push and update to 'target' branch at the\n'origin' repository, so you cannot rely on the topology among them\nto make any decision.  Other people may be force pushing to the\nbranch while you are preparing the commit to replace $TO_LOSE by\nforce pushing.\n\n> +\tThe check ensures that the commit at the tip of the remote-tracking\n> +\tbranch -- which may have been implicitly updated by tools that fetch\n> +\tremote refs by running linkgit:git-fetch[1] in the background -- has\n> +\tbeen integrated locally, when holding the \"lease\".\n\nThe problem with \"expect-less\" form is that it does not hold the\nlease at all.  The point of \"hold the lease\" by giving an explicit\n$TO_LOSE is to force a push that does not fast-forward, so if you\niterate over the remote-tracking branch at the time of \"push\", if\nyou find more commits built on top of $TO_LOSE because a background\nfetch updated the branch, or if you find NO commits on top of $TO_LOSE\nbecause no background fetch happened in the meantime, what would be\npushed would not fast-forward.  So I am not sure what the point of\niterating over reflog.\n\nIf we want an option to force a safer behaviour, I think adding a\nconfiguration variable to forbid the form of force-with-lease\nwithout <expect> would be one way to help.  Perhaps make it the\ndefault, while allowing those who want to live dangerously to\nexplicitly turn it off.\n\nThanks.\n\n"},{"id":"405137","messageId":"nycvar.QRO.7.76.6.2009072119350.56@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"20200904185147.77439-1-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-07T19:45:18Z","receivedAt":"2020-09-07T19:45:27Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Srinidhi,\n\nOn Sat, 5 Sep 2020, Srinidhi Kaushik wrote:\n\n> The `--force-with-lease` option in `git-push`, makes sure that\n> refs on remote aren't clobbered by unexpected changes when the\n> \"<expect>\" ref value is explicitly specified.\n>\n> For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip\n> of the remote tracking branch is populated as the \"<expect>\" value,\n> there is a possibility of allowing unwanted overwrites on the remote\n> side when some tools that implicitly fetch remote-tracking refs in\n> the background are used with the repository. If a remote-tracking ref\n> was updated when a rewrite is happening locally and if those changes\n> are pushed by omitting the \"<expect>\" value in `--force-with-lease`,\n> any new changes from the updated tip will be lost locally and will\n> be overwritten on the remote.\n>\n> This problem can be addressed by checking the `reflog` of the branch\n> that is being pushed and verify if there in a entry with the remote\n> tracking ref. By running this check, we can ensure that refs being\n> are fetched in the background while a \"lease\" is being held are not\n> overlooked before a push, and any new changes can be acknowledged\n> and (if necessary) integrated locally.\n>\n> The new check will cause `git-push` to fail if it detects the presence\n> of any updated refs that we do not have locally and reject the push\n> stating `implicit fetch` as the reason.\n>\n> An experimental configuration setting: `push.rejectImplicitFetch`\n> which defaults to `true` (when `features.experimental` is enabled)\n> has been added, to allow `git-push` to reject a push if the check\n> fails.\n>\n> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n> ---\n>\n> Hello,\n> I picked this up from #leftoverbits over at GitHub [1] from the open\n> issues list. This idea [2], for a safer `--force-with-lease` was\n> originally proposed by Johannes on the mailing list.\n>\n> [1]: https://github.com/gitgitgadget/git/issues/640\n> [2]: https://lore.kernel.org/git/nycvar.QRO.7.76.6.1808272306271.73@tvgsbejvaqbjf.bet/\n\nFirst of all: thank you for picking this up! The contribution is\npleasantly well-written, thank you also for that.\n\nNow, to be honest, I thought that this mode would merit a new option\nrather than piggy-backing on top of `--force-with-lease`. The reason is\nthat `--force-with-lease` targets a slightly different use case than mine:\nit makes sure that we do not overwrite remote refs unless we already had a\nchance to inspect them.\n\nIn contrast, my workflow uses `git pull --rebase` in two or more separate\nworktrees, e.g. when developing a patch on two different Operating\nSystems, I frequently forget to pull (to my public repository) on one\nside, and I want to avoid force-pushing in that case, even if VS Code (or\nI, via `git remote update`) fetched the ref (but failing to rebase the\nlocal branch on top of it).\n\nHowever, in other scenarios I very much do _not_ want to incorporate the\nremote ref. For example, I often fetch\nhttps://github.com/git-for-windows/git.wiki.git to check for the\noccasional bogus change. Whenever I see such a bogus change, and it is at\nthe tip of the branch, I want to force-push _without_ incorporating the\nbogus change into the local branch, yet I _do_ want to use\n`--force-with-lease` because an independent change could have come in via\nthe Wiki in the meantime.\n\nSo I think that the original `--force-with-lease` and the mode you\nimplemented target subtly different use cases that are both valid, and\ntherefore I would like to request a separate option for the latter.\n\nHowever, I have to admit that I could not think of a good name for that\noption. \"Implicit fetch\" seems a bit too vague here, because the local\nbranch was not fetched, and certainly not implicitly, yet the logic\nrevolves around the local branch having been rebased to the\nremote-tracking ref at some stage.\n\nEven if we went with the config option to modify `--force-with-lease`'s\nbehavior, I would recommend separating out the `feature.experimental`\nchanges into their own patch, so that they can be reverted easily in case\nthe experimental feature is made the default.\n\nA couple more comments:\n\n> @@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n>  \t\t * If the remote ref has moved and is now different\n>  \t\t * from what we expect, reject any push.\n>  \t\t *\n> -\t\t * It also is an error if the user told us to check\n> -\t\t * with the remote-tracking branch to find the value\n> -\t\t * to expect, but we did not have such a tracking\n> -\t\t * branch.\n> +\t\t * It also is an error if the user told us to check with the\n> +\t\t * remote-tracking branch to find the value to expect, but we\n> +\t\t * did not have such a tracking branch, or we have one that\n> +\t\t * has new changes.\n\nIf I were you, I would try to keep the original formatting, so that it\nbecomes more obvious that the part \", or we have [...]\" was appended.\n\n>  \t\tif (ref->expect_old_sha1) {\n>  \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n>  \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n> +\t\t\telse if (reject_implicit_fetch() && ref->implicit_fetch)\n> +\t\t\t\treject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;\n>  \t\t\telse\n> -\t\t\t\t/* If the ref isn't stale then force the update. */\n> +\t\t\t\t/*\n> +\t\t\t\t * If the ref isn't stale, or there was no\n\nShould this \"or\" not be an \"and\" instead?\n\n> +\t\t\t\t * implicit fetch, force the update.\n> +\t\t\t\t */\n>  \t\t\t\tforce_ref_update = 1;\n>  \t\t}\n> [...]\n>  static void apply_cas(struct push_cas_option *cas,\n>  \t\t      struct remote *remote,\n>  \t\t      struct ref *ref)\n>  {\n> -\tint i;\n> +\tint i, do_reflog_check = 0;\n> +\tstruct object_id oid;\n> +\tstruct ref *local_ref = get_local_ref(ref->name);\n>\n>  \t/* Find an explicit --<option>=<name>[:<value>] entry */\n>  \tfor (i = 0; i < cas->nr; i++) {\n>  \t\tstruct push_cas *entry = &cas->entry[i];\n>  \t\tif (!refname_match(entry->refname, ref->name))\n>  \t\t\tcontinue;\n> +\n>  \t\tref->expect_old_sha1 = 1;\n>  \t\tif (!entry->use_tracking)\n>  \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n>  \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n>  \t\t\toidclr(&ref->old_oid_expect);\n> -\t\treturn;\n> +\t\telse\n> +\t\t\tdo_reflog_check = 1;\n> +\n> +\t\tgoto reflog_check;\n\nHmm. I do not condemn `goto` statements in general, but this one makes the\nflow harder to follow. I would prefer something like this:\n\n-- snip --\n \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n \t\t\toidclr(&ref->old_oid_expect);\n+\t\telse if (local_ref && !read_ref(local_ref->name, &oid))\n+\t\t\tref->implicit_fetch =\n+\t\t\t\t!remote_ref_in_reflog(&ref->old_oid, &oid,\n+\t\t\t\t\t\t      local_ref->name);\n \t\treturn;\n-- snap --\n\nAgain, thank you so much for working on this!\n\nCiao,\nDscho\n"},{"id":"405162","messageId":"20200908165900.GC40807@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009072119350.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-08T16:59:00Z","receivedAt":"2020-09-08T16:59:31Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Johannes,\n\nOn 09/07/2020 21:45, Johannes Schindelin wrote:\n>> [...]\n> First of all: thank you for picking this up! The contribution is\n> pleasantly well-written, thank you also for that.\n\nThank you for the kind words.\n\n> Now, to be honest, I thought that this mode would merit a new option\n> rather than piggy-backing on top of `--force-with-lease`. The reason is\n> that `--force-with-lease` targets a slightly different use case than mine:\n> it makes sure that we do not overwrite remote refs unless we already had a\n> chance to inspect them.\n>\n> In contrast, my workflow uses `git pull --rebase` in two or more separate\n> worktrees, e.g. when developing a patch on two different Operating\n> Systems, I frequently forget to pull (to my public repository) on one\n> side, and I want to avoid force-pushing in that case, even if VS Code (or\n> I, via `git remote update`) fetched the ref (but failing to rebase the\n> local branch on top of it).\n>\n> However, in other scenarios I very much do _not_ want to incorporate the\n> remote ref. For example, I often fetch\n> https://github.com/git-for-windows/git.wiki.git to check for the\n> occasional bogus change. Whenever I see such a bogus change, and it is at\n> the tip of the branch, I want to force-push _without_ incorporating the\n> bogus change into the local branch, yet I _do_ want to use\n> `--force-with-lease` because an independent change could have come in via\n> the Wiki in the meantime.\n\nI realize that this new check would not be helpful if we deliberately\nchoose not to include an unwanted change from the updated remote's tip.\nIn that case, we would have to use `--force` make it work, and that\ndefeats the use of `--force-with-lease`.\n\n> So I think that the original `--force-with-lease` and the mode you\n> implemented target subtly different use cases that are both valid, and\n> therefore I would like to request a separate option for the latter.\n\nOK. So, I am assuming that you are suggesting to add a new function that\nis separate from `apply_push_cas()` and run the check on each of the\nremote refs. Would that be correct?\n\nIf that's the case, how does it work along with `--force-with-lease`?\nOn one hand we have `--force-with-lease` to ensure we rewrite the remote\nthat we have _already_ seen, and on the other, a new option that checks\nreflog of the local branch to see if it is missing any updates from the\nremote that may have happened in the meantime. If we pass both of them\nfor `push` and if the former doesn't complain, and the latter check\nfails, should the `push` still go through?\n\nI feel that this check included with `--force-with-lease` only when\nthe `use_tracking` or `use_tracking_for_rest` options are enabled\nwould give a heads-up the the user about the background fetch. If\nthey decide that they don't need new updates, then supplying the\nnew \"<expect>\" value in the next push would imply they've seen the\nnew update, and choose to overwrite it anyway. The check would not\nrun in this case. But again, I wonder if the this \"two-step\" process\nmakes `push` cumbersome.\n\n> However, I have to admit that I could not think of a good name for that\n> option. \"Implicit fetch\" seems a bit too vague here, because the local\n> branch was not fetched, and certainly not implicitly, yet the logic\n> revolves around the local branch having been rebased to the\n> remote-tracking ref at some stage.\n\nThe message \"implicit fetch\" was in context of the remote ref. But yes,\nthe current reject reason is not clear and implies that local branch\nwas fetched, which isn't the case.\n\n> Even if we went with the config option to modify `--force-with-lease`'s\n> behavior, I would recommend separating out the `feature.experimental`\n> changes into their own patch, so that they can be reverted easily in case\n> the experimental feature is made the default.\n\nGood idea!\n\n> A couple more comments:\n>\n>> @@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n>>                * If the remote ref has moved and is now different\n>>                * from what we expect, reject any push.\n>>                *\n>> -              * It also is an error if the user told us to check\n>> -              * with the remote-tracking branch to find the value\n>> -              * to expect, but we did not have such a tracking\n>> -              * branch.\n>> +              * It also is an error if the user told us to check with the\n>> +              * remote-tracking branch to find the value to expect, but we\n>> +              * did not have such a tracking branch, or we have one that\n>> +              * has new changes.\n>\n> If I were you, I would try to keep the original formatting, so that it\n> becomes more obvious that the part \", or we have [...]\" was appended.\n\nAlright, I will append the new comment in a new line instead.\n\n>\n>>               if (ref->expect_old_sha1) {\n>>                       if (!oideq(&ref->old_oid, &ref->old_oid_expect))\n>>                               reject_reason = REF_STATUS_REJECT_STALE;\n>> +                     else if (reject_implicit_fetch() && ref->implicit_fetch)\n>> +                             reject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;\n>>                       else\n>> -                             /* If the ref isn't stale then force the update. */\n>> +                             /*\n>> +                              * If the ref isn't stale, or there was no\n>\n> Should this \"or\" not be an \"and\" instead?\n\nD'oh, you are right. It should have been an \"and\".\n\n>\n>> +                              * implicit fetch, force the update.\n>> +                              */\n>>                               force_ref_update = 1;\n>>               }\n>> [...]\n>>  static void apply_cas(struct push_cas_option *cas,\n>>                     struct remote *remote,\n>>                     struct ref *ref)\n>>  {\n>> -     int i;\n>> +     int i, do_reflog_check = 0;\n>> +     struct object_id oid;\n>> +     struct ref *local_ref = get_local_ref(ref->name);\n>>\n>>       /* Find an explicit --<option>=<name>[:<value>] entry */\n>>       for (i = 0; i < cas->nr; i++) {\n>>               struct push_cas *entry = &cas->entry[i];\n>>               if (!refname_match(entry->refname, ref->name))\n>>                       continue;\n>> +\n>>               ref->expect_old_sha1 = 1;\n>>               if (!entry->use_tracking)\n>>                       oidcpy(&ref->old_oid_expect, &entry->expect);\n>>               else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n>>                       oidclr(&ref->old_oid_expect);\n>> -             return;\n>> +             else\n>> +                     do_reflog_check = 1;\n>> +\n>> +             goto reflog_check;\n>\n> Hmm. I do not condemn `goto` statements in general, but this one makes the\n> flow harder to follow. I would prefer something like this:\n>\n> -- snip --\n>               else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n>                       oidclr(&ref->old_oid_expect);\n> +             else if (local_ref && !read_ref(local_ref->name, &oid))\n> +                     ref->implicit_fetch =\n> +                             !remote_ref_in_reflog(&ref->old_oid, &oid,\n> +                                                   local_ref->name);\n>               return;\n> -- snap --\n\nAdding this condition looks cleaner instead of the `goto`. A similar\nsuggestion was made in the other thread [1] as well; this will be\naddressed in v2.\n\n> Again, thank you so much for working on this!\n\nThanks again, for taking the time to review this.\n\n[1]: https://public-inbox.org/git/624d9e35-29b8-4012-a3d6-e9b00a9e4485@gmail.com/\n--\nSrinidhi Kaushik\n"},{"id":"405174","messageId":"20200908160021.GB40807@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqmu211s43.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-08T16:00:21Z","receivedAt":"2020-09-08T19:23:38Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Junio,\nThanks for taking the time to review this patch.\n\nOn 09/07/2020 09:14, Junio C Hamano wrote:\n>> The `--force-with-lease` option in `git-push`, makes sure that\n>> refs on remote aren't clobbered by unexpected changes when the\n>> \"<expect>\" ref value is explicitly specified.\n>>\n>> For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip\n>> of the remote tracking branch is populated as the \"<expect>\" value,\n>> there is a possibility of allowing unwanted overwrites on the remote\n>> side when some tools that implicitly fetch remote-tracking refs in\n>> the background are used with the repository. If a remote-tracking ref\n>> was updated when a rewrite is happening locally and if those changes\n>> are pushed by omitting the \"<expect>\" value in `--force-with-lease`,\n>> any new changes from the updated tip will be lost locally and will\n>> be overwritten on the remote.\n>\n> Hmph, I am not sure if we are on the same page as the problem with\n> the form of force-with-lease without <expect>.\n>\n> In this sequence of end-user operation\n>\n>    $ git checkout --detach origin/target\n>    ... edit working tree files ...\n>    $ git commit --amend -a\n>    $ git push origin +HEAD:target\n>\n> the user wanted to fix the topmost commit at the 'target' branch at\n> the origin repository, and force-update it.\n>\n> The --force-with-lease is a way to make sure that the only commit\n> being lost by the force-update is the commit the user wanted to\n> amend.  If other users pushed to the 'target' branch in the\n> meantime, the forced push at the last step will lose it.\n>\n>    $ git checkout --detach origin/target\n>    $ TO_LOSE=$(git rev-parse HEAD)\n>    ... edit working tree files ...\n>    $ git commit --amend -a\n>    $ git push origin --force-with-lease=target:$TO_LOSE HEAD:target\n>\n> So we say \"I knew, when I started working on the replacement, I\n> started at the commit $TO_LOSE; please stop my forced push if the\n> tip of 'target' was moved by somebody else, away from $TO_LOSE\".\n>\n> The force-with-lease without the exact <expect> object name, i.e.\n>\n>    $ git push origin --force-with-lease=target HEAD:target\n>\n> would break if 'origin/target' was updated anytime between the time\n> when the first \"git checkout --detach\" step finishes and the time\n> the last \"git push\" is run, because 'origin/target' would be\n> different from $TO_LOSE and things that were pushed to 'origin/target'\n> by others in the meantime will be lost, in addition to $TO_LOSE commit\n> that the user is willing to discard and replace.\n\nSorry, that commit message should have been worded in a better way.\nWhat I originally meant to say was: losing any _new_ changes that\nwere made on the remote during a rewrite, i.e., when we have a starting\npoint ($TO_LOSE) from the remote-tracking ref that we want to base our\nrewrite on (after checkout) and then if the remote-tracking ref gets\nupdated by a push from another user, _and_ we don't use an \"<expect>\"\nfor `--force-with-lease`, there is a possibility that changes from the\nother push would be lost because we're basing our rewrite on an older\nversion (?) of the remote-tracking ref, and our push would overwrite\nits new changes because the value \"<expect>\" when omitted would point\nto the updated tip of the remote instead. The condition:\n\n\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n\nwould evaluate false since we're using `use_tracking`. This essentially\nreduces the behavior of `--force-with-lease=<ref>` to `--force` in this\nscenario.\n\n>> This problem can be addressed by checking the `reflog` of the branch\n>> that is being pushed and verify if there in a entry with the remote\n>> tracking ref.\n> Sorry, but it is unclear how reflog would help.\n>\n> Before the \"git checkout\" step in the example, there would have been\n> a \"git fetch\" from the origin that brought the remote-tracking\n> branch 'origin/target' to the current state with a reflog entry for\n> it.  If an automated background process makes another fetch while\n> the user is editing files in the working tree, such a fetch may also\n> add another reflog entry for that action.\n>\n> Unless you make a snapshot of the reflog state immediately after you\n> do \"git checkout\" in the example, you wouldn't know if there were\n> unexpected updates to the remote-tracking branch even if you check\n> the reflog.\n>\n> Besides, you do not control the parenthood relationship between the\n> commits _other_ people push and update to 'target' branch at the\n> 'origin' repository, so you cannot rely on the topology among them\n> to make any decision.  Other people may be force pushing to the\n> branch while you are preparing the commit to replace $TO_LOSE by\n> force pushing.\n>\n>> +     The check ensures that the commit at the tip of the remote-tracking\n>> +     branch -- which may have been implicitly updated by tools that fetch\n>> +     remote refs by running linkgit:git-fetch[1] in the background -- has\n>> +     been integrated locally, when holding the \"lease\".\n>\n> The problem with \"expect-less\" form is that it does not hold the\n> lease at all.  The point of \"hold the lease\" by giving an explicit\n> $TO_LOSE is to force a push that does not fast-forward, so if you\n> iterate over the remote-tracking branch at the time of \"push\", if\n> you find more commits built on top of $TO_LOSE because a background\n> fetch updated the branch, or if you find NO commits on top of $TO_LOSE\n> because no background fetch happened in the meantime, what would be\n> pushed would not fast-forward.  So I am not sure what the point of\n> iterating over reflog.\n\nRight, I agree with what is described above. But, in this patch, we are\nlooking at the reflog of the _local_ branch that is going to be updated\non the remote side. The point of going through the reflog is to see if\nthe current tip its remote-tracking branch is present in one of the\nreflog entries implying that any new changes (pushes from another user)\nin the meantime aren't ignored and overwritten with our push.\nWould that be an incorrect assumption?\n\n> If we want an option to force a safer behaviour, I think adding a\n> configuration variable to forbid the form of force-with-lease\n> without <expect> would be one way to help.  Perhaps make it the\n> default, while allowing those who want to live dangerously to\n> explicitly turn it off.\n\nYes, that sounds like a good way to mitigate this issue; but that being\nsaid, setups where `--force-with-lease` is being used as an alias for\n`--force` should probably be taken into consideration though.\n\nThanks.\n--\nSrinidhi Kaushik\n"},{"id":"405175","messageId":"xmqqwo14z2de.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009072119350.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-08T15:58:53Z","receivedAt":"2020-09-08T19:29:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Now, to be honest, I thought that this mode would merit a new option\n> rather than piggy-backing on top of `--force-with-lease`. The reason is\n> that `--force-with-lease` targets a slightly different use case than mine:\n> it makes sure that we do not overwrite remote refs unless we already had a\n> chance to inspect them.\n>\n> In contrast, my workflow uses `git pull --rebase` in two or more separate\n> worktrees, e.g. when developing a patch on two different Operating\n> Systems, I frequently forget to pull (to my public repository) on one\n> side, and I want to avoid force-pushing in that case, even if VS Code (or\n> I, via `git remote update`) fetched the ref (but failing to rebase the\n> local branch on top of it).\n>\n> So I think that the original `--force-with-lease` and the mode you\n> implemented target subtly different use cases that are both valid, and\n> therefore I would like to request a separate option for the latter.\n\nI tend to agree that that particular use case does not fit what the\n\"force with lease\" option is meant to solve.  It should be a different\noption, as you do not even want to be forcing.\n\nBut probably I am not getting your use case well enough to give a\ngood name suggestion.  \"git push\" without any form of \"force\" would\nsafely fail in your \"I rebased on one side, and the other one is now\nout of sync\" situation already, so...\n"},{"id":"405177","messageId":"xmqqr1rcxdsu.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009072119350.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-08T19:34:57Z","receivedAt":"2020-09-08T19:35:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Now, to be honest, I thought that this mode would merit a new option\n> rather than piggy-backing on top of `--force-with-lease`. The reason is\n> that `--force-with-lease` targets a slightly different use case than mine:\n> it makes sure that we do not overwrite remote refs unless we already had a\n> chance to inspect them.\n>\n> In contrast, my workflow uses `git pull --rebase` in two or more separate\n> worktrees, e.g. when developing a patch on two different Operating\n> Systems, I frequently forget to pull (to my public repository) on one\n> side, and I want to avoid force-pushing in that case, even if VS Code (or\n> I, via `git remote update`) fetched the ref (but failing to rebase the\n> local branch on top of it).\n> ...\n> So I think that the original `--force-with-lease` and the mode you\n> implemented target subtly different use cases that are both valid, and\n> therefore I would like to request a separate option for the latter.\n\nI agree that the use case in the second paragraph above does not fit\nwhat the \"force with lease\" option is meant to solve.  You do not\neven want to be forcing in the workflow so \"--force-with-anything\"\nis a bad name for the mode of operation, if I am reading you right.\n"},{"id":"405179","messageId":"20200908154859.GA40807@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"624d9e35-29b8-4012-a3d6-e9b00a9e4485@gmail.com","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-08T15:48:59Z","receivedAt":"2020-09-08T19:44:32Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Phillip,\n\nOn 09/07/2020 16:23, Phillip Wood wrote:\n> [...]\n> Thanks for working on this, making --force-with-lease safer would be a\n> valuable contribution\n\n:)\n\n> > The `--force-with-lease` option in `git-push`, makes sure that\n> > refs on remote aren't clobbered by unexpected changes when the\n> > \"<expect>\" ref value is explicitly specified.\n>\n> I think it would help to write out\n> `--force-with-lease[=<refname>[:<expect>]]` so readers know what\n> \"<expect>\" is referring to\n\nThat makes sense; noted.\n\n> > For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip\n> > of the remote tracking branch is populated as the \"<expect>\" value,\n> > there is a possibility of allowing unwanted overwrites on the remote\n> > side when some tools that implicitly fetch remote-tracking refs in\n> > the background are used with the repository. If a remote-tracking ref\n> > was updated when a rewrite is happening locally and if those changes\n> > are pushed by omitting the \"<expect>\" value in `--force-with-lease`,\n> > any new changes from the updated tip will be lost locally and will\n> > be overwritten on the remote.\n> >\n> > This problem can be addressed by checking the `reflog` of the branch\n> > that is being pushed and verify if there in a entry with the remote\n> > tracking ref. By running this check, we can ensure that refs being\n> > are fetched in the background while a \"lease\" is being held are not\n> > overlooked before a push, and any new changes can be acknowledged\n> > and (if necessary) integrated locally.\n\n> An addition safety measure would be to check the reflog of the local\n> commit and the tip of the remote tracking branch dates overlap.\n> Otherwise if there is an implicit fetch of a remote head that has been\n> rewound we still push the local branch when we shouldn't.\n\nThis sounds much better. My initial description of the check was perhaps\na bit confusing.\n\n> > The new check will cause `git-push` to fail if it detects the presence\n> > of any updated refs that we do not have locally and reject the push\n> > stating `implicit fetch` as the reason.\n>\n> 'implicit fetch' is a rather terse message - can we say something along\n> the lines of \"the remote has been updated since the last merge/push\"?\n\nI was going by the \"two-word\" approach like \"stale info\", \"fetch first\",\n\"no match\", and so on. But, I'll look into wording the reject reason\nalong those lines.\n\n> > An experimental configuration setting: `push.rejectImplicitFetch`\n> > which defaults to `true` (when `features.experimental` is enabled)\n> > has been added, to allow `git-push` to reject a push if the check\n> > fails.\n>\n> Making this available with features.experimental initially is probably a\n> good idea, I hope it will become the default if in future versions.\n\nI hope so. :)\n\n> > [...]\n> > +\t\tcase REF_STATUS_REJECT_IMPLICIT_FETCH:\n> > +\t\t\tres = \"error\";\n> > +\t\t\tmsg = \"implicit fetch\";\n> > +\t\t\tbreak;\n> > +\n> >   \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n> >   \t\t\tres = \"error\";\n> >   \t\t\tmsg = \"already exists\";\n> > diff --git a/remote.c b/remote.c\n> > index c5ed74f91c..ee2dedd15b 100644\n> > --- a/remote.c\n> > +++ b/remote.c\n> > @@ -49,6 +49,8 @@ static const char *pushremote_name;\n> >   static struct rewrites rewrites;\n> >   static struct rewrites rewrites_push;\n> >\n> > +static struct object_id cas_reflog_check_oid;\n> > +\n>\n> rather than using a global variable I think it would be better just to\n> pass this value around using the cb_data argument of the reflog callback\n> function\n\nI have to admit that I was hesitant to use the global variable when\nwriting this. For some reason I thought the callback data was used to\nstore results from the called function only and not pass arguments.\nWill fix that in v2.\n\n> > [...]\n> > +static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,\n> > +\t\t\t     const char *ident, timestamp_t timestamp, int tz,\n> > +\t\t\t     const char *message, void *cb_data)\n> > +{\n>\n> using the callback data we would have something like\n>\n> struct oid *remote_head = cb_data;\n> return oideq(noid, remote_head);\n\nGot it; this and the callback argument to the reflog entry function\nwill be updated accordingly.\n\n> > [...]\n> > +{\n> > +\tint ret = 0;\n> > +\tcas_reflog_check_oid = *r_oid;\n> > +\n> > +\tstruct commit *r_commit, *l_commit;\n>\n> Our coding style is to declare all variables before any statements, so\n> this should come above `cas_reflog_check_oid = *r_oid` but that line\n> wants to go away anyway.\n\nNoted; `cas_reflog_check_oid` will go away as suggested above.\n\n> > [...]\n> > +\tret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;\n> > +\tif (ret)\n> > +\t\tgoto skip;\n>\n> Rather than using a goto it would perhaps be better to do\n>\n> if (!ret)\n>\tret = for_each_reflog_...\n>\n>\n\nOK, yes. The `goto` can be avoided here.\n\n> > +\n> > +\tret = for_each_reflog_ent_reverse(local_ref_name,\n> > +\t\t\t\t\t  oid_in_reflog_ent,\n> > +\t\t\t\t\t  NULL);\n>\n> using the callback data we'd pass r_oid rather than NULL as the last\n> argument\n\n\n> > [...]\n> >   \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n> >   \t\t\toidclr(&ref->old_oid_expect);\n> > -\t\treturn;\n> > +\t\telse\n> > +\t\t\tdo_reflog_check = 1;\n> > +\n> > +\t\tgoto reflog_check;\n>\n> I'm not too keen in jumping here, can't we just check `do_reflog_check`\n> below?\n\nYes, of course. I was trying conserve the original flow of returning\nfrom the function right after the loop. Since this is just one condition\nto check if `do_reflog_check` is set to 1; we can get rid of the `goto`\nand use a `break` instead.\n\n> [...]\n\nThanks for a thorough review.\n--\nSrinidhi Kaushik\n"},{"id":"405199","messageId":"xmqqmu20x9tn.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200908160021.GB40807@mail.clickyotomy.dev","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-08T21:00:52Z","receivedAt":"2020-09-08T21:01:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> to the updated tip of the remote instead. The condition:\n>\n> \tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n>\n> would evaluate false since we're using `use_tracking`. This essentially\n> reduces the behavior of `--force-with-lease=<ref>` to `--force` in this\n> scenario.\n\nYes, that is exactly why I have kept saying that the form without\n<expect> is not safe and cannot be made safer especially when\nauto-fetching is involved.\n\n> Right, I agree with what is described above. But, in this patch, we are\n> looking at the reflog of the _local_ branch that is going to be updated\n> on the remote side. The point of going through the reflog is to see if\n> the current tip its remote-tracking branch is present in one of the\n> reflog entries implying that any new changes (pushes from another user)\n> in the meantime aren't ignored and overwritten with our push.\n> Would that be an incorrect assumption?\n\nI am afraid it is.  You may have looked at it and even kept it\nlocally, to be looked at later, without letting it affect what you\nare going ot force-push in any way.  You might later come back and\nresurrect their change on top of what you force pushed, or more\nlikely you may simply forget, especially if the forced push goes\nthrough without failing.\n\n> Yes, that sounds like a good way to mitigate this issue; but that being\n> said, setups where `--force-with-lease` is being used as an alias for\n> `--force` should probably be taken into consideration though.\n\nWe cannot help, and it is not our job to dispel, misconfigurations\nand misconceptions caused by following bad pieces of advice other\npeople gave our users, though.\n"},{"id":"405230","messageId":"nycvar.QRO.7.76.6.2009090525120.54@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"xmqqwo14z2de.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-09T03:40:10Z","receivedAt":"2020-09-09T08:20:06Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 8 Sep 2020, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n>\n> > Now, to be honest, I thought that this mode would merit a new option\n> > rather than piggy-backing on top of `--force-with-lease`. The reason is\n> > that `--force-with-lease` targets a slightly different use case than mine:\n> > it makes sure that we do not overwrite remote refs unless we already had a\n> > chance to inspect them.\n> >\n> > In contrast, my workflow uses `git pull --rebase` in two or more separate\n> > worktrees, e.g. when developing a patch on two different Operating\n> > Systems, I frequently forget to pull (to my public repository) on one\n> > side, and I want to avoid force-pushing in that case, even if VS Code (or\n> > I, via `git remote update`) fetched the ref (but failing to rebase the\n> > local branch on top of it).\n> >\n> > So I think that the original `--force-with-lease` and the mode you\n> > implemented target subtly different use cases that are both valid, and\n> > therefore I would like to request a separate option for the latter.\n>\n> I tend to agree that that particular use case does not fit what the\n> \"force with lease\" option is meant to solve.  It should be a different\n> option, as you do not even want to be forcing.\n\nActually, I _do_ want to be forcing, but _only_ if the recorded remote tip\nhas been integrated into the local branch at some point (even if it has\nbeen rebased or amended away).\n\nAnd even then, I want to only force when the recorded remote tip still\nmatches the actual remote tip, i.e. I want to force \"with lease\".\n\n> But probably I am not getting your use case well enough to give a\n> good name suggestion.  \"git push\" without any form of \"force\" would\n> safely fail in your \"I rebased on one side, and the other one is now\n> out of sync\" situation already, so...\n\nYes, it would safely fail, but it would also fail in situations where I do\n_not_ want it to fail.\n\nImagine that you and I work together on a patch series, and we actually\nshare the same public repository. Let's say that we also want to allow for\nsome crossing emails that might be missed by the other when pushing our\nchanges.\n\nSince we are working on a patch series, there will be a lot of amending\nand rewording and rebasing going on.\n\nSo let's assume that you just did another round of amending and rewording\nand want to push the branch. What you will want to avoid is to overwrite\ncommits that I might have just pushed. `--force-with-lease` to the rescue!\n\nSince you did not fetch in between pulling and amending, that is safe.\n\nBut what if, maybe even by mistake, you called `git fetch` in between? And\nthen you _did_ get my updates, but only in the remote-tracking branch (the\n\"lease\"), yet those updates never made it into the local branch. And since\n`git push --force-with-lease` did not fail, you failed to notice that I\nhad made some edits, didn't `range-diff` and took care of amending your\nlocal changes accordingly, and therefore you overwrote my changes.\n\nThat is the scenario this new mode wants to address.\n\nNote that a similar scenario can occur if you work on a patch series on\nLinux, then push it to your public repository, then try to make it work\non, say, an HP/UX machine you happen to have access to (maybe because your\nemployer needs you to make sure that this particular patch series solves a\nparticular problem on that particular machine), and you need to amend the\nchanges there. If you're very disciplined, you will only work on one\nmachine until you're done, then push, then fetch on the other machine.\nHowever, if you're less disciplined (or \"just need to check out something\nreal quick on the other machine\"), this new mode will come in quite handy.\n\nCiao,\nDscho\n"},{"id":"405231","messageId":"nycvar.QRO.7.76.6.2009090542430.54@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"xmqqr1rcxdsu.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-09T03:44:33Z","receivedAt":"2020-09-09T08:24:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 8 Sep 2020, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n>\n> > Now, to be honest, I thought that this mode would merit a new option\n> > rather than piggy-backing on top of `--force-with-lease`. The reason is\n> > that `--force-with-lease` targets a slightly different use case than mine:\n> > it makes sure that we do not overwrite remote refs unless we already had a\n> > chance to inspect them.\n> >\n> > In contrast, my workflow uses `git pull --rebase` in two or more separate\n> > worktrees, e.g. when developing a patch on two different Operating\n> > Systems, I frequently forget to pull (to my public repository) on one\n> > side, and I want to avoid force-pushing in that case, even if VS Code (or\n> > I, via `git remote update`) fetched the ref (but failing to rebase the\n> > local branch on top of it).\n> > ...\n> > So I think that the original `--force-with-lease` and the mode you\n> > implemented target subtly different use cases that are both valid, and\n> > therefore I would like to request a separate option for the latter.\n>\n> I agree that the use case in the second paragraph above does not fit\n> what the \"force with lease\" option is meant to solve.  You do not\n> even want to be forcing in the workflow so \"--force-with-anything\"\n> is a bad name for the mode of operation, if I am reading you right.\n\nNo, you _have_ to force the push.\n\nIf you don't have to force the push, i.e. if it is a fast-forward, there\nabsolutely is no need for any of this.\n\nIn contrast, when you want to make sure that you _actually_ incorporated\nthe revision that is currently the remote tip, e.g. via `git pull\n--rebase` with a possible additional rebase on top that makes this _not_ a\nfast-forward, you totally have to force the push, otherwise it won't work.\n\nCiao,\nDscho\n"},{"id":"405327","messageId":"nycvar.QRO.7.76.6.2009101220400.56@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009090542430.54@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-10T10:22:58Z","receivedAt":"2020-09-10T13:24:19Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 9 Sep 2020, Johannes Schindelin wrote:\n\n> On Tue, 8 Sep 2020, Junio C Hamano wrote:\n>\n> > Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> >\n> > > Now, to be honest, I thought that this mode would merit a new option\n> > > rather than piggy-backing on top of `--force-with-lease`. The reason is\n> > > that `--force-with-lease` targets a slightly different use case than mine:\n> > > it makes sure that we do not overwrite remote refs unless we already had a\n> > > chance to inspect them.\n> > >\n> > > In contrast, my workflow uses `git pull --rebase` in two or more separate\n> > > worktrees, e.g. when developing a patch on two different Operating\n> > > Systems, I frequently forget to pull (to my public repository) on one\n> > > side, and I want to avoid force-pushing in that case, even if VS Code (or\n> > > I, via `git remote update`) fetched the ref (but failing to rebase the\n> > > local branch on top of it).\n> > > ...\n> > > So I think that the original `--force-with-lease` and the mode you\n> > > implemented target subtly different use cases that are both valid, and\n> > > therefore I would like to request a separate option for the latter.\n> >\n> > I agree that the use case in the second paragraph above does not fit\n> > what the \"force with lease\" option is meant to solve.  You do not\n> > even want to be forcing in the workflow so \"--force-with-anything\"\n> > is a bad name for the mode of operation, if I am reading you right.\n>\n> No, you _have_ to force the push.\n>\n> If you don't have to force the push, i.e. if it is a fast-forward, there\n> absolutely is no need for any of this.\n>\n> In contrast, when you want to make sure that you _actually_ incorporated\n> the revision that is currently the remote tip, e.g. via `git pull\n> --rebase` with a possible additional rebase on top that makes this _not_ a\n> fast-forward, you totally have to force the push, otherwise it won't work.\n\nMaybe `--force-if-incorporated`? Originally, I had in mind to call it\n`--safe-force`, but that might be too vague.\n\nBTW I think the patch needs to cover a bit more, still: after I run `git\npull --rebase`, the local branch will never have been at the same revision\nas the fetched one: `git rebase` moves to an unnamed branch before\nreplaying the patches. So I think we need to see whether the remote tip\nwas _reachable_ from (not necessarily identical to) any of the reflog's\nrevisions.\n\nCiao,\nDscho\n"},{"id":"405378","messageId":"xmqqsgbpr8oc.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009101220400.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-10T14:46:43Z","receivedAt":"2020-09-10T21:00:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n>> In contrast, when you want to make sure that you _actually_ incorporated\n>> the revision that is currently the remote tip, e.g. via `git pull\n>> --rebase` with a possible additional rebase on top that makes this _not_ a\n>> fast-forward, you totally have to force the push, otherwise it won't work.\n>\n> Maybe `--force-if-incorporated`? Originally, I had in mind to call it\n> `--safe-force`, but that might be too vague.\n\nYup.  \"safe force\" indeed feels like a misnomer.  The assumption of\nsafety relies heavily on the workflow.\n\nI might even say --force-if-merged even if the way the to-be-lost\nchanges have become part of what you are pushing out is not\ntechnically a merge, but there may be shorter and sweeter way to\nexpress it than 'merge' and 'incorporate'.\n"},{"id":"405379","messageId":"20200910144456.GA25125@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009101220400.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-10T14:44:56Z","receivedAt":"2020-09-10T21:01:56Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hello,\n\nOn 09/10/2020 12:22, Johannes Schindelin wrote:\n> Hi\n>> [...] \n> Maybe `--force-if-incorporated`? Originally, I had in mind to call it\n> `--safe-force`, but that might be too vague.\n\nThat's nice. I haven't been able to come up with a good name for this\noption. So far, I have: `--check-updated-remote-refs` which is really\nlong and is probably confusing.\n \n> BTW I think the patch needs to cover a bit more, still: after I run `git\n> pull --rebase`, the local branch will never have been at the same revision\n> as the fetched one: `git rebase` moves to an unnamed branch before\n> replaying the patches. So I think we need to see whether the remote tip\n> was _reachable_ from (not necessarily identical to) any of the reflog's\n> revisions.\n\nGood catch. Would adding in_merge_bases() along with checking if OIDs\nare equal for each reflog entry in oid_in_reflog_ent() address the\nproblem? That way, we would check if remote ref is reachable from\none of the entries?\n\nThanks.\n\n-- >8 --\n+ static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,\n+\t\t\t     const char *ident, timestamp_t timestamp, int tz,\n+\t\t\t     const char *message, void *cb_data)\n+ {\n+\tstruct object_id *remote_oid = cb_data;\n+\tstruct commit *a = lookup_commit_reference(the_repository, noid);\n+\tstruct commit *b = lookup_commit_reference(the_repository, remote_oid);\n+\treturn oideq(noid, remote_oid) || in_merge_bases(b, a);\n+ }\n\n"},{"id":"405471","messageId":"20200912150459.8282-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200904185147.77439-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v2 0/2] push: make \"--force-with-lease\" safer","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-12T15:04:57Z","receivedAt":"2020-09-12T15:05:28Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"The `--force-with-lease[=<refname>[:<expect]]` option in `git-push`\nmakes sure that refs on remote aren't clobbered by unexpected changes\nwhen the \"<refname>\" and \"<expect>\" ref values are explicitly specified.\n\nFor setups where the remote-tracking refs are implicitly updated by\ntools that run in the background, not specifying an \"<expect>\" value\nfor force updates may result in loss of remote updates.\n\nLet's say, we have a local branch that is based on a remote ref\nthat may be updated implicitly with a background fetch. If we decide\nto rewrite changes on the remote, and base our local branch on the\ncurrent tip of the remote-tracking ref. If the remote ref was updated\nby a push from another user, and it was fetched -- right after the\ncheckout, during rewrite or before push -- in the background, and if\nwe decide to force update our rewritten local changes on the remote\nwith `--force-with-lease[=<refname>]` (i.e, without specifying an\n\"<expect>\" value) to `git-push`, the remote changes pushed from\nanother user during our rewrite may be lost.\n\nThe new option `--force-if-includes` will allow forcing an update only\nif the tip of the remote-tracking ref has been integrated locally.\nUsing this along with `--force-with-lease`, during the time of push\ncan help preventing unintended remote overwrites.\n\nSrinidhi Kaushik (2):\n  push: add \"--[no-]force-if-includes\"\n  push: enable \"forceIfIncludesWithLease\" by default\n\n Documentation/config/advice.txt   |   4 +\n Documentation/config/feature.txt  |   6 ++\n Documentation/config/push.txt     |   8 ++\n Documentation/git-push.txt        |  22 +++++\n advice.c                          |   3 +\n advice.h                          |   2 +\n builtin/push.c                    |  38 +++++++-\n builtin/send-pack.c               |  13 ++-\n remote.c                          | 129 ++++++++++++++++++++++++---\n remote.h                          |  14 ++-\n send-pack.c                       |   1 +\n t/t5533-push-cas.sh               |  53 ++++++++++++\n t/t5549-push-force-if-includes.sh | 139 ++++++++++++++++++++++++++++++\n transport-helper.c                |   5 ++\n transport.c                       |  24 +++++-\n transport.h                       |  12 +--\n 16 files changed, 451 insertions(+), 22 deletions(-)\n create mode 100755 t/t5549-push-force-if-includes.sh\n\nbase-commit: 54e85e7af1ac9e9a92888060d6811ae767fea1bc\n--\n2.28.0\n"},{"id":"405472","messageId":"20200912150459.8282-2-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200912150459.8282-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v2 1/2] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-12T15:04:58Z","receivedAt":"2020-09-12T15:05:49Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new option: `--force-if-includes` to `git-push` where forced\nupdates are allowed only if the tip of the remote-tracking ref has\nbeen integrated locally, by verifying if the tip of the remote-tracking\nref on which a local branch has based on (for a rewrite), is reachable\nfrom at least one of the `reflog` entries of the local branch about\nto be updated by force on the remote.\n\nThis option can also be used with `--force-with-lease` in setups\nwhere the remote-tracking refs of the repository are implicitly\nupdated in the background.\n\nIf a local branch is based on a remote ref for a rewrite, and if that\nremote-tracking ref is updated by a push from another repository after\nit has been checked out locally, force updating that branch to remote\nwith `--force-with-lease[=<refname>[:expect]]` without specifying the\n\"<refname>\" or \"<expect>\" values, can cause the update that happened\nin-between the checkout and forced push to be lost.\n\nSpecifying `--force-with-includes` with `--force-with-lease` as an\nancillary argument at the time of push, ensures that any new updates\nto the remote-tracking refs are integrated locally before allowing a\nforced update. This behavior can enabled by default if the configuration\noption `push.forceIfIncludesWithLease` is set to `true`.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/advice.txt   |   4 +\n Documentation/config/push.txt     |   8 ++\n Documentation/git-push.txt        |  22 +++++\n advice.c                          |   3 +\n advice.h                          |   2 +\n builtin/push.c                    |  27 +++++-\n builtin/send-pack.c               |  13 ++-\n remote.c                          | 129 ++++++++++++++++++++++++---\n remote.h                          |  14 ++-\n send-pack.c                       |   1 +\n t/t5533-push-cas.sh               |  53 ++++++++++++\n t/t5549-push-force-if-includes.sh | 139 ++++++++++++++++++++++++++++++\n transport-helper.c                |   5 ++\n transport.c                       |  24 +++++-\n transport.h                       |  12 +--\n 15 files changed, 434 insertions(+), 22 deletions(-)\n create mode 100755 t/t5549-push-force-if-includes.sh\n\ndiff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\nindex bdd37c3eaa..f48aed49e8 100644\n--- a/Documentation/config/advice.txt\n+++ b/Documentation/config/advice.txt\n@@ -41,6 +41,10 @@ advice.*::\n \t\twe can still suggest that the user push to either\n \t\trefs/heads/* or refs/tags/* based on the type of the\n \t\tsource object.\n+\tpushRefNeedsUpdate::\n+\t\tShown when linkgit:git-push[1] rejects a forced update of\n+\t\ta branch when its remote-tracking ref has updates that we\n+\t\tdo not have locally.\n \tstatusAheadBehind::\n \t\tShown when linkgit:git-status[1] computes the ahead/behind\n \t\tcounts for a local ref compared to its remote tracking ref,\ndiff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\nindex f5e5b38c68..1b4948faa0 100644\n--- a/Documentation/config/push.txt\n+++ b/Documentation/config/push.txt\n@@ -114,3 +114,11 @@ push.recurseSubmodules::\n \tspecifying '--recurse-submodules=check|on-demand|no'.\n \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n \tset (in which case a 'true' value means 'on-demand').\n+\n+push.forceIfIncludesWithLease::\n+\tIf set to `true`, adds `--force-if-includes` as an ancillary argument\n+\tto `--force-with-lease[=<refname>[:<expect>]]`, when \"<refname>\" or\n+\t\"<expect>\" values are unspecified at the time of push.\n++\n+Note: Specifying `--no-force-if-includes` to linkgit:git-push[1] as an\n+argument during the time of push does _not_ override this configuration.\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..199c601bd4 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -320,6 +320,15 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, specifying `--force-if-includes` an an ancillary option along\n+with `--force-with-lease[=<refname>[:expect]]` (when \"<refname>\" or \"<expect>\"\n+values are unspecified) at the time of `push` will verify if updates from\n+the remote-tracking refs that may have been implicitly updated in the\n+background (via linkgit:git-fetch[1], and the like) are integrated locally\n+before allowing a forced update. This behavior can be enabled by default if\n+the configuration option `push.forceIfIncludesWithLease` to `true`\n+in linkgit:git-config[1].\n \n -f::\n --force::\n@@ -341,6 +350,19 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n origin +master` to force a push to the `master` branch). See the\n `<refspec>...` section above for details.\n \n+--[no-]force-if-includes::\n+\tForce an update only if the tip of the remote-tracking ref\n+\thas been integrated locally.\n++\n+This option verifies if the tip of the remote-tracking ref on which\n+a local branch has based on (for a rewrite), is reachable from at\n+least one of the `reflog` entries of the local branch about to be\n+updated by force on the remote. The check ensures that any updates\n+from the remote have been incorporated locally by rejecting a push\n+if that is not the case.\n++\n+Specifying `--no-force-if-includes` disables this behavior.\n+\n --repo=<repository>::\n \tThis option is equivalent to the <repository> argument. If both\n \tare specified, the command-line argument takes precedence.\ndiff --git a/advice.c b/advice.c\nindex f0a3d32d20..164742305f 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;\n int advice_push_fetch_first = 1;\n int advice_push_needs_force = 1;\n int advice_push_unqualified_ref_name = 1;\n+int advice_push_ref_needs_update = 1;\n int advice_status_hints = 1;\n int advice_status_u_option = 1;\n int advice_status_ahead_behind_warning = 1;\n@@ -72,6 +73,7 @@ static struct {\n \t{ \"pushFetchFirst\", &advice_push_fetch_first },\n \t{ \"pushNeedsForce\", &advice_push_needs_force },\n \t{ \"pushUnqualifiedRefName\", &advice_push_unqualified_ref_name },\n+\t{ \"pushRefNeedsUpdate\", &advice_push_ref_needs_update },\n \t{ \"statusHints\", &advice_status_hints },\n \t{ \"statusUoption\", &advice_status_u_option },\n \t{ \"statusAheadBehindWarning\", &advice_status_ahead_behind_warning },\n@@ -116,6 +118,7 @@ static struct {\n \t[ADVICE_PUSH_ALREADY_EXISTS]\t\t\t= { \"pushAlreadyExists\", 1 },\n \t[ADVICE_PUSH_FETCH_FIRST]\t\t\t= { \"pushFetchFirst\", 1 },\n \t[ADVICE_PUSH_NEEDS_FORCE]\t\t\t= { \"pushNeedsForce\", 1 },\n+\t[ADVICE_PUSH_REF_NEEDS_UPDATE]\t\t\t= { \"pushRefNeedsUpdate\", 1 },\n \n \t/* make this an alias for backward compatibility */\n \t[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]\t\t= { \"pushNonFastForward\", 1 },\ndiff --git a/advice.h b/advice.h\nindex 16f2c11642..bc2432980a 100644\n--- a/advice.h\n+++ b/advice.h\n@@ -11,6 +11,7 @@ extern int advice_push_already_exists;\n extern int advice_push_fetch_first;\n extern int advice_push_needs_force;\n extern int advice_push_unqualified_ref_name;\n+extern int advice_push_ref_needs_update;\n extern int advice_status_hints;\n extern int advice_status_u_option;\n extern int advice_status_ahead_behind_warning;\n@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;\n \tADVICE_PUSH_UNQUALIFIED_REF_NAME,\n \tADVICE_PUSH_UPDATE_REJECTED_ALIAS,\n \tADVICE_PUSH_UPDATE_REJECTED,\n+\tADVICE_PUSH_REF_NEEDS_UPDATE,\n \tADVICE_RESET_QUIET_WARNING,\n \tADVICE_RESOLVE_CONFLICT,\n \tADVICE_RM_HINTS,\ndiff --git a/builtin/push.c b/builtin/push.c\nindex bc94078e72..7fb07eb38e 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -300,6 +300,12 @@ static const char message_advice_ref_needs_force[] =\n \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n \t   \"without using the '--force' option.\\n\");\n \n+static const char message_advice_ref_needs_update[] =\n+\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n+\t   \"branch has been updated since the last checkout. You may want\\n\"\n+\t   \"to integrate those changes locally (e.g., 'git rebase ...')\\n\"\n+\t   \"before forcing an update.\\n\");\n+\n static void advise_pull_before_push(void)\n {\n \tif (!advice_push_non_ff_current || !advice_push_update_rejected)\n@@ -335,6 +341,13 @@ static void advise_ref_needs_force(void)\n \tadvise(_(message_advice_ref_needs_force));\n }\n \n+static void advise_ref_needs_update(void)\n+{\n+\tif (!advice_push_ref_needs_update || !advice_push_update_rejected)\n+\t\treturn;\n+\tadvise(_(message_advice_ref_needs_update));\n+}\n+\n static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\t\t     int flags)\n {\n@@ -384,8 +397,9 @@ static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\tadvise_ref_fetch_first();\n \t} else if (reject_reasons & REJECT_NEEDS_FORCE) {\n \t\tadvise_ref_needs_force();\n+\t} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {\n+\t\tadvise_ref_needs_update();\n \t}\n-\n \treturn 1;\n }\n \n@@ -520,8 +534,14 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\tif (!v)\n \t\t\treturn config_error_nonbool(k);\n \t\treturn color_parse(v, push_colors[slot]);\n-\t}\n+\t} else if (!strcmp(k, \"push.forceifincludeswithlease\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\t*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\telse\n+\t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n \n+\t\treturn 0;\n+\t}\n \treturn git_default_config(k, v, NULL);\n }\n \n@@ -551,6 +571,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t\t       N_(\"require old value of ref to be at this value\"),\n \t\t\t       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),\n+\t\tOPT_BIT(0, \"force-if-includes\", &flags,\n+\t\t\tN_(\"require remote updates to be integrated locally\"),\n+\t\t\tTRANSPORT_PUSH_FORCE_IF_INCLUDES),\n \t\tOPT_CALLBACK(0, \"recurse-submodules\", &recurse_submodules, \"(check|on-demand|no)\",\n \t\t\t     N_(\"control recursive pushing of submodules\"), option_parse_recurse_submodules),\n \t\tOPT_BOOL_F( 0 , \"thin\", &thin, N_(\"use thin pack\"), PARSE_OPT_NOCOMPLETE),\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 2b9610f121..ee5d7af00c 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n \n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"remote updated since checkout\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\n@@ -155,6 +160,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n \tstruct packet_reader reader;\n+\tunsigned int force_if_inc;\n \n \tstruct option options[] = {\n \t\tOPT__VERBOSITY(&verbose),\n@@ -179,6 +185,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t  N_(\"require old value of ref to be at this value\"),\n \t\t  PARSE_OPT_OPTARG, parseopt_push_cas_option),\n+\t\tOPT_BOOL(0, \"force-if-includes\", &force_if_inc,\n+\t\t\t N_(\"require remote updates to be integrated locally\")),\n \t\tOPT_END()\n \t};\n \n@@ -278,7 +286,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\treturn -1;\n \n \tif (!is_empty_cas(&cas))\n-\t\tapply_push_cas(&cas, remote, remote_refs);\n+\t\tapply_push_cas(&cas, remote, remote_refs, force_if_inc);\n+\n+\tif (is_empty_cas(&cas) && force_if_inc)\n+\t\trun_local_reflog_check(remote_refs);\n \n \tset_ref_status_for_push(remote_refs, args.send_mirror,\n \t\targs.force_update);\ndiff --git a/remote.c b/remote.c\nindex 420150837b..71af6d3073 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -1484,6 +1484,36 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n \n+\t\t/*\n+\t\t * If the tip of the remote-tracking ref is unreachable\n+\t\t * from any reflog entry of its local ref indicating a\n+\t\t * possible update since checkout; reject the push.\n+\t\t *\n+\t\t * There is no need to check for reachability, if the\n+\t\t * ref is marked for deletion.\n+\t\t */\n+\t\tif (ref->if_includes && !ref->deletion) {\n+\t\t\t/*\n+\t\t\t * If `force_ref_update' was previously set by\n+\t\t\t * \"compare-and-swap\", and we have to run this\n+\t\t\t * check, reset it back to the original value\n+\t\t\t * and update it depending on the status of this\n+\t\t\t * check.\n+\t\t\t */\n+\t\t\tforce_ref_update = ref->force || force_update;\n+\n+\t\t\tif (ref->unreachable)\n+\t\t\t\treject_reason =\n+\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\telse\n+\t\t\t\t/*\n+\t\t\t\t * If updates from the remote-tracking ref\n+\t\t\t\t * have been integrated locally; force the\n+\t\t\t\t * update.\n+\t\t\t\t */\n+\t\t\t\tforce_ref_update = 1;\n+\t\t}\n+\n \t\t/*\n \t\t * If the update isn't already rejected then check\n \t\t * the usual \"must fast-forward\" rules.\n@@ -2272,11 +2302,76 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \treturn 0;\n }\n \n+static int ref_reachable(struct object_id *o_oid, struct object_id *n_oid,\n+\t\t\t const char *ident, timestamp_t timestamp, int tz,\n+\t\t\t const char *message, void *cb_data)\n+{\n+\tint ret = 0;\n+\tstruct object_id *r_oid = cb_data;\n+\n+\tret = oideq(n_oid, r_oid);\n+\tif (!ret) {\n+\t\tstruct commit *loc = lookup_commit_reference(the_repository,\n+\t\t\t\t\t\t\t     n_oid);\n+\t\tstruct commit *rem = lookup_commit_reference(the_repository,\n+\t\t\t\t\t\t\t     r_oid);\n+\t\tret = (loc && rem) ? in_merge_bases(rem, loc) : 0;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Iterate through the reflog of a local branch and check\n+ * if the tip of the remote-tracking branch is reachable\n+ * from one of the entries.\n+ */\n+static int ref_reachable_from_reflog(const struct object_id *r_oid,\n+\t\t\t\t     const struct object_id *l_oid,\n+\t\t\t\t     const char *local_ref_name)\n+{\n+\tint ret = 0;\n+\tstruct commit *r_commit, *l_commit;\n+\n+\tl_commit = lookup_commit_reference(the_repository, l_oid);\n+\tr_commit = lookup_commit_reference(the_repository, r_oid);\n+\n+\t/*\n+\t * If the remote-tracking ref is an ancestor of the local\n+\t * ref (a merge, for instance) there is no need to iterate\n+\t * through the reflog entries to ensure reachability; it\n+\t * can be skipped to return early instead.\n+\t */\n+\tret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;\n+\tif (!ret)\n+\t\tret = for_each_reflog_ent_reverse(local_ref_name, ref_reachable,\n+\t\t\t\t\t\t  (struct object_id *)r_oid);\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Check for reachability of a remote-tracking ref in its local\n+ * ref's reflog entries.\n+ */\n+void check_reflog_for_ref(struct ref *r_ref)\n+{\n+\tstruct ref *l_ref = get_local_ref(r_ref->name);\n+\tstruct object_id r_oid;\n+\n+\tr_ref->if_includes = 1;\n+\tif (l_ref && !read_ref(l_ref->name, &r_oid))\n+\t\tr_ref->unreachable = !ref_reachable_from_reflog(&r_ref->old_oid,\n+\t\t\t\t\t\t\t\t&r_oid,\n+\t\t\t\t\t\t\t\tl_ref->name);\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n-\t\t      struct ref *ref)\n+\t\t      struct ref *ref,\n+\t\t      int if_includes)\n {\n-\tint i;\n+\tint i, check_reflog = 0;\n \n \t/* Find an explicit --<option>=<name>[:<value>] entry */\n \tfor (i = 0; i < cas->nr; i++) {\n@@ -2288,23 +2383,37 @@ static void apply_cas(struct push_cas_option *cas,\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n \t\t\toidclr(&ref->old_oid_expect);\n-\t\treturn;\n+\t\telse\n+\t\t\tcheck_reflog = 1;\n+\t\tbreak;\n \t}\n \n \t/* Are we using \"--<option>\" to cover all? */\n-\tif (!cas->use_tracking_for_rest)\n-\t\treturn;\n+\tif (cas->use_tracking_for_rest) {\n+\t\tref->expect_old_sha1 = 1;\n+\t\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\t\t\toidclr(&ref->old_oid_expect);\n+\t\telse\n+\t\t\tcheck_reflog = 1;\n+\t}\n \n-\tref->expect_old_sha1 = 1;\n-\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n-\t\toidclr(&ref->old_oid_expect);\n+\tif (if_includes && check_reflog)\n+\t\tcheck_reflog_for_ref(ref);\n }\n \n void apply_push_cas(struct push_cas_option *cas,\n \t\t    struct remote *remote,\n-\t\t    struct ref *remote_refs)\n+\t\t    struct ref *remote_refs,\n+\t\t    int if_includes)\n+{\n+\tstruct ref *ref;\n+\tfor (ref = remote_refs; ref; ref = ref->next)\n+\t\tapply_cas(cas, remote, ref, if_includes);\n+}\n+\n+void run_local_reflog_check(struct ref *remote_refs)\n {\n \tstruct ref *ref;\n \tfor (ref = remote_refs; ref; ref = ref->next)\n-\t\tapply_cas(cas, remote, ref);\n+\t\tcheck_reflog_for_ref(ref);\n }\ndiff --git a/remote.h b/remote.h\nindex 5e3ea5a26d..c4a648ed6d 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -104,7 +104,9 @@ struct ref {\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\tif_includes:1, /* For \"--force-if-includes\". */\n+\t\tunreachable:1; /* Used by \"if_includes\". */\n \n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -134,6 +136,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n \t\tREF_STATUS_REJECT_SHALLOW,\n+\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT,\n@@ -344,6 +347,13 @@ struct push_cas_option {\n int parseopt_push_cas_option(const struct option *, const char *arg, int unset);\n \n int is_empty_cas(const struct push_cas_option *);\n-void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);\n+void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *, int);\n+\n+/*\n+ * Check if the remote-tracking ref was updated (since checkout)\n+ * implicitly in the background and verify that changes from the\n+ * updated tip have been integrated locally, before pushing.\n+ */\n+void run_local_reflog_check(struct ref*);\n \n #endif\ndiff --git a/send-pack.c b/send-pack.c\nindex 632f1580ca..956306e8e8 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..dee422acaa 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -256,4 +256,57 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n \n+test_expect_success 'background updates of REMOTE can be mitigated with \"--force-if-includes\"' '\n+\trm -rf src dst &&\n+\tgit init --bare src.bare &&\n+\ttest_when_finished \"rm -rf src.bare\" &&\n+\tgit clone --no-local src.bare dst &&\n+\ttest_when_finished \"rm -rf dst\" &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit G &&\n+\t\tgit push origin master:master\n+\t) &&\n+\tgit clone --no-local src.bare dst2 &&\n+\ttest_when_finished \"rm -rf dst2\" &&\n+\t(\n+\t\tcd dst2 &&\n+\t\ttest_commit H &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit I &&\n+\t\tgit fetch origin &&\n+\t\ttest_must_fail git push --force-with-lease --force-if-includes origin\n+\t)\n+'\n+\n+test_expect_success 'background updates of REMOTE can be mitigated with \"push.forceIfIncludesWithLease\"' '\n+\trm -rf src dst &&\n+\tgit init --bare src.bare &&\n+\ttest_when_finished \"rm -rf src.bare\" &&\n+\tgit clone --no-local src.bare dst &&\n+\ttest_when_finished \"rm -rf dst\" &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit G &&\n+\t\tgit push origin master:master\n+\t) &&\n+\tgit clone --no-local src.bare dst2 &&\n+\ttest_when_finished \"rm -rf dst2\" &&\n+\t(\n+\t\tcd dst2 &&\n+\t\ttest_commit H &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit I &&\n+\t\tgit fetch origin &&\n+\t\tgit config --local push.forceIfIncludesWithLease true &&\n+\t\ttest_must_fail git push --force-with-lease origin\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t5549-push-force-if-includes.sh b/t/t5549-push-force-if-includes.sh\nnew file mode 100755\nindex 0000000000..2e2ca79266\n--- /dev/null\n+++ b/t/t5549-push-force-if-includes.sh\n@@ -0,0 +1,139 @@\n+test_description='Test push \"--force-if-includes\" forced update safety.'\n+\n+. ./test-lib.sh\n+\n+setup_src_dup_dst () {\n+\trm -fr src dup dst &&\n+\tgit init --bare dst &&\n+\tgit clone --no-local dst src &&\n+\tgit clone --no-local dst dup\n+\t(\n+\t\tcd src &&\n+\t\ttest_commit A &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit fetch &&\n+\t\tgit merge origin/master &&\n+\t\ttest_commit B &&\n+\t\tgit switch -c branch master~1 &&\n+\t\ttest_commit C &&\n+\t\ttest_commit D &&\n+\t\tgit push --all\n+\t) &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch master &&\n+\t\tgit fetch --all &&\n+\t\tgit branch branch --track origin/branch &&\n+\t\tgit rebase origin/master\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit switch master &&\n+\t\ttest_commit E &&\n+\t\tgit switch branch &&\n+\t\ttest_commit F &&\n+\t\tgit push origin --all\n+\t)\n+}\n+\n+test_expect_success 'reject push if remote changes are not integrated locally (protected, all refs)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit switch master &&\n+\t\ttest_commit I &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes --all\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success 'reject push if remote changes are not integrated locally (protected, specific ref)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit switch master &&\n+\t\ttest_commit I &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes origin master\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\ttest_cmp expect.master actual.master\n+'\n+\n+test_expect_success 'allow force push if \"--force\" is specified' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit switch master &&\n+\t\ttest_commit I &&\n+\t\tgit fetch --all &&\n+\t\tgit push --force --force-if-includes origin --all 2>err &&\n+\t\tgrep \"forced update\" err\n+\t)\n+'\n+\n+test_expect_success 'allow force push if \"--delete\" is specified' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit switch master &&\n+\t\ttest_commit I &&\n+\t\tgit fetch --all &&\n+\t\tgit push --delete --force-if-includes origin branch 2>err &&\n+\t\tgrep \"deleted\" err\n+\t)\n+'\n+\n+test_expect_success 'honor specified refspecs (force)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit switch master &&\n+\t\ttest_commit I &&\n+\t\tgit fetch --all &&\n+\t\tgit push --force-if-includes origin +branch 2>err &&\n+\t\tgrep \"forced update\" err\n+\t)\n+'\n+\n+test_expect_success 'honor specified refspecs (delete)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit switch master &&\n+\t\ttest_commit I &&\n+\t\tgit fetch --all &&\n+\t\tgit push --force-if-includes origin :branch 2>err &&\n+\t\tgrep \"deleted\" err\n+\t)\n+'\n+\n+test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c52c99d829..6fd74c2c65 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"remote updated since checkout\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\n \t\t\t\tstring_list_clear(&cas_options, 0);\ndiff --git a/transport.c b/transport.c\nindex 43e24bf1e5..3f9e697cd1 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"stale info\", porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"remote updated since checkout\",\n+\t\t\t\t porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -659,6 +664,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t\t*reject_reasons |= REJECT_FETCH_FIRST;\n \t\t} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {\n \t\t\t*reject_reasons |= REJECT_NEEDS_FORCE;\n+\t\t} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {\n+\t\t\t*reject_reasons |= REJECT_REF_NEEDS_UPDATE;\n \t\t}\n \t}\n \tfree(head);\n@@ -1101,6 +1108,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n \n \t\tstrbuf_reset(&buf);\n@@ -1151,6 +1159,8 @@ int transport_push(struct repository *r,\n \t\tint pretend = flags & TRANSPORT_PUSH_DRY_RUN;\n \t\tint push_ret, ret, err;\n \t\tstruct strvec ref_prefixes = STRVEC_INIT;\n+\t\tint empty_cas = 1;\n+\t\tint if_includes = flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n \n \t\tif (check_push_refs(local_refs, rs) < 0)\n \t\t\treturn -1;\n@@ -1178,9 +1188,19 @@ int transport_push(struct repository *r,\n \n \t\tif (transport->smart_options &&\n \t\t    transport->smart_options->cas &&\n-\t\t    !is_empty_cas(transport->smart_options->cas))\n+\t\t    !is_empty_cas(transport->smart_options->cas)) {\n+\t\t\tempty_cas = 0;\n \t\t\tapply_push_cas(transport->smart_options->cas,\n-\t\t\t\t       transport->remote, remote_refs);\n+\t\t\t\t       transport->remote, remote_refs,\n+\t\t\t\t       if_includes);\n+\t\t}\n+\n+\t\t/*\n+\t\t * Run the check for all refs when \"--force-if-includes\"\n+\t\t * is specified without \"--force-with-lease\".\n+\t\t */\n+\t\tif (empty_cas && if_includes)\n+\t\t\trun_local_reflog_check(remote_refs);\n \n \t\tset_ref_status_for_push(remote_refs,\n \t\t\tflags & TRANSPORT_PUSH_MIRROR,\ndiff --git a/transport.h b/transport.h\nindex ca409ea1e4..729bb6979e 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -136,6 +136,7 @@ struct transport {\n #define TRANSPORT_PUSH_ATOMIC\t\t\t(1<<13)\n #define TRANSPORT_PUSH_OPTIONS\t\t\t(1<<14)\n #define TRANSPORT_RECURSE_SUBMODULES_ONLY\t(1<<15)\n+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES\t(1<<16)\n \n int transport_summary_width(const struct ref *refs);\n \n@@ -217,11 +218,12 @@ int transport_set_option(struct transport *transport, const char *name,\n void transport_set_verbosity(struct transport *transport, int verbosity,\n \tint force_progress);\n \n-#define REJECT_NON_FF_HEAD     0x01\n-#define REJECT_NON_FF_OTHER    0x02\n-#define REJECT_ALREADY_EXISTS  0x04\n-#define REJECT_FETCH_FIRST     0x08\n-#define REJECT_NEEDS_FORCE     0x10\n+#define REJECT_NON_FF_HEAD      0x01\n+#define REJECT_NON_FF_OTHER     0x02\n+#define REJECT_ALREADY_EXISTS   0x04\n+#define REJECT_FETCH_FIRST      0x08\n+#define REJECT_NEEDS_FORCE      0x10\n+#define REJECT_REF_NEEDS_UPDATE 0x20\n \n int transport_push(struct repository *repo,\n \t\t   struct transport *connection,\n-- \n2.28.0\n\n"},{"id":"405473","messageId":"20200912150459.8282-3-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200912150459.8282-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v2 2/2] push: enable \"forceIfIncludesWithLease\" by default","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-12T15:04:59Z","receivedAt":"2020-09-12T15:05:53Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Set `push.forceIfIncludesWithLease` to `true` if experimental\nfeatures are enabled.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/feature.txt |  6 ++++++\n builtin/push.c                   | 11 +++++++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/Documentation/config/feature.txt b/Documentation/config/feature.txt\nindex c0cbf2bb1c..4a8a386132 100644\n--- a/Documentation/config/feature.txt\n+++ b/Documentation/config/feature.txt\n@@ -18,6 +18,12 @@ skipping more commits at a time, reducing the number of round trips.\n * `protocol.version=2` speeds up fetches from repositories with many refs by\n allowing the client to specify which refs to list before the server lists\n them.\n++\n+* `push.forceIfIncludesWithLease=true` adds `--force-if-includes` as an\n+additional option along with `--force-with-lease[=<refname>[:<expect>]`\n+when \"<refname>\" or \"<expect>\" valus are unspecified for linkgit:git-push[1].\n+Allows forced updates only if the local branch has incorporated changes\n+from the remote-tracking ref.\n \n feature.manyFiles::\n \tEnable config options that optimize for repos with many files in the\ndiff --git a/builtin/push.c b/builtin/push.c\nindex 7fb07eb38e..658694edff 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -540,6 +540,17 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n \n+\t\treturn 0;\n+\t} else if (!strcmp(k, \"feature.experimental\")) {\n+\t\t/*\n+\t\t * Set `push.forceIfIncludesWithLease` to true,\n+\t\t * if experimental features are enabled.\n+\t\t */\n+\t\tif (git_config_bool(k, v))\n+\t\t\t*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\telse\n+\t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\n \t\treturn 0;\n \t}\n \treturn git_default_config(k, v, NULL);\n-- \n2.28.0\n\n"},{"id":"405474","messageId":"xmqqa6xuoo8h.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200912150459.8282-1-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v2 0/2] push: make \"--force-with-lease\" safer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-12T18:15:42Z","receivedAt":"2020-09-12T18:15:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> The `--force-with-lease[=<refname>[:<expect]]` option in `git-push`\n> makes sure that refs on remote aren't clobbered by unexpected changes\n> when the \"<refname>\" and \"<expect>\" ref values are explicitly specified.\n\nIf you did a feature with different semantics to satisfy Dscho's\nneed, then this is no longer \"make force-with-lease safer\", I would\nthink.  Hopefully it is just the cover letter.\n\n> The new option `--force-if-includes` will allow forcing an update only\n> if the tip of the remote-tracking ref has been integrated locally.\n> Using this along with `--force-with-lease`, during the time of push\n> can help preventing unintended remote overwrites.\n\n\"if-includes\" sounds quite sensible.  I think you want to lose the\nword \"lease\" from the configuration variable name.  I do not think\nit should be on by default, though.\n\nThanks.\n"},{"id":"405475","messageId":"xmqq5z8ioo15.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200912150459.8282-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v2 1/2] push: add \"--[no-]force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-12T18:20:06Z","receivedAt":"2020-09-12T18:20:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Add a new option: `--force-if-includes` to `git-push` where forced\n> updates are allowed only if the tip of the remote-tracking ref has\n> been integrated locally, by verifying if the tip of the remote-tracking\n> ref on which a local branch has based on (for a rewrite), is reachable\n> from at least one of the `reflog` entries of the local branch about\n> to be updated by force on the remote.\n>\n> This option can also be used with `--force-with-lease` in setups\n> where the remote-tracking refs of the repository are implicitly\n> updated in the background.\n>\n> If a local branch is based on a remote ref for a rewrite, and if that\n> remote-tracking ref is updated by a push from another repository after\n> it has been checked out locally, force updating that branch to remote\n> with `--force-with-lease[=<refname>[:expect]]` without specifying the\n> \"<refname>\" or \"<expect>\" values, can cause the update that happened\n> in-between the checkout and forced push to be lost.\n>\n> Specifying `--force-with-includes` with `--force-with-lease` as an\n> ancillary argument at the time of push, ensures that any new updates\n> to the remote-tracking refs are integrated locally before allowing a\n> forced update. This behavior can enabled by default if the configuration\n> option `push.forceIfIncludesWithLease` is set to `true`.\n\nThis step seems to do too many things at once.  Split into logical\nprogression of improvements, or nobody can sensibly review it, I am\nafraid.\n\n> diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\n> index f5e5b38c68..1b4948faa0 100644\n> --- a/Documentation/config/push.txt\n> +++ b/Documentation/config/push.txt\n> @@ -114,3 +114,11 @@ push.recurseSubmodules::\n>  \tspecifying '--recurse-submodules=check|on-demand|no'.\n>  \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n>  \tset (in which case a 'true' value means 'on-demand').\n> +\n> +push.forceIfIncludesWithLease::\n> +\tIf set to `true`, adds `--force-if-includes` as an ancillary argument\n> +\tto `--force-with-lease[=<refname>[:<expect>]]`, when \"<refname>\" or\n> +\t\"<expect>\" values are unspecified at the time of push.\n> ++\n> +Note: Specifying `--no-force-if-includes` to linkgit:git-push[1] as an\n> +argument during the time of push does _not_ override this configuration.\n\nI do not see why you still want to link these two unrelated\nfeatures.  I may want to do forced push with lease when I know I am\nrewinding, and I may want to do a if-included force when needed, but\nI do not see why I want to ask the former and implicitly see it also\ntrigger the latter.\n\nI haven't seen the details, but if we severe the (unnecessary?)\nentanglement of these two features, perhaps this patch will become a\nlot smaller and more focused?  I dunno.\n"},{"id":"405476","messageId":"xmqq1rj6onxu.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200912150459.8282-3-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v2 2/2] push: enable \"forceIfIncludesWithLease\" by default","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-12T18:22:05Z","receivedAt":"2020-09-12T18:22:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Set `push.forceIfIncludesWithLease` to `true` if experimental\n> features are enabled.\n\nWhy?  \n\nWe enable something to the guinea-pig audience if we consider that\nit should become the default eventually for everybody.  I did not\nsee enough justification why this should be on for everybody\nsomeday.\n"},{"id":"405483","messageId":"20200912210307.GA55022@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqa6xuoo8h.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 0/2] push: make \"--force-with-lease\" safer","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-12T21:03:07Z","receivedAt":"2020-09-12T21:03:18Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Junio,\n\nOn 09/12/2020 11:15, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> > The `--force-with-lease[=<refname>[:<expect]]` option in `git-push`\n> > makes sure that refs on remote aren't clobbered by unexpected changes\n> > when the \"<refname>\" and \"<expect>\" ref values are explicitly specified.\n> \n> If you did a feature with different semantics to satisfy Dscho's\n> need, then this is no longer \"make force-with-lease safer\", I would\n> think.  Hopefully it is just the cover letter.\n\nYes, this patch is about the new option, but I thought of keeping the\noriginal reason for introducing it  in the cover letter for context.\nI will add this as a note and change subjject cover letter in v3.\n \n> > The new option `--force-if-includes` will allow forcing an update only\n> > if the tip of the remote-tracking ref has been integrated locally.\n> > Using this along with `--force-with-lease`, during the time of push\n> > can help preventing unintended remote overwrites.\n> \n> \"if-includes\" sounds quite sensible.  I think you want to lose the\n> word \"lease\" from the configuration variable name.  I do not think\n> it should be on by default, though.\n\nThanks; that makes sense. I am thinking of  just adding the option\nas a command line argument without a configuration option. Will change\nthis in the next patch-set.\n\n> [...]\n\nThanks again, for reviewing this.\n-- \nSrinidhi Kaushik\n"},{"id":"405484","messageId":"20200912212524.GB55022@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqq5z8ioo15.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/2] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-12T21:25:24Z","receivedAt":"2020-09-12T21:27:34Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"On 09/12/2020 11:20, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> > Add a new option: `--force-if-includes` to `git-push` where forced\n> > updates are allowed only if the tip of the remote-tracking ref has\n> > been integrated locally, by verifying if the tip of the remote-tracking\n> > ref on which a local branch has based on (for a rewrite), is reachable\n> > from at least one of the `reflog` entries of the local branch about\n> > to be updated by force on the remote.\n> >\n> > This option can also be used with `--force-with-lease` in setups\n> > where the remote-tracking refs of the repository are implicitly\n> > updated in the background.\n> >\n> > If a local branch is based on a remote ref for a rewrite, and if that\n> > remote-tracking ref is updated by a push from another repository after\n> > it has been checked out locally, force updating that branch to remote\n> > with `--force-with-lease[=<refname>[:expect]]` without specifying the\n> > \"<refname>\" or \"<expect>\" values, can cause the update that happened\n> > in-between the checkout and forced push to be lost.\n> >\n> > Specifying `--force-with-includes` with `--force-with-lease` as an\n> > ancillary argument at the time of push, ensures that any new updates\n> > to the remote-tracking refs are integrated locally before allowing a\n> > forced update. This behavior can enabled by default if the configuration\n> > option `push.forceIfIncludesWithLease` is set to `true`.\n> \n> This step seems to do too many things at once.  Split into logical\n> progression of improvements, or nobody can sensibly review it, I am\n> afraid.\n\nOK, I will break the commit up into smaller change-sets in the next\npatch.\n \n> > diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\n> > index f5e5b38c68..1b4948faa0 100644\n> > --- a/Documentation/config/push.txt\n> > +++ b/Documentation/config/push.txt\n> > @@ -114,3 +114,11 @@ push.recurseSubmodules::\n> >  \tspecifying '--recurse-submodules=check|on-demand|no'.\n> >  \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n> >  \tset (in which case a 'true' value means 'on-demand').\n> > +\n> > +push.forceIfIncludesWithLease::\n> > +\tIf set to `true`, adds `--force-if-includes` as an ancillary argument\n> > +\tto `--force-with-lease[=<refname>[:<expect>]]`, when \"<refname>\" or\n> > +\t\"<expect>\" values are unspecified at the time of push.\n> > ++\n> > +Note: Specifying `--no-force-if-includes` to linkgit:git-push[1] as an\n> > +argument during the time of push does _not_ override this configuration.\n> \n> I do not see why you still want to link these two unrelated\n> features.  I may want to do forced push with lease when I know I am\n> rewinding, and I may want to do a if-included force when needed, but\n> I do not see why I want to ask the former and implicitly see it also\n> trigger the latter.\n\nI am actually thinking of getting rid of the configuration option\naltogether. Sadly, I realized that after I sent out the patch.\n\n> I haven't seen the details, but if we severe the (unnecessary?)\n> entanglement of these two features, perhaps this patch will become a\n> lot smaller and more focused?  I dunno.\n\nOther than the configuration setting (which will be removed), tthe only\nparts that overlap are checking to see if \"--force-with-lease\" specified\nwith `use_tracking` and `use_tracking_for_test` modes only. The new\noption does not modify the \"strict\" version of \"compare-and-swap\"\nbecause in that case, it is not needed at all.\n\nHowever, I think it can be cleaned up a bit more; will look into it.\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"405491","messageId":"20200913145413.18351-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200912150459.8282-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v3 0/7] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-13T14:54:06Z","receivedAt":"2020-09-13T14:54:36Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new option: \"--force-if-includes\" to `git-push` where forced\nupdates are allowed only if the tip of the remote-tracking ref has\nbeen integrated locally, by verifying if the tip of the remote-tracking\nref -- on which a local branch has based on -- is reachable from at\nleast one of the \"reflog\" entries of the branch about to be updated\nby force on the remote.\n\nThis option can also be used with `--force-with-lease` with setups\nwhere the remote-tracking refs of the repository are implicitly\nupdated in the background to help prevent unintended remote\noverwrites.\n\nIf a local branch is based on a remote ref for a rewrite, and if that\nremote-tracking ref is updated by a push from another repository after\nit has been checked out locally, force updating that branch to remote\nwith `--force-with-lease[=<refname>[:expect]]` without specifying the\n\"<refname>\" or \"<expect>\" values, can cause the update that happened\nin-between the checkout and forced push to be lost.\n\nChanges since v2:\n  * Removed configuration option for setting \"--force-if-includes\"\n    with \"--force-with-lease\".\n  * Broke up the patch into smaller commits.\n\nbase-commit: 54e85e7af1ac9e9a92888060d6811ae767fea1bc\n\nSrinidhi Kaushik (7):\n  remote: add reflog check for \"force-if-includes\"\n  transport: add flag for \"--[no-]force-if-includes\"\n  send-pack: check ref status for \"force-if-includes\"\n  transport-helper: update ref status for \"force-if-includes\"\n  builtin/push: add option \"--[no-]force-if-includes\"\n  doc: add reference for \"--[no-]force-if-includes\"\n  t: add tests for \"force-if-includes\"\n\n Documentation/config/advice.txt   |   9 +-\n Documentation/git-push.txt        |  19 ++++\n advice.c                          |   3 +\n advice.h                          |   2 +\n builtin/push.c                    |  20 +++-\n builtin/send-pack.c               |   5 +\n remote.c                          | 135 +++++++++++++++++++++++--\n remote.h                          |  14 ++-\n send-pack.c                       |   1 +\n t/t5533-push-cas.sh               |  26 +++++\n t/t5549-push-force-if-includes.sh | 161 ++++++++++++++++++++++++++++++\n transport-helper.c                |   5 +\n transport.c                       |  18 +++-\n transport.h                       |  12 ++-\n 14 files changed, 411 insertions(+), 19 deletions(-)\n create mode 100755 t/t5549-push-force-if-includes.sh\n\n--\n2.28.0\n"},{"id":"405492","messageId":"20200913145413.18351-2-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200913145413.18351-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v3 1/7] remote: add reflog check for \"force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-13T14:54:07Z","receivedAt":"2020-09-13T14:54:57Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a check to verify if the remote-tracking ref of the local branch is\nreachable from one of its \"reflog\" entries; `set_ref_status_for_push`\nupdated to add a reject reason and disallow the forced push if the\ncheck fails.\n\nWhen a local branch that is based on a remote ref, has been rewound and\nis to be force pushed on the remote, \"apply_push_force_if_includes()\"\nruns a check that ensure any updates to remote-tracking refs that may\nhave happened (by push from another repository) in-between the time of\nthe last checkout, and right before the time of push by rejecting the\nforced update.\n\nThe struct \"ref\" has three new bit-fields:\n  * if_includes: Set when we have to run the new check on the ref.\n  * is_tracking: Set when the remote ref was marked as \"use_tracking\"\n                 or \"use_tracking_for_rest\" by compare-and-swap.\n  * unreachable: Set if the ref is unreachable from any of the \"reflog\"\n                 entries of its local counterpart.\n\nWhen \"--force-with-includes\" is used along with \"--force-with-lease\",\nthe check is run only for refs marked as \"is_tracking\".\n\nThe enum \"status\" updated to include \"REF_STATUS_REJECT_REMOTE_UPDATED\"\nto imply that the ref failed the check.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n remote.c | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++++---\n remote.h |  14 +++++-\n 2 files changed, 141 insertions(+), 8 deletions(-)\n\ndiff --git a/remote.c b/remote.c\nindex 420150837b..e4b2d85a6f 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -1484,6 +1484,36 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n \n+\t\t/*\n+\t\t * If the tip of the remote-tracking ref is unreachable\n+\t\t * from any reflog entry of its local ref indicating a\n+\t\t * possible update since checkout; reject the push.\n+\t\t *\n+\t\t * There is no need to check for reachability, if the\n+\t\t * ref is marked for deletion.\n+\t\t */\n+\t\tif (ref->if_includes && !ref->deletion) {\n+\t\t\t/*\n+\t\t\t * If `force_ref_update' was previously set by\n+\t\t\t * \"compare-and-swap\", and we have to run this\n+\t\t\t * check, reset it back to the original value\n+\t\t\t * and update it depending on the status of this\n+\t\t\t * check.\n+\t\t\t */\n+\t\t\tforce_ref_update = ref->force || force_update;\n+\n+\t\t\tif (ref->unreachable)\n+\t\t\t\treject_reason =\n+\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\telse\n+\t\t\t\t/*\n+\t\t\t\t * If updates from the remote-tracking ref\n+\t\t\t\t * have been integrated locally; force the\n+\t\t\t\t * update.\n+\t\t\t\t */\n+\t\t\t\tforce_ref_update = 1;\n+\t\t}\n+\n \t\t/*\n \t\t * If the update isn't already rejected then check\n \t\t * the usual \"must fast-forward\" rules.\n@@ -2272,11 +2302,74 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \treturn 0;\n }\n \n+static int ref_reachable(struct object_id *o_oid, struct object_id *n_oid,\n+\t\t\t const char *ident, timestamp_t timestamp, int tz,\n+\t\t\t const char *message, void *cb_data)\n+{\n+\tint ret = 0;\n+\tstruct object_id *r_oid = cb_data;\n+\n+\tret = oideq(n_oid, r_oid);\n+\tif (!ret) {\n+\t\tstruct commit *loc = lookup_commit_reference(the_repository,\n+\t\t\t\t\t\t\t     n_oid);\n+\t\tstruct commit *rem = lookup_commit_reference(the_repository,\n+\t\t\t\t\t\t\t     r_oid);\n+\t\tret = (loc && rem) ? in_merge_bases(rem, loc) : 0;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Iterate through the reflog of a local branch and check\n+ * if the tip of the remote-tracking branch is reachable\n+ * from one of the entries.\n+ */\n+static int ref_reachable_from_reflog(const struct object_id *r_oid,\n+\t\t\t\t     const struct object_id *l_oid,\n+\t\t\t\t     const char *local_ref_name)\n+{\n+\tint ret = 0;\n+\tstruct commit *r_commit, *l_commit;\n+\n+\tl_commit = lookup_commit_reference(the_repository, l_oid);\n+\tr_commit = lookup_commit_reference(the_repository, r_oid);\n+\n+\t/*\n+\t * If the remote-tracking ref is an ancestor of the local\n+\t * ref (a merge, for instance) there is no need to iterate\n+\t * through the reflog entries to ensure reachability; it\n+\t * can be skipped to return early instead.\n+\t */\n+\tret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;\n+\tif (!ret)\n+\t\tret = for_each_reflog_ent_reverse(local_ref_name, ref_reachable,\n+\t\t\t\t\t\t  (struct object_id *)r_oid);\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Check for reachability of a remote-tracking\n+ * ref in the reflog entries of its local ref.\n+ */\n+void check_reflog_for_ref(struct ref *r_ref)\n+{\n+\tstruct object_id r_oid;\n+\tstruct ref *l_ref = get_local_ref(r_ref->name);\n+\n+\tif (r_ref->if_includes && l_ref && !read_ref(l_ref->name, &r_oid))\n+\t\tr_ref->unreachable = !ref_reachable_from_reflog(&r_ref->old_oid,\n+\t\t\t\t\t\t\t\t&r_oid,\n+\t\t\t\t\t\t\t\tl_ref->name);\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n \t\t      struct ref *ref)\n {\n-\tint i;\n+\tint i, is_tracking = 0;\n \n \t/* Find an explicit --<option>=<name>[:<value>] entry */\n \tfor (i = 0; i < cas->nr; i++) {\n@@ -2288,16 +2381,26 @@ static void apply_cas(struct push_cas_option *cas,\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n \t\t\toidclr(&ref->old_oid_expect);\n-\t\treturn;\n+\t\telse\n+\t\t\tis_tracking = 1;\n+\t\tbreak;\n \t}\n \n \t/* Are we using \"--<option>\" to cover all? */\n-\tif (!cas->use_tracking_for_rest)\n-\t\treturn;\n+\tif (cas->use_tracking_for_rest) {\n+\t\tref->expect_old_sha1 = 1;\n+\t\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\t\t\toidclr(&ref->old_oid_expect);\n+\t\telse\n+\t\t\tis_tracking = 1;\n+\t}\n+\n+\t/*\n+\t * Mark this ref to be checked if \"--force-if-includes\" is\n+\t * specified as an argument along with \"compare-and-swap\".\n+\t */\n+\tref->is_tracking = is_tracking;\n \n-\tref->expect_old_sha1 = 1;\n-\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n-\t\toidclr(&ref->old_oid_expect);\n }\n \n void apply_push_cas(struct push_cas_option *cas,\n@@ -2308,3 +2411,21 @@ void apply_push_cas(struct push_cas_option *cas,\n \tfor (ref = remote_refs; ref; ref = ref->next)\n \t\tapply_cas(cas, remote, ref);\n }\n+\n+void apply_push_force_if_includes(struct ref *remote_refs, int used_with_cas)\n+{\n+\tstruct ref *ref;\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n+\t\t/*\n+\t\t * If \"compare-and-swap\" is used along with option, run the\n+\t\t * check on refs that have been marked to do so. Otherwise,\n+\t\t * all refs will be checked.\n+\t\t */\n+\t\tif (used_with_cas)\n+\t\t\tref->if_includes = ref->is_tracking;\n+\t\telse\n+\t\t\tref->if_includes = 1;\n+\n+\t\tcheck_reflog_for_ref(ref);\n+\t}\n+}\ndiff --git a/remote.h b/remote.h\nindex 5e3ea5a26d..1618ba892b 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -104,7 +104,10 @@ struct ref {\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\tif_includes:1, /* If \"--force-with-includes\" was specified.  */\n+\t\tis_tracking:1, /* If \"use_tracking[_for_rest]\" is set (CAS). */\n+\t\tunreachable:1; /* For \"if_includes\"; unreachable in reflog.  */\n \n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -134,6 +137,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n \t\tREF_STATUS_REJECT_SHALLOW,\n+\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT,\n@@ -346,4 +350,12 @@ int parseopt_push_cas_option(const struct option *, const char *arg, int unset);\n int is_empty_cas(const struct push_cas_option *);\n void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);\n \n+/*\n+ * Runs when \"--force-if-includes\" is specified.\n+ * Checks if the remote-tracking ref was updated (since checkout)\n+ * implicitly in the background and verify that changes from the\n+ * updated tip have been integrated locally, before pushing.\n+ */\n+void apply_push_force_if_includes(struct ref*, int);\n+\n #endif\n-- \n2.28.0\n\n"},{"id":"405493","messageId":"20200913145413.18351-4-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200913145413.18351-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v3 3/7] send-pack: check ref status for \"force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-13T14:54:09Z","receivedAt":"2020-09-13T14:55:00Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Update \"print_helper_status\" in send-pack to catch the ref\nstatus (\"REF_STATUS_REJECT_REMOTE_UPDATED\") that is set if\nchecks for \"--force-if-includes\" fail, and add a string\nequivalent of the ref status for printing.\n\nUpdate \"check_to_send_update\" to catch the new ref status\nwhen setting \"CHECK_REF_STATUS_REJECTED\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n builtin/send-pack.c | 5 +++++\n send-pack.c         | 1 +\n 2 files changed, 6 insertions(+)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 2b9610f121..4d76727edb 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n \n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"remote ref updated since checkout\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\ndiff --git a/send-pack.c b/send-pack.c\nindex 632f1580ca..956306e8e8 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\n-- \n2.28.0\n\n"},{"id":"405494","messageId":"20200913145413.18351-3-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200913145413.18351-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v3 2/7] transport: add flag for \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-13T14:54:08Z","receivedAt":"2020-09-13T14:55:05Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new flag: \"TRANSPORT_PUSH_FORCE_IF_INCLUDES\", which is\nset if \"--force-if-includes\" is specified in the command line\nand calls \"apply_push_force_if_includes()\" if set.\n\nUpdate \"print_one_push_status()\" to catch and print the ref\nstatus if set to \"REF_STATUS_REJECT_REMOTE_UPDATED\"; add a\nreject reason: \"REJECT_REF_NEEDS_UPDATE\" to map it new ref\nstatus.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n transport.c | 18 +++++++++++++++++-\n transport.h | 12 +++++++-----\n 2 files changed, 24 insertions(+), 6 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex 43e24bf1e5..7f5020bba3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"stale info\", porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"remote ref updated since checkout\",\n+\t\t\t\t porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -659,6 +664,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t\t*reject_reasons |= REJECT_FETCH_FIRST;\n \t\t} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {\n \t\t\t*reject_reasons |= REJECT_NEEDS_FORCE;\n+\t\t} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {\n+\t\t\t*reject_reasons |= REJECT_REF_NEEDS_UPDATE;\n \t\t}\n \t}\n \tfree(head);\n@@ -1101,6 +1108,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n \n \t\tstrbuf_reset(&buf);\n@@ -1151,6 +1159,8 @@ int transport_push(struct repository *r,\n \t\tint pretend = flags & TRANSPORT_PUSH_DRY_RUN;\n \t\tint push_ret, ret, err;\n \t\tstruct strvec ref_prefixes = STRVEC_INIT;\n+\t\tint is_cas = 0;\n+\t\tint if_includes = flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n \n \t\tif (check_push_refs(local_refs, rs) < 0)\n \t\t\treturn -1;\n@@ -1178,9 +1188,15 @@ int transport_push(struct repository *r,\n \n \t\tif (transport->smart_options &&\n \t\t    transport->smart_options->cas &&\n-\t\t    !is_empty_cas(transport->smart_options->cas))\n+\t\t    !is_empty_cas(transport->smart_options->cas)) {\n+\t\t\tis_cas = 1;\n \t\t\tapply_push_cas(transport->smart_options->cas,\n \t\t\t\t       transport->remote, remote_refs);\n+\t\t}\n+\n+\t\t/* When \"--force-if-includes\" is specified. */\n+\t\tif (if_includes)\n+\t\t\tapply_push_force_if_includes(remote_refs, is_cas);\n \n \t\tset_ref_status_for_push(remote_refs,\n \t\t\tflags & TRANSPORT_PUSH_MIRROR,\ndiff --git a/transport.h b/transport.h\nindex ca409ea1e4..729bb6979e 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -136,6 +136,7 @@ struct transport {\n #define TRANSPORT_PUSH_ATOMIC\t\t\t(1<<13)\n #define TRANSPORT_PUSH_OPTIONS\t\t\t(1<<14)\n #define TRANSPORT_RECURSE_SUBMODULES_ONLY\t(1<<15)\n+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES\t(1<<16)\n \n int transport_summary_width(const struct ref *refs);\n \n@@ -217,11 +218,12 @@ int transport_set_option(struct transport *transport, const char *name,\n void transport_set_verbosity(struct transport *transport, int verbosity,\n \tint force_progress);\n \n-#define REJECT_NON_FF_HEAD     0x01\n-#define REJECT_NON_FF_OTHER    0x02\n-#define REJECT_ALREADY_EXISTS  0x04\n-#define REJECT_FETCH_FIRST     0x08\n-#define REJECT_NEEDS_FORCE     0x10\n+#define REJECT_NON_FF_HEAD      0x01\n+#define REJECT_NON_FF_OTHER     0x02\n+#define REJECT_ALREADY_EXISTS   0x04\n+#define REJECT_FETCH_FIRST      0x08\n+#define REJECT_NEEDS_FORCE      0x10\n+#define REJECT_REF_NEEDS_UPDATE 0x20\n \n int transport_push(struct repository *repo,\n \t\t   struct transport *connection,\n-- \n2.28.0\n\n"},{"id":"405495","messageId":"20200913145413.18351-5-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200913145413.18351-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v3 4/7] transport-helper: update ref status for \"force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-13T14:54:10Z","receivedAt":"2020-09-13T14:55:08Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Update \"push_update_refs_status\" to map the string equivalent\nof the ref status \"REF_STATUS_REJECT_REMOTE_UPDATED\" when set\nif a `git-push` fails for \"--force-if-includes\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n transport-helper.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c52c99d829..e547e21199 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"remote ref updated since checkout\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\n \t\t\t\tstring_list_clear(&cas_options, 0);\n-- \n2.28.0\n\n"},{"id":"405496","messageId":"20200913145413.18351-6-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200913145413.18351-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v3 5/7] builtin/push: add option \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-13T14:54:11Z","receivedAt":"2020-09-13T14:55:10Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Update the option parser to set the flag for \"force-if-includes\";\nadd an (optional) advice message to print when the push fails and\nthe reject reason is \"REJECT_REF_NEEDS_UPDATE\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n advice.c       |  3 +++\n advice.h       |  2 ++\n builtin/push.c | 20 ++++++++++++++++++--\n 3 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex f0a3d32d20..164742305f 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;\n int advice_push_fetch_first = 1;\n int advice_push_needs_force = 1;\n int advice_push_unqualified_ref_name = 1;\n+int advice_push_ref_needs_update = 1;\n int advice_status_hints = 1;\n int advice_status_u_option = 1;\n int advice_status_ahead_behind_warning = 1;\n@@ -72,6 +73,7 @@ static struct {\n \t{ \"pushFetchFirst\", &advice_push_fetch_first },\n \t{ \"pushNeedsForce\", &advice_push_needs_force },\n \t{ \"pushUnqualifiedRefName\", &advice_push_unqualified_ref_name },\n+\t{ \"pushRefNeedsUpdate\", &advice_push_ref_needs_update },\n \t{ \"statusHints\", &advice_status_hints },\n \t{ \"statusUoption\", &advice_status_u_option },\n \t{ \"statusAheadBehindWarning\", &advice_status_ahead_behind_warning },\n@@ -116,6 +118,7 @@ static struct {\n \t[ADVICE_PUSH_ALREADY_EXISTS]\t\t\t= { \"pushAlreadyExists\", 1 },\n \t[ADVICE_PUSH_FETCH_FIRST]\t\t\t= { \"pushFetchFirst\", 1 },\n \t[ADVICE_PUSH_NEEDS_FORCE]\t\t\t= { \"pushNeedsForce\", 1 },\n+\t[ADVICE_PUSH_REF_NEEDS_UPDATE]\t\t\t= { \"pushRefNeedsUpdate\", 1 },\n \n \t/* make this an alias for backward compatibility */\n \t[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]\t\t= { \"pushNonFastForward\", 1 },\ndiff --git a/advice.h b/advice.h\nindex 16f2c11642..bc2432980a 100644\n--- a/advice.h\n+++ b/advice.h\n@@ -11,6 +11,7 @@ extern int advice_push_already_exists;\n extern int advice_push_fetch_first;\n extern int advice_push_needs_force;\n extern int advice_push_unqualified_ref_name;\n+extern int advice_push_ref_needs_update;\n extern int advice_status_hints;\n extern int advice_status_u_option;\n extern int advice_status_ahead_behind_warning;\n@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;\n \tADVICE_PUSH_UNQUALIFIED_REF_NAME,\n \tADVICE_PUSH_UPDATE_REJECTED_ALIAS,\n \tADVICE_PUSH_UPDATE_REJECTED,\n+\tADVICE_PUSH_REF_NEEDS_UPDATE,\n \tADVICE_RESET_QUIET_WARNING,\n \tADVICE_RESOLVE_CONFLICT,\n \tADVICE_RM_HINTS,\ndiff --git a/builtin/push.c b/builtin/push.c\nindex bc94078e72..d23b4678b4 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -300,6 +300,12 @@ static const char message_advice_ref_needs_force[] =\n \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n \t   \"without using the '--force' option.\\n\");\n \n+static const char message_advice_ref_needs_update[] =\n+\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n+\t   \"branch has been updated since the last checkout. You may want\\n\"\n+\t   \"to integrate those changes locally (e.g., 'git rebase ...')\\n\"\n+\t   \"before forcing an update.\\n\");\n+\n static void advise_pull_before_push(void)\n {\n \tif (!advice_push_non_ff_current || !advice_push_update_rejected)\n@@ -335,6 +341,13 @@ static void advise_ref_needs_force(void)\n \tadvise(_(message_advice_ref_needs_force));\n }\n \n+static void advise_ref_needs_update(void)\n+{\n+\tif (!advice_push_ref_needs_update || !advice_push_update_rejected)\n+\t\treturn;\n+\tadvise(_(message_advice_ref_needs_update));\n+}\n+\n static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\t\t     int flags)\n {\n@@ -384,8 +397,9 @@ static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\tadvise_ref_fetch_first();\n \t} else if (reject_reasons & REJECT_NEEDS_FORCE) {\n \t\tadvise_ref_needs_force();\n+\t} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {\n+\t\tadvise_ref_needs_update();\n \t}\n-\n \treturn 1;\n }\n \n@@ -521,7 +535,6 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\t\treturn config_error_nonbool(k);\n \t\treturn color_parse(v, push_colors[slot]);\n \t}\n-\n \treturn git_default_config(k, v, NULL);\n }\n \n@@ -551,6 +564,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t\t       N_(\"require old value of ref to be at this value\"),\n \t\t\t       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),\n+\t\tOPT_BIT(0, \"force-if-includes\", &flags,\n+\t\t\tN_(\"require remote updates to be integrated locally\"),\n+\t\t\tTRANSPORT_PUSH_FORCE_IF_INCLUDES),\n \t\tOPT_CALLBACK(0, \"recurse-submodules\", &recurse_submodules, \"(check|on-demand|no)\",\n \t\t\t     N_(\"control recursive pushing of submodules\"), option_parse_recurse_submodules),\n \t\tOPT_BOOL_F( 0 , \"thin\", &thin, N_(\"use thin pack\"), PARSE_OPT_NOCOMPLETE),\n-- \n2.28.0\n\n"},{"id":"405497","messageId":"20200913145413.18351-7-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200913145413.18351-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v3 6/7] doc: add reference for \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-13T14:54:12Z","receivedAt":"2020-09-13T14:55:20Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add documentation for using the new option; append notes for\n\"--force-with-lease\" about using the new option to prevent\nunintended remote overwrites when being used in setups where a\ntool implicitly updates remote-tracking refs in the background.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/advice.txt |  9 ++++++---\n Documentation/git-push.txt      | 19 +++++++++++++++++++\n 2 files changed, 25 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\nindex bdd37c3eaa..acbd0c09aa 100644\n--- a/Documentation/config/advice.txt\n+++ b/Documentation/config/advice.txt\n@@ -10,9 +10,8 @@ advice.*::\n \t\tthat the check is disabled.\n \tpushUpdateRejected::\n \t\tSet this variable to 'false' if you want to disable\n-\t\t'pushNonFFCurrent',\n-\t\t'pushNonFFMatching', 'pushAlreadyExists',\n-\t\t'pushFetchFirst', and 'pushNeedsForce'\n+\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n+\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n \t\tsimultaneously.\n \tpushNonFFCurrent::\n \t\tAdvice shown when linkgit:git-push[1] fails due to a\n@@ -41,6 +40,10 @@ advice.*::\n \t\twe can still suggest that the user push to either\n \t\trefs/heads/* or refs/tags/* based on the type of the\n \t\tsource object.\n+\tpushRefNeedsUpdate::\n+\t\tShown when linkgit:git-push[1] rejects a forced update of\n+\t\ta branch when its remote-tracking ref has updates that we\n+\t\tdo not have locally.\n \tstatusAheadBehind::\n \t\tShown when linkgit:git-status[1] computes the ahead/behind\n \t\tcounts for a local ref compared to its remote tracking ref,\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..b40fe7e7cf 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, specifying `--force-if-includes` an an ancillary option along\n+with `--force-with-lease[=<refname>[:expect]]` (when \"<refname>\" or \"<expect>\"\n+values are unspecified) at the time of `push` will verify if updates from the\n+remote-tracking refs that may have been implicitly updated in the background\n+are integrated locally before allowing a forced update.\n \n -f::\n --force::\n@@ -341,6 +347,19 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n origin +master` to force a push to the `master` branch). See the\n `<refspec>...` section above for details.\n \n+--[no-]force-if-includes::\n+\tForce an update only if the tip of the remote-tracking ref\n+\thas been integrated locally.\n++\n+This option verifies if the tip of the remote-tracking ref on which\n+a local branch has based on (for a rewrite), is reachable from at\n+least one of the `reflog` entries of the local branch about to be\n+updated by force on the remote. The check ensures that any updates\n+from the remote have been incorporated locally by rejecting a push\n+if that is not the case.\n++\n+Specifying `--no-force-if-includes` disables this behavior.\n+\n --repo=<repository>::\n \tThis option is equivalent to the <repository> argument. If both\n \tare specified, the command-line argument takes precedence.\n-- \n2.28.0\n\n"},{"id":"405498","messageId":"20200913145413.18351-8-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200913145413.18351-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v3 7/7] t: add tests for \"force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-13T14:54:13Z","receivedAt":"2020-09-13T14:55:27Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"t/5533:\n  * Add a test cases to verify when \"--force-if-includes\" is used\n    along with \"--force-with-lease[=<refname>[:expect]]\" (when the\n    \"<expect>\" value is unspecified) can help prevent unintended\n    remote overwrites when remote refs are updated in the background.\n\nt/t5549:\n  * Add test for the new option to cover the following scenarios:\n    - Reject forced updates to remote, if the remote ref is updated\n      in-between the time of checkout, rewrite and before the push,\n      with cases for a specific ref, and \"--all\".\n    - Allow forced updates for \"--force\", or if the refspec is\n      prepended with a \"+\".\n    - Allow deletes on the remote for \"--delete\", or if refspec is\n      specified as \":<ref>\".\n    - Skip the reflog check introduced by the new option if `git-push`\n      is specified with \"--force-with-lease=<refname>:<expect>\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n t/t5533-push-cas.sh               |  26 +++++\n t/t5549-push-force-if-includes.sh | 161 ++++++++++++++++++++++++++++++\n 2 files changed, 187 insertions(+)\n create mode 100755 t/t5549-push-force-if-includes.sh\n\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..6580aab49c 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -256,4 +256,30 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n \n+test_expect_success 'background updates of REMOTE can be mitigated with \"--force-if-includes\"' '\n+\trm -rf src dst &&\n+\tgit init --bare src.bare &&\n+\ttest_when_finished \"rm -rf src.bare\" &&\n+\tgit clone --no-local src.bare dst &&\n+\ttest_when_finished \"rm -rf dst\" &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit G &&\n+\t\tgit push origin master:master\n+\t) &&\n+\tgit clone --no-local src.bare dst2 &&\n+\ttest_when_finished \"rm -rf dst2\" &&\n+\t(\n+\t\tcd dst2 &&\n+\t\ttest_commit H &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit I &&\n+\t\tgit fetch origin &&\n+\t\ttest_must_fail git push --force-with-lease --force-if-includes origin\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t5549-push-force-if-includes.sh b/t/t5549-push-force-if-includes.sh\nnew file mode 100755\nindex 0000000000..e5d1675478\n--- /dev/null\n+++ b/t/t5549-push-force-if-includes.sh\n@@ -0,0 +1,161 @@\n+test_description='Test push \"--force-if-includes\" forced update safety.'\n+\n+. ./test-lib.sh\n+\n+setup_src_dup_dst () {\n+\trm -fr src dup dst &&\n+\tgit init --bare dst &&\n+\tgit clone --no-local dst src &&\n+\tgit clone --no-local dst dup\n+\t(\n+\t\tcd src &&\n+\t\ttest_commit foo &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit fetch &&\n+\t\tgit merge origin/master &&\n+\t\ttest_commit bar &&\n+\t\tgit switch -c branch master~1 &&\n+\t\ttest_commit baz &&\n+\t\ttest_commit D &&\n+\t\tgit push --all\n+\t) &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch master &&\n+\t\tgit fetch --all &&\n+\t\tgit branch branch --track origin/branch &&\n+\t\tgit rebase origin/master\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit switch master &&\n+\t\ttest_commit qux &&\n+\t\tgit switch branch &&\n+\t\ttest_commit quux &&\n+\t\tgit push origin --all\n+\t)\n+}\n+\n+test_expect_success 'reject push if remote changes are not integrated locally (protected, all refs)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit wobble &&\n+\t\tgit switch master &&\n+\t\ttest_commit wubble &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes --all\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success 'reject push if remote changes are not integrated locally (protected, specific ref)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit wobble &&\n+\t\tgit switch master &&\n+\t\ttest_commit wubble &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes origin master\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\ttest_cmp expect.master actual.master\n+'\n+\n+test_expect_success 'allow force push if \"--force\" is specified (forced, all refs)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit wobble &&\n+\t\tgit switch master &&\n+\t\ttest_commit wubble &&\n+\t\tgit fetch --all &&\n+\t\tgit push --force --force-if-includes origin --all 2>err &&\n+\t\tgrep \"forced update\" err\n+\t)\n+'\n+\n+test_expect_success 'allow force push if \"--delete\" is specified' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit wobble &&\n+\t\tgit switch master &&\n+\t\ttest_commit wubble &&\n+\t\tgit fetch --all &&\n+\t\tgit push --delete --force-if-includes origin branch 2>err &&\n+\t\tgrep \"deleted\" err\n+\t)\n+'\n+\n+test_expect_success 'allow forced updates if specified with refspec (forced, specific ref)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit wobble &&\n+\t\tgit switch master &&\n+\t\ttest_commit wubble &&\n+\t\tgit fetch --all &&\n+\t\tgit push --force-if-includes origin +branch 2>err &&\n+\t\tgrep \"forced update\" err\n+\t)\n+'\n+\n+test_expect_success 'allow deletes if specified with refspec (delete, specific ref)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit wobble &&\n+\t\tgit switch master &&\n+\t\ttest_commit wubble &&\n+\t\tgit fetch --all &&\n+\t\tgit push --force-if-includes origin :branch 2>err &&\n+\t\tgrep \"deleted\" err\n+\t)\n+'\n+\n+test_expect_success 'must be disabled for --force-with-lease=\"<ref>:<expect>\" (protected, specific ref)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit wobble &&\n+\t\tgit switch master &&\n+\t\ttest_commit wubble &&\n+\t\tr_head=\"$(git rev-parse refs/remotes/origin/master)\" &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes --force-with-lease=\"master:$r_head\" 2>err &&\n+\t\tgrep \"stale info\" err\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_done\n-- \n2.28.0\n\n"},{"id":"405516","messageId":"nycvar.QRO.7.76.6.2009120013070.56@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"20200910144456.GA25125@mail.clickyotomy.dev","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-11T22:16:10Z","receivedAt":"2020-09-14T10:30:07Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Srinidhi,\n\nOn Thu, 10 Sep 2020, Srinidhi Kaushik wrote:\n\n> On 09/10/2020 12:22, Johannes Schindelin wrote:\n>\n> > BTW I think the patch needs to cover a bit more, still: after I run `git\n> > pull --rebase`, the local branch will never have been at the same revision\n> > as the fetched one: `git rebase` moves to an unnamed branch before\n> > replaying the patches. So I think we need to see whether the remote tip\n> > was _reachable_ from (not necessarily identical to) any of the reflog's\n> > revisions.\n>\n> Good catch. Would adding in_merge_bases() along with checking if OIDs\n> are equal for each reflog entry in oid_in_reflog_ent() address the\n> problem? That way, we would check if remote ref is reachable from\n> one of the entries?\n>\n> Thanks.\n>\n> -- >8 --\n> + static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,\n> +\t\t\t     const char *ident, timestamp_t timestamp, int tz,\n> +\t\t\t     const char *message, void *cb_data)\n> + {\n> +\tstruct object_id *remote_oid = cb_data;\n> +\tstruct commit *a = lookup_commit_reference(the_repository, noid);\n> +\tstruct commit *b = lookup_commit_reference(the_repository, remote_oid);\n> +\treturn oideq(noid, remote_oid) || in_merge_bases(b, a);\n> + }\n\nSince `in_merge_bases()` is quite a bit more expensive than `oideq()`,\npersonally, I would actually walk the reflog with the `oideq()` check\nfirst (stopping early in case a match was found), and only fall back to\nlooking for a merge base in the reflog if the first reflog walk did not\nfind a match.\n\nCiao,\nDscho\n"},{"id":"405517","messageId":"nycvar.QRO.7.76.6.2009120016310.56@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"xmqqsgbpr8oc.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-11T22:17:07Z","receivedAt":"2020-09-14T10:31:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 10 Sep 2020, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n>\n> >> In contrast, when you want to make sure that you _actually_ incorporated\n> >> the revision that is currently the remote tip, e.g. via `git pull\n> >> --rebase` with a possible additional rebase on top that makes this _not_ a\n> >> fast-forward, you totally have to force the push, otherwise it won't work.\n> >\n> > Maybe `--force-if-incorporated`? Originally, I had in mind to call it\n> > `--safe-force`, but that might be too vague.\n>\n> Yup.  \"safe force\" indeed feels like a misnomer.  The assumption of\n> safety relies heavily on the workflow.\n>\n> I might even say --force-if-merged even if the way the to-be-lost\n> changes have become part of what you are pushing out is not\n> technically a merge, but there may be shorter and sweeter way to\n> express it than 'merge' and 'incorporate'.\n\nYou're right, `--force-if-merged` is a much better way to put it.\n\nThanks,\nDscho\n"},{"id":"405518","messageId":"20200914110638.GA32876@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009120013070.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-14T11:06:38Z","receivedAt":"2020-09-14T11:06:51Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Johannes,\n\nOn 09/12/2020 00:16, Johannes Schindelin wrote:\n> > + static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,\n> > +\t\t\t     const char *ident, timestamp_t timestamp, int tz,\n> > +\t\t\t     const char *message, void *cb_data)\n> > + {\n> > +\tstruct object_id *remote_oid = cb_data;\n> > +\tstruct commit *a = lookup_commit_reference(the_repository, noid);\n> > +\tstruct commit *b = lookup_commit_reference(the_repository, remote_oid);\n> > +\treturn oideq(noid, remote_oid) || in_merge_bases(b, a);\n> > + }\n> \n> Since `in_merge_bases()` is quite a bit more expensive than `oideq()`,\n> personally, I would actually walk the reflog with the `oideq()` check\n> first (stopping early in case a match was found), and only fall back to\n> looking for a merge base in the reflog if the first reflog walk did not\n> find a match.\n\nThanks for the suggestion; the updated the patch set (v3) - [1] does\nsomething similar to that, along with some other changes. Can you please\ntake a look?\n\n[1]: https://public-inbox.org/git/20200913145413.18351-2-shrinidhi.kaushik@gmail.com/\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"405525","messageId":"xmqqy2lcjf5g.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009120016310.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-14T20:07:39Z","receivedAt":"2020-09-14T20:08:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Hi Junio,\n>\n> On Thu, 10 Sep 2020, Junio C Hamano wrote:\n>\n>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n>>\n>> >> In contrast, when you want to make sure that you _actually_ incorporated\n>> >> the revision that is currently the remote tip, e.g. via `git pull\n>> >> --rebase` with a possible additional rebase on top that makes this _not_ a\n>> >> fast-forward, you totally have to force the push, otherwise it won't work.\n>> >\n>> > Maybe `--force-if-incorporated`? Originally, I had in mind to call it\n>> > `--safe-force`, but that might be too vague.\n>>\n>> Yup.  \"safe force\" indeed feels like a misnomer.  The assumption of\n>> safety relies heavily on the workflow.\n>>\n>> I might even say --force-if-merged even if the way the to-be-lost\n>> changes have become part of what you are pushing out is not\n>> technically a merge, but there may be shorter and sweeter way to\n>> express it than 'merge' and 'incorporate'.\n>\n> You're right, `--force-if-merged` is a much better way to put it.\n\nI am pretty happy that Srinidhi chose 'included', which is what\nseems the best description without being a white-lie that is\ntechnically incorrect.\n"},{"id":"405527","messageId":"xmqqk0wwjf4p.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009120013070.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-14T20:08:06Z","receivedAt":"2020-09-14T20:08:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Hi Srinidhi,\n>\n> On Thu, 10 Sep 2020, Srinidhi Kaushik wrote:\n>\n>> On 09/10/2020 12:22, Johannes Schindelin wrote:\n>>\n>> > BTW I think the patch needs to cover a bit more, still: after I run `git\n>> > pull --rebase`, the local branch will never have been at the same revision\n>> > as the fetched one: `git rebase` moves to an unnamed branch before\n>> > replaying the patches. So I think we need to see whether the remote tip\n>> > was _reachable_ from (not necessarily identical to) any of the reflog's\n>> > revisions.\n>>\n>> Good catch. Would adding in_merge_bases() along with checking if OIDs\n>> are equal for each reflog entry in oid_in_reflog_ent() address the\n>> problem? That way, we would check if remote ref is reachable from\n>> one of the entries?\n\nThat sounds very expensive.  \n\nIf we switched to check the reflog of HEAD (and not any particular\nlocal branch's reflog), then \"has this ever been checked out\", tests\nwould suffice, no?  We detach at the tip of the remote-tracking\nbranch and then reapply our work one by one in such a case, so we\nshould have the record of having their tip in the working tree at\nsome point (i.e. at the beginning of the \"rebase\" phase of the \"pull\n--rebase\" operation).\n\n"},{"id":"405529","messageId":"xmqq5z8gje18.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200913145413.18351-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v3 1/7] remote: add reflog check for \"force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-14T20:31:47Z","receivedAt":"2020-09-14T20:31:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> The struct \"ref\" has three new bit-fields:\n>   * if_includes: Set when we have to run the new check on the ref.\n>   * is_tracking: Set when the remote ref was marked as \"use_tracking\"\n>                  or \"use_tracking_for_rest\" by compare-and-swap.\n\n... meaning that --force-with-lease with an explicit \"the tip must\nstill be at this exact commit\" won't use the extra check to loosen\nthe condition?  That sounds sensible.\n\n>   * unreachable: Set if the ref is unreachable from any of the \"reflog\"\n>                  entries of its local counterpart.\n\nThe same comment applies on \"reflog of which branch should we\ncheck\"; I suspect that checking HEAD and without checking merge base\nmay prove to be a good way to go.\n\n> + */\n> +void apply_push_force_if_includes(struct ref*, int);\n\nSP between ref and '*'?\n"},{"id":"405531","messageId":"xmqqd02ojep5.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200913145413.18351-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v3 1/7] remote: add reflog check for \"force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-14T20:17:26Z","receivedAt":"2020-09-14T20:34:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Add a check to verify if the remote-tracking ref of the local branch is\n> reachable from one of its \"reflog\" entries; `set_ref_status_for_push`\n> updated to add a reject reason and disallow the forced push if the\n> check fails.\n\nI have to wonder (not objecting to, just wondering about) if it is a\ngood assumption that the current branch must be where we should have\nseen the tip of the other side we are about to lose.  I ask because\nwhen I do a large rewrite I often am on a detached HEAD most of the\ntime, and after everything looks sensible in the rewritten result,\nI \"checkout -B\" the local branch.\n\nWe could reduce the rate of false positive (\"no you've not looked at\nwhat you are about to discard, so we won't let you force\") by\nchecking reflogs of all the local branches and HEAD, but that may be\ntoo much.  I wonder if checking reflog entries only for HEAD (and\nnot any of the current local branches) would be a good compromise.\n"},{"id":"405535","messageId":"xmqqzh5s1398.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200913145413.18351-7-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v3 6/7] doc: add reference for \"--[no-]force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-14T21:01:55Z","receivedAt":"2020-09-14T21:02:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\n> index 3b8053447e..b40fe7e7cf 100644\n> --- a/Documentation/git-push.txt\n> +++ b/Documentation/git-push.txt\n> @@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally\n>  force push changes to `master` if the remote version is still at\n>  `base`, regardless of what your local `remotes/origin/master` has been\n>  updated to in the background.\n> ++\n> +Alternatively, specifying `--force-if-includes` an an ancillary option along\n> +with `--force-with-lease[=<refname>[:expect]]` (when \"<refname>\" or \"<expect>\"\n> +values are unspecified) at the time of `push` will verify if updates from the\n> +remote-tracking refs that may have been implicitly updated in the background\n> +are integrated locally before allowing a forced update.\n\nYou cannot omit <refname> without omitting <expect>, so\n\n\t... with \"--force-with-lease[=<refname>]\" (i.e. without\n\tsaying what exact commit the ref on the remote side must be\n\tpointing at, or which refs on the remote side are being\n\tprotected) at the time of ...\n\nwould be more appropriate.\n\n> +--[no-]force-if-includes::\n> +\tForce an update only if the tip of the remote-tracking ref\n> +\thas been integrated locally.\n> ++\n> +This option verifies if the tip of the remote-tracking ref on which\n> +a local branch has based on (for a rewrite), is reachable from at\n> +least one of the `reflog` entries of the local branch about to be\n\nIf we take the \"we don't have to look at a local branch's reflog;\njust check HEAD's and rebase will automatically be handled without\nexpensive merge-base\" approach, then\n\n    ... if the tip of the remote-tracking ref was once checked out\n    to the working tree (for a rewrite) by seeing if it appears in\n    the reflog of \"HEAD\" ...\n\n"},{"id":"405536","messageId":"xmqqsgbk12q3.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200913145413.18351-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v3 1/7] remote: add reflog check for \"force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-14T21:13:24Z","receivedAt":"2020-09-14T21:13:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> +/*\n> + * Check for reachability of a remote-tracking\n> + * ref in the reflog entries of its local ref.\n> + */\n> +void check_reflog_for_ref(struct ref *r_ref)\n\nMake it file-scope static.\n"},{"id":"405602","messageId":"20200916053141.GA29901@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqk0wwjf4p.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-16T05:31:41Z","receivedAt":"2020-09-16T05:31:57Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Junio,\n\nOn 09/14/2020 13:08, Junio C Hamano wrote:\n>>> [...]\n>>>\n>>> Good catch. Would adding in_merge_bases() along with checking if OIDs\n>>> are equal for each reflog entry in oid_in_reflog_ent() address the\n>>> problem? That way, we would check if remote ref is reachable from\n>>> one of the entries?\n> \n> That sounds very expensive.  \n\nYes, you're right about that.\n \n> If we switched to check the reflog of HEAD (and not any particular\n> local branch's reflog), then \"has this ever been checked out\", tests\n> would suffice, no?  We detach at the tip of the remote-tracking\n> branch and then reapply our work one by one in such a case, so we\n> should have the record of having their tip in the working tree at\n> some point (i.e. at the beginning of the \"rebase\" phase of the \"pull\n> --rebase\" operation).\n \nInteresting, I think that might work! Since HEAD moves around and\nrecords all the movements, if the remote was ever checked out there\nshould be an entry in the reflog. I guess we could stick to \"oideq()\"\nfor each entry if we're going this way. However, I think we should\nkeep that one \"in_merge_bases()\" call at the beginning to check if\nwe can can skip checking the reflog if the remote ref is an ancestor\nof the local ref.\n\n>>>> On 09/10/2020 12:22, Johannes Schindelin wrote:\n>>>>\n>>>> BTW I think the patch needs to cover a bit more, still: after I run `git\n>>>> pull --rebase`, the local branch will never have been at the same revision\n>>>> as the fetched one: `git rebase` moves to an unnamed branch before\n>>>> replaying the patches. So I think we need to see whether the remote tip\n>>>> was _reachable_ from (not necessarily identical to) any of the reflog's\n>>>> revision\n\nAlso, there would be an entry for what Johannes describes above -- when\na `git pull --rebase` is run. Will change to this in the next series.\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"405603","messageId":"20200916053543.GB29901@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqzh5s1398.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 6/7] doc: add reference for \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-16T05:35:43Z","receivedAt":"2020-09-16T05:35:57Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Junio,\n\nOn 09/14/2020 14:01, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> > diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\n> > index 3b8053447e..b40fe7e7cf 100644\n> > --- a/Documentation/git-push.txt\n> > +++ b/Documentation/git-push.txt\n> > @@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally\n> >  force push changes to `master` if the remote version is still at\n> >  `base`, regardless of what your local `remotes/origin/master` has been\n> >  updated to in the background.\n> > ++\n> > +Alternatively, specifying `--force-if-includes` an an ancillary option along\n> > +with `--force-with-lease[=<refname>[:expect]]` (when \"<refname>\" or \"<expect>\"\n> > +values are unspecified) at the time of `push` will verify if updates from the\n> > +remote-tracking refs that may have been implicitly updated in the background\n> > +are integrated locally before allowing a forced update.\n> \n> You cannot omit <refname> without omitting <expect>, so\n> \n> \t... with \"--force-with-lease[=<refname>]\" (i.e. without\n> \tsaying what exact commit the ref on the remote side must be\n> \tpointing at, or which refs on the remote side are being\n> \tprotected) at the time of ...\n> \n> would be more appropriate.\n\nOK, that makes sense, will update.\n \n> > +--[no-]force-if-includes::\n> > +\tForce an update only if the tip of the remote-tracking ref\n> > +\thas been integrated locally.\n> > ++\n> > +This option verifies if the tip of the remote-tracking ref on which\n> > +a local branch has based on (for a rewrite), is reachable from at\n> > +least one of the `reflog` entries of the local branch about to be\n> \n> If we take the \"we don't have to look at a local branch's reflog;\n> just check HEAD's and rebase will automatically be handled without\n> expensive merge-base\" approach, then\n> \n>     ... if the tip of the remote-tracking ref was once checked out\n>     to the working tree (for a rewrite) by seeing if it appears in\n>     the reflog of \"HEAD\" ...\n> \n\nRight, as mentioned in the other thread, we can go with this route,\nwill change in the next series.\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"405612","messageId":"nycvar.QRO.7.76.6.2009161214030.56@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"20200916053141.GA29901@mail.clickyotomy.dev","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-16T10:20:46Z","receivedAt":"2020-09-16T10:24:53Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 16 Sep 2020, Srinidhi Kaushik wrote:\n\n> On 09/14/2020 13:08, Junio C Hamano wrote:\n> >>> [...]\n> >>>\n> >>> Good catch. Would adding in_merge_bases() along with checking if OIDs\n> >>> are equal for each reflog entry in oid_in_reflog_ent() address the\n> >>> problem? That way, we would check if remote ref is reachable from\n> >>> one of the entries?\n> >\n> > That sounds very expensive.\n>\n> Yes, you're right about that.\n>\n> > If we switched to check the reflog of HEAD (and not any particular\n> > local branch's reflog), then \"has this ever been checked out\", tests\n> > would suffice, no?  We detach at the tip of the remote-tracking\n> > branch and then reapply our work one by one in such a case, so we\n> > should have the record of having their tip in the working tree at\n> > some point (i.e. at the beginning of the \"rebase\" phase of the \"pull\n> > --rebase\" operation).\n>\n> Interesting, I think that might work! Since HEAD moves around and\n> records all the movements, if the remote was ever checked out there\n> should be an entry in the reflog.\n\nNo, I don't think that would suffice. For several reasons:\n\n- the user could have checked out the remote-tracking branch directly\n  (`git switch --detach origin/my-branch`), as opposed to actually\n  integrating the revision into the branch at some stage. I know that I do\n  that sometimes.\n\n- even if the remote-tracking branch has not been checked out directly, it\n  might have been `git pull --rebase`d into a _different_ branch, in which\n  case the reflog of `HEAD` would say \"yep, I saw this commit!\" but that\n  would not mean that it was integrated into the branch we want to\n  (force-)push.\n\n- the reflog of the `HEAD` is worktree-specific, and if the user pushes\n  from a different worktree (e.g. to push multiple branches at the same\n  time, or because the push is scripted and run from a different working\n  directory), it would be missed.\n\n- and if we take a little step back, we do see that the reflog of `HEAD`\n  _does_ answer a different question from what we actually asked.\n\nSure, it would be faster, but is that worth the consequences?\n\nI really think we need to stick to looking at the reflog of the asked-for\nbranch. And to make that faster, we should have a first loop using\n`oideq()` and failing that check, run a second loop using\n`is_in_merge_bases()`.\n\nCiao,\nDscho\n"},{"id":"405647","messageId":"nycvar.QRO.7.76.6.2009161436120.56@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"20200913145413.18351-6-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v3 5/7] builtin/push: add option \"--[no-]force-if-includes\"","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-16T12:36:42Z","receivedAt":"2020-09-16T18:17:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Srinidhi,\n\nOn Sun, 13 Sep 2020, Srinidhi Kaushik wrote:\n\n> diff --git a/builtin/push.c b/builtin/push.c\n> index bc94078e72..d23b4678b4 100644\n> --- a/builtin/push.c\n> +++ b/builtin/push.c\n> @@ -300,6 +300,12 @@ static const char message_advice_ref_needs_force[] =\n>  \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n>  \t   \"without using the '--force' option.\\n\");\n>\n> +static const char message_advice_ref_needs_update[] =\n> +\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n> +\t   \"branch has been updated since the last checkout. You may want\\n\"\n> +\t   \"to integrate those changes locally (e.g., 'git rebase ...')\\n\"\n\nShouldn't this talk about `git pull` instead of `git rebase`?\n\nCiao,\nDscho\n\n"},{"id":"405648","messageId":"nycvar.QRO.7.76.6.2009161356560.56@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"20200913145413.18351-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v3 1/7] remote: add reflog check for \"force-if-includes\"","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-16T12:35:04Z","receivedAt":"2020-09-16T18:21:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Srinidhi,\n\nOn Sun, 13 Sep 2020, Srinidhi Kaushik wrote:\n\n> diff --git a/remote.c b/remote.c\n> index 420150837b..e4b2d85a6f 100644\n> --- a/remote.c\n> +++ b/remote.c\n> @@ -1484,6 +1484,36 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n>  \t\t\t\tforce_ref_update = 1;\n>  \t\t}\n>\n> +\t\t/*\n> +\t\t * If the tip of the remote-tracking ref is unreachable\n> +\t\t * from any reflog entry of its local ref indicating a\n> +\t\t * possible update since checkout; reject the push.\n> +\t\t *\n> +\t\t * There is no need to check for reachability, if the\n> +\t\t * ref is marked for deletion.\n> +\t\t */\n> +\t\tif (ref->if_includes && !ref->deletion) {\n> +\t\t\t/*\n> +\t\t\t * If `force_ref_update' was previously set by\n> +\t\t\t * \"compare-and-swap\", and we have to run this\n> +\t\t\t * check, reset it back to the original value\n> +\t\t\t * and update it depending on the status of this\n> +\t\t\t * check.\n> +\t\t\t */\n> +\t\t\tforce_ref_update = ref->force || force_update;\n> +\n> +\t\t\tif (ref->unreachable)\n> +\t\t\t\treject_reason =\n> +\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n> +\t\t\telse\n> +\t\t\t\t/*\n> +\t\t\t\t * If updates from the remote-tracking ref\n> +\t\t\t\t * have been integrated locally; force the\n> +\t\t\t\t * update.\n> +\t\t\t\t */\n> +\t\t\t\tforce_ref_update = 1;\n> +\t\t}\n> +\n>  \t\t/*\n>  \t\t * If the update isn't already rejected then check\n>  \t\t * the usual \"must fast-forward\" rules.\n> @@ -2272,11 +2302,74 @@ static int remote_tracking(struct remote *remote, const char *refname,\n>  \treturn 0;\n>  }\n>\n> +static int ref_reachable(struct object_id *o_oid, struct object_id *n_oid,\n> +\t\t\t const char *ident, timestamp_t timestamp, int tz,\n> +\t\t\t const char *message, void *cb_data)\n> +{\n> +\tint ret = 0;\n> +\tstruct object_id *r_oid = cb_data;\n> +\n> +\tret = oideq(n_oid, r_oid);\n> +\tif (!ret) {\n\nRather than having the largest part of the actual code statements in this\nfunction indented, it would make more sense to write\n\n\tif (oideq(n_oid, r_oid))\n\t\treturn 1;\n\n\tif (!(local = lookup...) ||\n\t    !(remote = lookup...))\n\t\treturn 0;\n\n\treturn in_merge_bases(remote, local);\n\n> +\t\tstruct commit *loc = lookup_commit_reference(the_repository,\n> +\t\t\t\t\t\t\t     n_oid);\n> +\t\tstruct commit *rem = lookup_commit_reference(the_repository,\n> +\t\t\t\t\t\t\t     r_oid);\n> +\t\tret = (loc && rem) ? in_merge_bases(rem, loc) : 0;\n> +\t}\n\nThis chooses the strategy of iterating over the reflog just once, and at\nevery step first testing whether the respective reflog entry is identical\nto the remote-tracking branch tip. Only when they are different do we test\nwhether the remote-tracking branch tip is at least reachable from the\nreflog entry.\n\nLet's assume that our local branch has 20 reflog entries, and the 4th one\nis identical to the current tip of the remote-tracking branch. Then we\ntested reachability 3 times. But that test is rather expensive.\n\nTherefore, I would have preferred to have a call to\n`for_each_reflog_ent_reverse()` with a callback function that only returns\nthe `oideq()` result, and only if the return value of that call is 0, I\nwould have wanted to see another call to `for_each_reflog_ent_reverse()`\nto go through, this time looking for reachability.\n\n> +\n> +\treturn ret;\n> +}\n> +\n> +/*\n> + * Iterate through the reflog of a local branch and check\n> + * if the tip of the remote-tracking branch is reachable\n> + * from one of the entries.\n> + */\n> +static int ref_reachable_from_reflog(const struct object_id *r_oid,\n> +\t\t\t\t     const struct object_id *l_oid,\n> +\t\t\t\t     const char *local_ref_name)\n> +{\n> +\tint ret = 0;\n> +\tstruct commit *r_commit, *l_commit;\n> +\n> +\tl_commit = lookup_commit_reference(the_repository, l_oid);\n> +\tr_commit = lookup_commit_reference(the_repository, r_oid);\n\nAt this point, we already LOOked up `r_commit`. But we don't pass that to\n`ref_reachable()` at any point (instead passing only `r_oid`), so we have\nto perform the lookup again.\n\nThat's wasteful. Shouldn't we pass `r_commit` directly?\n\nWith the two-pass strategy I outlined above, the first pass would use\n`r_oid`, and only when the second pass is necessary would we resort to\ncalling the expensive reachability check.\n\n> +\n> +\t/*\n> +\t * If the remote-tracking ref is an ancestor of the local\n> +\t * ref (a merge, for instance) there is no need to iterate\n> +\t * through the reflog entries to ensure reachability; it\n> +\t * can be skipped to return early instead.\n> +\t */\n> +\tret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;\n\nCorrect me if I am wrong, but isn't the first reflog entry\n(`<remote-branch>@{0}`) identical to `r_commit`? In that case, the first\niteration of the second pass over the reflog would trivially perform this\ncheck, and we do not need to duplicate the logic here.\n\n> +\tif (!ret)\n> +\t\tret = for_each_reflog_ent_reverse(local_ref_name, ref_reachable,\n> +\t\t\t\t\t\t  (struct object_id *)r_oid);\n> +\n> +\treturn ret;\n> +}\n> +\n> +/*\n> + * Check for reachability of a remote-tracking\n> + * ref in the reflog entries of its local ref.\n> + */\n> +void check_reflog_for_ref(struct ref *r_ref)\n> +{\n> +\tstruct object_id r_oid;\n> +\tstruct ref *l_ref = get_local_ref(r_ref->name);\n> +\n> +\tif (r_ref->if_includes && l_ref && !read_ref(l_ref->name, &r_oid))\n\nIf `r_ref->if_includes` is 0, we do not even have to get the local ref,\ncorrect? It would make `check_reflog_for_ref()` much easier to read for me\nif it was only called when that flag was already verified to be 1, and\nthen followed this structure:\n\n\tif (!l_ref)\n\t\treturn;\n\tif (read_ref(...))\n\t\twarning(_(\"ignoring stale remote branch information ...\"));\n\telse\n\t\tr_ref->unreachable = ...\n\nAlso, it might make a lot more sense to rename `check_reflog_for_ref()` to\n`check_if_includes_upstream()`, and to rename `r_ref` to `local` and\n`l_ref` to `remote_tracking` or something like that: nothing is inherently\n\"left\" or \"right\" about those refs.\n\n> +\t\tr_ref->unreachable = !ref_reachable_from_reflog(&r_ref->old_oid,\n> +\t\t\t\t\t\t\t\t&r_oid,\n> +\t\t\t\t\t\t\t\tl_ref->name);\n> +}\n> +\n>  static void apply_cas(struct push_cas_option *cas,\n>  \t\t      struct remote *remote,\n>  \t\t      struct ref *ref)\n>  {\n> -\tint i;\n> +\tint i, is_tracking = 0;\n>\n>  \t/* Find an explicit --<option>=<name>[:<value>] entry */\n>  \tfor (i = 0; i < cas->nr; i++) {\n> @@ -2288,16 +2381,26 @@ static void apply_cas(struct push_cas_option *cas,\n>  \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n>  \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n>  \t\t\toidclr(&ref->old_oid_expect);\n> -\t\treturn;\n> +\t\telse\n> +\t\t\tis_tracking = 1;\n\nAs part of `remote_tracking()`, we already looked up the branch name.\nSince we need it in the `is_tracking` case, maybe this should not be a\nBoolean anymore but store a copy of the remote-tracking branch name\ninstead?\n\nOh, following the code path all the way down to\n`match_name_with_pattern()`, it seems that `remote_tracking()`'s `dst`\nvariable _already_ contains a copy (which means that that memory is\nleaked, right?).\n\n> +\t\tbreak;\n>  \t}\n>\n>  \t/* Are we using \"--<option>\" to cover all? */\n> -\tif (!cas->use_tracking_for_rest)\n> -\t\treturn;\n> +\tif (cas->use_tracking_for_rest) {\n> +\t\tref->expect_old_sha1 = 1;\n> +\t\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n> +\t\t\toidclr(&ref->old_oid_expect);\n> +\t\telse\n> +\t\t\tis_tracking = 1;\n> +\t}\n> +\n> +\t/*\n> +\t * Mark this ref to be checked if \"--force-if-includes\" is\n> +\t * specified as an argument along with \"compare-and-swap\".\n> +\t */\n> +\tref->is_tracking = is_tracking;\n>\n> -\tref->expect_old_sha1 = 1;\n> -\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n> -\t\toidclr(&ref->old_oid_expect);\n>  }\n>\n>  void apply_push_cas(struct push_cas_option *cas,\n> @@ -2308,3 +2411,21 @@ void apply_push_cas(struct push_cas_option *cas,\n>  \tfor (ref = remote_refs; ref; ref = ref->next)\n>  \t\tapply_cas(cas, remote, ref);\n>  }\n> +\n> +void apply_push_force_if_includes(struct ref *remote_refs, int used_with_cas)\n> +{\n> +\tstruct ref *ref;\n> +\tfor (ref = remote_refs; ref; ref = ref->next) {\n> +\t\t/*\n> +\t\t * If \"compare-and-swap\" is used along with option, run the\n> +\t\t * check on refs that have been marked to do so. Otherwise,\n> +\t\t * all refs will be checked.\n> +\t\t */\n> +\t\tif (used_with_cas)\n> +\t\t\tref->if_includes = ref->is_tracking;\n> +\t\telse\n> +\t\t\tref->if_includes = 1;\n> +\n> +\t\tcheck_reflog_for_ref(ref);\n> +\t}\n> +}\n> diff --git a/remote.h b/remote.h\n> index 5e3ea5a26d..1618ba892b 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -104,7 +104,10 @@ struct ref {\n>  \t\tforced_update:1,\n>  \t\texpect_old_sha1:1,\n>  \t\texact_oid:1,\n> -\t\tdeletion:1;\n> +\t\tdeletion:1,\n> +\t\tif_includes:1, /* If \"--force-with-includes\" was specified.  */\n> +\t\tis_tracking:1, /* If \"use_tracking[_for_rest]\" is set (CAS). */\n> +\t\tunreachable:1; /* For \"if_includes\"; unreachable in reflog.  */\n>\n>  \tenum {\n>  \t\tREF_NOT_MATCHED = 0, /* initial value */\n> @@ -134,6 +137,7 @@ struct ref {\n>  \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n>  \t\tREF_STATUS_REJECT_STALE,\n>  \t\tREF_STATUS_REJECT_SHALLOW,\n> +\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n>  \t\tREF_STATUS_UPTODATE,\n>  \t\tREF_STATUS_REMOTE_REJECT,\n>  \t\tREF_STATUS_EXPECTING_REPORT,\n> @@ -346,4 +350,12 @@ int parseopt_push_cas_option(const struct option *, const char *arg, int unset);\n>  int is_empty_cas(const struct push_cas_option *);\n>  void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);\n>\n> +/*\n> + * Runs when \"--force-if-includes\" is specified.\n> + * Checks if the remote-tracking ref was updated (since checkout)\n> + * implicitly in the background and verify that changes from the\n> + * updated tip have been integrated locally, before pushing.\n> + */\n> +void apply_push_force_if_includes(struct ref*, int);\n\nThis function is not even hooked up in this patch, right? I don't think\nthat it makes sense to introduce it without a caller, in particular since\nit makes it harder to guess what those parameters might be used for.\n\nIn general, it appears to me as if the code worked way too hard to\naccomplish something that should be a lot simpler: when\n`--force-if-includes` is passed, it should piggy-back on top of the\n`--force-with-lease` code path, and just add yet another check on top.\n\nWith that in mind, I would have expected something more in line with this:\n\n-- snip --\n struct push_cas_option {\n \tunsigned use_tracking_for_rest:1;\n \tstruct push_cas {\n \t\tstruct object_id expect;\n-\t\tunsigned use_tracking:1;\n+\t\tenum {\n+\t\t\tPUSH_NO_CAS = 0,\n+\t\t\tPUSH_CAS_USE_TRACKING,\n+\t\t\tPUSH_CAS_IF_INCLUDED\n+\t\t} mode;\n \t\tchar *refname;\n \t} *entry;\n \tint nr;\n \tint alloc;\n };\n-- snap --\n\nand then adjusting the respective code paths accordingly.\n\nCiao,\nDscho\n\n> +\n>  #endif\n> --\n> 2.28.0\n>\n>\n"},{"id":"405679","messageId":"20200916105108.GA45240@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqd02ojep5.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 1/7] remote: add reflog check for \"force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-16T10:51:08Z","receivedAt":"2020-09-16T20:23:36Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hello,\n\nOn 09/14/2020 13:17, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> > Add a check to verify if the remote-tracking ref of the local branch is\n> > reachable from one of its \"reflog\" entries; `set_ref_status_for_push`\n> > updated to add a reject reason and disallow the forced push if the\n> > check fails.\n> \n> I have to wonder (not objecting to, just wondering about) if it is a\n> good assumption that the current branch must be where we should have\n> seen the tip of the other side we are about to lose.  I ask because\n> when I do a large rewrite I often am on a detached HEAD most of the\n> time, and after everything looks sensible in the rewritten result,\n> I \"checkout -B\" the local branch.\n> \n> We could reduce the rate of false positive (\"no you've not looked at\n> what you are about to discard, so we won't let you force\") by\n> checking reflogs of all the local branches and HEAD, but that may be\n> too much.  I wonder if checking reflog entries only for HEAD (and\n> not any of the current local branches) would be a good compromise.\n\nOne scenario I can think of is when there are multiple local branches\nthat track the same remote. Let's say we have two branches \"A\" and \"B\"\nand they both track \"A\" on the remote. If we are currently on \"A\",\nand then we decide to rebase on \"origin/A\" (after a push from another\nrepository). Then, if we (accidentally) switch to \"B\", and force update\nwith `--force-if-includes` it will _not_ be rejected because HEAD's\nreflog has a record of the checkout and there will be an overwrite\nif we check only HEAD's reflog.\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"405692","messageId":"nycvar.QRO.7.76.6.2009161437170.56@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"20200913145413.18351-1-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v3 0/7] push: add \"--[no-]force-if-includes\"","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-16T12:47:56Z","receivedAt":"2020-09-16T21:05:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Srinidhi,\n\nOn Sun, 13 Sep 2020, Srinidhi Kaushik wrote:\n\n> Add a new option: \"--force-if-includes\" to `git-push` where forced\n> updates are allowed only if the tip of the remote-tracking ref has\n> been integrated locally, by verifying if the tip of the remote-tracking\n> ref -- on which a local branch has based on -- is reachable from at\n> least one of the \"reflog\" entries of the branch about to be updated\n> by force on the remote.\n>\n> This option can also be used with `--force-with-lease` with setups\n> where the remote-tracking refs of the repository are implicitly\n> updated in the background to help prevent unintended remote\n> overwrites.\n>\n> If a local branch is based on a remote ref for a rewrite, and if that\n> remote-tracking ref is updated by a push from another repository after\n> it has been checked out locally, force updating that branch to remote\n> with `--force-with-lease[=<refname>[:expect]]` without specifying the\n> \"<refname>\" or \"<expect>\" values, can cause the update that happened\n> in-between the checkout and forced push to be lost.\n\nThank you for working on this! I gave this an incomplete look-over, and\noffered some suggestions that you hopefully find useful.\n\n> Changes since v2:\n>   * Removed configuration option for setting \"--force-if-includes\"\n>     with \"--force-with-lease\".\n>   * Broke up the patch into smaller commits.\n\nWhile the commits all seem to be compiling individually, I am not really a\nfan of introducing a function without a caller that shows how it is\nsupposed to work. I'd rather see some incremental story, and in this case,\nI think if _I_ were to submit this patch series, I would probably have\nonly two commits: one that extends the already-existing code path to turn\nthat `use_tracking` flag into that `enum`, and the second patch which\nwires up the option, adds the documentation and the tests.\n\nHowever, please do not let my tastes dictate how you want to present the\nwork, although I hope that my suggestion inspires you ;-)\n\nCiao,\nDscho\n"},{"id":"405695","messageId":"nycvar.QRO.7.76.6.2009161221210.56@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"20200908165900.GC40807@mail.clickyotomy.dev","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-16T11:55:02Z","receivedAt":"2020-09-16T21:09:48Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Srinidhi,\n\nOn Tue, 8 Sep 2020, Srinidhi Kaushik wrote:\n\n> On 09/07/2020 21:45, Johannes Schindelin wrote:\n> >> [...]\n>\n> > Now, to be honest, I thought that this mode would merit a new option\n> > rather than piggy-backing on top of `--force-with-lease`. The reason is\n> > that `--force-with-lease` targets a slightly different use case than mine:\n> > it makes sure that we do not overwrite remote refs unless we already had a\n> > chance to inspect them.\n> >\n> > In contrast, my workflow uses `git pull --rebase` in two or more separate\n> > worktrees, e.g. when developing a patch on two different Operating\n> > Systems, I frequently forget to pull (to my public repository) on one\n> > side, and I want to avoid force-pushing in that case, even if VS Code (or\n> > I, via `git remote update`) fetched the ref (but failing to rebase the\n> > local branch on top of it).\n> >\n> > However, in other scenarios I very much do _not_ want to incorporate the\n> > remote ref. For example, I often fetch\n> > https://github.com/git-for-windows/git.wiki.git to check for the\n> > occasional bogus change. Whenever I see such a bogus change, and it is at\n> > the tip of the branch, I want to force-push _without_ incorporating the\n> > bogus change into the local branch, yet I _do_ want to use\n> > `--force-with-lease` because an independent change could have come in via\n> > the Wiki in the meantime.\n>\n> I realize that this new check would not be helpful if we deliberately\n> choose not to include an unwanted change from the updated remote's tip.\n> In that case, we would have to use `--force` make it work, and that\n> defeats the use of `--force-with-lease`.\n\nI would characterize it as \"somewhat stricter\" than `--force-with-lease`.\nThe check would not make sense without the lease.\n\n> > So I think that the original `--force-with-lease` and the mode you\n> > implemented target subtly different use cases that are both valid, and\n> > therefore I would like to request a separate option for the latter.\n>\n> OK. So, I am assuming that you are suggesting to add a new function that\n> is separate from `apply_push_cas()` and run the check on each of the\n> remote refs. Would that be correct?\n\nOh, I don't really know the code well enough to make a suggestion. I guess\nif it is easy enough to modify the existing code path (e.g. extend the\nfunction signature in a minimal manner), then that's what I would go for,\nrather than a completely separate code path.\n\n> If that's the case, how does it work along with `--force-with-lease`?\n\nIn my mind, the new mode implies `--force-with-lease`.\n\nThanks,\nDscho\n\n> On one hand we have `--force-with-lease` to ensure we rewrite the remote\n> that we have _already_ seen, and on the other, a new option that checks\n> reflog of the local branch to see if it is missing any updates from the\n> remote that may have happened in the meantime. If we pass both of them\n> for `push` and if the former doesn't complain, and the latter check\n> fails, should the `push` still go through?\n>\n> I feel that this check included with `--force-with-lease` only when\n> the `use_tracking` or `use_tracking_for_rest` options are enabled\n> would give a heads-up the the user about the background fetch. If\n> they decide that they don't need new updates, then supplying the\n> new \"<expect>\" value in the next push would imply they've seen the\n> new update, and choose to overwrite it anyway. The check would not\n> run in this case. But again, I wonder if the this \"two-step\" process\n> makes `push` cumbersome.\n>\n> > However, I have to admit that I could not think of a good name for that\n> > option. \"Implicit fetch\" seems a bit too vague here, because the local\n> > branch was not fetched, and certainly not implicitly, yet the logic\n> > revolves around the local branch having been rebased to the\n> > remote-tracking ref at some stage.\n>\n> The message \"implicit fetch\" was in context of the remote ref. But yes,\n> the current reject reason is not clear and implies that local branch\n> was fetched, which isn't the case.\n>\n> > Even if we went with the config option to modify `--force-with-lease`'s\n> > behavior, I would recommend separating out the `feature.experimental`\n> > changes into their own patch, so that they can be reverted easily in case\n> > the experimental feature is made the default.\n>\n> Good idea!\n>\n> > A couple more comments:\n> >\n> >> @@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n> >>                * If the remote ref has moved and is now different\n> >>                * from what we expect, reject any push.\n> >>                *\n> >> -              * It also is an error if the user told us to check\n> >> -              * with the remote-tracking branch to find the value\n> >> -              * to expect, but we did not have such a tracking\n> >> -              * branch.\n> >> +              * It also is an error if the user told us to check with the\n> >> +              * remote-tracking branch to find the value to expect, but we\n> >> +              * did not have such a tracking branch, or we have one that\n> >> +              * has new changes.\n> >\n> > If I were you, I would try to keep the original formatting, so that it\n> > becomes more obvious that the part \", or we have [...]\" was appended.\n>\n> Alright, I will append the new comment in a new line instead.\n>\n> >\n> >>               if (ref->expect_old_sha1) {\n> >>                       if (!oideq(&ref->old_oid, &ref->old_oid_expect))\n> >>                               reject_reason = REF_STATUS_REJECT_STALE;\n> >> +                     else if (reject_implicit_fetch() && ref->implicit_fetch)\n> >> +                             reject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;\n> >>                       else\n> >> -                             /* If the ref isn't stale then force the update. */\n> >> +                             /*\n> >> +                              * If the ref isn't stale, or there was no\n> >\n> > Should this \"or\" not be an \"and\" instead?\n>\n> D'oh, you are right. It should have been an \"and\".\n>\n> >\n> >> +                              * implicit fetch, force the update.\n> >> +                              */\n> >>                               force_ref_update = 1;\n> >>               }\n> >> [...]\n> >>  static void apply_cas(struct push_cas_option *cas,\n> >>                     struct remote *remote,\n> >>                     struct ref *ref)\n> >>  {\n> >> -     int i;\n> >> +     int i, do_reflog_check = 0;\n> >> +     struct object_id oid;\n> >> +     struct ref *local_ref = get_local_ref(ref->name);\n> >>\n> >>       /* Find an explicit --<option>=<name>[:<value>] entry */\n> >>       for (i = 0; i < cas->nr; i++) {\n> >>               struct push_cas *entry = &cas->entry[i];\n> >>               if (!refname_match(entry->refname, ref->name))\n> >>                       continue;\n> >> +\n> >>               ref->expect_old_sha1 = 1;\n> >>               if (!entry->use_tracking)\n> >>                       oidcpy(&ref->old_oid_expect, &entry->expect);\n> >>               else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n> >>                       oidclr(&ref->old_oid_expect);\n> >> -             return;\n> >> +             else\n> >> +                     do_reflog_check = 1;\n> >> +\n> >> +             goto reflog_check;\n> >\n> > Hmm. I do not condemn `goto` statements in general, but this one makes the\n> > flow harder to follow. I would prefer something like this:\n> >\n> > -- snip --\n> >               else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n> >                       oidclr(&ref->old_oid_expect);\n> > +             else if (local_ref && !read_ref(local_ref->name, &oid))\n> > +                     ref->implicit_fetch =\n> > +                             !remote_ref_in_reflog(&ref->old_oid, &oid,\n> > +                                                   local_ref->name);\n> >               return;\n> > -- snap --\n>\n> Adding this condition looks cleaner instead of the `goto`. A similar\n> suggestion was made in the other thread [1] as well; this will be\n> addressed in v2.\n>\n> > Again, thank you so much for working on this!\n>\n> Thanks again, for taking the time to review this.\n>\n> [1]: https://public-inbox.org/git/624d9e35-29b8-4012-a3d6-e9b00a9e4485@gmail.com/\n> --\n> Srinidhi Kaushik\n>\n"},{"id":"405939","messageId":"20200919170114.GA81372@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009161356560.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 1/7] remote: add reflog check for \"force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-19T17:01:14Z","receivedAt":"2020-09-19T17:01:24Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Johannes,\n\nOn 09/16/2020 14:35, Johannes Schindelin wrote:\n\n> > [...]\n> > +\t\tstruct commit *loc = lookup_commit_reference(the_repository,\n> > +\t\t\t\t\t\t\t     n_oid);\n> > +\t\tstruct commit *rem = lookup_commit_reference(the_repository,\n> > +\t\t\t\t\t\t\t     r_oid);\n> > +\t\tret = (loc && rem) ? in_merge_bases(rem, loc) : 0;\n> > +\t}\n> \n> This chooses the strategy of iterating over the reflog just once, and at\n> every step first testing whether the respective reflog entry is identical\n> to the remote-tracking branch tip. Only when they are different do we test\n> whether the remote-tracking branch tip is at least reachable from the\n> reflog entry.\n> \n> Let's assume that our local branch has 20 reflog entries, and the 4th one\n> is identical to the current tip of the remote-tracking branch. Then we\n> tested reachability 3 times. But that test is rather expensive.\n> \n> Therefore, I would have preferred to have a call to\n> `for_each_reflog_ent_reverse()` with a callback function that only returns\n> the `oideq()` result, and only if the return value of that call is 0, I\n> would have wanted to see another call to `for_each_reflog_ent_reverse()`\n> to go through, this time looking for reachability.\n\nOK, you're right about this. Will update check to use the two-step\napproach.\n\n> > [...] \n> > +\tint ret = 0;\n> > +\tstruct commit *r_commit, *l_commit;\n> > +\n> > +\tl_commit = lookup_commit_reference(the_repository, l_oid);\n> > +\tr_commit = lookup_commit_reference(the_repository, r_oid);\n> \n> At this point, we already LOOked up `r_commit`. But we don't pass that to\n> `ref_reachable()` at any point (instead passing only `r_oid`), so we have\n> to perform the lookup again.\n> \n> That's wasteful. Shouldn't we pass `r_commit` directly?\n\nHmm, I suppose we can. Not sure why I did that in the first place. \n\n> With the two-pass strategy I outlined above, the first pass would use\n> `r_oid`, and only when the second pass is necessary would we resort to\n> calling the expensive reachability check.\n> \n> > +\n> > +\t/*\n> > +\t * If the remote-tracking ref is an ancestor of the local\n> > +\t * ref (a merge, for instance) there is no need to iterate\n> > +\t * through the reflog entries to ensure reachability; it\n> > +\t * can be skipped to return early instead.\n> > +\t */\n> > +\tret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;\n> \n> Correct me if I am wrong, but isn't the first reflog entry\n> (`<remote-branch>@{0}`) identical to `r_commit`? In that case, the first\n> iteration of the second pass over the reflog would trivially perform this\n> check, and we do not need to duplicate the logic here.\n\nThat's a correct assumption; the second pass will check for this.\n\n> > +\tif (!ret)\n> > +\t\tret = for_each_reflog_ent_reverse(local_ref_name, ref_reachable,\n> > +\t\t\t\t\t\t  (struct object_id *)r_oid);\n> > +\n> > +\treturn ret;\n> > +}\n> > +\n> > +/*\n> > + * Check for reachability of a remote-tracking\n> > + * ref in the reflog entries of its local ref.\n> > + */\n> > +void check_reflog_for_ref(struct ref *r_ref)\n> > +{\n> > +\tstruct object_id r_oid;\n> > +\tstruct ref *l_ref = get_local_ref(r_ref->name);\n> > +\n> > +\tif (r_ref->if_includes && l_ref && !read_ref(l_ref->name, &r_oid))\n> \n> If `r_ref->if_includes` is 0, we do not even have to get the local ref,\n> correct? It would make `check_reflog_for_ref()` much easier to read for me\n> if it was only called when that flag was already verified to be 1, and\n> then followed this structure:\n> \n> \tif (!l_ref)\n> \t\treturn;\n> \tif (read_ref(...))\n> \t\twarning(_(\"ignoring stale remote branch information ...\"));\n> \telse\n> \t\tr_ref->unreachable = ...\n> \n> Also, it might make a lot more sense to rename `check_reflog_for_ref()` to\n> `check_if_includes_upstream()`, and to rename `r_ref` to `local` and\n> `l_ref` to `remote_tracking` or something like that: nothing is inherently\n> \"left\" or \"right\" about those refs.\n\nNow that I think about it, we don't need te call to \"read_ref()\" at all\nbecause the reflog will have the OID we need for checking. As to\n\"{l,r}_ref\", they were meant to be [l]ocal and [r]emote. :)\n\n> > +\t\tr_ref->unreachable = !ref_reachable_from_reflog(&r_ref->old_oid,\n> > +\t\t\t\t\t\t\t\t&r_oid,\n> > +\t\t\t\t\t\t\t\tl_ref->name);\n> > +}\n> > +\n> >  static void apply_cas(struct push_cas_option *cas,\n> >  \t\t      struct remote *remote,\n> >  \t\t      struct ref *ref)\n> >  {\n> > -\tint i;\n> > +\tint i, is_tracking = 0;\n> >\n> >  \t/* Find an explicit --<option>=<name>[:<value>] entry */\n> >  \tfor (i = 0; i < cas->nr; i++) {\n> > @@ -2288,16 +2381,26 @@ static void apply_cas(struct push_cas_option *cas,\n> >  \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n> >  \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n> >  \t\t\toidclr(&ref->old_oid_expect);\n> > -\t\treturn;\n> > +\t\telse\n> > +\t\t\tis_tracking = 1;\n> \n> As part of `remote_tracking()`, we already looked up the branch name.\n> Since we need it in the `is_tracking` case, maybe this should not be a\n> Boolean anymore but store a copy of the remote-tracking branch name\n> instead?\n> \n> Oh, following the code path all the way down to\n> `match_name_with_pattern()`, it seems that `remote_tracking()`'s `dst`\n> variable _already_ contains a copy (which means that that memory is\n> leaked, right?).\n\nI'm not sure I understand what you mean. The call to \"remote_tracking()\"\ngets the OID of the remote ref and writes it to \"old_oid_expect\".\nWe don't need \"dst\" from there because \"old_oid\" is sufficient for\nchecking the reflog entries. However, calling \"get_local_ref()\" is\nnecessary to get the local branch name. Also, why does \"is_tracking\"\nhave to be a Boolean? We already have \"ref->name\", right?\n\nAnyway, I think we can get rid of \"is_tracking\" altogether and just use\n\"if_includes\".\n\n> > [...] \n> > +/*\n> > + * Runs when \"--force-if-includes\" is specified.\n> > + * Checks if the remote-tracking ref was updated (since checkout)\n> > + * implicitly in the background and verify that changes from the\n> > + * updated tip have been integrated locally, before pushing.\n> > + */\n> > +void apply_push_force_if_includes(struct ref*, int);\n> \n> This function is not even hooked up in this patch, right? I don't think\n> that it makes sense to introduce it without a caller, in particular since\n> it makes it harder to guess what those parameters might be used for.\n> \n> In general, it appears to me as if the code worked way too hard to\n> accomplish something that should be a lot simpler: when\n> `--force-if-includes` is passed, it should piggy-back on top of the\n> `--force-with-lease` code path, and just add yet another check on top.\n\nWell, it isn't included in this commit, because it was called from\n\"transport.c\" and \"send-pack.c\", I decided to put that in another\ncommit. I will fix that in the next patch series.\n\n> With that in mind, I would have expected something more in line with this:\n> \n> -- snip --\n>  struct push_cas_option {\n>  \tunsigned use_tracking_for_rest:1;\n>  \tstruct push_cas {\n>  \t\tstruct object_id expect;\n> -\t\tunsigned use_tracking:1;\n> +\t\tenum {\n> +\t\t\tPUSH_NO_CAS = 0,\n> +\t\t\tPUSH_CAS_USE_TRACKING,\n> +\t\t\tPUSH_CAS_IF_INCLUDED\n> +\t\t} mode;\n>  \t\tchar *refname;\n>  \t} *entry;\n>  \tint nr;\n>  \tint alloc;\n>  };\n> -- snap --\n> \n> and then adjusting the respective code paths accordingly.\n\nOK, I will clean it up and get rid of the redundant/wasteful\ncalls and data being passed around. Regarding the new \"enum\",\nI feel that just having a new bit-field \"use_force_if_includes\"\nmight be sufficient for \"push_cas_option\". The idea is to set it\nto \"1\" when \"--force-if-includes\" is specified along with\n\"--force-with-lease\", and \"apply_push_cas()\" will run the check\nand set \"ref->unreachable\" depending on the check status.\nTo clarify, you would have to specify something like this to\nenable the check:\n\n   git push --force-if-includes --force-with-lease=master [...]\n\nThen, having a configuration setting \"push.useForceIfIncludes\" would\nbe the same as running \"git-push\" like mentioned above. This new option\nwill be a \"no-op\" if specified otherwise. Would that be acceptable?\n\nThanks again, for a thorough review. \n-- \nSrinidhi Kaushik\n"},{"id":"405940","messageId":"20200919170316.5310-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200912150459.8282-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v4 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-19T17:03:13Z","receivedAt":"2020-09-19T17:03:41Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new option: \"--force-if-includes\" to \"git-push\" where forced\nupdates are allowed only if the tip of the remote-tracking ref has\nbeen integrated locally, by verifying if the tip of the remote-tracking\nref -- on which a local branch has based on -- is reachable from at\nleast one of the \"reflog\" entries of the branch about to be updated\nby force on the remote.\n\nThis option can be used with \"--force-with-lease\" with setups where\nthe remote-tracking refs of the repository are implicitly updated in\nthe background to help prevent unintended remote overwrites.\n\nIf a local branch is based on a remote ref for a rewrite, and if that\nremote-tracking ref is updated by a push from another repository after\nit has been checked out locally, force updating that branch to remote\nwith \"--force-with-lease[=<refname>[:<expect>]]\" without specifying\nthe \"<expect>\" value, can cause the update that happened in-between the\ncheckout and forced push to be lost.\n\nChanges since v3:\n  - Switch to the two-step approach: check the \"reflog\" the first\n    time to see if the remote ref exists; if not, go through it one\n    more time to see if it is reachable (using \"in_merge_bases()\")\n    from any of the entries. This is a necessary step as explained\n    here [1].\n\n  - Added back the configuration setting \"push.useForceIfIncludes\", now\n    that it does not affect the functionality of \"--force-with-lease\"\n    when the expected ref is specified.\n\n  - Update \"remote-curl\" and \"send-pack\" to recognize the new option;\n    this was missing in the previous patches.\n\n  - Broke up the patch-set into medium sized commits.\n\n  - Clean-up unnecessary function calls.\n\nSrinidhi Kaushik (3):\n  push: add reflog check for \"--force-if-includes\"\n  push: parse and set flag for \"--force-if-includes\"\n  t, doc: update tests, reference for \"--force-if-includes\"\n\n Documentation/config/advice.txt |  9 ++-\n Documentation/config/push.txt   |  5 ++\n Documentation/git-push.txt      | 20 +++++++\n advice.c                        |  3 +\n advice.h                        |  2 +\n builtin/push.c                  | 27 +++++++++\n builtin/send-pack.c             | 11 ++++\n remote-curl.c                   | 14 ++++-\n remote.c                        | 97 ++++++++++++++++++++++++++++++++-\n remote.h                        | 12 +++-\n send-pack.c                     |  1 +\n t/t5533-push-cas.sh             | 53 ++++++++++++++++++\n transport-helper.c              | 11 ++++\n transport.c                     |  8 +++\n transport.h                     | 15 +++--\n 15 files changed, 276 insertions(+), 12 deletions(-)\n\nbase-commit: 385c171a018f2747b329bcfa6be8eda1709e5abd\n\n[1]: https://public-inbox.org/git/nycvar.QRO.7.76.6.2009161214030.56@tvgsbejvaqbjf.bet/\n\n--\n2.28.0\n"},{"id":"405941","messageId":"20200919170316.5310-2-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200919170316.5310-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-19T17:03:14Z","receivedAt":"2020-09-19T17:03:47Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Adds a check to verify if the remote-tracking ref of the local branch\nis reachable from one of its \"reflog\" entries.\n\nWhen a local branch that is based on a remote ref, has been rewound\nand is to be force pushed on the remote, \"--force-if-includes\" runs\na check that ensures any updates to remote-tracking refs that may have\nhappened (by push from another repository) in-between the time of the\nlast checkout, and right before the time of push, have been integrated\nlocally before allowing a forced updated.\n\nA new field \"use_force_if_includes\" has been added to \"push_cas_option\",\nwhich is set to \"1\" when \"--force-if-includes\" is specified as an\nargument in the command line or set as a configuration option.\n\nThe struct \"ref\" has two new bit-fields:\n  - if_includes:\n    Set when we have to run the new check on the ref, and the remote\n    ref was marked as \"use_tracking\" or \"use_tracking_for_rest\" by\n    compare-and-swap (if the \"the remote tip must be at the expected\n    commit\" condition is not specified); \"apply_push_cas()\" has been\n    updated to check if this field is set and run the check.\n\n  - unreachable:\n    Set if the ref is unreachable from any of the \"reflog\" entries of\n    its local counterpart.\n\n\"REF_STATUS_REJECT_REMOTE_UPDATED\" has been added to the \"status\"\nenum to imply that the ref failed the check; \"case\" statements in\n\"send-pack\", \"transport\" and \"transport-helper\" have been updated\naccordingly to catch this status when set.\n\nWhen \"--force-with-includes\" is used along with \"--force-with-lease\",\nthe check runs only for refs marked as \"if_includes\". If the option\nis passed without specifying \"--force-with-lease\", or specified along\nwith \"--force-with-lease=<refname>:<expect>\" it is a \"no-op\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n builtin/send-pack.c |  5 +++\n remote.c            | 92 ++++++++++++++++++++++++++++++++++++++++++++-\n remote.h            |  6 ++-\n send-pack.c         |  1 +\n transport-helper.c  |  5 +++\n transport.c         |  6 +++\n 6 files changed, 112 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 2b9610f121..4d76727edb 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n \n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"remote ref updated since checkout\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\ndiff --git a/remote.c b/remote.c\nindex eafc14cbe7..60d681a885 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t * with the remote-tracking branch to find the value\n \t\t * to expect, but we did not have such a tracking\n \t\t * branch.\n+\t\t *\n+\t\t * If the tip of the remote-tracking ref is unreachable\n+\t\t * from any reflog entry of its local ref indicating a\n+\t\t * possible update since checkout; reject the push.\n \t\t */\n \t\tif (ref->expect_old_sha1) {\n \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n+\t\t\telse if (ref->if_includes && ref->unreachable)\n+\t\t\t\treject_reason =\n+\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n \t\t\telse\n-\t\t\t\t/* If the ref isn't stale then force the update. */\n+\t\t\t\t/*\n+\t\t\t\t * If the ref isn't stale, and is reachable\n+\t\t\t\t * from from one of the reflog entries of\n+\t\t\t\t * the local branch, force the update.\n+\t\t\t\t */\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n \n@@ -2268,6 +2279,70 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \treturn 0;\n }\n \n+/* Checks if the ref exists in the reflog entry. */\n+static int reflog_entry_exists(struct object_id *o_oid,\n+\t\t\t\t  struct object_id *n_oid,\n+\t\t\t\t  const char *ident, timestamp_t timestamp,\n+\t\t\t\t  int tz, const char *message, void *cb_data)\n+{\n+\tstruct object_id *remote_oid = cb_data;\n+\treturn oideq(n_oid, remote_oid);\n+}\n+\n+/* Checks if the ref is reachable from the reflog entry. */\n+static int reflog_entry_reachable(struct object_id *o_oid,\n+\t\t\t       struct object_id *n_oid,\n+\t\t\t       const char *ident, timestamp_t timestamp,\n+\t\t\t       int tz, const char *message, void *cb_data)\n+{\n+\tstruct commit *local_commit;\n+\tstruct commit *remote_commit = cb_data;\n+\n+\tlocal_commit = lookup_commit_reference(the_repository, n_oid);\n+\tif (local_commit)\n+\t\treturn in_merge_bases(remote_commit, local_commit);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Iterate through he reflog entries of the local branch to check\n+ * if the remote-tracking ref exists in on of the entries; if not,\n+ * go through the entries once more, but this time check if the\n+ * remote-tracking ref is reachable from any of the entries.\n+ */\n+static int is_reachable_in_reflog(const char *local_ref_name,\n+\t\t\t\t  const struct object_id *remote_oid)\n+{\n+\tstruct commit *remote_commit;\n+\n+\tif (for_each_reflog_ent_reverse(local_ref_name, reflog_entry_exists,\n+\t\t\t\t\t(struct object_id *)remote_oid))\n+\t\treturn 1;\n+\n+\tremote_commit = lookup_commit_reference(the_repository, remote_oid);\n+\tif (remote_commit)\n+\t\treturn for_each_reflog_ent_reverse(local_ref_name,\n+\t\t\t\t\t\t   reflog_entry_reachable,\n+\t\t\t\t\t\t   remote_commit);\n+\treturn 0;\n+}\n+\n+/*\n+ * Check for reachability of a remote-tracking\n+ * ref in the reflog entries of its local ref.\n+ */\n+static void check_if_includes_upstream(struct ref *remote_ref)\n+{\n+\tstruct ref *local_ref = get_local_ref(remote_ref->name);\n+\n+\tif (!local_ref)\n+\t\treturn;\n+\n+\tif (!is_reachable_in_reflog(local_ref->name, &remote_ref->old_oid))\n+\t\tremote_ref->unreachable = 1;\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n \t\t      struct ref *ref)\n@@ -2284,6 +2359,8 @@ static void apply_cas(struct push_cas_option *cas,\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n \t\t\toidclr(&ref->old_oid_expect);\n+\t\telse\n+\t\t\tref->if_includes = cas->use_force_if_includes;\n \t\treturn;\n \t}\n \n@@ -2294,6 +2371,8 @@ static void apply_cas(struct push_cas_option *cas,\n \tref->expect_old_sha1 = 1;\n \tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n \t\toidclr(&ref->old_oid_expect);\n+\telse\n+\t\tref->if_includes = cas->use_force_if_includes;\n }\n \n void apply_push_cas(struct push_cas_option *cas,\n@@ -2301,6 +2380,15 @@ void apply_push_cas(struct push_cas_option *cas,\n \t\t    struct ref *remote_refs)\n {\n \tstruct ref *ref;\n-\tfor (ref = remote_refs; ref; ref = ref->next)\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n \t\tapply_cas(cas, remote, ref);\n+\n+\t\t/*\n+\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n+\t\t * mode, and if \"--foce-if-includes\" was specified, run\n+\t\t * the check.\n+\t\t */\n+\t\tif (ref->if_includes)\n+\t\t\tcheck_if_includes_upstream(ref);\n+\t}\n }\ndiff --git a/remote.h b/remote.h\nindex 5e3ea5a26d..38ab8539e2 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -104,7 +104,9 @@ struct ref {\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\tif_includes:1, /* If \"--force-with-includes\" was specified. */\n+\t\tunreachable:1; /* For \"if_includes\"; unreachable in reflog. */\n \n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -134,6 +136,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n \t\tREF_STATUS_REJECT_SHALLOW,\n+\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT,\n@@ -332,6 +335,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);\n \n struct push_cas_option {\n \tunsigned use_tracking_for_rest:1;\n+\tunsigned use_force_if_includes:1;\n \tstruct push_cas {\n \t\tstruct object_id expect;\n \t\tunsigned use_tracking:1;\ndiff --git a/send-pack.c b/send-pack.c\nindex 632f1580ca..956306e8e8 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c52c99d829..e547e21199 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"remote ref updated since checkout\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\n \t\t\t\tstring_list_clear(&cas_options, 0);\ndiff --git a/transport.c b/transport.c\nindex 43e24bf1e5..99fe6233a3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"stale info\", porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"remote ref updated since checkout\",\n+\t\t\t\t porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -1101,6 +1106,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n \n \t\tstrbuf_reset(&buf);\n-- \n2.28.0\n\n"},{"id":"405942","messageId":"20200919170316.5310-3-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200919170316.5310-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v4 2/3] push: parse and set flag for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-19T17:03:15Z","receivedAt":"2020-09-19T17:03:50Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Adds a flag: \"TRANSPORT_PUSH_FORCE_IF_INCLUDES\" to indicate that the new\noption was passed from the command line of via configuration settings;\nupdate command line and configuration parsers to set the new flag\naccordingly.\n\nIntroduces a new configuration option \"push.useForceIfIncludes\", which\nis equivalent to setting \"--force-if-includes\" in the command line.\n\nUpdates \"remote-curl\" to recognize and pass this option to \"send-pack\"\nwhen enabled.\n\nUpdates \"advise\" to catch the reject reason \"REJECT_REF_NEEDS_UPDATE\",\nwhich is set when the ref status is \"REF_STATUS_REJECT_REMOTE_UPDATED\"\nand (optionally) print a help message when the push fails.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n advice.c            |  3 +++\n advice.h            |  2 ++\n builtin/push.c      | 27 +++++++++++++++++++++++++++\n builtin/send-pack.c |  6 ++++++\n remote-curl.c       | 14 +++++++++++++-\n remote.c            |  5 +++++\n remote.h            |  6 ++++++\n transport-helper.c  |  6 ++++++\n transport.c         |  2 ++\n transport.h         | 15 ++++++++++-----\n 10 files changed, 80 insertions(+), 6 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex f0a3d32d20..164742305f 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;\n int advice_push_fetch_first = 1;\n int advice_push_needs_force = 1;\n int advice_push_unqualified_ref_name = 1;\n+int advice_push_ref_needs_update = 1;\n int advice_status_hints = 1;\n int advice_status_u_option = 1;\n int advice_status_ahead_behind_warning = 1;\n@@ -72,6 +73,7 @@ static struct {\n \t{ \"pushFetchFirst\", &advice_push_fetch_first },\n \t{ \"pushNeedsForce\", &advice_push_needs_force },\n \t{ \"pushUnqualifiedRefName\", &advice_push_unqualified_ref_name },\n+\t{ \"pushRefNeedsUpdate\", &advice_push_ref_needs_update },\n \t{ \"statusHints\", &advice_status_hints },\n \t{ \"statusUoption\", &advice_status_u_option },\n \t{ \"statusAheadBehindWarning\", &advice_status_ahead_behind_warning },\n@@ -116,6 +118,7 @@ static struct {\n \t[ADVICE_PUSH_ALREADY_EXISTS]\t\t\t= { \"pushAlreadyExists\", 1 },\n \t[ADVICE_PUSH_FETCH_FIRST]\t\t\t= { \"pushFetchFirst\", 1 },\n \t[ADVICE_PUSH_NEEDS_FORCE]\t\t\t= { \"pushNeedsForce\", 1 },\n+\t[ADVICE_PUSH_REF_NEEDS_UPDATE]\t\t\t= { \"pushRefNeedsUpdate\", 1 },\n \n \t/* make this an alias for backward compatibility */\n \t[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]\t\t= { \"pushNonFastForward\", 1 },\ndiff --git a/advice.h b/advice.h\nindex 16f2c11642..bc2432980a 100644\n--- a/advice.h\n+++ b/advice.h\n@@ -11,6 +11,7 @@ extern int advice_push_already_exists;\n extern int advice_push_fetch_first;\n extern int advice_push_needs_force;\n extern int advice_push_unqualified_ref_name;\n+extern int advice_push_ref_needs_update;\n extern int advice_status_hints;\n extern int advice_status_u_option;\n extern int advice_status_ahead_behind_warning;\n@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;\n \tADVICE_PUSH_UNQUALIFIED_REF_NAME,\n \tADVICE_PUSH_UPDATE_REJECTED_ALIAS,\n \tADVICE_PUSH_UPDATE_REJECTED,\n+\tADVICE_PUSH_REF_NEEDS_UPDATE,\n \tADVICE_RESET_QUIET_WARNING,\n \tADVICE_RESOLVE_CONFLICT,\n \tADVICE_RM_HINTS,\ndiff --git a/builtin/push.c b/builtin/push.c\nindex 0eeb2c8dd5..c007b19360 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =\n \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n \t   \"without using the '--force' option.\\n\");\n \n+static const char message_advice_ref_needs_update[] =\n+\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n+\t   \"branch has been updated since the last checkout. You may want\\n\"\n+\t   \"to integrate those changes locally (e.g., 'git pull ...')\\n\"\n+\t   \"before forcing an update.\\n\");\n+\n static void advise_pull_before_push(void)\n {\n \tif (!advice_push_non_ff_current || !advice_push_update_rejected)\n@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)\n \tadvise(_(message_advice_ref_needs_force));\n }\n \n+static void advise_ref_needs_update(void)\n+{\n+\tif (!advice_push_ref_needs_update || !advice_push_update_rejected)\n+\t\treturn;\n+\tadvise(_(message_advice_ref_needs_update));\n+}\n+\n static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\t\t     int flags)\n {\n@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\tadvise_ref_fetch_first();\n \t} else if (reject_reasons & REJECT_NEEDS_FORCE) {\n \t\tadvise_ref_needs_force();\n+\t} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {\n+\t\tadvise_ref_needs_update();\n \t}\n \n \treturn 1;\n@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\tif (!v)\n \t\t\treturn config_error_nonbool(k);\n \t\treturn color_parse(v, push_colors[slot]);\n+\t} else if (!strcmp(k, \"push.useforceifincludes\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\t*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\telse\n+\t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\treturn 0;\n \t}\n \n \treturn git_default_config(k, v, NULL);\n@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t\t       N_(\"require old value of ref to be at this value\"),\n \t\t\t       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),\n+\t\tOPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,\n+\t\t\tN_(\"require remote updates to be integrated locally\"),\n+\t\t\tTRANSPORT_PUSH_FORCE_IF_INCLUDES),\n \t\tOPT_CALLBACK(0, \"recurse-submodules\", &recurse_submodules, \"(check|on-demand|no)\",\n \t\t\t     N_(\"control recursive pushing of submodules\"), option_parse_recurse_submodules),\n \t\tOPT_BOOL_F( 0 , \"thin\", &thin, N_(\"use thin pack\"), PARSE_OPT_NOCOMPLETE),\n@@ -593,6 +617,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tset_refspecs(argv + 1, argc - 1, repo);\n \t}\n \n+\tif (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)\n+\t\tpush_set_force_if_includes(&cas);\n+\n \tremote = pushremote_get(repo);\n \tif (!remote) {\n \t\tif (repo)\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 4d76727edb..9289c0eecb 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -159,6 +159,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tunsigned int force_if_includes = 0;\n \tstruct packet_reader reader;\n \n \tstruct option options[] = {\n@@ -184,6 +185,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t  N_(\"require old value of ref to be at this value\"),\n \t\t  PARSE_OPT_OPTARG, parseopt_push_cas_option),\n+\t\tOPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,\n+\t\t\t N_(\"require remote updates to be integrated locally\")),\n \t\tOPT_END()\n \t};\n \n@@ -282,6 +285,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tif (match_push_refs(local_refs, &remote_refs, &rs, flags))\n \t\treturn -1;\n \n+\tif (force_if_includes)\n+\t\tpush_set_force_if_includes(&cas);\n+\n \tif (!is_empty_cas(&cas))\n \t\tapply_push_cas(&cas, remote, remote_refs);\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 32cc4a0c55..0290b04891 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -44,7 +44,8 @@ struct options {\n \t\tfrom_promisor : 1,\n \n \t\tatomic : 1,\n-\t\tobject_format : 1;\n+\t\tobject_format : 1,\n+\t\tforce_if_includes : 1;\n \tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)\n \t\tstring_list_append(&cas_options, val.buf);\n \t\tstrbuf_release(&val);\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.force_if_includes = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.force_if_includes = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n \t} else if (!strcmp(name, \"cloning\")) {\n \t\tif (!strcmp(value, \"true\"))\n \t\t\toptions.cloning = 1;\n@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)\n \t\tstrvec_push(&args, cas_option->string);\n \tstrvec_push(&args, url.buf);\n \n+\tif (options.force_if_includes)\n+\t\tstrvec_push(&args, \"--force-if-includes\");\n+\n \tstrvec_push(&args, \"--stdin\");\n \tfor (i = 0; i < nr_spec; i++)\n \t\tpacket_buf_write(&preamble, \"%s\\n\", specs[i]);\ndiff --git a/remote.c b/remote.c\nindex 60d681a885..7679be9213 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -2255,6 +2255,11 @@ int parseopt_push_cas_option(const struct option *opt, const char *arg, int unse\n \treturn parse_push_cas_option(opt->value, arg, unset);\n }\n \n+void push_set_force_if_includes(struct push_cas_option *cas)\n+{\n+\tcas->use_force_if_includes = 1;\n+}\n+\n int is_empty_cas(const struct push_cas_option *cas)\n {\n \treturn !cas->use_tracking_for_rest && !cas->nr;\ndiff --git a/remote.h b/remote.h\nindex 38ab8539e2..72c374d539 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -350,4 +350,10 @@ int parseopt_push_cas_option(const struct option *, const char *arg, int unset);\n int is_empty_cas(const struct push_cas_option *);\n void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);\n \n+/*\n+ * Sets \"use_force_if_includes\" for \"compare-and-swap\"\n+ * when \"--force-if-includes\" is specified.\n+ */\n+void push_set_force_if_includes(struct push_cas_option *);\n+\n #endif\ndiff --git a/transport-helper.c b/transport-helper.c\nindex e547e21199..2a4436dd79 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -868,6 +868,12 @@ static void set_common_push_options(struct transport *transport,\n \t\tif (set_helper_option(transport, TRANS_OPT_ATOMIC, \"true\") != 0)\n \t\t\tdie(_(\"helper %s does not support --atomic\"), name);\n \n+\t/* If called with \"--force-if-includes\". */\n+\tif (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)\n+\t\tif (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, \"true\") != 0)\n+\t\t\tdie(_(\"helper %s does not support --%s\"),\n+\t\t\t    name, TRANS_OPT_FORCE_IF_INCLUDES);\n+\n \tif (flags & TRANSPORT_PUSH_OPTIONS) {\n \t\tstruct string_list_item *item;\n \t\tfor_each_string_list_item(item, transport->push_options)\ndiff --git a/transport.c b/transport.c\nindex 99fe6233a3..da98ebf50e 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -664,6 +664,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t\t*reject_reasons |= REJECT_FETCH_FIRST;\n \t\t} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {\n \t\t\t*reject_reasons |= REJECT_NEEDS_FORCE;\n+\t\t} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {\n+\t\t\t*reject_reasons |= REJECT_REF_NEEDS_UPDATE;\n \t\t}\n \t}\n \tfree(head);\ndiff --git a/transport.h b/transport.h\nindex ca409ea1e4..24558c027d 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -136,6 +136,7 @@ struct transport {\n #define TRANSPORT_PUSH_ATOMIC\t\t\t(1<<13)\n #define TRANSPORT_PUSH_OPTIONS\t\t\t(1<<14)\n #define TRANSPORT_RECURSE_SUBMODULES_ONLY\t(1<<15)\n+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES\t(1<<16)\n \n int transport_summary_width(const struct ref *refs);\n \n@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);\n /* Request atomic (all-or-nothing) updates when pushing */\n #define TRANS_OPT_ATOMIC \"atomic\"\n \n+/* Require remote changes to be integrated locally. */\n+#define TRANS_OPT_FORCE_IF_INCLUDES \"force-if-includes\"\n+\n /**\n  * Returns 0 if the option was used, non-zero otherwise. Prints a\n  * message to stderr if the option is not used.\n@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,\n void transport_set_verbosity(struct transport *transport, int verbosity,\n \tint force_progress);\n \n-#define REJECT_NON_FF_HEAD     0x01\n-#define REJECT_NON_FF_OTHER    0x02\n-#define REJECT_ALREADY_EXISTS  0x04\n-#define REJECT_FETCH_FIRST     0x08\n-#define REJECT_NEEDS_FORCE     0x10\n+#define REJECT_NON_FF_HEAD      0x01\n+#define REJECT_NON_FF_OTHER     0x02\n+#define REJECT_ALREADY_EXISTS   0x04\n+#define REJECT_FETCH_FIRST      0x08\n+#define REJECT_NEEDS_FORCE      0x10\n+#define REJECT_REF_NEEDS_UPDATE 0x20\n \n int transport_push(struct repository *repo,\n \t\t   struct transport *connection,\n-- \n2.28.0\n\n"},{"id":"405943","messageId":"20200919170316.5310-4-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200919170316.5310-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v4 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-19T17:03:16Z","receivedAt":"2020-09-19T17:03:53Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"* t/t5533-push-cas.sh:\n    Updates test cases for \"compare-and-swap\" when used along with\n    \"--force-if-includes\" helps mitigate overwrites when remote\n    ref are updated in the background.\n\n* Documentation:\n    Adds reference for the new option, configuration setting\n    (\"push.useForceIfIncludes\") and advise messages.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/advice.txt |  9 ++++--\n Documentation/config/push.txt   |  5 ++++\n Documentation/git-push.txt      | 20 +++++++++++++\n t/t5533-push-cas.sh             | 53 +++++++++++++++++++++++++++++++++\n 4 files changed, 84 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\nindex bdd37c3eaa..acbd0c09aa 100644\n--- a/Documentation/config/advice.txt\n+++ b/Documentation/config/advice.txt\n@@ -10,9 +10,8 @@ advice.*::\n \t\tthat the check is disabled.\n \tpushUpdateRejected::\n \t\tSet this variable to 'false' if you want to disable\n-\t\t'pushNonFFCurrent',\n-\t\t'pushNonFFMatching', 'pushAlreadyExists',\n-\t\t'pushFetchFirst', and 'pushNeedsForce'\n+\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n+\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n \t\tsimultaneously.\n \tpushNonFFCurrent::\n \t\tAdvice shown when linkgit:git-push[1] fails due to a\n@@ -41,6 +40,10 @@ advice.*::\n \t\twe can still suggest that the user push to either\n \t\trefs/heads/* or refs/tags/* based on the type of the\n \t\tsource object.\n+\tpushRefNeedsUpdate::\n+\t\tShown when linkgit:git-push[1] rejects a forced update of\n+\t\ta branch when its remote-tracking ref has updates that we\n+\t\tdo not have locally.\n \tstatusAheadBehind::\n \t\tShown when linkgit:git-status[1] computes the ahead/behind\n \t\tcounts for a local ref compared to its remote tracking ref,\ndiff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\nindex f5e5b38c68..6e57a27608 100644\n--- a/Documentation/config/push.txt\n+++ b/Documentation/config/push.txt\n@@ -114,3 +114,8 @@ push.recurseSubmodules::\n \tspecifying '--recurse-submodules=check|on-demand|no'.\n \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n \tset (in which case a 'true' value means 'on-demand').\n+\n+push.useForceIfIncludes::\n+\tIf set to \"true\", it is equivalent to specifying \"--force-if-includes\"\n+\tas an argument to linkgit:git-push[1]. Adding \"--no-force-if-includes\"\n+\tthe time of push overrides this configuration setting.\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..dc1215a5ad 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -320,6 +320,13 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, specifying \"--force-if-includes\" an an ancillary option along\n+with \"--force-with-lease[=<refname>]\" (i.e., without saying what exact commit\n+the ref on the remote side must be pointing at, or which refs on the remote\n+side are being protected) at the time of \"push\" will verify if updates from the\n+remote-tracking refs that may have been implicitly updated in the background\n+are integrated locally before allowing a forced update.\n \n -f::\n --force::\n@@ -341,6 +348,19 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n origin +master` to force a push to the `master` branch). See the\n `<refspec>...` section above for details.\n \n+--[no-]force-if-includes::\n+\tForce an update only if the tip of the remote-tracking ref\n+\thas been integrated locally.\n++\n+This option verifies if the tip of the remote-tracking ref on which\n+a local branch has based on (for a rewrite), is reachable from at\n+least one of the \"reflog\" entries of the local branch about to be\n+updated by force on the remote. The check ensures that any updates\n+from the remote have been incorporated locally by rejecting a push\n+if that is not the case.\n++\n+Specifying \"--no-force-if-includes\" disables this behavior.\n+\n --repo=<repository>::\n \tThis option is equivalent to the <repository> argument. If both\n \tare specified, the command-line argument takes precedence.\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..60dd7d2f1c 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -256,4 +256,57 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n \n+test_expect_success 'background updates of REMOTE can be mitigated with \"--force-if-includes\"' '\n+\trm -rf src dst &&\n+\tgit init --bare src.bare &&\n+\ttest_when_finished \"rm -rf src.bare\" &&\n+\tgit clone --no-local src.bare dst &&\n+\ttest_when_finished \"rm -rf dst\" &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit G &&\n+\t\tgit push origin master:master\n+\t) &&\n+\tgit clone --no-local src.bare dst2 &&\n+\ttest_when_finished \"rm -rf dst2\" &&\n+\t(\n+\t\tcd dst2 &&\n+\t\ttest_commit H &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit I &&\n+\t\tgit fetch origin &&\n+\t\ttest_must_fail git push --force-with-lease --force-if-includes origin\n+\t)\n+'\n+\n+test_expect_success 'background updates of REMOTE can be mitigated with \"push.useForceIfIncludes\"' '\n+\trm -rf src dst &&\n+\tgit init --bare src.bare &&\n+\ttest_when_finished \"rm -rf src.bare\" &&\n+\tgit clone --no-local src.bare dst &&\n+\ttest_when_finished \"rm -rf dst\" &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit G &&\n+\t\tgit push origin master:master\n+\t) &&\n+\tgit clone --no-local src.bare dst2 &&\n+\ttest_when_finished \"rm -rf dst2\" &&\n+\t(\n+\t\tcd dst2 &&\n+\t\ttest_commit H &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit I &&\n+\t\tgit fetch origin &&\n+\t\tgit config --local push.useForceIfIncludes \"true\" &&\n+\t\ttest_must_fail git push --force-if-includes origin\n+\t)\n+'\n+\n test_done\n-- \n2.28.0\n\n"},{"id":"405944","messageId":"xmqqsgbdk69b.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2009161214030.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-19T17:48:00Z","receivedAt":"2020-09-19T17:48:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> No, I don't think that would suffice. For several reasons:\n>\n> - the user could have checked out the remote-tracking branch directly\n>   (`git switch --detach origin/my-branch`), as opposed to actually\n>   integrating the revision into the branch at some stage. I know that I do\n>   that sometimes.\n>\n> - even if the remote-tracking branch has not been checked out directly, it\n>   might have been `git pull --rebase`d into a _different_ branch, in which\n>   case the reflog of `HEAD` would say \"yep, I saw this commit!\" but that\n>   would not mean that it was integrated into the branch we want to\n>   (force-)push.\n>\n> - the reflog of the `HEAD` is worktree-specific, and if the user pushes\n>   from a different worktree (e.g. to push multiple branches at the same\n>   time, or because the push is scripted and run from a different working\n>   directory), it would be missed.\n>\n> - and if we take a little step back, we do see that the reflog of `HEAD`\n>   _does_ answer a different question from what we actually asked.\n>\n> Sure, it would be faster, but is that worth the consequences?\n\nIt is not about \"would be faster\" but is trying to be more inclusive\nof different workflows.\n\nOne minor downside consequence of not looking at the reflog of HEAD\nis that it does not make the new option useful for those who could\nbenefit from it.\n\nThey detach HEAD at the tip of the remote-tracking branch, work on\nthe replacement history while on the detached HEAD, and then force\nupdate the local branch from there once they are done (this is a\nvariant to the first one you said you \"do sometimes\").  This is a\nhandy technique to allow them to always compare the previous round\nof their effort (which is untouched while they prepare the new\niteration on detached HEAD) with what they achieved so far on HEAD.\n\nAfter finishing, they would go on to deal with other branches, so\nthe local branch prepared to be force-pushed may no longer be the\ncurrent one.  Without checking with the reflog of HEAD, there is no\nway to notice that the local branch to be force-pushed (which no\nlonger is the current branch) was built using the commit at the tip\nof the remote-tracking branch we are losing, because the branch's\nreflog would have only one entry that records where it was before\nadjusting for the updated remote-tracking branch, and where it is\nnow after rebuilding.\n\nIt is inevitable for a heuristics like the one that was proposed to\nwork only for those with some workflows while excluding those with\nsome other workflows, and that is why I said it is \"minor\" downside\nthat it does not make the feature useful for some people who could\npotentially benefit and also that is why I think it is perfectly OK\nto draw the line there and not support them.  After all, the line\nhas to be drawn somewhere, and the looser the heuristics are, the\nwider the potential audience *and* at the same time the wider the\npotential of false positives.\n\nWhere we drew the line and its consequences must be communicated\nclearly in the documentation, though.\n\nThanks.\n"},{"id":"405950","messageId":"xmqqft7djzz0.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200919170316.5310-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-19T20:03:47Z","receivedAt":"2020-09-19T20:04:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Adds a check to verify if the remote-tracking ref of the local branch\n> is reachable from one of its \"reflog\" entries.\n\ns/Adds/Add/\n\n> When \"--force-with-includes\" is used along with \"--force-with-lease\",\n\nA misspelt name for the new option is found here.\n\n> +/* Checks if the ref is reachable from the reflog entry. */\n> +static int reflog_entry_reachable(struct object_id *o_oid,\n> +\t\t\t       struct object_id *n_oid,\n> +\t\t\t       const char *ident, timestamp_t timestamp,\n> +\t\t\t       int tz, const char *message, void *cb_data)\n> +{\n> +\tstruct commit *local_commit;\n> +\tstruct commit *remote_commit = cb_data;\n> +\n> +\tlocal_commit = lookup_commit_reference(the_repository, n_oid);\n> +\tif (local_commit)\n> +\t\treturn in_merge_bases(remote_commit, local_commit);\n> +\n> +\treturn 0;\n> +}\n\nMakes me wonder, if in_merge_bases() is so expensive that it makes\nsense to split the \"were we exactly at the tip?\" and \"is one of the\ncommits we were at a descendant of the tip?\" into separate phases,\nif this part should be calling in_merge_bases() one by one.\n\nWould it make more sense to iterate over reflog entries from newer\nto older, collect them in an array of pointers to \"struct commit\" in\na batch of say 8 commits or less, and then ask in_merge_bases_many()\nif the remote_commit is an ancestor of one of these local commits?\n\nThe traversal cost to start from one \"local commit\" to see if\nremote_commit is an ancestor of it using in_merge_bases() and\nin_merge_bases_many() should be the same and an additional traversal\ncost to start from more local commits should be negligible compared\nto the traversal itself, so making a call to in_merge_bases() for\neach local_commit smells somewhat suboptimal.\n\nIf we were talking about older parts of the history, optional\ngeneration numbers could change the equation somewhat, but the\ncommon case for the workflow this series is trying to help is that\nthese local commits ane the remote tip are relatively new and it is\nnot unlikely that the generation numbers have not been computed for\nthem, which is why I suspect that in_merges_many may be a win.\n\n> @@ -2301,6 +2380,15 @@ void apply_push_cas(struct push_cas_option *cas,\n>  \t\t    struct ref *remote_refs)\n>  {\n>  \tstruct ref *ref;\n> -\tfor (ref = remote_refs; ref; ref = ref->next)\n> +\tfor (ref = remote_refs; ref; ref = ref->next) {\n>  \t\tapply_cas(cas, remote, ref);\n> +\n> +\t\t/*\n> +\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n> +\t\t * mode, and if \"--foce-if-includes\" was specified, run\n> +\t\t * the check.\n> +\t\t */\n> +\t\tif (ref->if_includes)\n> +\t\t\tcheck_if_includes_upstream(ref);\n\ns/foce/force/; \n\nI can see that the code is checking \"and if force-if-includes was\nspecified\" part, but it is not immediately clear where the code\nchecks if \"--force-with-lease\" is used with \"tracking\" and not with\n\"the other side must be exactly this commit\" mode here.\n\n    ... goes and looks ...\n\nAh, ok, I found out. \n\nThe field name \"if_includes\", and the comment for the field in\nremote.h, are both misleading.  It gives an impression that the\nfield being true means \"--force-if-included is in use\", but in\nreality the field means a lot more.  When it is true, it signals\nthat \"--force-if-included\" is in use *and* for this ref we were told\nto use the \"--force-with-lease\" without an exact object name.  And\nthat logic is not here, but has already happened in apply_cas().\n\nWhich makes the above comment correct.  We however need a better\nname for this field and/or an explanation for the field in the\nheader file, or both, to avoid misleading readers.\n\n> diff --git a/remote.h b/remote.h\n> index 5e3ea5a26d..38ab8539e2 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -104,7 +104,9 @@ struct ref {\n>  \t\tforced_update:1,\n>  \t\texpect_old_sha1:1,\n>  \t\texact_oid:1,\n> -\t\tdeletion:1;\n> +\t\tdeletion:1,\n> +\t\tif_includes:1, /* If \"--force-with-includes\" was specified. */\n\nThe description needs to be tightened.\n\n> +\t\tunreachable:1; /* For \"if_includes\"; unreachable in reflog. */\n\n\nThanks.\n"},{"id":"405951","messageId":"xmqq7dspjywh.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200919170316.5310-3-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v4 2/3] push: parse and set flag for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-19T20:26:54Z","receivedAt":"2020-09-19T20:27:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Adds a flag: \"TRANSPORT_PUSH_FORCE_IF_INCLUDES\" to indicate that the new\n> option was passed from the command line of via configuration settings;\n> update command line and configuration parsers to set the new flag\n> accordingly.\n\ns/Adds/Add/;\n\n> Introduces a new configuration option \"push.useForceIfIncludes\", which\n> is equivalent to setting \"--force-if-includes\" in the command line.\n\ns/Introduces/Introduce/; (I won't repeat).\n\n>\n> Updates \"remote-curl\" to recognize and pass this option to \"send-pack\"\n> when enabled.\n>\n> Updates \"advise\" to catch the reject reason \"REJECT_REF_NEEDS_UPDATE\",\n> which is set when the ref status is \"REF_STATUS_REJECT_REMOTE_UPDATED\"\n> and (optionally) print a help message when the push fails.\n\nAll of the above say what were done.  A summarizing sentence before\nall of the above would make the proposed commit log message perfect,\nperhaps:\n\n    The previous step added the necessary machinery to implement the\n    \"--force-if-includes\" protection, when \"--force-with-lease\" is\n    used without giving exact object the remote still ought to have.\n    Surface the feature by adding a command line option and a\n    configuration variable to enable it.\n\n    - Add a flag ... to indicate that ...\n\n    - Introduce a configuration option ...\n\n    - Update 'remote-curl' to ...\n\n    ...\n\n\nAlso, in the proposed log message for [1/3], especially near its\nend, how \"--force-if-includes\" interacts with \"--force-with-lease\"\nwas described.  The description should be added to the log message\nof this change, as it is what introduces the end-user facing\nfeature.  The description can also be in the log for [1/3] as well,\nbut not having it here for [2/3] is unfriendly to the readers.\n\n> diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n> index 4d76727edb..9289c0eecb 100644\n> --- a/builtin/send-pack.c\n> +++ b/builtin/send-pack.c\n> @@ -159,6 +159,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n>  \tint progress = -1;\n>  \tint from_stdin = 0;\n>  \tstruct push_cas_option cas = {0};\n> +\tunsigned int force_if_includes = 0;\n\nI think OPT_BOOL takes a pointer to int, not unsigned, as it is\nOPT_SET_INT in disguise, and you can see that a near-by 'progress'\nthat also is fed to OPT_BOOL() is 'int' so you can mimic it.\n\n> @@ -184,6 +185,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n>  \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n>  \t\t  N_(\"require old value of ref to be at this value\"),\n>  \t\t  PARSE_OPT_OPTARG, parseopt_push_cas_option),\n> +\t\tOPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,\n> +\t\t\t N_(\"require remote updates to be integrated locally\")),\n>  \t\tOPT_END()\n>  \t};\n\n> diff --git a/remote.h b/remote.h\n> index 38ab8539e2..72c374d539 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -350,4 +350,10 @@ int parseopt_push_cas_option(const struct option *, const char *arg, int unset);\n>  int is_empty_cas(const struct push_cas_option *);\n>  void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);\n>  \n> +/*\n> + * Sets \"use_force_if_includes\" for \"compare-and-swap\"\n> + * when \"--force-if-includes\" is specified.\n> + */\n> +void push_set_force_if_includes(struct push_cas_option *);\n\nLet's not add this helper function.  Instead just open-code a single\nliner at its two callers.  It makes it easier to read and understand\nthe flow and the logic in cmd_push() and cmd_send_pack().\n\n> diff --git a/transport-helper.c b/transport-helper.c\n> index e547e21199..2a4436dd79 100644\n> --- a/transport-helper.c\n> +++ b/transport-helper.c\n> @@ -868,6 +868,12 @@ static void set_common_push_options(struct transport *transport,\n>  \t\tif (set_helper_option(transport, TRANS_OPT_ATOMIC, \"true\") != 0)\n>  \t\t\tdie(_(\"helper %s does not support --atomic\"), name);\n>  \n> +\t/* If called with \"--force-if-includes\". */\n\nThe comment does not add any value as you are already using a\ndescriptive constant name.  Drop it to follow suit of existing if\nstatements nearby.\n\n> +\tif (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)\n> +\t\tif (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, \"true\") != 0)\n> +\t\t\tdie(_(\"helper %s does not support --%s\"),\n> +\t\t\t    name, TRANS_OPT_FORCE_IF_INCLUDES);\n> +\n>  \tif (flags & TRANSPORT_PUSH_OPTIONS) {\n>  \t\tstruct string_list_item *item;\n>  \t\tfor_each_string_list_item(item, transport->push_options)\n\nThanks.\n"},{"id":"405952","messageId":"xmqq363djy6l.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200919170316.5310-4-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v4 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-19T20:42:26Z","receivedAt":"2020-09-19T20:42:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Subject: Re: [PATCH v4 3/3] t, doc: update tests, reference for \"--force-if-includes\"\n\ngood.\n\n> * t/t5533-push-cas.sh:\n>     Updates test cases for \"compare-and-swap\" when used along with\n>     \"--force-if-includes\" helps mitigate overwrites when remote\n>     ref are updated in the background.\n>\n> * Documentation:\n>     Adds reference for the new option, configuration setting\n>     (\"push.useForceIfIncludes\") and advise messages.\n\ns/Updates/Update/; s/Adds/Add/;\n\n> +push.useForceIfIncludes::\n> +\tIf set to \"true\", it is equivalent to specifying \"--force-if-includes\"\n> +\tas an argument to linkgit:git-push[1]. Adding \"--no-force-if-includes\"\n\ns/as an argument to/on the command line of/ would be better.\n\nSome readers differenciate arguments and options and we are\ndiscussing an option, not an argument.\n\n> +Alternatively, specifying \"--force-if-includes\" an an ancillary option along\n> +with \"--force-with-lease[=<refname>]\" (i.e., without saying what exact commit\n> +the ref on the remote side must be pointing at, or which refs on the remote\n> +side are being protected) at the time of \"push\" will verify if updates from the\n> +remote-tracking refs that may have been implicitly updated in the background\n> +are integrated locally before allowing a forced update.\n\nOK.  A user who wants to know more can refer to --force-if-includes\nfrom here pretty easily.\n\n> @@ -341,6 +348,19 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n>  origin +master` to force a push to the `master` branch). See the\n>  `<refspec>...` section above for details.\n>  \n> +--[no-]force-if-includes::\n> +\tForce an update only if the tip of the remote-tracking ref\n> +\thas been integrated locally.\n> ++\n> +This option verifies if the tip of the remote-tracking ref on which\n> +a local branch has based on (for a rewrite), is reachable from at\n> +least one of the \"reflog\" entries of the local branch about to be\n> +updated by force on the remote.\n\nThe latter half of this sentence is quite a mouthful, and after\nreading it three times, it is not quite clear.\n\n> +The check ensures that any updates\n> +from the remote have been incorporated locally by rejecting a push\n> +if that is not the case.\n\nOK.\n\n> ++\n> +Specifying \"--no-force-if-includes\" disables this behavior.\n\nDo we want to add:\n\n    It is a no-op unless \"--force-with-lease[=<refname>]\" without exact\n    object name is used at the same time.\n\nhere or somewhere nearby?\n\n\n> +test_expect_success 'background updates of REMOTE can be mitigated with \"--force-if-includes\"' '\n> +\trm -rf src dst &&\n> +\tgit init --bare src.bare &&\n> +\ttest_when_finished \"rm -rf src.bare\" &&\n> +\tgit clone --no-local src.bare dst &&\n> +\ttest_when_finished \"rm -rf dst\" &&\n> +\t(\n> +\t\tcd dst &&\n> +\t\ttest_commit G &&\n> +\t\tgit push origin master:master\n> +\t) &&\n> +\tgit clone --no-local src.bare dst2 &&\n> +\ttest_when_finished \"rm -rf dst2\" &&\n> +\t(\n> +\t\tcd dst2 &&\n> +\t\ttest_commit H &&\n> +\t\tgit push\n> +\t) &&\n> +\t(\n> +\t\tcd dst &&\n> +\t\ttest_commit I &&\n> +\t\tgit fetch origin &&\n> +\t\ttest_must_fail git push --force-with-lease --force-if-includes origin\n\nI briefly wondered if it makes sense to also check if\n--force-with-lease alone (or with --no-force-if-includes)\nsuccessfully pushes in this case, but I think we are OK without such\na test.  After all, we won't test \"--force\" alone, either, as we\nexpect that to work (and should be tested elsewhere).\n\n> +\t)\n> +'\n> +\n> +test_expect_success 'background updates of REMOTE can be mitigated with \"push.useForceIfIncludes\"' '\n> +\trm -rf src dst &&\n> +\tgit init --bare src.bare &&\n> +\ttest_when_finished \"rm -rf src.bare\" &&\n> +\tgit clone --no-local src.bare dst &&\n> +\ttest_when_finished \"rm -rf dst\" &&\n> +\t(\n> +\t\tcd dst &&\n> +\t\ttest_commit G &&\n> +\t\tgit push origin master:master\n> +\t) &&\n> +\tgit clone --no-local src.bare dst2 &&\n> +\ttest_when_finished \"rm -rf dst2\" &&\n> +\t(\n> +\t\tcd dst2 &&\n> +\t\ttest_commit H &&\n> +\t\tgit push\n> +\t) &&\n> +\t(\n> +\t\tcd dst &&\n> +\t\ttest_commit I &&\n> +\t\tgit fetch origin &&\n> +\t\tgit config --local push.useForceIfIncludes \"true\" &&\n> +\t\ttest_must_fail git push --force-if-includes origin\n\nI am not sure what is tested here.  I thought with or without the\nconfiguration variable, the feature is a no-op unless a lazy\nforce-with-lease is in use?\n\nPerhaps you meant to test\n\n\t\ttest_must_fail git push --force-with-lease origin\n\ninstead?\n\n> +\t)\n> +'\n> +\n>  test_done\n\nThanks.\n"},{"id":"406016","messageId":"20200921084231.GA64896@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqft7djzz0.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-21T08:42:31Z","receivedAt":"2020-09-21T08:42:41Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Junio,\n\nOn 09/19/2020 13:03, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> > Adds a check to verify if the remote-tracking ref of the local branch\n> > is reachable from one of its \"reflog\" entries.\n> \n> s/Adds/Add/\n\nGotcha, I will reword the commit messages for all three commits.\nin the patch series.\n\n> > When \"--force-with-includes\" is used along with \"--force-with-lease\",\n> \n> A misspelt name for the new option is found here.\n\n*Facepalm.* Thanks, will update.\n\n> > [...]\n> Makes me wonder, if in_merge_bases() is so expensive that it makes\n> sense to split the \"were we exactly at the tip?\" and \"is one of the\n> commits we were at a descendant of the tip?\" into separate phases,\n> if this part should be calling in_merge_bases() one by one.\n> \n> Would it make more sense to iterate over reflog entries from newer\n> to older, collect them in an array of pointers to \"struct commit\" in\n> a batch of say 8 commits or less, and then ask in_merge_bases_many()\n> if the remote_commit is an ancestor of one of these local commits?\n>\n> The traversal cost to start from one \"local commit\" to see if\n> remote_commit is an ancestor of it using in_merge_bases() and\n> in_merge_bases_many() should be the same and an additional traversal\n> cost to start from more local commits should be negligible compared\n> to the traversal itself, so making a call to in_merge_bases() for\n> each local_commit smells somewhat suboptimal.\n> \n> If we were talking about older parts of the history, optional\n> generation numbers could change the equation somewhat, but the\n> common case for the workflow this series is trying to help is that\n> these local commits ane the remote tip are relatively new and it is\n> not unlikely that the generation numbers have not been computed for\n> them, which is why I suspect that in_merges_many may be a win.\n\nNice! We can definitely try batching commits from the reflog and\npass it along to \"in_merge_bases_many()\". As for being faster than\ncalling \"in_merge_bases()\" for each commit entry in the reflog --\nI am not familiar with how the former works. Do we still keep the\n\"reflog_entry_exists()\" part? It might still be faster to go through\nthe entries once to check with \"oideq()\" in the first run.\n\nAlso, I was wondering if it is worth considering this:\n  - check if the reflog of the HEAD has the remote ref\n  - check if the reflog of the local branch has the remote ref\n  - check if the remote ref is reachable from any of the local ref's\n    \"reflog\" entries using \"in_merge_bases_many()\" in batches as\n    suggested here.\n\nThe first two (we can even skip the second one) runs are relatively\nfast, and the third one might be faster than checking \"in_merge_bases()\"\nfor each reflog entry. I suppose adding these three steps would make\nthe process slower overall, though. For context, I was referring to\nyour message [1] on the other thread regarding checking the HEAD's\nreflog.\n\n> > [...]\n> > +\t\t/*\n> > +\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n> > +\t\t * mode, and if \"--foce-if-includes\" was specified, run\n> > +\t\t * the check.\n> > +\t\t */\n> > +\t\tif (ref->if_includes)\n> > +\t\t\tcheck_if_includes_upstream(ref);\n> \n> s/foce/force/; \n\nYes, sorry about that; will update.\n \n> I can see that the code is checking \"and if force-if-includes was\n> specified\" part, but it is not immediately clear where the code\n> checks if \"--force-with-lease\" is used with \"tracking\" and not with\n> \"the other side must be exactly this commit\" mode here.\n> \n>     ... goes and looks ...\n> \n> Ah, ok, I found out. \n> \n> The field name \"if_includes\", and the comment for the field in\n> remote.h, are both misleading.  It gives an impression that the\n> field being true means \"--force-if-included is in use\", but in\n> reality the field means a lot more.  When it is true, it signals\n> that \"--force-if-included\" is in use *and* for this ref we were told\n> to use the \"--force-with-lease\" without an exact object name.  And\n> that logic is not here, but has already happened in apply_cas().\n> \n> Which makes the above comment correct.  We however need a better\n> name for this field and/or an explanation for the field in the\n> header file, or both, to avoid misleading readers.\n>\n> > diff --git a/remote.h b/remote.h\n> > index 5e3ea5a26d..38ab8539e2 100644\n> > --- a/remote.h\n> > +++ b/remote.h\n> > @@ -104,7 +104,9 @@ struct ref {\n> >  \t\tforced_update:1,\n> >  \t\texpect_old_sha1:1,\n> >  \t\texact_oid:1,\n> > -\t\tdeletion:1;\n> > +\t\tdeletion:1,\n> > +\t\tif_includes:1, /* If \"--force-with-includes\" was specified. */\n> \n> The description needs to be tightened.\n> \n> > +\t\tunreachable:1; /* For \"if_includes\"; unreachable in reflog. */\n\nOK, you're right. Perhaps, we could rename it to something like\n\"if_includes_for_tracking\" and update the comment description\nwith saying something along the lines of:\n\n+  /*\n+   * Set when \"--force-if-includes\" is enabled, and\n+   * if \"compare-and-swap\" is not provided with the\n+   * exact commit to be expected on the remote (in\n+   * \"use_tracking\" or use_tracking_for_rest\" mode).\n+   */\n\n\n[1]: https://public-inbox.org/git/xmqqsgbdk69b.fsf@gitster.c.googlers.com\n\nThanks again, for taking the time to review this.\n-- \nSrinidhi Kaushik\n\n"},{"id":"406042","messageId":"f6bb2b1f-0f1b-f196-59f1-893580430cf2@gmail.com","threadId":"54190","inReplyTo":"xmqqft7djzz0.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-09-21T13:19:24Z","receivedAt":"2020-09-21T13:19:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 19/09/2020 21:03, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n>> +/* Checks if the ref is reachable from the reflog entry. */\n>> +static int reflog_entry_reachable(struct object_id *o_oid,\n>> +\t\t\t       struct object_id *n_oid,\n>> +\t\t\t       const char *ident, timestamp_t timestamp,\n>> +\t\t\t       int tz, const char *message, void *cb_data)\n>> +{\n>> +\tstruct commit *local_commit;\n>> +\tstruct commit *remote_commit = cb_data;\n>> +\n>> +\tlocal_commit = lookup_commit_reference(the_repository, n_oid);\n>> +\tif (local_commit)\n>> +\t\treturn in_merge_bases(remote_commit, local_commit);\n>> +\n>> +\treturn 0;\n>> +}\n> \n> Makes me wonder, if in_merge_bases() is so expensive that it makes\n> sense to split the \"were we exactly at the tip?\" and \"is one of the\n> commits we were at a descendant of the tip?\" into separate phases,\n> if this part should be calling in_merge_bases() one by one.\n> \n> Would it make more sense to iterate over reflog entries from newer\n> to older, collect them in an array of pointers to \"struct commit\" in\n> a batch of say 8 commits or less, and then ask in_merge_bases_many()\n> if the remote_commit is an ancestor of one of these local commits?\n\nAs I said before[1] I think we should also be checking the reflog dates \nso that we do not look at any local reflog entries that are older than \nthe most recent reflog entry for the remote tracking branch. This \nprotects against a background fetch when the remote has been rewound and \nit would also reduce the number of calls to in_merge_bases().\n\nBest Wishes\n\nPhillip\n\n[1] \nhttps://lore.kernel.org/git/624d9e35-29b8-4012-a3d6-e9b00a9e4485@gmail.com/\n\n> The traversal cost to start from one \"local commit\" to see if\n> remote_commit is an ancestor of it using in_merge_bases() and\n> in_merge_bases_many() should be the same and an additional traversal\n> cost to start from more local commits should be negligible compared\n> to the traversal itself, so making a call to in_merge_bases() for\n> each local_commit smells somewhat suboptimal.\n> \n> If we were talking about older parts of the history, optional\n> generation numbers could change the equation somewhat, but the\n> common case for the workflow this series is trying to help is that\n> these local commits ane the remote tip are relatively new and it is\n> not unlikely that the generation numbers have not been computed for\n> them, which is why I suspect that in_merges_many may be a win.\n> \n>> @@ -2301,6 +2380,15 @@ void apply_push_cas(struct push_cas_option *cas,\n>>   \t\t    struct ref *remote_refs)\n>>   {\n>>   \tstruct ref *ref;\n>> -\tfor (ref = remote_refs; ref; ref = ref->next)\n>> +\tfor (ref = remote_refs; ref; ref = ref->next) {\n>>   \t\tapply_cas(cas, remote, ref);\n>> +\n>> +\t\t/*\n>> +\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n>> +\t\t * mode, and if \"--foce-if-includes\" was specified, run\n>> +\t\t * the check.\n>> +\t\t */\n>> +\t\tif (ref->if_includes)\n>> +\t\t\tcheck_if_includes_upstream(ref);\n> \n> s/foce/force/;\n> \n> I can see that the code is checking \"and if force-if-includes was\n> specified\" part, but it is not immediately clear where the code\n> checks if \"--force-with-lease\" is used with \"tracking\" and not with\n> \"the other side must be exactly this commit\" mode here.\n> \n>      ... goes and looks ...\n> \n> Ah, ok, I found out.\n> \n> The field name \"if_includes\", and the comment for the field in\n> remote.h, are both misleading.  It gives an impression that the\n> field being true means \"--force-if-included is in use\", but in\n> reality the field means a lot more.  When it is true, it signals\n> that \"--force-if-included\" is in use *and* for this ref we were told\n> to use the \"--force-with-lease\" without an exact object name.  And\n> that logic is not here, but has already happened in apply_cas().\n> \n> Which makes the above comment correct.  We however need a better\n> name for this field and/or an explanation for the field in the\n> header file, or both, to avoid misleading readers.\n> \n>> diff --git a/remote.h b/remote.h\n>> index 5e3ea5a26d..38ab8539e2 100644\n>> --- a/remote.h\n>> +++ b/remote.h\n>> @@ -104,7 +104,9 @@ struct ref {\n>>   \t\tforced_update:1,\n>>   \t\texpect_old_sha1:1,\n>>   \t\texact_oid:1,\n>> -\t\tdeletion:1;\n>> +\t\tdeletion:1,\n>> +\t\tif_includes:1, /* If \"--force-with-includes\" was specified. */\n> \n> The description needs to be tightened.\n> \n>> +\t\tunreachable:1; /* For \"if_includes\"; unreachable in reflog. */\n> \n> \n> Thanks.\n> \n"},{"id":"406052","messageId":"xmqqv9g7f6r9.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"f6bb2b1f-0f1b-f196-59f1-893580430cf2@gmail.com","subject":"Re: [PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-21T16:12:58Z","receivedAt":"2020-09-21T16:13:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood <phillip.wood123@gmail.com> writes:\n\n> As I said before[1] I think we should also be checking the reflog\n> dates so that we do not look at any local reflog entries that are\n> older than the most recent reflog entry for the remote tracking\n> branch. This protects against a background fetch when the remote has\n> been rewound and it would also reduce the number of calls to\n> in_merge_bases().\n\nMeaning we first check the timestamp of the topmost reflog entry of\nremote-tracking branch (i.e. the time *we* acquired the tip commit\nthat we are about to lose), and leverage on the fact that no local\ncommit older than that timestamp can possibly be written with the\nknowledge of that remote work?  Assuming that local timestamp is\nmonotonically increasing, it is a quite valid optimization (and the\nclock skew we often talk about in the context of revision traversal\nare often discrepancy between matchines).\n\nHaving said that.\n\nThe new generation number based on (adjusted) timestamp is being\nworked in, and that work is supposed to bring such an optimization\nto us automatically (at least on the reachability's side, i.e. logic\nthat uses get_merge_bases()), I think, so we probably do *not* want\nto add such a heuristics specifically for this codepath.\n\nThanks.\n"},{"id":"406061","messageId":"xmqqmu1jf19d.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"xmqqv9g7f6r9.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-21T18:11:42Z","receivedAt":"2020-09-21T18:11:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Meaning we first check the timestamp of the topmost reflog entry of\n> remote-tracking branch (i.e. the time *we* acquired the tip commit\n> that we are about to lose), and leverage on the fact that no local\n> commit older than that timestamp can possibly be written with the\n> knowledge of that remote work?  Assuming that local timestamp is\n> monotonically increasing, it is a quite valid optimization (and the\n> clock skew we often talk about in the context of revision traversal\n> are often discrepancy between matchines).\n>\n> Having said that.\n>\n> The new generation number based on (adjusted) timestamp is being\n> worked in, and that work is supposed to bring such an optimization\n> to us automatically (at least on the reachability's side, i.e. logic\n> that uses get_merge_bases()), I think, so we probably do *not* want\n> to add such a heuristics specifically for this codepath.\n\nEh, I spoke too soon before I thought it through.  I do not think we\nwill gain \"assume that any commit whose timestamp is older than this\nexternally given one will never reach the other commit\" even with\nthe reachability index based on (adjusted) timestamp.  At least,\nstopping the traversal of reflog entries of the local side at the\ntimestamp of the topmost reflog entry of remote-tracking branch in\nquestion would be an easy, worthwhile and sensible optimization.\n\nThanks.\n"},{"id":"406064","messageId":"xmqqimc7ezk2.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200921084231.GA64896@mail.clickyotomy.dev","subject":"Re: [PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-21T18:48:29Z","receivedAt":"2020-09-21T18:48:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n>> If we were talking about older parts of the history, optional\n>> generation numbers could change the equation somewhat, but the\n>> common case for the workflow this series is trying to help is that\n>> these local commits ane the remote tip are relatively new and it is\n>> not unlikely that the generation numbers have not been computed for\n>> them, which is why I suspect that in_merges_many may be a win.\n>\n> Nice! We can definitely try batching commits from the reflog and\n> pass it along to \"in_merge_bases_many()\". As for being faster than\n> calling \"in_merge_bases()\" for each commit entry in the reflog --\n> I am not familiar with how the former works. Do we still keep the\n> \"reflog_entry_exists()\" part? It might still be faster to go through\n> the entries once to check with \"oideq()\" in the first run.\n\nThat is what I meant.  You go through local reflog entries until you\nfind one that is older than the timestamp of the reflog entry of the\nremote-tracking branch, check with oideq() to see if the tip was ever\ndirectly checked out.  Then, using these same local reflog entries,\nyou can make in_merge_bases_many() tranversal to see if any of them\nreach the tip.  I suspect that the number of local reflog entries you\nneed to examine would not be too many, so if you can put them all in\na single array of \"struct commit *\" pointers in the first \"oideq()\"\nphase, you may be able to do just a single in_merge_bases_many() batch\nto check for the reachability.\n\n> Also, I was wondering if it is worth considering this:\n>   - check if the reflog of the HEAD has the remote ref\n\nIt would help the workflow I had in mind, but it would raise the\nrisk of false positives according to Dscho and I tend to agree, so\nI do not know if it is overall a good idea.\n\n>   - check if the reflog of the local branch has the remote ref\n\nIsn't that the oideq() test?\n\n>   - check if the remote ref is reachable from any of the local ref's\n>     \"reflog\" entries using \"in_merge_bases_many()\" in batches as\n>     suggested here.\n\nI think it amounts to the same as \"does any reflog entry of HEAD\nreach it?\" and shares the same issues with false positives as the\nfirst one.\n\n>> > +\t\tdeletion:1,\n>> > +\t\tif_includes:1, /* If \"--force-with-includes\" was specified. */\n>> \n>> The description needs to be tightened.\n>> \n>> > +\t\tunreachable:1; /* For \"if_includes\"; unreachable in reflog. */\n>\n> OK, you're right. Perhaps, we could rename it to something like\n> \"if_includes_for_tracking\" and update the comment description\n> with saying something along the lines of:\n\nThat is overlong.  Let me try:\n\n\t\t/* need to check if local reflog reaches the remote tip */\n\t\tcheck_reachable:1,\n\n\t\t/* local reflog does not reach the remote tip */\n\t\tunreachable:1;\n\nThans.\n"},{"id":"406177","messageId":"20200923073022.61293-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200919170316.5310-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v5 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-23T07:30:19Z","receivedAt":"2020-09-23T07:30:34Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new option: \"--force-if-includes\" to \"git-push\" where forced\nupdates are allowed only if the tip of the remote-tracking ref has\nbeen integrated locally, by verifying if the tip of the remote-tracking\nref -- on which a local branch has based on -- is reachable from at\nleast one of the \"reflog\" entries of the branch about to be updated\nby force on the remote.\n\nThis option can be used with \"--force-with-lease\" with setups where\nthe remote-tracking refs of the repository are implicitly updated in\nthe background to help prevent unintended remote overwrites.\n\nIf a local branch is based on a remote ref for a rewrite, and if that\nremote-tracking ref is updated by a push from another repository after\nit has been checked out locally, force updating that branch to remote\nwith \"--force-with-lease[=<refname>[:<expect>]]\" without specifying\nthe \"<expect>\" value, can cause the update that happened in-between the\ncheckout and forced push to be lost.\n\nChanges since v4:\n  - Collect commits during the reflog iteration step and use\n    \"in_merge_bases_many()\" once on the list of collected commits\n    instead of using \"in_merge_bases()\" for every entry [1].\n\n  - Include reflog timestamps to reduce the number of reflog entry\n    iterations by stopping early if we see an entry that is older\n    than the remote tip's commit date [2]. This, coupled with a\n    single call of \"in_merge_bases_many()\" should help make the\n    check faster [3].\n\n  - Reword commit messages, documentation; add a note to explain\n    when the option is a \"no-op\".\n\n  - Fix a mistake in the test script (t5533).\n\nSrinidhi Kaushik (3):\n  push: add reflog check for \"--force-if-includes\"\n  push: parse and set flag for \"--force-if-includes\"\n  t, doc: update tests, reference for \"--force-if-includes\"\n\n Documentation/config/advice.txt |   9 +-\n Documentation/config/push.txt   |   6 ++\n Documentation/git-push.txt      |  26 +++++-\n advice.c                        |   3 +\n advice.h                        |   2 +\n builtin/push.c                  |  27 ++++++\n builtin/send-pack.c             |  12 +++\n remote-curl.c                   |  14 +++-\n remote.c                        | 140 +++++++++++++++++++++++++++++++-\n remote.h                        |   8 +-\n send-pack.c                     |   1 +\n t/t5533-push-cas.sh             |  53 ++++++++++++\n transport-helper.c              |  10 +++\n transport.c                     |   8 ++\n transport.h                     |  15 ++--\n 15 files changed, 321 insertions(+), 13 deletions(-)\n\nbase-commit: 385c171a018f2747b329bcfa6be8eda1709e5abd\n\nThanks to Johannes, Junio and Phillip for continually suggesting\nimprovements and patiently reviewing the patch series.\n\n[1]: https://public-inbox.org/git/xmqqft7djzz0.fsf@gitster.c.googlers.com/\n[2]: https://public-inbox.org/git/f6bb2b1f-0f1b-f196-59f1-893580430cf2@gmail.com/\n[3]: https://public-inbox.org/git/xmqqimc7ezk2.fsf@gitster.c.googlers.com/\n--\n2.28.0\n"},{"id":"406178","messageId":"20200923073022.61293-2-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200923073022.61293-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v5 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-23T07:30:20Z","receivedAt":"2020-09-23T07:30:36Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a check to verify if the remote-tracking ref of the local branch\nis reachable from one of its \"reflog\" entries.\n\nWhen a local branch that is based on a remote ref, has been rewound\nand is to be force pushed on the remote, \"--force-if-includes\" runs\na check that ensures any updates to remote-tracking refs that may have\nhappened (by push from another repository) in-between the time of the\nlast checkout, and right before the time of push, have been integrated\nlocally before allowing a forced updated.\n\nA new field \"use_force_if_includes\" has been added to \"push_cas_option\",\nwhich is set to \"1\" when \"--force-if-includes\" is specified as an\noption in the command line or as a configuration setting.\n\nThe struct \"ref\" has two new bit-fields:\n  - check_reachable:\n    Set when we have to run the new check on the ref, and the remote\n    ref was marked as \"use_tracking\" or \"use_tracking_for_rest\" by\n    compare-and-swap (if the \"the remote tip must be at the expected\n    commit\" condition is not specified); \"apply_push_cas()\" has been\n    updated to check if this field is set and run the check.\n\n  - unreachable:\n    Set if the ref is unreachable from any of the \"reflog\" entries of\n    its local counterpart.\n\n\"REF_STATUS_REJECT_REMOTE_UPDATED\" has been added to the \"status\"\nenum to imply that the ref failed the check; \"case\" statements in\n\"send-pack\", \"transport\" and \"transport-helper\" have been updated\naccordingly to catch this status when set.\n\nWhen \"--force-is-includes\" is used along with \"--force-with-lease\",\nthe check runs only for refs marked as \"if_includes\". If the option\nis passed without specifying \"--force-with-lease\", or specified along\nwith \"--force-with-lease=<refname>:<expect>\" it is a \"no-op\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n builtin/send-pack.c |   5 ++\n remote.c            | 140 +++++++++++++++++++++++++++++++++++++++++++-\n remote.h            |   8 ++-\n send-pack.c         |   1 +\n transport-helper.c  |   5 ++\n transport.c         |   6 ++\n 6 files changed, 162 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 2b9610f121..4d76727edb 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n \n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"remote ref updated since checkout\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\ndiff --git a/remote.c b/remote.c\nindex eafc14cbe7..0dcac4ab8e 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t * with the remote-tracking branch to find the value\n \t\t * to expect, but we did not have such a tracking\n \t\t * branch.\n+\t\t *\n+\t\t * If the tip of the remote-tracking ref is unreachable\n+\t\t * from any reflog entry of its local ref indicating a\n+\t\t * possible update since checkout; reject the push.\n \t\t */\n \t\tif (ref->expect_old_sha1) {\n \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n+\t\t\telse if (ref->check_reachable && ref->unreachable)\n+\t\t\t\treject_reason =\n+\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n \t\t\telse\n-\t\t\t\t/* If the ref isn't stale then force the update. */\n+\t\t\t\t/*\n+\t\t\t\t * If the ref isn't stale, and is reachable\n+\t\t\t\t * from from one of the reflog entries of\n+\t\t\t\t * the local branch, force the update.\n+\t\t\t\t */\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n \n@@ -2268,6 +2279,118 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \treturn 0;\n }\n \n+/*\n+ * The struct \"reflog_commit_list\" and related helper functions\n+ * for list manipulation are used for collecting commits into a\n+ * list during reflog traversals in \"if_exists_or_grab_until()\".\n+ */\n+struct reflog_commit_list {\n+\tstruct commit **items;\n+\tsize_t nr, alloc;\n+};\n+\n+/* Adds a commit to list. */\n+static void add_commit(struct reflog_commit_list *list, struct commit *commit)\n+{\n+\tALLOC_GROW(list->items, list->nr + 1, list->alloc);\n+\tlist->items[list->nr++] = commit;\n+}\n+\n+/* Free and reset the list. */\n+static void free_reflog_commit_list(struct reflog_commit_list *list)\n+{\n+\tFREE_AND_NULL(list->items);\n+\tlist->nr = list->alloc = 0;\n+}\n+\n+struct check_and_collect_until_cb_data {\n+\tstruct commit *remote_commit;\n+\tstruct reflog_commit_list *local_commits;\n+};\n+\n+\n+static int check_and_collect_until(struct object_id *o_oid,\n+\t\t\t\t   struct object_id *n_oid,\n+\t\t\t\t   const char *ident, timestamp_t timestamp,\n+\t\t\t\t   int tz, const char *message, void *cb_data)\n+{\n+\tstruct commit *commit;\n+\tstruct check_and_collect_until_cb_data *cb = cb_data;\n+\n+\t/*\n+\t * If the reflog entry timestamp is older than the\n+\t * remote commit date, there is no need to check or\n+\t * collect entries older than this one.\n+\t */\n+\tif (timestamp < cb->remote_commit->date)\n+\t\treturn -1;\n+\n+\t/* An entry was found. */\n+\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n+\t\treturn 1;\n+\n+\t/* Lookup the commit and append it to the list. */\n+\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n+\t\tadd_commit(cb->local_commits, commit);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Iterate through the reflog of a local ref to check if there is an entry for\n+ * the given remote-tracking ref (i.e., if it was checked out); runs until the\n+ * timestamp of an entry is older than the commit date of the remote-tracking\n+ * ref. Any commits that seen along the way are collected into a list to check\n+ * if the remote-tracking ref is reachable from any of them.\n+ */\n+static int is_reachable_in_reflog(const char *local_ref_name,\n+\t\t\t\t  const struct object_id *remote_oid)\n+{\n+\tstruct commit *remote_commit;\n+\tstruct reflog_commit_list list = { NULL, 0, 0 };\n+\tstruct check_and_collect_until_cb_data cb;\n+\tint ret = 0;\n+\n+\tremote_commit = lookup_commit_reference(the_repository, remote_oid);\n+\tif (!remote_commit)\n+\t\tgoto cleanup_return;\n+\n+\tcb.remote_commit = remote_commit;\n+\tcb.local_commits = &list;\n+\tret = for_each_reflog_ent_reverse(local_ref_name,\n+\t\t\t\t\t  check_and_collect_until, &cb);\n+\n+\t/* We found an entry in the reflog. */\n+\tif (ret > 0)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Check if \"remote_commit\" is reachable from\n+\t * any of the commits in the collected list.\n+\t */\n+\tif (list.nr > 0)\n+\t\tret = in_merge_bases_many(remote_commit, list.nr, list.items);\n+\n+cleanup_return:\n+\tfree_reflog_commit_list(&list);\n+\treturn ret;\n+}\n+\n+/*\n+ * Check for reachability of a remote-tracking\n+ * ref in the reflog entries of its local ref.\n+ */\n+static void check_if_includes_upstream(struct ref *remote_ref)\n+{\n+\tstruct ref *local_ref = get_local_ref(remote_ref->name);\n+\n+\tif (!local_ref)\n+\t\treturn;\n+\n+\tif (!is_reachable_in_reflog(local_ref->name, &remote_ref->old_oid))\n+\t\tremote_ref->unreachable = 1;\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n \t\t      struct ref *ref)\n@@ -2284,6 +2407,8 @@ static void apply_cas(struct push_cas_option *cas,\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n \t\t\toidclr(&ref->old_oid_expect);\n+\t\telse\n+\t\t\tref->check_reachable = cas->use_force_if_includes;\n \t\treturn;\n \t}\n \n@@ -2294,6 +2419,8 @@ static void apply_cas(struct push_cas_option *cas,\n \tref->expect_old_sha1 = 1;\n \tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n \t\toidclr(&ref->old_oid_expect);\n+\telse\n+\t\tref->check_reachable = cas->use_force_if_includes;\n }\n \n void apply_push_cas(struct push_cas_option *cas,\n@@ -2301,6 +2428,15 @@ void apply_push_cas(struct push_cas_option *cas,\n \t\t    struct ref *remote_refs)\n {\n \tstruct ref *ref;\n-\tfor (ref = remote_refs; ref; ref = ref->next)\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n \t\tapply_cas(cas, remote, ref);\n+\n+\t\t/*\n+\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n+\t\t * mode, and if \"--force-if-includes\" was specified, run\n+\t\t * the check.\n+\t\t */\n+\t\tif (ref->check_reachable)\n+\t\t\tcheck_if_includes_upstream(ref);\n+\t}\n }\ndiff --git a/remote.h b/remote.h\nindex 5e3ea5a26d..7c5e59770e 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -104,7 +104,11 @@ struct ref {\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\t/* Need to check if local reflog reaches the remote tip. */\n+\t\tcheck_reachable:1,\n+\t\t/* The local reflog does not reach the remote tip. */\n+\t\tunreachable:1;\n \n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -134,6 +138,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n \t\tREF_STATUS_REJECT_SHALLOW,\n+\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT,\n@@ -332,6 +337,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);\n \n struct push_cas_option {\n \tunsigned use_tracking_for_rest:1;\n+\tunsigned use_force_if_includes:1;\n \tstruct push_cas {\n \t\tstruct object_id expect;\n \t\tunsigned use_tracking:1;\ndiff --git a/send-pack.c b/send-pack.c\nindex 632f1580ca..956306e8e8 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c52c99d829..e547e21199 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"remote ref updated since checkout\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\n \t\t\t\tstring_list_clear(&cas_options, 0);\ndiff --git a/transport.c b/transport.c\nindex 43e24bf1e5..99fe6233a3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"stale info\", porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"remote ref updated since checkout\",\n+\t\t\t\t porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -1101,6 +1106,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n \n \t\tstrbuf_reset(&buf);\n-- \n2.28.0\n\n"},{"id":"406179","messageId":"20200923073022.61293-3-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200923073022.61293-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v5 2/3] push: parse and set flag for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-23T07:30:21Z","receivedAt":"2020-09-23T07:30:39Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"The previous commit added the necessary machinery to implement the\n\"--force-if-includes\" protection, when \"--force-with-lease\" is used\nwithout giving exact object the remote still ought to have. Surface\nthe feature by adding a command line option and a configuration\nvariable to enable it.\n\n - Add a flag: \"TRANSPORT_PUSH_FORCE_IF_INCLUDES\" to indicate that the\n   new option was passed from the command line of via configuration\n   settings; update command line and configuration parsers to set the\n   new flag accordingly.\n\n - Introduce a new configuration option \"push.useForceIfIncludes\", which\n   is equivalent to setting \"--force-if-includes\" in the command line.\n\n - Update \"remote-curl\" to recognize and pass this option to \"send-pack\"\n   when enabled.\n\n - Update \"advise\" to catch the reject reason \"REJECT_REF_NEEDS_UPDATE\",\n   set when the ref status is \"REF_STATUS_REJECT_REMOTE_UPDATED\" and\n   (optionally) print a help message when the push fails.\n\n - The new option is a \"no-op\" in the following scenarios:\n    * When used without \"--force-with-lease\".\n    * When used with \"--force-with-lease\", and if the expected commit\n      on the remote side is specified as an argument.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n advice.c            |  3 +++\n advice.h            |  2 ++\n builtin/push.c      | 27 +++++++++++++++++++++++++++\n builtin/send-pack.c |  7 +++++++\n remote-curl.c       | 14 +++++++++++++-\n transport-helper.c  |  5 +++++\n transport.c         |  2 ++\n transport.h         | 15 ++++++++++-----\n 8 files changed, 69 insertions(+), 6 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex f0a3d32d20..164742305f 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;\n int advice_push_fetch_first = 1;\n int advice_push_needs_force = 1;\n int advice_push_unqualified_ref_name = 1;\n+int advice_push_ref_needs_update = 1;\n int advice_status_hints = 1;\n int advice_status_u_option = 1;\n int advice_status_ahead_behind_warning = 1;\n@@ -72,6 +73,7 @@ static struct {\n \t{ \"pushFetchFirst\", &advice_push_fetch_first },\n \t{ \"pushNeedsForce\", &advice_push_needs_force },\n \t{ \"pushUnqualifiedRefName\", &advice_push_unqualified_ref_name },\n+\t{ \"pushRefNeedsUpdate\", &advice_push_ref_needs_update },\n \t{ \"statusHints\", &advice_status_hints },\n \t{ \"statusUoption\", &advice_status_u_option },\n \t{ \"statusAheadBehindWarning\", &advice_status_ahead_behind_warning },\n@@ -116,6 +118,7 @@ static struct {\n \t[ADVICE_PUSH_ALREADY_EXISTS]\t\t\t= { \"pushAlreadyExists\", 1 },\n \t[ADVICE_PUSH_FETCH_FIRST]\t\t\t= { \"pushFetchFirst\", 1 },\n \t[ADVICE_PUSH_NEEDS_FORCE]\t\t\t= { \"pushNeedsForce\", 1 },\n+\t[ADVICE_PUSH_REF_NEEDS_UPDATE]\t\t\t= { \"pushRefNeedsUpdate\", 1 },\n \n \t/* make this an alias for backward compatibility */\n \t[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]\t\t= { \"pushNonFastForward\", 1 },\ndiff --git a/advice.h b/advice.h\nindex 16f2c11642..bc2432980a 100644\n--- a/advice.h\n+++ b/advice.h\n@@ -11,6 +11,7 @@ extern int advice_push_already_exists;\n extern int advice_push_fetch_first;\n extern int advice_push_needs_force;\n extern int advice_push_unqualified_ref_name;\n+extern int advice_push_ref_needs_update;\n extern int advice_status_hints;\n extern int advice_status_u_option;\n extern int advice_status_ahead_behind_warning;\n@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;\n \tADVICE_PUSH_UNQUALIFIED_REF_NAME,\n \tADVICE_PUSH_UPDATE_REJECTED_ALIAS,\n \tADVICE_PUSH_UPDATE_REJECTED,\n+\tADVICE_PUSH_REF_NEEDS_UPDATE,\n \tADVICE_RESET_QUIET_WARNING,\n \tADVICE_RESOLVE_CONFLICT,\n \tADVICE_RM_HINTS,\ndiff --git a/builtin/push.c b/builtin/push.c\nindex 0eeb2c8dd5..908b557edb 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =\n \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n \t   \"without using the '--force' option.\\n\");\n \n+static const char message_advice_ref_needs_update[] =\n+\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n+\t   \"branch has been updated since the last checkout. You may want\\n\"\n+\t   \"to integrate those changes locally (e.g., 'git pull ...')\\n\"\n+\t   \"before forcing an update.\\n\");\n+\n static void advise_pull_before_push(void)\n {\n \tif (!advice_push_non_ff_current || !advice_push_update_rejected)\n@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)\n \tadvise(_(message_advice_ref_needs_force));\n }\n \n+static void advise_ref_needs_update(void)\n+{\n+\tif (!advice_push_ref_needs_update || !advice_push_update_rejected)\n+\t\treturn;\n+\tadvise(_(message_advice_ref_needs_update));\n+}\n+\n static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\t\t     int flags)\n {\n@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\tadvise_ref_fetch_first();\n \t} else if (reject_reasons & REJECT_NEEDS_FORCE) {\n \t\tadvise_ref_needs_force();\n+\t} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {\n+\t\tadvise_ref_needs_update();\n \t}\n \n \treturn 1;\n@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\tif (!v)\n \t\t\treturn config_error_nonbool(k);\n \t\treturn color_parse(v, push_colors[slot]);\n+\t} else if (!strcmp(k, \"push.useforceifincludes\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\t*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\telse\n+\t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\treturn 0;\n \t}\n \n \treturn git_default_config(k, v, NULL);\n@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t\t       N_(\"require old value of ref to be at this value\"),\n \t\t\t       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),\n+\t\tOPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,\n+\t\t\tN_(\"require remote updates to be integrated locally\"),\n+\t\t\tTRANSPORT_PUSH_FORCE_IF_INCLUDES),\n \t\tOPT_CALLBACK(0, \"recurse-submodules\", &recurse_submodules, \"(check|on-demand|no)\",\n \t\t\t     N_(\"control recursive pushing of submodules\"), option_parse_recurse_submodules),\n \t\tOPT_BOOL_F( 0 , \"thin\", &thin, N_(\"use thin pack\"), PARSE_OPT_NOCOMPLETE),\n@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \tif ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))\n \t\tdie(_(\"--all and --mirror are incompatible\"));\n \n+\tif (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))\n+\t\tcas.use_force_if_includes = 1;\n+\n \tfor_each_string_list_item(item, push_options)\n \t\tif (strchr(item->string, '\\n'))\n \t\t\tdie(_(\"push options must not have new line characters\"));\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 4d76727edb..c115a4a1ac 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -159,6 +159,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tint force_if_includes = 0;\n \tstruct packet_reader reader;\n \n \tstruct option options[] = {\n@@ -184,6 +185,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t  N_(\"require old value of ref to be at this value\"),\n \t\t  PARSE_OPT_OPTARG, parseopt_push_cas_option),\n+\t\tOPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,\n+\t\t\t N_(\"require remote updates to be integrated locally\")),\n \t\tOPT_END()\n \t};\n \n@@ -285,6 +288,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tif (!is_empty_cas(&cas))\n \t\tapply_push_cas(&cas, remote, remote_refs);\n \n+\tif (!is_empty_cas(&cas) && force_if_includes)\n+\t\tcas.use_force_if_includes = 1;\n+\n+\n \tset_ref_status_for_push(remote_refs, args.send_mirror,\n \t\targs.force_update);\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 32cc4a0c55..0290b04891 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -44,7 +44,8 @@ struct options {\n \t\tfrom_promisor : 1,\n \n \t\tatomic : 1,\n-\t\tobject_format : 1;\n+\t\tobject_format : 1,\n+\t\tforce_if_includes : 1;\n \tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)\n \t\tstring_list_append(&cas_options, val.buf);\n \t\tstrbuf_release(&val);\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.force_if_includes = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.force_if_includes = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n \t} else if (!strcmp(name, \"cloning\")) {\n \t\tif (!strcmp(value, \"true\"))\n \t\t\toptions.cloning = 1;\n@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)\n \t\tstrvec_push(&args, cas_option->string);\n \tstrvec_push(&args, url.buf);\n \n+\tif (options.force_if_includes)\n+\t\tstrvec_push(&args, \"--force-if-includes\");\n+\n \tstrvec_push(&args, \"--stdin\");\n \tfor (i = 0; i < nr_spec; i++)\n \t\tpacket_buf_write(&preamble, \"%s\\n\", specs[i]);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex e547e21199..298e814bf7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -868,6 +868,11 @@ static void set_common_push_options(struct transport *transport,\n \t\tif (set_helper_option(transport, TRANS_OPT_ATOMIC, \"true\") != 0)\n \t\t\tdie(_(\"helper %s does not support --atomic\"), name);\n \n+\tif (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)\n+\t\tif (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, \"true\") != 0)\n+\t\t\tdie(_(\"helper %s does not support --%s\"),\n+\t\t\t    name, TRANS_OPT_FORCE_IF_INCLUDES);\n+\n \tif (flags & TRANSPORT_PUSH_OPTIONS) {\n \t\tstruct string_list_item *item;\n \t\tfor_each_string_list_item(item, transport->push_options)\ndiff --git a/transport.c b/transport.c\nindex 99fe6233a3..da98ebf50e 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -664,6 +664,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t\t*reject_reasons |= REJECT_FETCH_FIRST;\n \t\t} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {\n \t\t\t*reject_reasons |= REJECT_NEEDS_FORCE;\n+\t\t} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {\n+\t\t\t*reject_reasons |= REJECT_REF_NEEDS_UPDATE;\n \t\t}\n \t}\n \tfree(head);\ndiff --git a/transport.h b/transport.h\nindex ca409ea1e4..24558c027d 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -136,6 +136,7 @@ struct transport {\n #define TRANSPORT_PUSH_ATOMIC\t\t\t(1<<13)\n #define TRANSPORT_PUSH_OPTIONS\t\t\t(1<<14)\n #define TRANSPORT_RECURSE_SUBMODULES_ONLY\t(1<<15)\n+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES\t(1<<16)\n \n int transport_summary_width(const struct ref *refs);\n \n@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);\n /* Request atomic (all-or-nothing) updates when pushing */\n #define TRANS_OPT_ATOMIC \"atomic\"\n \n+/* Require remote changes to be integrated locally. */\n+#define TRANS_OPT_FORCE_IF_INCLUDES \"force-if-includes\"\n+\n /**\n  * Returns 0 if the option was used, non-zero otherwise. Prints a\n  * message to stderr if the option is not used.\n@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,\n void transport_set_verbosity(struct transport *transport, int verbosity,\n \tint force_progress);\n \n-#define REJECT_NON_FF_HEAD     0x01\n-#define REJECT_NON_FF_OTHER    0x02\n-#define REJECT_ALREADY_EXISTS  0x04\n-#define REJECT_FETCH_FIRST     0x08\n-#define REJECT_NEEDS_FORCE     0x10\n+#define REJECT_NON_FF_HEAD      0x01\n+#define REJECT_NON_FF_OTHER     0x02\n+#define REJECT_ALREADY_EXISTS   0x04\n+#define REJECT_FETCH_FIRST      0x08\n+#define REJECT_NEEDS_FORCE      0x10\n+#define REJECT_REF_NEEDS_UPDATE 0x20\n \n int transport_push(struct repository *repo,\n \t\t   struct transport *connection,\n-- \n2.28.0\n\n"},{"id":"406180","messageId":"20200923073022.61293-4-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200923073022.61293-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v5 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-23T07:30:22Z","receivedAt":"2020-09-23T07:30:41Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Update test cases for the new option, and document its usage\nand update related references.\n\n - t/t5533-push-cas.sh:\n   Update test cases for \"compare-and-swap\" when used along with\n   \"--force-if-includes\" helps mitigate overwrites when remote\n   ref are updated in the background.\n\n - Documentation:\n   Add reference for the new option, configuration setting\n   (\"push.useForceIfIncludes\") and advise messages.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/advice.txt |  9 ++++--\n Documentation/config/push.txt   |  6 ++++\n Documentation/git-push.txt      | 26 +++++++++++++++-\n t/t5533-push-cas.sh             | 53 +++++++++++++++++++++++++++++++++\n 4 files changed, 90 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\nindex bdd37c3eaa..acbd0c09aa 100644\n--- a/Documentation/config/advice.txt\n+++ b/Documentation/config/advice.txt\n@@ -10,9 +10,8 @@ advice.*::\n \t\tthat the check is disabled.\n \tpushUpdateRejected::\n \t\tSet this variable to 'false' if you want to disable\n-\t\t'pushNonFFCurrent',\n-\t\t'pushNonFFMatching', 'pushAlreadyExists',\n-\t\t'pushFetchFirst', and 'pushNeedsForce'\n+\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n+\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n \t\tsimultaneously.\n \tpushNonFFCurrent::\n \t\tAdvice shown when linkgit:git-push[1] fails due to a\n@@ -41,6 +40,10 @@ advice.*::\n \t\twe can still suggest that the user push to either\n \t\trefs/heads/* or refs/tags/* based on the type of the\n \t\tsource object.\n+\tpushRefNeedsUpdate::\n+\t\tShown when linkgit:git-push[1] rejects a forced update of\n+\t\ta branch when its remote-tracking ref has updates that we\n+\t\tdo not have locally.\n \tstatusAheadBehind::\n \t\tShown when linkgit:git-status[1] computes the ahead/behind\n \t\tcounts for a local ref compared to its remote tracking ref,\ndiff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\nindex f5e5b38c68..fd981f7808 100644\n--- a/Documentation/config/push.txt\n+++ b/Documentation/config/push.txt\n@@ -114,3 +114,9 @@ push.recurseSubmodules::\n \tspecifying '--recurse-submodules=check|on-demand|no'.\n \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n \tset (in which case a 'true' value means 'on-demand').\n+\n+push.useForceIfIncludes::\n+\tIf set to \"true\", it is equivalent to specifying\n+\t\"--force-if-includes\" as an option to linkgit:git-push[1]\n+\tin the command line. Adding \"--no-force-if-includes\" at the\n+\ttime of push overrides this configuration setting.\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..706380d263 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]\n \t   [-u | --set-upstream] [-o <string> | --push-option=<string>]\n \t   [--[no-]signed|--signed=(true|false|if-asked)]\n-\t   [--force-with-lease[=<refname>[:<expect>]]]\n+\t   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]\n \t   [--no-verify] [<repository> [<refspec>...]]\n \n DESCRIPTION\n@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, specifying \"--force-if-includes\" an an ancillary option\n+along with \"--force-with-lease[=<refname>]\" (i.e., without saying what\n+exact commit the ref on the remote side must be pointing at, or which\n+refs on the remote side are being protected) at the time of \"push\" will\n+verify if updates from the remote-tracking refs that may have been\n+implicitly updated in the background are integrated locally before\n+allowing a forced update.\n \n -f::\n --force::\n@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n origin +master` to force a push to the `master` branch). See the\n `<refspec>...` section above for details.\n \n+--[no-]force-if-includes::\n+\tForce an update only if the tip of the remote-tracking ref\n+\thas been integrated locally.\n++\n+This option enables a check that verifies if the tip of the\n+remote-tracking ref is reachable from one of the \"reflog\" entries of\n+the local branch based in it for a rewrite. The check ensures that any\n+updates from the remote have been incorporated locally by rejecting the\n+forced update if that is not the case.\n++\n+If the option is passed without specifying \"--force-with-lease\", or\n+specified along with \"--force-with-lease=<refname>:<expect>\", it is\n+a \"no-op\".\n++\n+Specifying \"--no-force-if-includes\" disables this behavior.\n+\n --repo=<repository>::\n \tThis option is equivalent to the <repository> argument. If both\n \tare specified, the command-line argument takes precedence.\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..620d101f50 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -256,4 +256,57 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n \n+test_expect_success 'background updates of REMOTE can be mitigated with \"--force-if-includes\"' '\n+\trm -rf src dst &&\n+\tgit init --bare src.bare &&\n+\ttest_when_finished \"rm -rf src.bare\" &&\n+\tgit clone --no-local src.bare dst &&\n+\ttest_when_finished \"rm -rf dst\" &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit G &&\n+\t\tgit push origin master:master\n+\t) &&\n+\tgit clone --no-local src.bare dst2 &&\n+\ttest_when_finished \"rm -rf dst2\" &&\n+\t(\n+\t\tcd dst2 &&\n+\t\ttest_commit H &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit I &&\n+\t\tgit fetch origin &&\n+\t\ttest_must_fail git push --force-with-lease --force-if-includes origin\n+\t)\n+'\n+\n+test_expect_success 'background updates of REMOTE can be mitigated with \"push.useForceIfIncludes\"' '\n+\trm -rf src dst &&\n+\tgit init --bare src.bare &&\n+\ttest_when_finished \"rm -rf src.bare\" &&\n+\tgit clone --no-local src.bare dst &&\n+\ttest_when_finished \"rm -rf dst\" &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit G &&\n+\t\tgit push origin master:master\n+\t) &&\n+\tgit clone --no-local src.bare dst2 &&\n+\ttest_when_finished \"rm -rf dst2\" &&\n+\t(\n+\t\tcd dst2 &&\n+\t\ttest_commit H &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dst &&\n+\t\ttest_commit I &&\n+\t\tgit fetch origin &&\n+\t\tgit config --local push.useForceIfIncludes \"true\" &&\n+\t\ttest_must_fail git push --force-with-lease origin\n+\t)\n+'\n+\n test_done\n-- \n2.28.0\n\n"},{"id":"406188","messageId":"3c254220-de10-8e17-1175-f88e790c17a6@gmail.com","threadId":"54190","inReplyTo":"20200923073022.61293-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v5 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-09-23T10:18:43Z","receivedAt":"2020-09-23T10:18:49Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Srinidhi\n\nI think this is moving forward in the right direction, I've got a couple \nof comments below. Note I've only looked at the first part of the patch \nas I'm not that familiar with the rest of the code.\n\nOn 23/09/2020 08:30, Srinidhi Kaushik wrote:\n> Add a check to verify if the remote-tracking ref of the local branch\n> is reachable from one of its \"reflog\" entries.\n> \n> When a local branch that is based on a remote ref, has been rewound\n> and is to be force pushed on the remote, \"--force-if-includes\" runs\n> a check that ensures any updates to remote-tracking refs that may have\n\nnit pick - there is only one remote tracking ref for each local ref\n\n> happened (by push from another repository) in-between the time of the\n\ns/push/fetch/ ?\n\n> last checkout,\n\nmore generally it is the last time we updated the local branch to \nincorporate any fetched changes in the remote tracking branch, this \nincludes `pull --rebase` `pull --merge` as well as checking out the \nremote ref\n\n> and right before the time of push, have been integrated\n> locally before allowing a forced updated.\n> \n> A new field \"use_force_if_includes\" has been added to \"push_cas_option\",\n> which is set to \"1\" when \"--force-if-includes\" is specified as an\n> option in the command line or as a configuration setting.\n> \n> The struct \"ref\" has two new bit-fields:\n>    - check_reachable:\n>      Set when we have to run the new check on the ref, and the remote\n>      ref was marked as \"use_tracking\" or \"use_tracking_for_rest\" by\n>      compare-and-swap (if the \"the remote tip must be at the expected\n>      commit\" condition is not specified); \"apply_push_cas()\" has been\n>      updated to check if this field is set and run the check.\n> \n>    - unreachable:\n>      Set if the ref is unreachable from any of the \"reflog\" entries of\n>      its local counterpart.\n> \n> \"REF_STATUS_REJECT_REMOTE_UPDATED\" has been added to the \"status\"\n> enum to imply that the ref failed the check; \"case\" statements in\n> \"send-pack\", \"transport\" and \"transport-helper\" have been updated\n> accordingly to catch this status when set.\n\nThis is quite a long description of the implementation, I think it would \nbe more helpful to the reader to concentrate on what the new feature is \nand why it is useful.\n\n> \n> When \"--force-is-includes\" is used along with \"--force-with-lease\",\n\ns/-is-/-if-/\n\n> the check runs only for refs marked as \"if_includes\". If the option\n> is passed without specifying \"--force-with-lease\", or specified along\n> with \"--force-with-lease=<refname>:<expect>\" it is a \"no-op\".\n\nIf I've understood this correctly `--force-if-includes` does nothing on \nits own - I had hoped it would imply --force-with-lease\n\n> \n> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n> ---\n>   builtin/send-pack.c |   5 ++\n>   remote.c            | 140 +++++++++++++++++++++++++++++++++++++++++++-\n>   remote.h            |   8 ++-\n>   send-pack.c         |   1 +\n>   transport-helper.c  |   5 ++\n>   transport.c         |   6 ++\n>   6 files changed, 162 insertions(+), 3 deletions(-)\n> \n> diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n> index 2b9610f121..4d76727edb 100644\n> --- a/builtin/send-pack.c\n> +++ b/builtin/send-pack.c\n> @@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)\n>   \t\t\tmsg = \"stale info\";\n>   \t\t\tbreak;\n>   \n> +\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n> +\t\t\tres = \"error\";\n> +\t\t\tmsg = \"remote ref updated since checkout\";\n> +\t\t\tbreak;\n> +\n>   \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n>   \t\t\tres = \"error\";\n>   \t\t\tmsg = \"already exists\";\n> diff --git a/remote.c b/remote.c\n> index eafc14cbe7..0dcac4ab8e 100644\n> --- a/remote.c\n> +++ b/remote.c\n> @@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n>   \t\t * with the remote-tracking branch to find the value\n>   \t\t * to expect, but we did not have such a tracking\n>   \t\t * branch.\n> +\t\t *\n> +\t\t * If the tip of the remote-tracking ref is unreachable\n> +\t\t * from any reflog entry of its local ref indicating a\n> +\t\t * possible update since checkout; reject the push.\n>   \t\t */\n>   \t\tif (ref->expect_old_sha1) {\n>   \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n>   \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n> +\t\t\telse if (ref->check_reachable && ref->unreachable)\n> +\t\t\t\treject_reason =\n> +\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n>   \t\t\telse\n> -\t\t\t\t/* If the ref isn't stale then force the update. */\n> +\t\t\t\t/*\n> +\t\t\t\t * If the ref isn't stale, and is reachable\n> +\t\t\t\t * from from one of the reflog entries of\n> +\t\t\t\t * the local branch, force the update.\n> +\t\t\t\t */\n>   \t\t\t\tforce_ref_update = 1;\n>   \t\t}\n>   \n> @@ -2268,6 +2279,118 @@ static int remote_tracking(struct remote *remote, const char *refname,\n>   \treturn 0;\n>   }\n>   \n> +/*\n> + * The struct \"reflog_commit_list\" and related helper functions\n> + * for list manipulation are used for collecting commits into a\n> + * list during reflog traversals in \"if_exists_or_grab_until()\".\n> + */\n> +struct reflog_commit_list {\n> +\tstruct commit **items;\n> +\tsize_t nr, alloc;\n> +};\n> +\n> +/* Adds a commit to list. */\n> +static void add_commit(struct reflog_commit_list *list, struct commit *commit)\n> +{\n> +\tALLOC_GROW(list->items, list->nr + 1, list->alloc);\n> +\tlist->items[list->nr++] = commit;\n> +}\n> +\n> +/* Free and reset the list. */\n> +static void free_reflog_commit_list(struct reflog_commit_list *list)\n> +{\n> +\tFREE_AND_NULL(list->items);\n> +\tlist->nr = list->alloc = 0;\n> +}\n> +\n> +struct check_and_collect_until_cb_data {\n> +\tstruct commit *remote_commit;\n> +\tstruct reflog_commit_list *local_commits;\n> +};\n> +\n> +\n> +static int check_and_collect_until(struct object_id *o_oid,\n> +\t\t\t\t   struct object_id *n_oid,\n> +\t\t\t\t   const char *ident, timestamp_t timestamp,\n> +\t\t\t\t   int tz, const char *message, void *cb_data)\n> +{\n> +\tstruct commit *commit;\n> +\tstruct check_and_collect_until_cb_data *cb = cb_data;\n> +\n> +\t/*\n> +\t * If the reflog entry timestamp is older than the\n> +\t * remote commit date, there is no need to check or\n> +\t * collect entries older than this one.\n> +\t */\n> +\tif (timestamp < cb->remote_commit->date)\n\nIt's great that you've incorporated a date check, however I think we \nneed to check the local reflog timestamp against the last time the \nremote ref was updated (i.e. the remote reflog timestamp), not the \ncommit date of the commit that the remote ref points too. We are \ninterested in whether the local branch has incorporated the remote \nbranch since the last time the remote branch was updated.\n\n> +\t\treturn -1;\n> +\n> +\t/* An entry was found. */\n> +\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n> +\t\treturn 1;\n> +\n> +\t/* Lookup the commit and append it to the list. */\n> +\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n> +\t\tadd_commit(cb->local_commits, commit);\n\nI think Junio suggested batching the commits for the merge base check \ninto small groups, rather than checking them all at once\n\nBest Wishes\n\nPhillip\n\n> +\n> +\treturn 0;\n> +}\n> +\n> +/*\n> + * Iterate through the reflog of a local ref to check if there is an entry for\n> + * the given remote-tracking ref (i.e., if it was checked out); runs until the\n> + * timestamp of an entry is older than the commit date of the remote-tracking\n> + * ref. Any commits that seen along the way are collected into a list to check\n> + * if the remote-tracking ref is reachable from any of them.\n> + */\n> +static int is_reachable_in_reflog(const char *local_ref_name,\n> +\t\t\t\t  const struct object_id *remote_oid)\n> +{\n> +\tstruct commit *remote_commit;\n> +\tstruct reflog_commit_list list = { NULL, 0, 0 };\n> +\tstruct check_and_collect_until_cb_data cb;\n> +\tint ret = 0;\n> +\n> +\tremote_commit = lookup_commit_reference(the_repository, remote_oid);\n> +\tif (!remote_commit)\n> +\t\tgoto cleanup_return;\n> +\n> +\tcb.remote_commit = remote_commit;\n> +\tcb.local_commits = &list;\n> +\tret = for_each_reflog_ent_reverse(local_ref_name,\n> +\t\t\t\t\t  check_and_collect_until, &cb);\n> +\n> +\t/* We found an entry in the reflog. */\n> +\tif (ret > 0)\n> +\t\tgoto cleanup_return;\n> +\n> +\t/*\n> +\t * Check if \"remote_commit\" is reachable from\n> +\t * any of the commits in the collected list.\n> +\t */\n> +\tif (list.nr > 0)\n> +\t\tret = in_merge_bases_many(remote_commit, list.nr, list.items);\n> +\n> +cleanup_return:\n> +\tfree_reflog_commit_list(&list);\n> +\treturn ret;\n> +}\n> +\n> +/*\n> + * Check for reachability of a remote-tracking\n> + * ref in the reflog entries of its local ref.\n> + */\n> +static void check_if_includes_upstream(struct ref *remote_ref)\n> +{\n> +\tstruct ref *local_ref = get_local_ref(remote_ref->name);\n> +\n> +\tif (!local_ref)\n> +\t\treturn;\n> +\n> +\tif (!is_reachable_in_reflog(local_ref->name, &remote_ref->old_oid))\n> +\t\tremote_ref->unreachable = 1;\n> +}\n> +\n>   static void apply_cas(struct push_cas_option *cas,\n>   \t\t      struct remote *remote,\n>   \t\t      struct ref *ref)\n> @@ -2284,6 +2407,8 @@ static void apply_cas(struct push_cas_option *cas,\n>   \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n>   \t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n>   \t\t\toidclr(&ref->old_oid_expect);\n> +\t\telse\n> +\t\t\tref->check_reachable = cas->use_force_if_includes;\n>   \t\treturn;\n>   \t}\n>   \n> @@ -2294,6 +2419,8 @@ static void apply_cas(struct push_cas_option *cas,\n>   \tref->expect_old_sha1 = 1;\n>   \tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n>   \t\toidclr(&ref->old_oid_expect);\n> +\telse\n> +\t\tref->check_reachable = cas->use_force_if_includes;\n>   }\n>   \n>   void apply_push_cas(struct push_cas_option *cas,\n> @@ -2301,6 +2428,15 @@ void apply_push_cas(struct push_cas_option *cas,\n>   \t\t    struct ref *remote_refs)\n>   {\n>   \tstruct ref *ref;\n> -\tfor (ref = remote_refs; ref; ref = ref->next)\n> +\tfor (ref = remote_refs; ref; ref = ref->next) {\n>   \t\tapply_cas(cas, remote, ref);\n> +\n> +\t\t/*\n> +\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n> +\t\t * mode, and if \"--force-if-includes\" was specified, run\n> +\t\t * the check.\n> +\t\t */\n> +\t\tif (ref->check_reachable)\n> +\t\t\tcheck_if_includes_upstream(ref);\n> +\t}\n>   }\n> diff --git a/remote.h b/remote.h\n> index 5e3ea5a26d..7c5e59770e 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -104,7 +104,11 @@ struct ref {\n>   \t\tforced_update:1,\n>   \t\texpect_old_sha1:1,\n>   \t\texact_oid:1,\n> -\t\tdeletion:1;\n> +\t\tdeletion:1,\n> +\t\t/* Need to check if local reflog reaches the remote tip. */\n> +\t\tcheck_reachable:1,\n> +\t\t/* The local reflog does not reach the remote tip. */\n> +\t\tunreachable:1;\n>   \n>   \tenum {\n>   \t\tREF_NOT_MATCHED = 0, /* initial value */\n> @@ -134,6 +138,7 @@ struct ref {\n>   \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n>   \t\tREF_STATUS_REJECT_STALE,\n>   \t\tREF_STATUS_REJECT_SHALLOW,\n> +\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n>   \t\tREF_STATUS_UPTODATE,\n>   \t\tREF_STATUS_REMOTE_REJECT,\n>   \t\tREF_STATUS_EXPECTING_REPORT,\n> @@ -332,6 +337,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);\n>   \n>   struct push_cas_option {\n>   \tunsigned use_tracking_for_rest:1;\n> +\tunsigned use_force_if_includes:1;\n>   \tstruct push_cas {\n>   \t\tstruct object_id expect;\n>   \t\tunsigned use_tracking:1;\n> diff --git a/send-pack.c b/send-pack.c\n> index 632f1580ca..956306e8e8 100644\n> --- a/send-pack.c\n> +++ b/send-pack.c\n> @@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n>   \tcase REF_STATUS_REJECT_FETCH_FIRST:\n>   \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n>   \tcase REF_STATUS_REJECT_STALE:\n> +\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n>   \tcase REF_STATUS_REJECT_NODELETE:\n>   \t\treturn CHECK_REF_STATUS_REJECTED;\n>   \tcase REF_STATUS_UPTODATE:\n> diff --git a/transport-helper.c b/transport-helper.c\n> index c52c99d829..e547e21199 100644\n> --- a/transport-helper.c\n> +++ b/transport-helper.c\n> @@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,\n>   \t\t\tstatus = REF_STATUS_REJECT_STALE;\n>   \t\t\tFREE_AND_NULL(msg);\n>   \t\t}\n> +\t\telse if (!strcmp(msg, \"remote ref updated since checkout\")) {\n> +\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n> +\t\t\tFREE_AND_NULL(msg);\n> +\t\t}\n>   \t\telse if (!strcmp(msg, \"forced update\")) {\n>   \t\t\tforced = 1;\n>   \t\t\tFREE_AND_NULL(msg);\n> @@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,\n>   \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n>   \t\tcase REF_STATUS_REJECT_STALE:\n>   \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n> +\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n>   \t\t\tif (atomic) {\n>   \t\t\t\treject_atomic_push(remote_refs, mirror);\n>   \t\t\t\tstring_list_clear(&cas_options, 0);\n> diff --git a/transport.c b/transport.c\n> index 43e24bf1e5..99fe6233a3 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,\n>   \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n>   \t\t\t\t \"stale info\", porcelain, summary_width);\n>   \t\tbreak;\n> +\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n> +\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n> +\t\t\t\t \"remote ref updated since checkout\",\n> +\t\t\t\t porcelain, summary_width);\n> +\t\tbreak;\n>   \tcase REF_STATUS_REJECT_SHALLOW:\n>   \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n>   \t\t\t\t \"new shallow roots not allowed\",\n> @@ -1101,6 +1106,7 @@ static int run_pre_push_hook(struct transport *transport,\n>   \t\tif (!r->peer_ref) continue;\n>   \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n>   \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n> +\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n>   \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n>   \n>   \t\tstrbuf_reset(&buf);\n> \n\n"},{"id":"406189","messageId":"20200923102215.GA70669@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqimc7ezk2.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-23T10:22:15Z","receivedAt":"2020-09-23T10:22:30Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Junio,\n\nOn 09/21/2020 11:48, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> >> If we were talking about older parts of the history, optional\n> >> generation numbers could change the equation somewhat, but the\n> >> common case for the workflow this series is trying to help is that\n> >> these local commits ane the remote tip are relatively new and it is\n> >> not unlikely that the generation numbers have not been computed for\n> >> them, which is why I suspect that in_merges_many may be a win.\n> >\n> > Nice! We can definitely try batching commits from the reflog and\n> > pass it along to \"in_merge_bases_many()\". As for being faster than\n> > calling \"in_merge_bases()\" for each commit entry in the reflog --\n> > I am not familiar with how the former works. Do we still keep the\n> > \"reflog_entry_exists()\" part? It might still be faster to go through\n> > the entries once to check with \"oideq()\" in the first run.\n> \n> That is what I meant.  You go through local reflog entries until you\n> find one that is older than the timestamp of the reflog entry of the\n> remote-tracking branch, check with oideq() to see if the tip was ever\n> directly checked out.  Then, using these same local reflog entries,\n> you can make in_merge_bases_many() tranversal to see if any of them\n> reach the tip.  I suspect that the number of local reflog entries you\n> need to examine would not be too many, so if you can put them all in\n> a single array of \"struct commit *\" pointers in the first \"oideq()\"\n> phase, you may be able to do just a single in_merge_bases_many() batch\n> to check for the reachability.\n\nGotcha.\n\n> > Also, I was wondering if it is worth considering this:\n> >   - check if the reflog of the HEAD has the remote ref\n> \n> It would help the workflow I had in mind, but it would raise the\n> risk of false positives according to Dscho and I tend to agree, so\n> I do not know if it is overall a good idea.\n\nOh, right. This doesn't work when a \"git pull --rebase\" is run on\na different branch (and a few other cases, as mentioned by Johannes).\n\n> >   - check if the reflog of the local branch has the remote ref\n> \n> Isn't that the oideq() test?\n\nYes.\n\n> >   - check if the remote ref is reachable from any of the local ref's\n> >     \"reflog\" entries using \"in_merge_bases_many()\" in batches as\n> >     suggested here.\n> \n> I think it amounts to the same as \"does any reflog entry of HEAD\n> reach it?\" and shares the same issues with false positives as the\n> first one.\n\nHmm, isn't this the same as what was mentioned by you earlier (without\nthe timestamp:\n\n> [...] Then, using these same local reflog entries,\n> you can make in_merge_bases_many() tranversal to see if any of them\n> reach the tip.\n\nIn v5 (the new patch) [1], the check does this:\n  - go through the local reflog until it hits an entry with a timestamp\n    older than the remote commit, and doing an \"oideq()\" check and\n    collecting commits into a list along the way.\n\n  - if an exact entry was found, the test passes; otherwise use\n    the commit list and make a call to \"in_merge_bases_many()\" to\n    check for reachability, and report it.\n\n> >> > +\t\tdeletion:1,\n> >> > +\t\tif_includes:1, /* If \"--force-with-includes\" was specified. */\n> >> \n> >> The description needs to be tightened.\n> >> \n> >> > +\t\tunreachable:1; /* For \"if_includes\"; unreachable in reflog. */\n> >\n> > OK, you're right. Perhaps, we could rename it to something like\n> > \"if_includes_for_tracking\" and update the comment description\n> > with saying something along the lines of:\n> \n> That is overlong.  Let me try:\n> \n> \t\t/* need to check if local reflog reaches the remote tip */\n> \t\tcheck_reachable:1,\n> \n> \t\t/* local reflog does not reach the remote tip */\n> \t\tunreachable:1;\n> \n\nI have updated the description in v5 [1]; thanks!\n\n[1]: https://public-inbox.org/git/20200923073022.61293-1-shrinidhi.kaushik@gmail.com/\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"406190","messageId":"65574677-faad-f13c-810f-68e0d07479b1@gmail.com","threadId":"54190","inReplyTo":"20200923073022.61293-4-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v5 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-09-23T10:24:26Z","receivedAt":"2020-09-23T10:24:30Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Srinidhi\n\nOn 23/09/2020 08:30, Srinidhi Kaushik wrote:\n> Update test cases for the new option, and document its usage\n> and update related references.\n> \n>   - t/t5533-push-cas.sh:\n>     Update test cases for \"compare-and-swap\" when used along with\n>     \"--force-if-includes\" helps mitigate overwrites when remote\n>     ref are updated in the background.\n> \n>   - Documentation:\n>     Add reference for the new option, configuration setting\n>     (\"push.useForceIfIncludes\") and advise messages.\n> \n> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n> ---\n>   Documentation/config/advice.txt |  9 ++++--\n>   Documentation/config/push.txt   |  6 ++++\n>   Documentation/git-push.txt      | 26 +++++++++++++++-\n>   t/t5533-push-cas.sh             | 53 +++++++++++++++++++++++++++++++++\n>   4 files changed, 90 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\n> index bdd37c3eaa..acbd0c09aa 100644\n> --- a/Documentation/config/advice.txt\n> +++ b/Documentation/config/advice.txt\n> @@ -10,9 +10,8 @@ advice.*::\n>   \t\tthat the check is disabled.\n>   \tpushUpdateRejected::\n>   \t\tSet this variable to 'false' if you want to disable\n> -\t\t'pushNonFFCurrent',\n> -\t\t'pushNonFFMatching', 'pushAlreadyExists',\n> -\t\t'pushFetchFirst', and 'pushNeedsForce'\n> +\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n> +\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n>   \t\tsimultaneously.\n>   \tpushNonFFCurrent::\n>   \t\tAdvice shown when linkgit:git-push[1] fails due to a\n> @@ -41,6 +40,10 @@ advice.*::\n>   \t\twe can still suggest that the user push to either\n>   \t\trefs/heads/* or refs/tags/* based on the type of the\n>   \t\tsource object.\n> +\tpushRefNeedsUpdate::\n> +\t\tShown when linkgit:git-push[1] rejects a forced update of\n> +\t\ta branch when its remote-tracking ref has updates that we\n> +\t\tdo not have locally.\n>   \tstatusAheadBehind::\n>   \t\tShown when linkgit:git-status[1] computes the ahead/behind\n>   \t\tcounts for a local ref compared to its remote tracking ref,\n> diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\n> index f5e5b38c68..fd981f7808 100644\n> --- a/Documentation/config/push.txt\n> +++ b/Documentation/config/push.txt\n> @@ -114,3 +114,9 @@ push.recurseSubmodules::\n>   \tspecifying '--recurse-submodules=check|on-demand|no'.\n>   \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n>   \tset (in which case a 'true' value means 'on-demand').\n> +\n> +push.useForceIfIncludes::\n> +\tIf set to \"true\", it is equivalent to specifying\n> +\t\"--force-if-includes\" as an option to linkgit:git-push[1]\n\nI think we normally use backquotes for option names in the documentation \nso this would be `--force-if-includes`\n\n> +\tin the command line. Adding \"--no-force-if-includes\" at the\n> +\ttime of push overrides this configuration setting.\n> diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\n> index 3b8053447e..706380d263 100644\n> --- a/Documentation/git-push.txt\n> +++ b/Documentation/git-push.txt\n> @@ -13,7 +13,7 @@ SYNOPSIS\n>   \t   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]\n>   \t   [-u | --set-upstream] [-o <string> | --push-option=<string>]\n>   \t   [--[no-]signed|--signed=(true|false|if-asked)]\n> -\t   [--force-with-lease[=<refname>[:<expect>]]]\n> +\t   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]\n>   \t   [--no-verify] [<repository> [<refspec>...]]\n>   \n>   DESCRIPTION\n> @@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally\n>   force push changes to `master` if the remote version is still at\n>   `base`, regardless of what your local `remotes/origin/master` has been\n>   updated to in the background.\n> ++\n> +Alternatively, specifying \"--force-if-includes\" an an ancillary option\n> +along with \"--force-with-lease[=<refname>]\" (i.e., without saying what\n> +exact commit the ref on the remote side must be pointing at, or which\n> +refs on the remote side are being protected) at the time of \"push\" will\n> +verify if updates from the remote-tracking refs that may have been\n> +implicitly updated in the background are integrated locally before\n> +allowing a forced update.\n>   \n>   -f::\n>   --force::\n> @@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n>   origin +master` to force a push to the `master` branch). See the\n>   `<refspec>...` section above for details.\n>   \n> +--[no-]force-if-includes::\n> +\tForce an update only if the tip of the remote-tracking ref\n> +\thas been integrated locally.\n> ++\n> +This option enables a check that verifies if the tip of the\n> +remote-tracking ref is reachable from one of the \"reflog\" entries of\n> +the local branch based in it for a rewrite. The check ensures that any\n> +updates from the remote have been incorporated locally by rejecting the\n> +forced update if that is not the case.\n> ++\n> +If the option is passed without specifying \"--force-with-lease\", or\n> +specified along with \"--force-with-lease=<refname>:<expect>\", it is\n> +a \"no-op\".\n> ++\n> +Specifying \"--no-force-if-includes\" disables this behavior.\n> +\n>   --repo=<repository>::\n>   \tThis option is equivalent to the <repository> argument. If both\n>   \tare specified, the command-line argument takes precedence.\n> diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\n> index 0b0eb1d025..620d101f50 100755\n> --- a/t/t5533-push-cas.sh\n> +++ b/t/t5533-push-cas.sh\n> @@ -256,4 +256,57 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n>   \t)\n>   '\n\nIt's great to see a couple of tests, however I think it would be useful \nto have some more - one to check that we do not force when the remote is \nrewound and a couple to show that we do force when we should. In \nparticular we should still force if we `pull --rebase` and then rewrite \nthe local history using `rebase -i` so that the remote ref is no longer \nan ancestor of the local HEAD.\n\nBest Wishes\n\nPhillip\n\n> +test_expect_success 'background updates of REMOTE can be mitigated with \"--force-if-includes\"' '\n> +\trm -rf src dst &&\n> +\tgit init --bare src.bare &&\n> +\ttest_when_finished \"rm -rf src.bare\" &&\n> +\tgit clone --no-local src.bare dst &&\n> +\ttest_when_finished \"rm -rf dst\" &&\n> +\t(\n> +\t\tcd dst &&\n> +\t\ttest_commit G &&\n> +\t\tgit push origin master:master\n> +\t) &&\n> +\tgit clone --no-local src.bare dst2 &&\n> +\ttest_when_finished \"rm -rf dst2\" &&\n> +\t(\n> +\t\tcd dst2 &&\n> +\t\ttest_commit H &&\n> +\t\tgit push\n> +\t) &&\n> +\t(\n> +\t\tcd dst &&\n> +\t\ttest_commit I &&\n> +\t\tgit fetch origin &&\n> +\t\ttest_must_fail git push --force-with-lease --force-if-includes origin\n> +\t)\n> +'\n> +\n> +test_expect_success 'background updates of REMOTE can be mitigated with \"push.useForceIfIncludes\"' '\n> +\trm -rf src dst &&\n> +\tgit init --bare src.bare &&\n> +\ttest_when_finished \"rm -rf src.bare\" &&\n> +\tgit clone --no-local src.bare dst &&\n> +\ttest_when_finished \"rm -rf dst\" &&\n> +\t(\n> +\t\tcd dst &&\n> +\t\ttest_commit G &&\n> +\t\tgit push origin master:master\n> +\t) &&\n> +\tgit clone --no-local src.bare dst2 &&\n> +\ttest_when_finished \"rm -rf dst2\" &&\n> +\t(\n> +\t\tcd dst2 &&\n> +\t\ttest_commit H &&\n> +\t\tgit push\n> +\t) &&\n> +\t(\n> +\t\tcd dst &&\n> +\t\ttest_commit I &&\n> +\t\tgit fetch origin &&\n> +\t\tgit config --local push.useForceIfIncludes \"true\" &&\n> +\t\ttest_must_fail git push --force-with-lease origin\n> +\t)\n> +'\n> +\n>   test_done\n> \n\n"},{"id":"406191","messageId":"20200923102711.GB70669@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"f6bb2b1f-0f1b-f196-59f1-893580430cf2@gmail.com","subject":"Re: [PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-23T10:27:11Z","receivedAt":"2020-09-23T10:27:19Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Phillip,\n\nOn 09/21/2020 14:19, Phillip Wood wrote:\n> As I said before[1] I think we should also be checking the reflog dates so\n> that we do not look at any local reflog entries that are older than the most\n> recent reflog entry for the remote tracking branch. This protects against a\n> background fetch when the remote has been rewound and it would also reduce\n> the number of calls to in_merge_bases().\n\nThanks for suggesting this; I must have missed it earlier (sorry).\nThe latest patch [1], has been updated to take timestamps into\naccount which helps reduce the number of reflog iterations.\n\n[1]: https://public-inbox.org/git/20200923073022.61293-1-shrinidhi.kaushik@gmail.com/\n\nThanks.\n--\nSrinidhi Kaushik\n"},{"id":"406192","messageId":"20200923112606.GA71705@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"3c254220-de10-8e17-1175-f88e790c17a6@gmail.com","subject":"Re: [PATCH v5 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-23T11:26:06Z","receivedAt":"2020-09-23T11:26:17Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Phillip,\n\nOn 09/23/2020 11:18, Phillip Wood wrote:\n> Hi Srinidhi\n> \n> I think this is moving forward in the right direction, I've got a couple of\n> comments below. Note I've only looked at the first part of the patch as I'm\n> not that familiar with the rest of the code.\n> \n> On 23/09/2020 08:30, Srinidhi Kaushik wrote:\n> > Add a check to verify if the remote-tracking ref of the local branch\n> > is reachable from one of its \"reflog\" entries.\n> > \n> > When a local branch that is based on a remote ref, has been rewound\n> > and is to be force pushed on the remote, \"--force-if-includes\" runs\n> > a check that ensures any updates to remote-tracking refs that may have\n> \n> nit pick - there is only one remote tracking ref for each local ref\n\nYou're right, I will correct that. \n> > happened (by push from another repository) in-between the time of the\n> \n> s/push/fetch/ ?\n\nWell, I was hoping it implies that there was a push made by another\nperson to the ref while a rewrite is happening locally. \n \n> > last checkout,\n>\n> more generally it is the last time we updated the local branch to\n> incorporate any fetched changes in the remote tracking branch, this includes\n> `pull --rebase` `pull --merge` as well as checking out the remote ref\n> \n> > and right before the time of push, have been integrated\n> > locally before allowing a forced updated.\n> > \n> > A new field \"use_force_if_includes\" has been added to \"push_cas_option\",\n> > which is set to \"1\" when \"--force-if-includes\" is specified as an\n> > option in the command line or as a configuration setting.\n> > \n> > The struct \"ref\" has two new bit-fields:\n> >    - check_reachable:\n> >      Set when we have to run the new check on the ref, and the remote\n> >      ref was marked as \"use_tracking\" or \"use_tracking_for_rest\" by\n> >      compare-and-swap (if the \"the remote tip must be at the expected\n> >      commit\" condition is not specified); \"apply_push_cas()\" has been\n> >      updated to check if this field is set and run the check.\n> > \n> >    - unreachable:\n> >      Set if the ref is unreachable from any of the \"reflog\" entries of\n> >      its local counterpart.\n> > \n> > \"REF_STATUS_REJECT_REMOTE_UPDATED\" has been added to the \"status\"\n> > enum to imply that the ref failed the check; \"case\" statements in\n> > \"send-pack\", \"transport\" and \"transport-helper\" have been updated\n> > accordingly to catch this status when set.\n> \n> This is quite a long description of the implementation, I think it would be\n> more helpful to the reader to concentrate on what the new feature is and why\n> it is useful.\n\nOK, I will try to minimize the description a bit further.\n \n> > \n> > When \"--force-is-includes\" is used along with \"--force-with-lease\",\n> \n> s/-is-/-if-/\n\nOh, this happened again. Sorry!\n\n> > the check runs only for refs marked as \"if_includes\". If the option\n> > is passed without specifying \"--force-with-lease\", or specified along\n> > with \"--force-with-lease=<refname>:<expect>\" it is a \"no-op\".\n\nThere's another typo: s/if_includes/check_reachable/.\n \n> If I've understood this correctly `--force-if-includes` does nothing on its\n> own - I had hoped it would imply --force-with-lease\n\nYes, it does nothing if not used with  \"--force-with-lease\", or when\nused with it and the exact commit to be expected is specified for\n\"--force-with-lease\" (i.e., with --force-with-lease=<ref>:<expect>).\n\n> > [...]\n> It's great that you've incorporated a date check, however I think we need to\n> check the local reflog timestamp against the last time the remote ref was\n> updated (i.e. the remote reflog timestamp), not the commit date of the\n> commit that the remote ref points too. We are interested in whether the\n> local branch has incorporated the remote branch since the last time the\n> remote branch was updated.\n\nGot it. I misread the requirement here; will update.\n\n> > +\t\treturn -1;\n> > +\n> > +\t/* An entry was found. */\n> > +\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n> > +\t\treturn 1;\n> > +\n> > +\t/* Lookup the commit and append it to the list. */\n> > +\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n> > +\t\tadd_commit(cb->local_commits, commit);\n> \n> I think Junio suggested batching the commits for the merge base check into\n> small groups, rather than checking them all at once\n\nDoesn't stopping the iteration early with the date check collect fewer\ncommits? Would batching still be necessary?\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"406198","messageId":"xmqqimc4jwac.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"3c254220-de10-8e17-1175-f88e790c17a6@gmail.com","subject":"Re: [PATCH v5 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-23T16:24:43Z","receivedAt":"2020-09-23T16:24:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood <phillip.wood123@gmail.com> writes:\n\n> Hi Srinidhi\n>\n> I think this is moving forward in the right direction, I've got a\n> couple of comments below. Note I've only looked at the first part of\n> the patch as I'm not that familiar with the rest of the code.\n>\n> On 23/09/2020 08:30, Srinidhi Kaushik wrote:\n>> Add a check to verify if the remote-tracking ref of the local branch\n>> is reachable from one of its \"reflog\" entries.\n>> When a local branch that is based on a remote ref, has been rewound\n>> and is to be force pushed on the remote, \"--force-if-includes\" runs\n>> a check that ensures any updates to remote-tracking refs that may have\n>\n> nit pick - there is only one remote tracking ref for each local ref\n\nYup, \"updates to the remote-tracking ref\".\n\n> This is quite a long description of the implementation, I think it\n> would be more helpful to the reader to concentrate on what the new\n> feature is and why it is useful.\n\nThanks for pointing it out.\n\n> It's great that you've incorporated a date check, however I think we\n> need to check the local reflog timestamp against the last time the \n> remote ref was updated (i.e. the remote reflog timestamp), not the\n> commit date of the commit that the remote ref points too. We are \n> interested in whether the local branch has incorporated the remote\n> branch since the last time the remote branch was updated.\n\nYes, exactly.\n\n>\n>> +\t\treturn -1;\n>> +\n>> +\t/* An entry was found. */\n>> +\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n>> +\t\treturn 1;\n>> +\n>> +\t/* Lookup the commit and append it to the list. */\n>> +\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n>> +\t\tadd_commit(cb->local_commits, commit);\n>\n> I think Junio suggested batching the commits for the merge base check\n> into small groups, rather than checking them all at once\n\nThat was before you suggested to cut the reflog traversal using the\nreflog timestamp.  The code in earlier rounds of this series wanted\nto read the reflog entries all the way down to the beginning of time,\nand it made sense to batch in order to limit the traversal.\n\nNow we found out that using only the local reflog entries more\nrecent than the latest update to the remote-tracking branch, and\nbecause these local reflog entries will already be read in the above\ncode anyway, I do not see a point in batching.\n\nEven if we were to batch calls to git_merge_bases_many(), it won't\nhappen in the quoted part of the code, where it first tries to see\nany of the recent local reflog entries directly point at the object\nwith oideq().  While it is doing so, the only thing it does for the\nlater merge-base phase is to collect them in local_commits[] array.\nChunking can, if desired, be done by carving the array into pieces\nand calling git_merge_bases_many() in is_reachable_in_reflog()\nfunction.\n\nThanks for a careful review.\n"},{"id":"406199","messageId":"xmqqeemsjw2u.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200923073022.61293-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v5 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-23T16:29:13Z","receivedAt":"2020-09-23T16:29:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> diff --git a/remote.h b/remote.h\n> index 5e3ea5a26d..7c5e59770e 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -104,7 +104,11 @@ struct ref {\n>  \t\tforced_update:1,\n>  \t\texpect_old_sha1:1,\n>  \t\texact_oid:1,\n> -\t\tdeletion:1;\n> +\t\tdeletion:1,\n> +\t\t/* Need to check if local reflog reaches the remote tip. */\n> +\t\tcheck_reachable:1,\n> +\t\t/* The local reflog does not reach the remote tip. */\n> +\t\tunreachable:1;\n\nI know the comment text above is what I suggested, but it may be\nmore useful to readers to explain the latter as\n\n\t\t/* ... the result of the above check */\n\t\tunreachable:1;\n\nThanks.\n"},{"id":"406202","messageId":"xmqq1risjv8z.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200923102215.GA70669@mail.clickyotomy.dev","subject":"Re: [PATCH v4 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-23T16:47:08Z","receivedAt":"2020-09-23T16:47:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n>> >   - check if the remote ref is reachable from any of the local ref's\n>> >     \"reflog\" entries using \"in_merge_bases_many()\" in batches as\n>> >     suggested here.\n>> \n>> I think it amounts to the same as \"does any reflog entry of HEAD\n>> reach it?\" and shares the same issues with false positives as the\n>> first one.\n>\n> Hmm, isn't this the same as what was mentioned by you earlier (without\n> the timestamp:\n\nI misunderstood what you meant with the \"any of the local ref\" part\nof the sentence and mistook it as \"enumerate all the local refs and\nHEAD, and collect all reflog entries of these local refs, and see if\nany of them reach the remote ref\".  Re-reading the sentence with\nthat \"any of\" refers to \"entries\" (of a single local ref) in mind,\nit is pretty much the same as we agreed is a good way to go.\n\nThanks.\n"},{"id":"406415","messageId":"20200926101324.73457-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200923073022.61293-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v6 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-26T10:13:21Z","receivedAt":"2020-09-26T10:13:37Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new option: \"--force-if-includes\" to \"git-push\" where forced\nupdates are allowed only if the tip of the remote-tracking ref has\nbeen integrated locally, by verifying if the tip of the remote-tracking\nref -- on which a local branch has based on -- is reachable from at\nleast one of the \"reflog\" entries of the branch about to be updated\nby force on the remote.\n\nThis option can be used with \"--force-with-lease\" with setups where\nthe remote-tracking refs of the repository are implicitly updated in\nthe background to help prevent unintended remote overwrites.\n\nIf a local branch is based on a remote ref for a rewrite, and if that\nremote-tracking ref is updated by a push from another repository after\nit has been checked out locally, force updating that branch to remote\nwith \"--force-with-lease[=<refname>[:<expect>]]\" without specifying\nthe \"<expect>\" value, can cause the update that happened in-between\nthe checkout and forced push to be lost.\n\nChanges since v5:\n  - In the previous patch, the stopping condition for \"reflog\"\n    iteration used the commit date of the remote ref. That has\n    been changed to use the the timestamp of the latest entry\n    of the remote ref's \"reflog\"; calls \"in_merge_bases_many()\"\n    are batched when the number of commits collected in the list\n    is more than 8.\n\n  - Add more tests for t5533 to include scenarios where forced\n    updates should be allowed for \"pull --rebase\" and local\n    rebase operations.\n\n  - Reword commit messages, fix typos, and update documentation\n    to use \"`\" where necessary.\n\nSrinidhi Kaushik (3):\n  push: add reflog check for \"--force-if-includes\"\n  push: parse and set flag for \"--force-if-includes\"\n  t, doc: update tests, reference for \"--force-if-includes\"\n\n Documentation/config/advice.txt |   9 +-\n Documentation/config/push.txt   |   6 ++\n Documentation/git-push.txt      |  26 ++++-\n advice.c                        |   3 +\n advice.h                        |   2 +\n builtin/push.c                  |  27 +++++\n builtin/send-pack.c             |  12 +++\n remote-curl.c                   |  14 ++-\n remote.c                        | 185 ++++++++++++++++++++++++++++++--\n remote.h                        |  12 ++-\n send-pack.c                     |   1 +\n t/t5533-push-cas.sh             | 129 ++++++++++++++++++++++\n transport-helper.c              |  10 ++\n transport.c                     |   8 ++\n transport.h                     |  15 ++-\n 15 files changed, 442 insertions(+), 17 deletions(-)\n\nbase-commit: 9bc233ae1cf19a49e51842c7959d80a675dbd1c0\n--\n2.28.0\n"},{"id":"406416","messageId":"20200926101324.73457-2-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200926101324.73457-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v6 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-26T10:13:22Z","receivedAt":"2020-09-26T10:13:39Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a check to verify if the remote-tracking ref of the local branch\nis reachable from one of its \"reflog\" entries.\n\nThe check iterates through the local ref's reflog to see if there\nis an entry for the remote-tracking ref and collecting any commits\nthat are seen, into a list; the iteration stops if an entry in the\nreflog matches the remote ref or if the entry timestamp is older\nthe latest entry of the remote ref's \"reflog\". If there wasn't an\nentry found for the remote ref, \"in_merge_bases_many()\" is called\nto check if it is reachable from the list of collected commits.\n\nWhen a local branch that is based on a remote ref, has been rewound\nand is to be force pushed on the remote, \"--force-if-includes\" runs\na check that ensures any updates to the remote-tracking ref that may\nhave happened (by push from another repository) in-between the time\nof the last update to the local branch (via \"git-pull\", for instance)\nand right before the time of push, have been integrated locally\nbefore allowing a forced update.\n\nIf the new option is passed without specifying \"--force-with-lease\",\nor specified along with \"--force-with-lease=<refname>:<expect>\" it\nis a \"no-op\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n builtin/send-pack.c |   5 ++\n remote.c            | 185 ++++++++++++++++++++++++++++++++++++++++++--\n remote.h            |  12 ++-\n send-pack.c         |   1 +\n transport-helper.c  |   5 ++\n transport.c         |   6 ++\n 6 files changed, 207 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 7af148d733..516cba7336 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n \n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"remote ref updated since checkout\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\ndiff --git a/remote.c b/remote.c\nindex eafc14cbe7..a696243247 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t * with the remote-tracking branch to find the value\n \t\t * to expect, but we did not have such a tracking\n \t\t * branch.\n+\t\t *\n+\t\t * If the tip of the remote-tracking ref is unreachable\n+\t\t * from any reflog entry of its local ref indicating a\n+\t\t * possible update since checkout; reject the push.\n \t\t */\n \t\tif (ref->expect_old_sha1) {\n \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n+\t\t\telse if (ref->check_reachable && ref->unreachable)\n+\t\t\t\treject_reason =\n+\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n \t\t\telse\n-\t\t\t\t/* If the ref isn't stale then force the update. */\n+\t\t\t\t/*\n+\t\t\t\t * If the ref isn't stale, and is reachable\n+\t\t\t\t * from from one of the reflog entries of\n+\t\t\t\t * the local branch, force the update.\n+\t\t\t\t */\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n \n@@ -2252,11 +2263,11 @@ int is_empty_cas(const struct push_cas_option *cas)\n /*\n  * Look at remote.fetch refspec and see if we have a remote\n  * tracking branch for the refname there.  Fill its current\n- * value in sha1[].\n+ * value in sha1[], and as a string.\n  * If we cannot do so, return negative to signal an error.\n  */\n static int remote_tracking(struct remote *remote, const char *refname,\n-\t\t\t   struct object_id *oid)\n+\t\t\t   struct object_id *oid, char **dst_refname)\n {\n \tchar *dst;\n \n@@ -2265,9 +2276,154 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \t\treturn -1; /* no tracking ref for refname at remote */\n \tif (read_ref(dst, oid))\n \t\treturn -1; /* we know what the tracking ref is but we cannot read it */\n+\n+\t*dst_refname = dst;\n+\treturn 0;\n+}\n+\n+/*\n+ * The struct \"reflog_commit_list\" and related helper functions\n+ * for list manipulation are used for collecting commits into a\n+ * list during reflog traversals in \"if_exists_or_grab_until()\".\n+ */\n+struct reflog_commit_list {\n+\tstruct commit **items;\n+\tsize_t nr, alloc;\n+};\n+\n+/* Add a commit to list. */\n+static void add_commit(struct reflog_commit_list *list, struct commit *commit)\n+{\n+\tALLOC_GROW(list->items, list->nr + 1, list->alloc);\n+\tlist->items[list->nr++] = commit;\n+}\n+\n+/* Free and reset the list. */\n+static void free_reflog_commit_list(struct reflog_commit_list *list)\n+{\n+\tFREE_AND_NULL(list->items);\n+\tlist->nr = list->alloc = 0;\n+}\n+\n+struct check_and_collect_until_cb_data {\n+\tstruct commit *remote_commit;\n+\tstruct reflog_commit_list *local_commits;\n+\ttimestamp_t remote_reflog_timestamp;\n+};\n+\n+/* Get the timestamp of the latest entry. */\n+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,\n+\t\t       const char *ident, timestamp_t timestamp,\n+\t\t       int tz, const char *message, void *cb_data)\n+{\n+\ttimestamp_t *ts = cb_data;\n+\t*ts = timestamp;\n+\treturn 1;\n+}\n+\n+static int check_and_collect_until(struct object_id *o_oid,\n+\t\t\t\t   struct object_id *n_oid,\n+\t\t\t\t   const char *ident, timestamp_t timestamp,\n+\t\t\t\t   int tz, const char *message, void *cb_data)\n+{\n+\tstruct commit *commit;\n+\tstruct check_and_collect_until_cb_data *cb = cb_data;\n+\n+\t/*\n+\t * If the reflog entry timestamp is older than the remote ref's\n+\t * latest reflog entry, there is no need to check or collect\n+\t * entries older than this one.\n+\t */\n+\tif (timestamp < cb->remote_reflog_timestamp)\n+\t\treturn -1;\n+\n+\t/* An entry was found. */\n+\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n+\t\treturn 1;\n+\n+\t/* Look-up the commit and append it to the list. */\n+\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n+\t\tadd_commit(cb->local_commits, commit);\n+\n \treturn 0;\n }\n \n+#define MERGE_BASES_BATCH_SIZE 8\n+\n+/*\n+ * Iterate through the reflog of the local ref to check if there is an entry\n+ * for the given remote-tracking ref; runs until the timestamp of an entry is\n+ * older than latest timestamp of remote-tracking ref's reflog. Any commits\n+ * are that seen along the way are collected into a list to check if the\n+ * remote-tracking ref is reachable from any of them.\n+ */\n+static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n+{\n+\ttimestamp_t date;\n+\tstruct commit *commit;\n+\tstruct commit **chunk;\n+\tstruct check_and_collect_until_cb_data cb;\n+\tstruct reflog_commit_list list = { NULL, 0, 0 };\n+\tsize_t count = 0, batch_size = 0;\n+\tint ret = 0;\n+\n+\tcommit = lookup_commit_reference(the_repository, &remote->old_oid);\n+\tif (!commit)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Get the timestamp from the latest entry\n+\t * of the remote-tracking ref's reflog.\n+\t */\n+\tfor_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);\n+\n+\tcb.remote_commit = commit;\n+\tcb.local_commits = &list;\n+\tcb.remote_reflog_timestamp = date;\n+\tret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);\n+\n+\t/* We found an entry in the reflog. */\n+\tif (ret > 0)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Check if the remote commit is reachable from any\n+\t * of the commits in the collected list, in batches.\n+\t */\n+\tchunk = list.items;\n+\twhile (count < list.nr) {\n+\t\tbatch_size = MERGE_BASES_BATCH_SIZE;\n+\n+\t\t/* For any leftover entries. */\n+\t\tif ((count + MERGE_BASES_BATCH_SIZE) > list.nr)\n+\t\t\tbatch_size = list.nr - count;\n+\n+\t\tif ((ret = in_merge_bases_many(commit, batch_size, chunk)))\n+\t\t\tbreak;\n+\n+\t\tchunk += batch_size;\n+\t\tcount += MERGE_BASES_BATCH_SIZE;\n+\t}\n+\n+cleanup_return:\n+\tfree_reflog_commit_list(&list);\n+\treturn ret;\n+}\n+\n+/*\n+ * Check for reachability of a remote-tracking\n+ * ref in the reflog entries of its local ref.\n+ */\n+static void check_if_includes_upstream(struct ref *remote)\n+{\n+\tstruct ref *local = get_local_ref(remote->name);\n+\tif (!local)\n+\t\treturn;\n+\n+\tif (is_reachable_in_reflog(local->name, remote) <= 0)\n+\t\tremote->unreachable = 1;\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n \t\t      struct ref *ref)\n@@ -2282,8 +2438,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\tref->expect_old_sha1 = 1;\n \t\tif (!entry->use_tracking)\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n-\t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\t\telse if (remote_tracking(remote, ref->name,\n+\t\t\t\t\t &ref->old_oid_expect,\n+\t\t\t\t\t &ref->tracking_ref))\n \t\t\toidclr(&ref->old_oid_expect);\n+\t\telse\n+\t\t\tref->check_reachable = cas->use_force_if_includes;\n \t\treturn;\n \t}\n \n@@ -2292,8 +2452,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\treturn;\n \n \tref->expect_old_sha1 = 1;\n-\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\tif (remote_tracking(remote, ref->name,\n+\t\t\t    &ref->old_oid_expect,\n+\t\t\t    &ref->tracking_ref))\n \t\toidclr(&ref->old_oid_expect);\n+\telse\n+\t\tref->check_reachable = cas->use_force_if_includes;\n }\n \n void apply_push_cas(struct push_cas_option *cas,\n@@ -2301,6 +2465,15 @@ void apply_push_cas(struct push_cas_option *cas,\n \t\t    struct ref *remote_refs)\n {\n \tstruct ref *ref;\n-\tfor (ref = remote_refs; ref; ref = ref->next)\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n \t\tapply_cas(cas, remote, ref);\n+\n+\t\t/*\n+\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n+\t\t * mode, and if \"--force-if-includes\" was specified, run\n+\t\t * the check.\n+\t\t */\n+\t\tif (ref->check_reachable)\n+\t\t\tcheck_if_includes_upstream(ref);\n+\t}\n }\ndiff --git a/remote.h b/remote.h\nindex eb62a47044..2d5391d281 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -107,12 +107,20 @@ struct ref {\n \tstruct object_id new_oid;\n \tstruct object_id old_oid_expect; /* used by expect-old */\n \tchar *symref;\n+\tchar *tracking_ref;\n \tunsigned int\n \t\tforce:1,\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\t/* Need to check if local reflog reaches the remote tip. */\n+\t\tcheck_reachable:1,\n+\t\t/*\n+\t\t * Store the result of the check enabled by \"check_reachable\";\n+\t\t * implies the local reflog does not reach the remote tip.\n+\t\t */\n+\t\tunreachable:1;\n \n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -142,6 +150,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n \t\tREF_STATUS_REJECT_SHALLOW,\n+\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT,\n@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);\n \n struct push_cas_option {\n \tunsigned use_tracking_for_rest:1;\n+\tunsigned use_force_if_includes:1;\n \tstruct push_cas {\n \t\tstruct object_id expect;\n \t\tunsigned use_tracking:1;\ndiff --git a/send-pack.c b/send-pack.c\nindex c9698070fc..eb4a44270b 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\ndiff --git a/transport-helper.c b/transport-helper.c\nindex b573b6c730..6157de30c7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"remote ref updated since checkout\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\n \t\t\t\tstring_list_clear(&cas_options, 0);\ndiff --git a/transport.c b/transport.c\nindex ffe2115845..75f1786aca 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,\n \t\t\t\t \"stale info\",\n \t\t\t\t report, porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"remote ref updated since checkout\",\n+\t\t\t\t porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n \n \t\tstrbuf_reset(&buf);\n-- \n2.28.0\n\n"},{"id":"406417","messageId":"20200926101324.73457-4-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200926101324.73457-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v6 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-26T10:13:24Z","receivedAt":"2020-09-26T10:13:43Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Update test cases for the new option, and document its usage\nand update related references.\n\nUpdate test cases for the new option, and document its usage\nand update related references.\n\n - t/t5533-push-cas.sh:\n   Update test cases for \"compare-and-swap\" when used along with\n   \"--force-if-includes\" helps mitigate overwrites when remote\n   refs are updated in the background; allows forced updates when\n   changes from remote are integrated locally.\n\n - Documentation:\n   Add reference for the new option, configuration setting\n   (\"push.useForceIfIncludes\") and advise messages.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/advice.txt |   9 ++-\n Documentation/config/push.txt   |   6 ++\n Documentation/git-push.txt      |  26 ++++++-\n t/t5533-push-cas.sh             | 129 ++++++++++++++++++++++++++++++++\n 4 files changed, 166 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\nindex bdd37c3eaa..acbd0c09aa 100644\n--- a/Documentation/config/advice.txt\n+++ b/Documentation/config/advice.txt\n@@ -10,9 +10,8 @@ advice.*::\n \t\tthat the check is disabled.\n \tpushUpdateRejected::\n \t\tSet this variable to 'false' if you want to disable\n-\t\t'pushNonFFCurrent',\n-\t\t'pushNonFFMatching', 'pushAlreadyExists',\n-\t\t'pushFetchFirst', and 'pushNeedsForce'\n+\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n+\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n \t\tsimultaneously.\n \tpushNonFFCurrent::\n \t\tAdvice shown when linkgit:git-push[1] fails due to a\n@@ -41,6 +40,10 @@ advice.*::\n \t\twe can still suggest that the user push to either\n \t\trefs/heads/* or refs/tags/* based on the type of the\n \t\tsource object.\n+\tpushRefNeedsUpdate::\n+\t\tShown when linkgit:git-push[1] rejects a forced update of\n+\t\ta branch when its remote-tracking ref has updates that we\n+\t\tdo not have locally.\n \tstatusAheadBehind::\n \t\tShown when linkgit:git-status[1] computes the ahead/behind\n \t\tcounts for a local ref compared to its remote tracking ref,\ndiff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\nindex f5e5b38c68..21b256e0a4 100644\n--- a/Documentation/config/push.txt\n+++ b/Documentation/config/push.txt\n@@ -114,3 +114,9 @@ push.recurseSubmodules::\n \tspecifying '--recurse-submodules=check|on-demand|no'.\n \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n \tset (in which case a 'true' value means 'on-demand').\n+\n+push.useForceIfIncludes::\n+\tIf set to \"true\", it is equivalent to specifying\n+\t`--force-if-includes` as an option to linkgit:git-push[1]\n+\tin the command line. Adding `--no-force-if-includes` at the\n+\ttime of push overrides this configuration setting.\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..fb3a220386 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]\n \t   [-u | --set-upstream] [-o <string> | --push-option=<string>]\n \t   [--[no-]signed|--signed=(true|false|if-asked)]\n-\t   [--force-with-lease[=<refname>[:<expect>]]]\n+\t   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]\n \t   [--no-verify] [<repository> [<refspec>...]]\n \n DESCRIPTION\n@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, specifying `--force-if-includes` an an ancillary option\n+along with `--force-with-lease[=<refname>]` (i.e., without saying what\n+exact commit the ref on the remote side must be pointing at, or which\n+refs on the remote side are being protected) at the time of \"push\" will\n+verify if updates from the remote-tracking refs that may have been\n+implicitly updated in the background are integrated locally before\n+allowing a forced update.\n \n -f::\n --force::\n@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n origin +master` to force a push to the `master` branch). See the\n `<refspec>...` section above for details.\n \n+--[no-]force-if-includes::\n+\tForce an update only if the tip of the remote-tracking ref\n+\thas been integrated locally.\n++\n+This option enables a check that verifies if the tip of the\n+remote-tracking ref is reachable from one of the \"reflog\" entries of\n+the local branch based in it for a rewrite. The check ensures that any\n+updates from the remote have been incorporated locally by rejecting the\n+forced update if that is not the case.\n++\n+If the option is passed without specifying `--force-with-lease`, or\n+specified along with `--force-with-lease=<refname>:<expect>`, it is\n+a \"no-op\".\n++\n+Specifying `--no-force-if-includes` disables this behavior.\n+\n --repo=<repository>::\n \tThis option is equivalent to the <repository> argument. If both\n \tare specified, the command-line argument takes precedence.\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..ed44ddc654 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -13,6 +13,46 @@ setup_srcdst_basic () {\n \t)\n }\n \n+# For tests with \"--force-if-includes\".\n+setup_src_dup_dst () {\n+\trm -fr src dup dst &&\n+\tgit init --bare dst &&\n+\tgit clone --no-local dst src &&\n+\tgit clone --no-local dst dup\n+\t(\n+\t\tcd src &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\ttest_commit C &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit fetch &&\n+\t\tgit merge origin/master &&\n+\t\tgit switch -c branch master~2 &&\n+\t\ttest_commit D &&\n+\t\ttest_commit E &&\n+\t\tgit push --all\n+\t) &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch master &&\n+\t\tgit fetch --all &&\n+\t\tgit branch branch --track origin/branch &&\n+\t\tgit rebase origin/master\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit switch master &&\n+\t\ttest_commit F &&\n+\t\ttest_commit G &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit push origin --all\n+\t)\n+}\n+\n test_expect_success setup '\n \t# create template repository\n \ttest_commit A &&\n@@ -256,4 +296,93 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n \n+test_expect_success 'background updates to remote can be mitigated with \"--force-if-includes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-with-lease --force-if-includes --all\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success 'background updates to remote can be mitigated with \"push.useForceIfIncludes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\tgit config --local push.useForceIfIncludes true &&\n+\t\ttest_must_fail git push --force-with-lease=master origin master\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\ttest_cmp expect.master actual.master\n+'\n+\n+test_expect_success '\"--force-if-includes\" should be disabled for --force-with-lease=\"<refname>:<expect>\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tremote_head=\"$(git rev-parse refs/remotes/origin/master)\" &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes --force-with-lease=\"master:$remote_head\" 2>err &&\n+\t\tgrep \"stale info\" err\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\")' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\", local rebase)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit rebase --onto HEAD~4 HEAD~1 &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n test_done\n-- \n2.28.0\n\n"},{"id":"406418","messageId":"20200926101324.73457-3-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200926101324.73457-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v6 2/3] push: parse and set flag for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-26T10:13:23Z","receivedAt":"2020-09-26T10:13:46Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"The previous commit added the necessary machinery to implement the\n\"--force-if-includes\" protection, when \"--force-with-lease\" is used\nwithout giving exact object the remote still ought to have. Surface\nthe feature by adding a command line option and a configuration\nvariable to enable it.\n\n - Add a flag: \"TRANSPORT_PUSH_FORCE_IF_INCLUDES\" to indicate that the\n   new option was passed from the command line of via configuration\n   settings; update command line and configuration parsers to set the\n   new flag accordingly.\n\n - Introduce a new configuration option \"push.useForceIfIncludes\", which\n   is equivalent to setting \"--force-if-includes\" in the command line.\n\n - Update \"remote-curl\" to recognize and pass this option to \"send-pack\"\n   when enabled.\n\n - Update \"advise\" to catch the reject reason \"REJECT_REF_NEEDS_UPDATE\",\n   set when the ref status is \"REF_STATUS_REJECT_REMOTE_UPDATED\" and\n   (optionally) print a help message when the push fails.\n\n - The new option is a \"no-op\" in the following scenarios:\n    * When used without \"--force-with-lease\".\n    * When used with \"--force-with-lease\", and if the expected commit\n      on the remote side is specified as an argument.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n advice.c            |  3 +++\n advice.h            |  2 ++\n builtin/push.c      | 27 +++++++++++++++++++++++++++\n builtin/send-pack.c |  7 +++++++\n remote-curl.c       | 14 +++++++++++++-\n transport-helper.c  |  5 +++++\n transport.c         |  2 ++\n transport.h         | 15 ++++++++++-----\n 8 files changed, 69 insertions(+), 6 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex f0a3d32d20..164742305f 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;\n int advice_push_fetch_first = 1;\n int advice_push_needs_force = 1;\n int advice_push_unqualified_ref_name = 1;\n+int advice_push_ref_needs_update = 1;\n int advice_status_hints = 1;\n int advice_status_u_option = 1;\n int advice_status_ahead_behind_warning = 1;\n@@ -72,6 +73,7 @@ static struct {\n \t{ \"pushFetchFirst\", &advice_push_fetch_first },\n \t{ \"pushNeedsForce\", &advice_push_needs_force },\n \t{ \"pushUnqualifiedRefName\", &advice_push_unqualified_ref_name },\n+\t{ \"pushRefNeedsUpdate\", &advice_push_ref_needs_update },\n \t{ \"statusHints\", &advice_status_hints },\n \t{ \"statusUoption\", &advice_status_u_option },\n \t{ \"statusAheadBehindWarning\", &advice_status_ahead_behind_warning },\n@@ -116,6 +118,7 @@ static struct {\n \t[ADVICE_PUSH_ALREADY_EXISTS]\t\t\t= { \"pushAlreadyExists\", 1 },\n \t[ADVICE_PUSH_FETCH_FIRST]\t\t\t= { \"pushFetchFirst\", 1 },\n \t[ADVICE_PUSH_NEEDS_FORCE]\t\t\t= { \"pushNeedsForce\", 1 },\n+\t[ADVICE_PUSH_REF_NEEDS_UPDATE]\t\t\t= { \"pushRefNeedsUpdate\", 1 },\n \n \t/* make this an alias for backward compatibility */\n \t[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]\t\t= { \"pushNonFastForward\", 1 },\ndiff --git a/advice.h b/advice.h\nindex 16f2c11642..bc2432980a 100644\n--- a/advice.h\n+++ b/advice.h\n@@ -11,6 +11,7 @@ extern int advice_push_already_exists;\n extern int advice_push_fetch_first;\n extern int advice_push_needs_force;\n extern int advice_push_unqualified_ref_name;\n+extern int advice_push_ref_needs_update;\n extern int advice_status_hints;\n extern int advice_status_u_option;\n extern int advice_status_ahead_behind_warning;\n@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;\n \tADVICE_PUSH_UNQUALIFIED_REF_NAME,\n \tADVICE_PUSH_UPDATE_REJECTED_ALIAS,\n \tADVICE_PUSH_UPDATE_REJECTED,\n+\tADVICE_PUSH_REF_NEEDS_UPDATE,\n \tADVICE_RESET_QUIET_WARNING,\n \tADVICE_RESOLVE_CONFLICT,\n \tADVICE_RM_HINTS,\ndiff --git a/builtin/push.c b/builtin/push.c\nindex 0eeb2c8dd5..908b557edb 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =\n \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n \t   \"without using the '--force' option.\\n\");\n \n+static const char message_advice_ref_needs_update[] =\n+\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n+\t   \"branch has been updated since the last checkout. You may want\\n\"\n+\t   \"to integrate those changes locally (e.g., 'git pull ...')\\n\"\n+\t   \"before forcing an update.\\n\");\n+\n static void advise_pull_before_push(void)\n {\n \tif (!advice_push_non_ff_current || !advice_push_update_rejected)\n@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)\n \tadvise(_(message_advice_ref_needs_force));\n }\n \n+static void advise_ref_needs_update(void)\n+{\n+\tif (!advice_push_ref_needs_update || !advice_push_update_rejected)\n+\t\treturn;\n+\tadvise(_(message_advice_ref_needs_update));\n+}\n+\n static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\t\t     int flags)\n {\n@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\tadvise_ref_fetch_first();\n \t} else if (reject_reasons & REJECT_NEEDS_FORCE) {\n \t\tadvise_ref_needs_force();\n+\t} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {\n+\t\tadvise_ref_needs_update();\n \t}\n \n \treturn 1;\n@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\tif (!v)\n \t\t\treturn config_error_nonbool(k);\n \t\treturn color_parse(v, push_colors[slot]);\n+\t} else if (!strcmp(k, \"push.useforceifincludes\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\t*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\telse\n+\t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\treturn 0;\n \t}\n \n \treturn git_default_config(k, v, NULL);\n@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t\t       N_(\"require old value of ref to be at this value\"),\n \t\t\t       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),\n+\t\tOPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,\n+\t\t\tN_(\"require remote updates to be integrated locally\"),\n+\t\t\tTRANSPORT_PUSH_FORCE_IF_INCLUDES),\n \t\tOPT_CALLBACK(0, \"recurse-submodules\", &recurse_submodules, \"(check|on-demand|no)\",\n \t\t\t     N_(\"control recursive pushing of submodules\"), option_parse_recurse_submodules),\n \t\tOPT_BOOL_F( 0 , \"thin\", &thin, N_(\"use thin pack\"), PARSE_OPT_NOCOMPLETE),\n@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \tif ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))\n \t\tdie(_(\"--all and --mirror are incompatible\"));\n \n+\tif (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))\n+\t\tcas.use_force_if_includes = 1;\n+\n \tfor_each_string_list_item(item, push_options)\n \t\tif (strchr(item->string, '\\n'))\n \t\t\tdie(_(\"push options must not have new line characters\"));\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 516cba7336..a284ada051 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tint force_if_includes = 0;\n \tstruct packet_reader reader;\n \n \tstruct option options[] = {\n@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t  N_(\"require old value of ref to be at this value\"),\n \t\t  PARSE_OPT_OPTARG, parseopt_push_cas_option),\n+\t\tOPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,\n+\t\t\t N_(\"require remote updates to be integrated locally\")),\n \t\tOPT_END()\n \t};\n \n@@ -304,6 +307,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tif (!is_empty_cas(&cas))\n \t\tapply_push_cas(&cas, remote, remote_refs);\n \n+\tif (!is_empty_cas(&cas) && force_if_includes)\n+\t\tcas.use_force_if_includes = 1;\n+\n+\n \tset_ref_status_for_push(remote_refs, args.send_mirror,\n \t\targs.force_update);\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 32cc4a0c55..0290b04891 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -44,7 +44,8 @@ struct options {\n \t\tfrom_promisor : 1,\n \n \t\tatomic : 1,\n-\t\tobject_format : 1;\n+\t\tobject_format : 1,\n+\t\tforce_if_includes : 1;\n \tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)\n \t\tstring_list_append(&cas_options, val.buf);\n \t\tstrbuf_release(&val);\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.force_if_includes = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.force_if_includes = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n \t} else if (!strcmp(name, \"cloning\")) {\n \t\tif (!strcmp(value, \"true\"))\n \t\t\toptions.cloning = 1;\n@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)\n \t\tstrvec_push(&args, cas_option->string);\n \tstrvec_push(&args, url.buf);\n \n+\tif (options.force_if_includes)\n+\t\tstrvec_push(&args, \"--force-if-includes\");\n+\n \tstrvec_push(&args, \"--stdin\");\n \tfor (i = 0; i < nr_spec; i++)\n \t\tpacket_buf_write(&preamble, \"%s\\n\", specs[i]);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 6157de30c7..5f6e0b3bd8 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,\n \t\tif (set_helper_option(transport, TRANS_OPT_ATOMIC, \"true\") != 0)\n \t\t\tdie(_(\"helper %s does not support --atomic\"), name);\n \n+\tif (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)\n+\t\tif (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, \"true\") != 0)\n+\t\t\tdie(_(\"helper %s does not support --%s\"),\n+\t\t\t    name, TRANS_OPT_FORCE_IF_INCLUDES);\n+\n \tif (flags & TRANSPORT_PUSH_OPTIONS) {\n \t\tstruct string_list_item *item;\n \t\tfor_each_string_list_item(item, transport->push_options)\ndiff --git a/transport.c b/transport.c\nindex 75f1786aca..5cfb7360e9 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t\t*reject_reasons |= REJECT_FETCH_FIRST;\n \t\t} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {\n \t\t\t*reject_reasons |= REJECT_NEEDS_FORCE;\n+\t\t} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {\n+\t\t\t*reject_reasons |= REJECT_REF_NEEDS_UPDATE;\n \t\t}\n \t}\n \tfree(head);\ndiff --git a/transport.h b/transport.h\nindex ca409ea1e4..24558c027d 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -136,6 +136,7 @@ struct transport {\n #define TRANSPORT_PUSH_ATOMIC\t\t\t(1<<13)\n #define TRANSPORT_PUSH_OPTIONS\t\t\t(1<<14)\n #define TRANSPORT_RECURSE_SUBMODULES_ONLY\t(1<<15)\n+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES\t(1<<16)\n \n int transport_summary_width(const struct ref *refs);\n \n@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);\n /* Request atomic (all-or-nothing) updates when pushing */\n #define TRANS_OPT_ATOMIC \"atomic\"\n \n+/* Require remote changes to be integrated locally. */\n+#define TRANS_OPT_FORCE_IF_INCLUDES \"force-if-includes\"\n+\n /**\n  * Returns 0 if the option was used, non-zero otherwise. Prints a\n  * message to stderr if the option is not used.\n@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,\n void transport_set_verbosity(struct transport *transport, int verbosity,\n \tint force_progress);\n \n-#define REJECT_NON_FF_HEAD     0x01\n-#define REJECT_NON_FF_OTHER    0x02\n-#define REJECT_ALREADY_EXISTS  0x04\n-#define REJECT_FETCH_FIRST     0x08\n-#define REJECT_NEEDS_FORCE     0x10\n+#define REJECT_NON_FF_HEAD      0x01\n+#define REJECT_NON_FF_OTHER     0x02\n+#define REJECT_ALREADY_EXISTS   0x04\n+#define REJECT_FETCH_FIRST      0x08\n+#define REJECT_NEEDS_FORCE      0x10\n+#define REJECT_REF_NEEDS_UPDATE 0x20\n \n int transport_push(struct repository *repo,\n \t\t   struct transport *connection,\n-- \n2.28.0\n\n"},{"id":"406419","messageId":"20200926102116.GA76247@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"20200926101324.73457-1-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v6 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-26T10:21:16Z","receivedAt":"2020-09-26T10:21:24Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hello,\nPlease disregard this patch series. It does not compile with the\nlatest commit on \"master\" (9bc233ae1cf19a49e51842c7959d80a675dbd1c0).\nI will fix those errors and retest in v7.\n\nApologies for the noise.\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"406420","messageId":"20200926114626.28823-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200926101324.73457-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v7 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-26T11:46:23Z","receivedAt":"2020-09-26T11:46:44Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new option: \"--force-if-includes\" to \"git-push\" where forced\nupdates are allowed only if the tip of the remote-tracking ref has\nbeen integrated locally, by verifying if the tip of the remote-tracking\nref -- on which a local branch has based on -- is reachable from at\nleast one of the \"reflog\" entries of the branch about to be updated\nby force on the remote.\n\nThis option can be used with \"--force-with-lease\" with setups where\nthe remote-tracking refs of the repository are implicitly updated in\nthe background to help prevent unintended remote overwrites.\n\nIf a local branch is based on a remote ref for a rewrite, and if that\nremote-tracking ref is updated by a push from another repository after\nit has been checked out locally, force updating that branch to remote\nwith \"--force-with-lease[=<refname>[:<expect>]]\" without specifying\nthe \"<expect>\" value, can cause the update that happened in-between\nthe checkout and forced push to be lost.\n\nChanges since v5:\n  - In the previous patch, the stopping condition for \"reflog\"\n    iteration used the commit date of the remote ref. That has\n    been changed to use the the timestamp of the latest entry\n    of the remote ref's \"reflog\"; calls \"in_merge_bases_many()\"\n    are batched when the number of commits collected in the list\n    is more than 8.\n\n  - Add more tests for t5533 to include scenarios where forced\n    updates should be allowed for \"pull --rebase\" and local\n    rebase operations.\n\n  - Reword commit messages, fix typos, and update documentation\n    to use \"`\" where necessary.\n\nChanges since v6:\n  - Compilation failed because there was an argument missing when\n    calling \"print_ref_status()\"; it has been updated.\n\nSrinidhi Kaushik (3):\n  push: add reflog check for \"--force-if-includes\"\n  push: parse and set flag for \"--force-if-includes\"\n  t, doc: update tests, reference for \"--force-if-includes\"\n\n Documentation/config/advice.txt |   9 +-\n Documentation/config/push.txt   |   6 ++\n Documentation/git-push.txt      |  26 ++++-\n advice.c                        |   3 +\n advice.h                        |   2 +\n builtin/push.c                  |  27 +++++\n builtin/send-pack.c             |  12 +++\n remote-curl.c                   |  14 ++-\n remote.c                        | 185 ++++++++++++++++++++++++++++++--\n remote.h                        |  12 ++-\n send-pack.c                     |   1 +\n t/t5533-push-cas.sh             | 129 ++++++++++++++++++++++\n transport-helper.c              |  10 ++\n transport.c                     |   8 ++\n transport.h                     |  15 ++-\n 15 files changed, 442 insertions(+), 17 deletions(-)\n\nbase-commit: 9bc233ae1cf19a49e51842c7959d80a675dbd1c0\n--\n2.28.0\n"},{"id":"406421","messageId":"20200926114626.28823-2-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200926114626.28823-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v7 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-26T11:46:24Z","receivedAt":"2020-09-26T11:46:44Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a check to verify if the remote-tracking ref of the local branch\nis reachable from one of its \"reflog\" entries.\n\nThe check iterates through the local ref's reflog to see if there\nis an entry for the remote-tracking ref and collecting any commits\nthat are seen, into a list; the iteration stops if an entry in the\nreflog matches the remote ref or if the entry timestamp is older\nthe latest entry of the remote ref's \"reflog\". If there wasn't an\nentry found for the remote ref, \"in_merge_bases_many()\" is called\nto check if it is reachable from the list of collected commits.\n\nWhen a local branch that is based on a remote ref, has been rewound\nand is to be force pushed on the remote, \"--force-if-includes\" runs\na check that ensures any updates to the remote-tracking ref that may\nhave happened (by push from another repository) in-between the time\nof the last update to the local branch (via \"git-pull\", for instance)\nand right before the time of push, have been integrated locally\nbefore allowing a forced update.\n\nIf the new option is passed without specifying \"--force-with-lease\",\nor specified along with \"--force-with-lease=<refname>:<expect>\" it\nis a \"no-op\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n builtin/send-pack.c |   5 ++\n remote.c            | 185 ++++++++++++++++++++++++++++++++++++++++++--\n remote.h            |  12 ++-\n send-pack.c         |   1 +\n transport-helper.c  |   5 ++\n transport.c         |   6 ++\n 6 files changed, 207 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 7af148d733..516cba7336 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n \n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"remote ref updated since checkout\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\ndiff --git a/remote.c b/remote.c\nindex eafc14cbe7..a696243247 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t * with the remote-tracking branch to find the value\n \t\t * to expect, but we did not have such a tracking\n \t\t * branch.\n+\t\t *\n+\t\t * If the tip of the remote-tracking ref is unreachable\n+\t\t * from any reflog entry of its local ref indicating a\n+\t\t * possible update since checkout; reject the push.\n \t\t */\n \t\tif (ref->expect_old_sha1) {\n \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n+\t\t\telse if (ref->check_reachable && ref->unreachable)\n+\t\t\t\treject_reason =\n+\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n \t\t\telse\n-\t\t\t\t/* If the ref isn't stale then force the update. */\n+\t\t\t\t/*\n+\t\t\t\t * If the ref isn't stale, and is reachable\n+\t\t\t\t * from from one of the reflog entries of\n+\t\t\t\t * the local branch, force the update.\n+\t\t\t\t */\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n \n@@ -2252,11 +2263,11 @@ int is_empty_cas(const struct push_cas_option *cas)\n /*\n  * Look at remote.fetch refspec and see if we have a remote\n  * tracking branch for the refname there.  Fill its current\n- * value in sha1[].\n+ * value in sha1[], and as a string.\n  * If we cannot do so, return negative to signal an error.\n  */\n static int remote_tracking(struct remote *remote, const char *refname,\n-\t\t\t   struct object_id *oid)\n+\t\t\t   struct object_id *oid, char **dst_refname)\n {\n \tchar *dst;\n \n@@ -2265,9 +2276,154 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \t\treturn -1; /* no tracking ref for refname at remote */\n \tif (read_ref(dst, oid))\n \t\treturn -1; /* we know what the tracking ref is but we cannot read it */\n+\n+\t*dst_refname = dst;\n+\treturn 0;\n+}\n+\n+/*\n+ * The struct \"reflog_commit_list\" and related helper functions\n+ * for list manipulation are used for collecting commits into a\n+ * list during reflog traversals in \"if_exists_or_grab_until()\".\n+ */\n+struct reflog_commit_list {\n+\tstruct commit **items;\n+\tsize_t nr, alloc;\n+};\n+\n+/* Add a commit to list. */\n+static void add_commit(struct reflog_commit_list *list, struct commit *commit)\n+{\n+\tALLOC_GROW(list->items, list->nr + 1, list->alloc);\n+\tlist->items[list->nr++] = commit;\n+}\n+\n+/* Free and reset the list. */\n+static void free_reflog_commit_list(struct reflog_commit_list *list)\n+{\n+\tFREE_AND_NULL(list->items);\n+\tlist->nr = list->alloc = 0;\n+}\n+\n+struct check_and_collect_until_cb_data {\n+\tstruct commit *remote_commit;\n+\tstruct reflog_commit_list *local_commits;\n+\ttimestamp_t remote_reflog_timestamp;\n+};\n+\n+/* Get the timestamp of the latest entry. */\n+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,\n+\t\t       const char *ident, timestamp_t timestamp,\n+\t\t       int tz, const char *message, void *cb_data)\n+{\n+\ttimestamp_t *ts = cb_data;\n+\t*ts = timestamp;\n+\treturn 1;\n+}\n+\n+static int check_and_collect_until(struct object_id *o_oid,\n+\t\t\t\t   struct object_id *n_oid,\n+\t\t\t\t   const char *ident, timestamp_t timestamp,\n+\t\t\t\t   int tz, const char *message, void *cb_data)\n+{\n+\tstruct commit *commit;\n+\tstruct check_and_collect_until_cb_data *cb = cb_data;\n+\n+\t/*\n+\t * If the reflog entry timestamp is older than the remote ref's\n+\t * latest reflog entry, there is no need to check or collect\n+\t * entries older than this one.\n+\t */\n+\tif (timestamp < cb->remote_reflog_timestamp)\n+\t\treturn -1;\n+\n+\t/* An entry was found. */\n+\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n+\t\treturn 1;\n+\n+\t/* Look-up the commit and append it to the list. */\n+\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n+\t\tadd_commit(cb->local_commits, commit);\n+\n \treturn 0;\n }\n \n+#define MERGE_BASES_BATCH_SIZE 8\n+\n+/*\n+ * Iterate through the reflog of the local ref to check if there is an entry\n+ * for the given remote-tracking ref; runs until the timestamp of an entry is\n+ * older than latest timestamp of remote-tracking ref's reflog. Any commits\n+ * are that seen along the way are collected into a list to check if the\n+ * remote-tracking ref is reachable from any of them.\n+ */\n+static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n+{\n+\ttimestamp_t date;\n+\tstruct commit *commit;\n+\tstruct commit **chunk;\n+\tstruct check_and_collect_until_cb_data cb;\n+\tstruct reflog_commit_list list = { NULL, 0, 0 };\n+\tsize_t count = 0, batch_size = 0;\n+\tint ret = 0;\n+\n+\tcommit = lookup_commit_reference(the_repository, &remote->old_oid);\n+\tif (!commit)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Get the timestamp from the latest entry\n+\t * of the remote-tracking ref's reflog.\n+\t */\n+\tfor_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);\n+\n+\tcb.remote_commit = commit;\n+\tcb.local_commits = &list;\n+\tcb.remote_reflog_timestamp = date;\n+\tret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);\n+\n+\t/* We found an entry in the reflog. */\n+\tif (ret > 0)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Check if the remote commit is reachable from any\n+\t * of the commits in the collected list, in batches.\n+\t */\n+\tchunk = list.items;\n+\twhile (count < list.nr) {\n+\t\tbatch_size = MERGE_BASES_BATCH_SIZE;\n+\n+\t\t/* For any leftover entries. */\n+\t\tif ((count + MERGE_BASES_BATCH_SIZE) > list.nr)\n+\t\t\tbatch_size = list.nr - count;\n+\n+\t\tif ((ret = in_merge_bases_many(commit, batch_size, chunk)))\n+\t\t\tbreak;\n+\n+\t\tchunk += batch_size;\n+\t\tcount += MERGE_BASES_BATCH_SIZE;\n+\t}\n+\n+cleanup_return:\n+\tfree_reflog_commit_list(&list);\n+\treturn ret;\n+}\n+\n+/*\n+ * Check for reachability of a remote-tracking\n+ * ref in the reflog entries of its local ref.\n+ */\n+static void check_if_includes_upstream(struct ref *remote)\n+{\n+\tstruct ref *local = get_local_ref(remote->name);\n+\tif (!local)\n+\t\treturn;\n+\n+\tif (is_reachable_in_reflog(local->name, remote) <= 0)\n+\t\tremote->unreachable = 1;\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n \t\t      struct ref *ref)\n@@ -2282,8 +2438,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\tref->expect_old_sha1 = 1;\n \t\tif (!entry->use_tracking)\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n-\t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\t\telse if (remote_tracking(remote, ref->name,\n+\t\t\t\t\t &ref->old_oid_expect,\n+\t\t\t\t\t &ref->tracking_ref))\n \t\t\toidclr(&ref->old_oid_expect);\n+\t\telse\n+\t\t\tref->check_reachable = cas->use_force_if_includes;\n \t\treturn;\n \t}\n \n@@ -2292,8 +2452,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\treturn;\n \n \tref->expect_old_sha1 = 1;\n-\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\tif (remote_tracking(remote, ref->name,\n+\t\t\t    &ref->old_oid_expect,\n+\t\t\t    &ref->tracking_ref))\n \t\toidclr(&ref->old_oid_expect);\n+\telse\n+\t\tref->check_reachable = cas->use_force_if_includes;\n }\n \n void apply_push_cas(struct push_cas_option *cas,\n@@ -2301,6 +2465,15 @@ void apply_push_cas(struct push_cas_option *cas,\n \t\t    struct ref *remote_refs)\n {\n \tstruct ref *ref;\n-\tfor (ref = remote_refs; ref; ref = ref->next)\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n \t\tapply_cas(cas, remote, ref);\n+\n+\t\t/*\n+\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n+\t\t * mode, and if \"--force-if-includes\" was specified, run\n+\t\t * the check.\n+\t\t */\n+\t\tif (ref->check_reachable)\n+\t\t\tcheck_if_includes_upstream(ref);\n+\t}\n }\ndiff --git a/remote.h b/remote.h\nindex eb62a47044..2d5391d281 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -107,12 +107,20 @@ struct ref {\n \tstruct object_id new_oid;\n \tstruct object_id old_oid_expect; /* used by expect-old */\n \tchar *symref;\n+\tchar *tracking_ref;\n \tunsigned int\n \t\tforce:1,\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\t/* Need to check if local reflog reaches the remote tip. */\n+\t\tcheck_reachable:1,\n+\t\t/*\n+\t\t * Store the result of the check enabled by \"check_reachable\";\n+\t\t * implies the local reflog does not reach the remote tip.\n+\t\t */\n+\t\tunreachable:1;\n \n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -142,6 +150,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n \t\tREF_STATUS_REJECT_SHALLOW,\n+\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT,\n@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);\n \n struct push_cas_option {\n \tunsigned use_tracking_for_rest:1;\n+\tunsigned use_force_if_includes:1;\n \tstruct push_cas {\n \t\tstruct object_id expect;\n \t\tunsigned use_tracking:1;\ndiff --git a/send-pack.c b/send-pack.c\nindex c9698070fc..eb4a44270b 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\ndiff --git a/transport-helper.c b/transport-helper.c\nindex b573b6c730..6157de30c7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"remote ref updated since checkout\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\n \t\t\t\tstring_list_clear(&cas_options, 0);\ndiff --git a/transport.c b/transport.c\nindex ffe2115845..65fcd22b20 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,\n \t\t\t\t \"stale info\",\n \t\t\t\t report, porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"remote ref updated since checkout\",\n+\t\t\t\t report, porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n \n \t\tstrbuf_reset(&buf);\n-- \n2.28.0\n\n"},{"id":"406422","messageId":"20200926114626.28823-3-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200926114626.28823-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v7 2/3] push: parse and set flag for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-26T11:46:25Z","receivedAt":"2020-09-26T11:46:44Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"The previous commit added the necessary machinery to implement the\n\"--force-if-includes\" protection, when \"--force-with-lease\" is used\nwithout giving exact object the remote still ought to have. Surface\nthe feature by adding a command line option and a configuration\nvariable to enable it.\n\n - Add a flag: \"TRANSPORT_PUSH_FORCE_IF_INCLUDES\" to indicate that the\n   new option was passed from the command line of via configuration\n   settings; update command line and configuration parsers to set the\n   new flag accordingly.\n\n - Introduce a new configuration option \"push.useForceIfIncludes\", which\n   is equivalent to setting \"--force-if-includes\" in the command line.\n\n - Update \"remote-curl\" to recognize and pass this option to \"send-pack\"\n   when enabled.\n\n - Update \"advise\" to catch the reject reason \"REJECT_REF_NEEDS_UPDATE\",\n   set when the ref status is \"REF_STATUS_REJECT_REMOTE_UPDATED\" and\n   (optionally) print a help message when the push fails.\n\n - The new option is a \"no-op\" in the following scenarios:\n    * When used without \"--force-with-lease\".\n    * When used with \"--force-with-lease\", and if the expected commit\n      on the remote side is specified as an argument.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n advice.c            |  3 +++\n advice.h            |  2 ++\n builtin/push.c      | 27 +++++++++++++++++++++++++++\n builtin/send-pack.c |  7 +++++++\n remote-curl.c       | 14 +++++++++++++-\n transport-helper.c  |  5 +++++\n transport.c         |  2 ++\n transport.h         | 15 ++++++++++-----\n 8 files changed, 69 insertions(+), 6 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex f0a3d32d20..164742305f 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;\n int advice_push_fetch_first = 1;\n int advice_push_needs_force = 1;\n int advice_push_unqualified_ref_name = 1;\n+int advice_push_ref_needs_update = 1;\n int advice_status_hints = 1;\n int advice_status_u_option = 1;\n int advice_status_ahead_behind_warning = 1;\n@@ -72,6 +73,7 @@ static struct {\n \t{ \"pushFetchFirst\", &advice_push_fetch_first },\n \t{ \"pushNeedsForce\", &advice_push_needs_force },\n \t{ \"pushUnqualifiedRefName\", &advice_push_unqualified_ref_name },\n+\t{ \"pushRefNeedsUpdate\", &advice_push_ref_needs_update },\n \t{ \"statusHints\", &advice_status_hints },\n \t{ \"statusUoption\", &advice_status_u_option },\n \t{ \"statusAheadBehindWarning\", &advice_status_ahead_behind_warning },\n@@ -116,6 +118,7 @@ static struct {\n \t[ADVICE_PUSH_ALREADY_EXISTS]\t\t\t= { \"pushAlreadyExists\", 1 },\n \t[ADVICE_PUSH_FETCH_FIRST]\t\t\t= { \"pushFetchFirst\", 1 },\n \t[ADVICE_PUSH_NEEDS_FORCE]\t\t\t= { \"pushNeedsForce\", 1 },\n+\t[ADVICE_PUSH_REF_NEEDS_UPDATE]\t\t\t= { \"pushRefNeedsUpdate\", 1 },\n \n \t/* make this an alias for backward compatibility */\n \t[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]\t\t= { \"pushNonFastForward\", 1 },\ndiff --git a/advice.h b/advice.h\nindex 16f2c11642..bc2432980a 100644\n--- a/advice.h\n+++ b/advice.h\n@@ -11,6 +11,7 @@ extern int advice_push_already_exists;\n extern int advice_push_fetch_first;\n extern int advice_push_needs_force;\n extern int advice_push_unqualified_ref_name;\n+extern int advice_push_ref_needs_update;\n extern int advice_status_hints;\n extern int advice_status_u_option;\n extern int advice_status_ahead_behind_warning;\n@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;\n \tADVICE_PUSH_UNQUALIFIED_REF_NAME,\n \tADVICE_PUSH_UPDATE_REJECTED_ALIAS,\n \tADVICE_PUSH_UPDATE_REJECTED,\n+\tADVICE_PUSH_REF_NEEDS_UPDATE,\n \tADVICE_RESET_QUIET_WARNING,\n \tADVICE_RESOLVE_CONFLICT,\n \tADVICE_RM_HINTS,\ndiff --git a/builtin/push.c b/builtin/push.c\nindex 0eeb2c8dd5..908b557edb 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =\n \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n \t   \"without using the '--force' option.\\n\");\n \n+static const char message_advice_ref_needs_update[] =\n+\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n+\t   \"branch has been updated since the last checkout. You may want\\n\"\n+\t   \"to integrate those changes locally (e.g., 'git pull ...')\\n\"\n+\t   \"before forcing an update.\\n\");\n+\n static void advise_pull_before_push(void)\n {\n \tif (!advice_push_non_ff_current || !advice_push_update_rejected)\n@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)\n \tadvise(_(message_advice_ref_needs_force));\n }\n \n+static void advise_ref_needs_update(void)\n+{\n+\tif (!advice_push_ref_needs_update || !advice_push_update_rejected)\n+\t\treturn;\n+\tadvise(_(message_advice_ref_needs_update));\n+}\n+\n static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\t\t     int flags)\n {\n@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\tadvise_ref_fetch_first();\n \t} else if (reject_reasons & REJECT_NEEDS_FORCE) {\n \t\tadvise_ref_needs_force();\n+\t} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {\n+\t\tadvise_ref_needs_update();\n \t}\n \n \treturn 1;\n@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\tif (!v)\n \t\t\treturn config_error_nonbool(k);\n \t\treturn color_parse(v, push_colors[slot]);\n+\t} else if (!strcmp(k, \"push.useforceifincludes\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\t*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\telse\n+\t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\treturn 0;\n \t}\n \n \treturn git_default_config(k, v, NULL);\n@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t\t       N_(\"require old value of ref to be at this value\"),\n \t\t\t       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),\n+\t\tOPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,\n+\t\t\tN_(\"require remote updates to be integrated locally\"),\n+\t\t\tTRANSPORT_PUSH_FORCE_IF_INCLUDES),\n \t\tOPT_CALLBACK(0, \"recurse-submodules\", &recurse_submodules, \"(check|on-demand|no)\",\n \t\t\t     N_(\"control recursive pushing of submodules\"), option_parse_recurse_submodules),\n \t\tOPT_BOOL_F( 0 , \"thin\", &thin, N_(\"use thin pack\"), PARSE_OPT_NOCOMPLETE),\n@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \tif ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))\n \t\tdie(_(\"--all and --mirror are incompatible\"));\n \n+\tif (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))\n+\t\tcas.use_force_if_includes = 1;\n+\n \tfor_each_string_list_item(item, push_options)\n \t\tif (strchr(item->string, '\\n'))\n \t\t\tdie(_(\"push options must not have new line characters\"));\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 516cba7336..a284ada051 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tint force_if_includes = 0;\n \tstruct packet_reader reader;\n \n \tstruct option options[] = {\n@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t  N_(\"require old value of ref to be at this value\"),\n \t\t  PARSE_OPT_OPTARG, parseopt_push_cas_option),\n+\t\tOPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,\n+\t\t\t N_(\"require remote updates to be integrated locally\")),\n \t\tOPT_END()\n \t};\n \n@@ -304,6 +307,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tif (!is_empty_cas(&cas))\n \t\tapply_push_cas(&cas, remote, remote_refs);\n \n+\tif (!is_empty_cas(&cas) && force_if_includes)\n+\t\tcas.use_force_if_includes = 1;\n+\n+\n \tset_ref_status_for_push(remote_refs, args.send_mirror,\n \t\targs.force_update);\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 32cc4a0c55..0290b04891 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -44,7 +44,8 @@ struct options {\n \t\tfrom_promisor : 1,\n \n \t\tatomic : 1,\n-\t\tobject_format : 1;\n+\t\tobject_format : 1,\n+\t\tforce_if_includes : 1;\n \tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)\n \t\tstring_list_append(&cas_options, val.buf);\n \t\tstrbuf_release(&val);\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.force_if_includes = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.force_if_includes = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n \t} else if (!strcmp(name, \"cloning\")) {\n \t\tif (!strcmp(value, \"true\"))\n \t\t\toptions.cloning = 1;\n@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)\n \t\tstrvec_push(&args, cas_option->string);\n \tstrvec_push(&args, url.buf);\n \n+\tif (options.force_if_includes)\n+\t\tstrvec_push(&args, \"--force-if-includes\");\n+\n \tstrvec_push(&args, \"--stdin\");\n \tfor (i = 0; i < nr_spec; i++)\n \t\tpacket_buf_write(&preamble, \"%s\\n\", specs[i]);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 6157de30c7..5f6e0b3bd8 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,\n \t\tif (set_helper_option(transport, TRANS_OPT_ATOMIC, \"true\") != 0)\n \t\t\tdie(_(\"helper %s does not support --atomic\"), name);\n \n+\tif (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)\n+\t\tif (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, \"true\") != 0)\n+\t\t\tdie(_(\"helper %s does not support --%s\"),\n+\t\t\t    name, TRANS_OPT_FORCE_IF_INCLUDES);\n+\n \tif (flags & TRANSPORT_PUSH_OPTIONS) {\n \t\tstruct string_list_item *item;\n \t\tfor_each_string_list_item(item, transport->push_options)\ndiff --git a/transport.c b/transport.c\nindex 65fcd22b20..47da955e4f 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t\t*reject_reasons |= REJECT_FETCH_FIRST;\n \t\t} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {\n \t\t\t*reject_reasons |= REJECT_NEEDS_FORCE;\n+\t\t} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {\n+\t\t\t*reject_reasons |= REJECT_REF_NEEDS_UPDATE;\n \t\t}\n \t}\n \tfree(head);\ndiff --git a/transport.h b/transport.h\nindex ca409ea1e4..24558c027d 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -136,6 +136,7 @@ struct transport {\n #define TRANSPORT_PUSH_ATOMIC\t\t\t(1<<13)\n #define TRANSPORT_PUSH_OPTIONS\t\t\t(1<<14)\n #define TRANSPORT_RECURSE_SUBMODULES_ONLY\t(1<<15)\n+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES\t(1<<16)\n \n int transport_summary_width(const struct ref *refs);\n \n@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);\n /* Request atomic (all-or-nothing) updates when pushing */\n #define TRANS_OPT_ATOMIC \"atomic\"\n \n+/* Require remote changes to be integrated locally. */\n+#define TRANS_OPT_FORCE_IF_INCLUDES \"force-if-includes\"\n+\n /**\n  * Returns 0 if the option was used, non-zero otherwise. Prints a\n  * message to stderr if the option is not used.\n@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,\n void transport_set_verbosity(struct transport *transport, int verbosity,\n \tint force_progress);\n \n-#define REJECT_NON_FF_HEAD     0x01\n-#define REJECT_NON_FF_OTHER    0x02\n-#define REJECT_ALREADY_EXISTS  0x04\n-#define REJECT_FETCH_FIRST     0x08\n-#define REJECT_NEEDS_FORCE     0x10\n+#define REJECT_NON_FF_HEAD      0x01\n+#define REJECT_NON_FF_OTHER     0x02\n+#define REJECT_ALREADY_EXISTS   0x04\n+#define REJECT_FETCH_FIRST      0x08\n+#define REJECT_NEEDS_FORCE      0x10\n+#define REJECT_REF_NEEDS_UPDATE 0x20\n \n int transport_push(struct repository *repo,\n \t\t   struct transport *connection,\n-- \n2.28.0\n\n"},{"id":"406423","messageId":"20200926114626.28823-4-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200926114626.28823-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v7 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-26T11:46:26Z","receivedAt":"2020-09-26T11:46:44Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Update test cases for the new option, and document its usage\nand update related references.\n\nUpdate test cases for the new option, and document its usage\nand update related references.\n\n - t/t5533-push-cas.sh:\n   Update test cases for \"compare-and-swap\" when used along with\n   \"--force-if-includes\" helps mitigate overwrites when remote\n   refs are updated in the background; allows forced updates when\n   changes from remote are integrated locally.\n\n - Documentation:\n   Add reference for the new option, configuration setting\n   (\"push.useForceIfIncludes\") and advise messages.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/advice.txt |   9 ++-\n Documentation/config/push.txt   |   6 ++\n Documentation/git-push.txt      |  26 ++++++-\n t/t5533-push-cas.sh             | 129 ++++++++++++++++++++++++++++++++\n 4 files changed, 166 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\nindex bdd37c3eaa..acbd0c09aa 100644\n--- a/Documentation/config/advice.txt\n+++ b/Documentation/config/advice.txt\n@@ -10,9 +10,8 @@ advice.*::\n \t\tthat the check is disabled.\n \tpushUpdateRejected::\n \t\tSet this variable to 'false' if you want to disable\n-\t\t'pushNonFFCurrent',\n-\t\t'pushNonFFMatching', 'pushAlreadyExists',\n-\t\t'pushFetchFirst', and 'pushNeedsForce'\n+\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n+\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n \t\tsimultaneously.\n \tpushNonFFCurrent::\n \t\tAdvice shown when linkgit:git-push[1] fails due to a\n@@ -41,6 +40,10 @@ advice.*::\n \t\twe can still suggest that the user push to either\n \t\trefs/heads/* or refs/tags/* based on the type of the\n \t\tsource object.\n+\tpushRefNeedsUpdate::\n+\t\tShown when linkgit:git-push[1] rejects a forced update of\n+\t\ta branch when its remote-tracking ref has updates that we\n+\t\tdo not have locally.\n \tstatusAheadBehind::\n \t\tShown when linkgit:git-status[1] computes the ahead/behind\n \t\tcounts for a local ref compared to its remote tracking ref,\ndiff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\nindex f5e5b38c68..21b256e0a4 100644\n--- a/Documentation/config/push.txt\n+++ b/Documentation/config/push.txt\n@@ -114,3 +114,9 @@ push.recurseSubmodules::\n \tspecifying '--recurse-submodules=check|on-demand|no'.\n \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n \tset (in which case a 'true' value means 'on-demand').\n+\n+push.useForceIfIncludes::\n+\tIf set to \"true\", it is equivalent to specifying\n+\t`--force-if-includes` as an option to linkgit:git-push[1]\n+\tin the command line. Adding `--no-force-if-includes` at the\n+\ttime of push overrides this configuration setting.\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..fb3a220386 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]\n \t   [-u | --set-upstream] [-o <string> | --push-option=<string>]\n \t   [--[no-]signed|--signed=(true|false|if-asked)]\n-\t   [--force-with-lease[=<refname>[:<expect>]]]\n+\t   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]\n \t   [--no-verify] [<repository> [<refspec>...]]\n \n DESCRIPTION\n@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, specifying `--force-if-includes` an an ancillary option\n+along with `--force-with-lease[=<refname>]` (i.e., without saying what\n+exact commit the ref on the remote side must be pointing at, or which\n+refs on the remote side are being protected) at the time of \"push\" will\n+verify if updates from the remote-tracking refs that may have been\n+implicitly updated in the background are integrated locally before\n+allowing a forced update.\n \n -f::\n --force::\n@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n origin +master` to force a push to the `master` branch). See the\n `<refspec>...` section above for details.\n \n+--[no-]force-if-includes::\n+\tForce an update only if the tip of the remote-tracking ref\n+\thas been integrated locally.\n++\n+This option enables a check that verifies if the tip of the\n+remote-tracking ref is reachable from one of the \"reflog\" entries of\n+the local branch based in it for a rewrite. The check ensures that any\n+updates from the remote have been incorporated locally by rejecting the\n+forced update if that is not the case.\n++\n+If the option is passed without specifying `--force-with-lease`, or\n+specified along with `--force-with-lease=<refname>:<expect>`, it is\n+a \"no-op\".\n++\n+Specifying `--no-force-if-includes` disables this behavior.\n+\n --repo=<repository>::\n \tThis option is equivalent to the <repository> argument. If both\n \tare specified, the command-line argument takes precedence.\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..ed44ddc654 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -13,6 +13,46 @@ setup_srcdst_basic () {\n \t)\n }\n \n+# For tests with \"--force-if-includes\".\n+setup_src_dup_dst () {\n+\trm -fr src dup dst &&\n+\tgit init --bare dst &&\n+\tgit clone --no-local dst src &&\n+\tgit clone --no-local dst dup\n+\t(\n+\t\tcd src &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\ttest_commit C &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit fetch &&\n+\t\tgit merge origin/master &&\n+\t\tgit switch -c branch master~2 &&\n+\t\ttest_commit D &&\n+\t\ttest_commit E &&\n+\t\tgit push --all\n+\t) &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch master &&\n+\t\tgit fetch --all &&\n+\t\tgit branch branch --track origin/branch &&\n+\t\tgit rebase origin/master\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit switch master &&\n+\t\ttest_commit F &&\n+\t\ttest_commit G &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit push origin --all\n+\t)\n+}\n+\n test_expect_success setup '\n \t# create template repository\n \ttest_commit A &&\n@@ -256,4 +296,93 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n \n+test_expect_success 'background updates to remote can be mitigated with \"--force-if-includes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-with-lease --force-if-includes --all\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success 'background updates to remote can be mitigated with \"push.useForceIfIncludes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\tgit config --local push.useForceIfIncludes true &&\n+\t\ttest_must_fail git push --force-with-lease=master origin master\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\ttest_cmp expect.master actual.master\n+'\n+\n+test_expect_success '\"--force-if-includes\" should be disabled for --force-with-lease=\"<refname>:<expect>\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tremote_head=\"$(git rev-parse refs/remotes/origin/master)\" &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes --force-with-lease=\"master:$remote_head\" 2>err &&\n+\t\tgrep \"stale info\" err\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\")' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\", local rebase)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit rebase --onto HEAD~4 HEAD~1 &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n test_done\n-- \n2.28.0\n\n"},{"id":"406462","messageId":"xmqqwo0ggl65.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200926114626.28823-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v7 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-26T23:42:10Z","receivedAt":"2020-09-26T23:42:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> @@ -2252,11 +2263,11 @@ int is_empty_cas(const struct push_cas_option *cas)\n>  /*\n>   * Look at remote.fetch refspec and see if we have a remote\n>   * tracking branch for the refname there.  Fill its current\n> - * value in sha1[].\n> + * value in sha1[], and as a string.\n\nI think the array being referred to was renamed to oid[] sometime\nago.  \"and as a string\" makes it sound as if sha1[] gets the value\nas 40-hex object name text, but that is not what is being done.\n\n    Fill the name of the remote-tracking branch in *dst_refname,\n    and the name of the commit object at tis tip in oid[].\n\nperhaps?\n\n>   * If we cannot do so, return negative to signal an error.\n>   */\n>  static int remote_tracking(struct remote *remote, const char *refname,\n> -\t\t\t   struct object_id *oid)\n> +\t\t\t   struct object_id *oid, char **dst_refname)\n>  {\n>  \tchar *dst;\n>  \n> @@ -2265,9 +2276,154 @@ static int remote_tracking(struct remote *remote, const char *refname,\n>  \t\treturn -1; /* no tracking ref for refname at remote */\n>  \tif (read_ref(dst, oid))\n>  \t\treturn -1; /* we know what the tracking ref is but we cannot read it */\n> +\n> +\t*dst_refname = dst;\n> +\treturn 0;\n> +}\n> +\n> +/*\n> + * The struct \"reflog_commit_list\" and related helper functions\n> + * for list manipulation are used for collecting commits into a\n> + * list during reflog traversals in \"if_exists_or_grab_until()\".\n\nHas the name of that function changed since this comment was\nwritten?\n\n> + */\n> +struct reflog_commit_list {\n> +\tstruct commit **items;\n\nName an array in singular when its primary use is to work on an\nelement at a time---that will let you say item[4] to call the 4-th\nitem, instead of items[4] that smells awkward.\n\nAn array that is used mostly to pass around a collection as a whole\nis easier to think about when given a plural name, though.\n\n> +\tsize_t nr, alloc;\n> +};\n> +\n> +/* Add a commit to list. */\n> +static void add_commit(struct reflog_commit_list *list, struct commit *commit)\n> +{\n> +\tALLOC_GROW(list->items, list->nr + 1, list->alloc);\n> +\tlist->items[list->nr++] = commit;\n> +}\n> +\n> +/* Free and reset the list. */\n> +static void free_reflog_commit_list(struct reflog_commit_list *list)\n> +{\n> +\tFREE_AND_NULL(list->items);\n> +\tlist->nr = list->alloc = 0;\n> +}\n> +\n> +struct check_and_collect_until_cb_data {\n> +\tstruct commit *remote_commit;\n> +\tstruct reflog_commit_list *local_commits;\n> +\ttimestamp_t remote_reflog_timestamp;\n> +};\n> +\n> +/* Get the timestamp of the latest entry. */\n> +static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,\n> +\t\t       const char *ident, timestamp_t timestamp,\n> +\t\t       int tz, const char *message, void *cb_data)\n> +{\n> +\ttimestamp_t *ts = cb_data;\n> +\t*ts = timestamp;\n> +\treturn 1;\n> +}\n\nThe idea is to use a callback that immediately says \"no more\" to\ngrab the data from the first item in the iteration.  It feels\nsomewhat awkward but because there is no \"give us the Nth entry\" API\nfunction, it is the cleanest way we can do this.\n\n> +static int check_and_collect_until(struct object_id *o_oid,\n> +\t\t\t\t   struct object_id *n_oid,\n> +\t\t\t\t   const char *ident, timestamp_t timestamp,\n> +\t\t\t\t   int tz, const char *message, void *cb_data)\n> +{\n> +\tstruct commit *commit;\n> +\tstruct check_and_collect_until_cb_data *cb = cb_data;\n> +\n> +\t/*\n> +\t * If the reflog entry timestamp is older than the remote ref's\n> +\t * latest reflog entry, there is no need to check or collect\n> +\t * entries older than this one.\n> +\t */\n> +\tif (timestamp < cb->remote_reflog_timestamp)\n> +\t\treturn -1;\n> +\n> +\t/* An entry was found. */\n> +\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n> +\t\treturn 1;\n> +\n> +\t/* Look-up the commit and append it to the list. */\n> +\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n> +\t\tadd_commit(cb->local_commits, commit);\n\nThis is merely a minor naming thing, but if you rename add_commit()\nto append_commit(), you probably do not even need the comment before\nthis statement.\n\n>  \treturn 0;\n>  }\n>  \n> +#define MERGE_BASES_BATCH_SIZE 8\n\nHmph.  Do we still need batching?\n\n> +/*\n> + * Iterate through the reflog of the local ref to check if there is an entry\n> + * for the given remote-tracking ref; runs until the timestamp of an entry is\n> + * older than latest timestamp of remote-tracking ref's reflog. Any commits\n> + * are that seen along the way are collected into a list to check if the\n> + * remote-tracking ref is reachable from any of them.\n> + */\n> +static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n> +{\n> +\ttimestamp_t date;\n> +\tstruct commit *commit;\n> +\tstruct commit **chunk;\n> +\tstruct check_and_collect_until_cb_data cb;\n> +\tstruct reflog_commit_list list = { NULL, 0, 0 };\n> +\tsize_t count = 0, batch_size = 0;\n> +\tint ret = 0;\n> +\n> +\tcommit = lookup_commit_reference(the_repository, &remote->old_oid);\n> +\tif (!commit)\n> +\t\tgoto cleanup_return;\n> +\n> +\t/*\n> +\t * Get the timestamp from the latest entry\n> +\t * of the remote-tracking ref's reflog.\n> +\t */\n> +\tfor_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);\n> +\n> +\tcb.remote_commit = commit;\n> +\tcb.local_commits = &list;\n> +\tcb.remote_reflog_timestamp = date;\n> +\tret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);\n> +\n> +\t/* We found an entry in the reflog. */\n> +\tif (ret > 0)\n> +\t\tgoto cleanup_return;\n\nGood.  So '1' from the callback is \"we found one, no need to look\nfurther and no need to do merge-base\", and '-1' from the callback is\n\"we looked at all entries that are young enough to matter and we\ndidn't find exact match\".  Makes sense.\n\n> +\t/*\n> +\t * Check if the remote commit is reachable from any\n> +\t * of the commits in the collected list, in batches.\n> +\t */\n\nI do not know if batching would help (have you measured it?), but if\nwe were to batch, it is more common to arrange the loop like this:\n\n\tfor (chunk = list.items;\n             chunk < list.items + list.nr;\n\t     chunk += size) {\n             \tsize = list.items + list.nr - chunk;\n                if (MERGE_BASES_BATCH_SIZE < size)\n\t\t\tsize = MERGE_BASES_BATCH_SIZE;\n\t\t... use chunk[0..size] ...\n\t\tchunk += size;\n\t}\n\nThat is, assume that we can grab everything during this round, and\nif that bites off too many, clamp it to the maximum value.  If you\nare not comfortable with pointer arithmetic, it is also fine to use\nan auxiliary variable 'count', but ...\n\n> +\tchunk = list.items;\n> +\twhile (count < list.nr) {\n> +\t\tbatch_size = MERGE_BASES_BATCH_SIZE;\n> +\n> +\t\t/* For any leftover entries. */\n> +\t\tif ((count + MERGE_BASES_BATCH_SIZE) > list.nr)\n> +\t\t\tbatch_size = list.nr - count;\n> +\n> +\t\tif ((ret = in_merge_bases_many(commit, batch_size, chunk)))\n> +\t\t\tbreak;\n> +\n> +\t\tchunk += batch_size;\n> +\t\tcount += MERGE_BASES_BATCH_SIZE;\n\n... you are risking chunk and count to go out of sync here.\n\nIt does not matter within this loop (count will point beyond the end\nof list.item[] while chunk will never go past the array), but future\ndevelopers can be confused into thinking that they can use chunk and\ncount interchangeably after this loop exits, and at that point the\ndiscrepancy may start to matter.\n\nBut all of the above matters if it is a good idea to batch.  Does it\nmake a difference?\n\n    ... goes and looks at in_merge_bases_many() ...\n\nAh, it probably would.  \n\nI thought in_merge_bases_many() would stop early as soon as any of\nthe traversal from chunk[] reaches commit, but it uses a rather more\ngeneric paint_down_to_common() so extra items in chunk[] that are\ntopologically older than commit would result in additional traversal\nfrom commit down to them, which would not contribute much to the end\nresult.  It may be a good #leftovebit idea for future improvement to\nteach in_merge_bases_many() to use a custom replacement for\npaint_down_to_common() that stops early as soon as we find the\nanswer is true.\n\n> +\t}\n> +\n> +cleanup_return:\n> +\tfree_reflog_commit_list(&list);\n> +\treturn ret;\n> +}\n> +\n\nThanks.\n"},{"id":"406486","messageId":"20200927122745.GA6641@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqwo0ggl65.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v7 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-27T12:27:45Z","receivedAt":"2020-09-27T12:27:53Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Junio,\n\nOn 09/26/2020 16:42, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> > @@ -2252,11 +2263,11 @@ int is_empty_cas(const struct push_cas_option *cas)\n> >  /*\n> >   * Look at remote.fetch refspec and see if we have a remote\n> >   * tracking branch for the refname there.  Fill its current\n> > - * value in sha1[].\n> > + * value in sha1[], and as a string.\n> \n> I think the array being referred to was renamed to oid[] sometime\n> ago.  \"and as a string\" makes it sound as if sha1[] gets the value\n> as 40-hex object name text, but that is not what is being done.\n> \n>     Fill the name of the remote-tracking branch in *dst_refname,\n>     and the name of the commit object at tis tip in oid[].\n> \n> perhaps?\n\nOf course, that sounds better; will update.\n \n> > + * The struct \"reflog_commit_list\" and related helper functions\n> > + * for list manipulation are used for collecting commits into a\n> > + * list during reflog traversals in \"if_exists_or_grab_until()\".\n> \n> Has the name of that function changed since this comment was\n> written?\n\nHeh, it sure has. It should have been \"check_and_collect_until()\".\n \n> > + */\n> > +struct reflog_commit_list {\n> > +\tstruct commit **items;\n> \n> Name an array in singular when its primary use is to work on an\n> element at a time---that will let you say item[4] to call the 4-th\n> item, instead of items[4] that smells awkward.\n> \n> An array that is used mostly to pass around a collection as a whole\n> is easier to think about when given a plural name, though.\n\nYup.\n\n> > +\n> > +/* Get the timestamp of the latest entry. */\n> > +static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,\n> > +\t\t       const char *ident, timestamp_t timestamp,\n> > +\t\t       int tz, const char *message, void *cb_data)\n> > +{\n> > +\ttimestamp_t *ts = cb_data;\n> > +\t*ts = timestamp;\n> > +\treturn 1;\n> > +}\n> \n> The idea is to use a callback that immediately says \"no more\" to\n> grab the data from the first item in the iteration.  It feels\n> somewhat awkward but because there is no \"give us the Nth entry\" API\n> function, it is the cleanest way we can do this.\n\nI considered using \"grab_1st_entry_timestamp()\" briefy, but\n\"peek_reflog\" is shorter compared to that.\n\n> > +\t/* Look-up the commit and append it to the list. */\n> > +\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n> > +\t\tadd_commit(cb->local_commits, commit);\n> \n> This is merely a minor naming thing, but if you rename add_commit()\n> to append_commit(), you probably do not even need the comment before\n> this statement.\n\nWill do.\n\n> >  \treturn 0;\n> >  }\n> >  \n> > +#define MERGE_BASES_BATCH_SIZE 8\n> \n> Hmph.  Do we still need batching?\n> \n> > +/*\n> > + * Iterate through the reflog of the local ref to check if there is an entry\n> > + * for the given remote-tracking ref; runs until the timestamp of an entry is\n> > + * older than latest timestamp of remote-tracking ref's reflog. Any commits\n> > + * are that seen along the way are collected into a list to check if the\n> > + * remote-tracking ref is reachable from any of them.\n> > + */\n> > +static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n> > +{\n> > +\ttimestamp_t date;\n> > +\tstruct commit *commit;\n> > +\tstruct commit **chunk;\n> > +\tstruct check_and_collect_until_cb_data cb;\n> > +\tstruct reflog_commit_list list = { NULL, 0, 0 };\n> > +\tsize_t count = 0, batch_size = 0;\n> > +\tint ret = 0;\n> > +\n> > +\tcommit = lookup_commit_reference(the_repository, &remote->old_oid);\n> > +\tif (!commit)\n> > +\t\tgoto cleanup_return;\n> > +\n> > +\t/*\n> > +\t * Get the timestamp from the latest entry\n> > +\t * of the remote-tracking ref's reflog.\n> > +\t */\n> > +\tfor_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);\n> > +\n> > +\tcb.remote_commit = commit;\n> > +\tcb.local_commits = &list;\n> > +\tcb.remote_reflog_timestamp = date;\n> > +\tret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);\n> > +\n> > +\t/* We found an entry in the reflog. */\n> > +\tif (ret > 0)\n> > +\t\tgoto cleanup_return;\n> \n> Good.  So '1' from the callback is \"we found one, no need to look\n> further and no need to do merge-base\", and '-1' from the callback is\n> \"we looked at all entries that are young enough to matter and we\n> didn't find exact match\".  Makes sense.\n> \n> > +\t/*\n> > +\t * Check if the remote commit is reachable from any\n> > +\t * of the commits in the collected list, in batches.\n> > +\t */\n> \n> I do not know if batching would help (have you measured it?), but if\n> we were to batch, it is more common to arrange the loop like this:\n> \n> \tfor (chunk = list.items;\n>              chunk < list.items + list.nr;\n> \t     chunk += size) {\n>              \tsize = list.items + list.nr - chunk;\n>                 if (MERGE_BASES_BATCH_SIZE < size)\n> \t\t\tsize = MERGE_BASES_BATCH_SIZE;\n> \t\t... use chunk[0..size] ...\n> \t\tchunk += size;\n> \t}\n> \n> That is, assume that we can grab everything during this round, and\n> if that bites off too many, clamp it to the maximum value.  If you\n> are not comfortable with pointer arithmetic, it is also fine to use\n> an auxiliary variable 'count', but ...\n\nActually, the \"for\" version looks much cleaner and avoids the use\nof \"count\". However, I think ...\n\n>               chunk += size;\n\n... should be skipped because \"for ( ... ; chunk += size)\" is already\ndoing it for us; otherwise we would offset 16 entries instead of 8\nper iteration, no?\n\n> > +\tchunk = list.items;\n> > +\twhile (count < list.nr) {\n> > +\t\tbatch_size = MERGE_BASES_BATCH_SIZE;\n> > +\n> > +\t\t/* For any leftover entries. */\n> > +\t\tif ((count + MERGE_BASES_BATCH_SIZE) > list.nr)\n> > +\t\t\tbatch_size = list.nr - count;\n> > +\n> > +\t\tif ((ret = in_merge_bases_many(commit, batch_size, chunk)))\n> > +\t\t\tbreak;\n> > +\n> > +\t\tchunk += batch_size;\n> > +\t\tcount += MERGE_BASES_BATCH_SIZE;\n> \n> ... you are risking chunk and count to go out of sync here.\n> \n> It does not matter within this loop (count will point beyond the end\n> of list.item[] while chunk will never go past the array), but future\n> developers can be confused into thinking that they can use chunk and\n> count interchangeably after this loop exits, and at that point the\n> discrepancy may start to matter.\n\nI agree, it should have been \"count += batch_size;\". But, I think the\n\"for\" version looks cleaner; I will change it to that the next set.\n \n> But all of the above matters if it is a good idea to batch.  Does it\n> make a difference?\n> \n>     ... goes and looks at in_merge_bases_many() ...\n> \n> Ah, it probably would.  \n> \n> I thought in_merge_bases_many() would stop early as soon as any of\n> the traversal from chunk[] reaches commit, but it uses a rather more\n> generic paint_down_to_common() so extra items in chunk[] that are\n> topologically older than commit would result in additional traversal\n> from commit down to them, which would not contribute much to the end\n> result.  It may be a good #leftovebit idea for future improvement to\n> teach in_merge_bases_many() to use a custom replacement for\n> paint_down_to_common() that stops early as soon as we find the\n> answer is true.\n\nIf we consider the amount of time it takes when \"in_merge_bases_many()\"\nhas to be run for all the entries, there isn't much of a difference in\nperformance between batching and non-batching -- they took about the\nsame. But, as you said if the remote is reachable in the first few\nentries, batching would help with returning early if a descendant is\nfound.\n\nMaking the function stop early when a descendent is found\ndoes sound like a good #leftoverbits idea. :)\n\nThanks again, for a detailed review.\n-- \nSrinidhi Kaushik\n"},{"id":"406498","messageId":"20200927141747.78047-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200926114626.28823-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v8 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-27T14:17:44Z","receivedAt":"2020-09-27T14:18:37Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new option: \"--force-if-includes\" to \"git-push\" where forced\nupdates are allowed only if the tip of the remote-tracking ref has\nbeen integrated locally, by verifying if the tip of the remote-tracking\nref -- on which a local branch has based on -- is reachable from at\nleast one of the \"reflog\" entries of the branch about to be updated\nby force on the remote.\n\nThis option can be used with \"--force-with-lease\" with setups where\nthe remote-tracking refs of the repository are implicitly updated in\nthe background to help prevent unintended remote overwrites.\n\nIf a local branch is based on a remote ref for a rewrite, and if that\nremote-tracking ref is updated by a push from another repository after\nit has been checked out locally, force updating that branch to remote\nwith \"--force-with-lease[=<refname>[:<expect>]]\" without specifying\nthe \"<expect>\" value, can cause the update that happened in-between\nthe checkout and forced push to be lost.\n\nChanges since v7:\n  - Clean up the way batching is done for \"in_merge_bases_many()\".\n\n  - The timestamp check during \"reflog\" iteration has been moved\n    to the end of the function because we should stop collecting\n    entries older than the current one (i.e., in the next round).\n\n  - Add a test case to show deletes should be allowed with the\n    new option.\n\n  - Minor changes to comments and function names.\n\nSrinidhi Kaushik (3):\n  push: add reflog check for \"--force-if-includes\"\n  push: parse and set flag for \"--force-if-includes\"\n  t, doc: update tests, reference for \"--force-if-includes\"\n\n Documentation/config/advice.txt |   9 +-\n Documentation/config/push.txt   |   6 +\n Documentation/git-push.txt      |  26 ++++-\n advice.c                        |   3 +\n advice.h                        |   2 +\n builtin/push.c                  |  27 +++++\n builtin/send-pack.c             |  12 ++\n remote-curl.c                   |  14 ++-\n remote.c                        | 188 ++++++++++++++++++++++++++++++--\n remote.h                        |  12 +-\n send-pack.c                     |   1 +\n t/t5533-push-cas.sh             | 140 ++++++++++++++++++++++++\n transport-helper.c              |  10 ++\n transport.c                     |   8 ++\n transport.h                     |  15 ++-\n 15 files changed, 455 insertions(+), 18 deletions(-)\n\nbase-commit: 9bc233ae1cf19a49e51842c7959d80a675dbd1c0\n--\n2.28.0\n"},{"id":"406499","messageId":"20200927141747.78047-2-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200927141747.78047-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v8 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-27T14:17:45Z","receivedAt":"2020-09-27T14:18:41Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a check to verify if the remote-tracking ref of the local branch\nis reachable from one of its \"reflog\" entries.\n\nThe check iterates through the local ref's reflog to see if there\nis an entry for the remote-tracking ref and collecting any commits\nthat are seen, into a list; the iteration stops if an entry in the\nreflog matches the remote ref or if the entry timestamp is older\nthe latest entry of the remote ref's \"reflog\". If there wasn't an\nentry found for the remote ref, \"in_merge_bases_many()\" is called\nto check if it is reachable from the list of collected commits.\n\nWhen a local branch that is based on a remote ref, has been rewound\nand is to be force pushed on the remote, \"--force-if-includes\" runs\na check that ensures any updates to the remote-tracking ref that may\nhave happened (by push from another repository) in-between the time\nof the last update to the local branch (via \"git-pull\", for instance)\nand right before the time of push, have been integrated locally\nbefore allowing a forced update.\n\nIf the new option is passed without specifying \"--force-with-lease\",\nor specified along with \"--force-with-lease=<refname>:<expect>\" it\nis a \"no-op\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n builtin/send-pack.c |   5 ++\n remote.c            | 188 ++++++++++++++++++++++++++++++++++++++++++--\n remote.h            |  12 ++-\n send-pack.c         |   1 +\n transport-helper.c  |   5 ++\n transport.c         |   6 ++\n 6 files changed, 209 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 7af148d733..516cba7336 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n \n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"remote ref updated since checkout\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\ndiff --git a/remote.c b/remote.c\nindex eafc14cbe7..4781680409 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t * with the remote-tracking branch to find the value\n \t\t * to expect, but we did not have such a tracking\n \t\t * branch.\n+\t\t *\n+\t\t * If the tip of the remote-tracking ref is unreachable\n+\t\t * from any reflog entry of its local ref indicating a\n+\t\t * possible update since checkout; reject the push.\n \t\t */\n \t\tif (ref->expect_old_sha1) {\n \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n+\t\t\telse if (ref->check_reachable && ref->unreachable)\n+\t\t\t\treject_reason =\n+\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n \t\t\telse\n-\t\t\t\t/* If the ref isn't stale then force the update. */\n+\t\t\t\t/*\n+\t\t\t\t * If the ref isn't stale, and is reachable\n+\t\t\t\t * from from one of the reflog entries of\n+\t\t\t\t * the local branch, force the update.\n+\t\t\t\t */\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n \n@@ -2251,12 +2262,13 @@ int is_empty_cas(const struct push_cas_option *cas)\n \n /*\n  * Look at remote.fetch refspec and see if we have a remote\n- * tracking branch for the refname there.  Fill its current\n- * value in sha1[].\n+ * tracking branch for the refname there. Fill the name of\n+ * the remote-tracking branch in *dst_refname, and the name\n+ * of the commit object at its tip in oid[].\n  * If we cannot do so, return negative to signal an error.\n  */\n static int remote_tracking(struct remote *remote, const char *refname,\n-\t\t\t   struct object_id *oid)\n+\t\t\t   struct object_id *oid, char **dst_refname)\n {\n \tchar *dst;\n \n@@ -2265,9 +2277,154 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \t\treturn -1; /* no tracking ref for refname at remote */\n \tif (read_ref(dst, oid))\n \t\treturn -1; /* we know what the tracking ref is but we cannot read it */\n+\n+\t*dst_refname = dst;\n+\treturn 0;\n+}\n+\n+/*\n+ * The struct \"reflog_commit_list\" and related helper functions\n+ * for list manipulation are used for collecting commits into a\n+ * list during reflog traversals in \"check_and_collect_until()\".\n+ */\n+struct reflog_commit_list {\n+\tstruct commit **items;\n+\tsize_t nr, alloc;\n+};\n+\n+/* Append a commit to the list. */\n+static void append_commit(struct reflog_commit_list *list,\n+\t\t\t  struct commit *commit)\n+{\n+\tALLOC_GROW(list->items, list->nr + 1, list->alloc);\n+\tlist->items[list->nr++] = commit;\n+}\n+\n+/* Free and reset the list. */\n+static void free_reflog_commit_list(struct reflog_commit_list *list)\n+{\n+\tFREE_AND_NULL(list->items);\n+\tlist->nr = list->alloc = 0;\n+}\n+\n+struct check_and_collect_until_cb_data {\n+\tstruct commit *remote_commit;\n+\tstruct reflog_commit_list *local_commits;\n+\ttimestamp_t remote_reflog_timestamp;\n+};\n+\n+/* Get the timestamp of the latest entry. */\n+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,\n+\t\t       const char *ident, timestamp_t timestamp,\n+\t\t       int tz, const char *message, void *cb_data)\n+{\n+\ttimestamp_t *ts = cb_data;\n+\t*ts = timestamp;\n+\treturn 1;\n+}\n+\n+static int check_and_collect_until(struct object_id *o_oid,\n+\t\t\t\t   struct object_id *n_oid,\n+\t\t\t\t   const char *ident, timestamp_t timestamp,\n+\t\t\t\t   int tz, const char *message, void *cb_data)\n+{\n+\tstruct commit *commit;\n+\tstruct check_and_collect_until_cb_data *cb = cb_data;\n+\n+\t/* An entry was found. */\n+\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n+\t\treturn 1;\n+\n+\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n+\t\tappend_commit(cb->local_commits, commit);\n+\n+\t/*\n+\t * If the reflog entry timestamp is older than the remote ref's\n+\t * latest reflog entry, there is no need to check or collect\n+\t * entries older than this one.\n+\t */\n+\tif (timestamp < cb->remote_reflog_timestamp)\n+\t\treturn -1;\n+\n \treturn 0;\n }\n \n+#define MERGE_BASES_BATCH_SIZE 8\n+\n+/*\n+ * Iterate through the reflog of the local ref to check if there is an entry\n+ * for the given remote-tracking ref; runs until the timestamp of an entry is\n+ * older than latest timestamp of remote-tracking ref's reflog. Any commits\n+ * are that seen along the way are collected into a list to check if the\n+ * remote-tracking ref is reachable from any of them.\n+ */\n+static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n+{\n+\ttimestamp_t date;\n+\tstruct commit *commit;\n+\tstruct commit **chunk;\n+\tstruct check_and_collect_until_cb_data cb;\n+\tstruct reflog_commit_list list = { NULL, 0, 0 };\n+\tsize_t size = 0;\n+\tint ret = 0;\n+\n+\tcommit = lookup_commit_reference(the_repository, &remote->old_oid);\n+\tif (!commit)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Get the timestamp from the latest entry\n+\t * of the remote-tracking ref's reflog.\n+\t */\n+\tfor_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);\n+\n+\tcb.remote_commit = commit;\n+\tcb.local_commits = &list;\n+\tcb.remote_reflog_timestamp = date;\n+\tret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);\n+\n+\t/* We found an entry in the reflog. */\n+\tif (ret > 0)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Check if the remote commit is reachable from any\n+\t * of the commits in the collected list, in batches.\n+\t */\n+\tfor (chunk = list.items; chunk < list.items + list.nr; chunk += size) {\n+\t\tsize = list.items + list.nr - chunk;\n+\n+\t\t/*\n+\t\t * If the number of remaining entries is more than\n+\t\t * the batch size, clamp the number of items passed\n+\t\t * to the maximum batch size.\n+\t\t */\n+\t\tif (MERGE_BASES_BATCH_SIZE < size)\n+\t\t\tsize = MERGE_BASES_BATCH_SIZE;\n+\n+\t\tif ((ret = in_merge_bases_many(commit, size, chunk)))\n+\t\t\tbreak;\n+\t}\n+\n+cleanup_return:\n+\tfree_reflog_commit_list(&list);\n+\treturn ret;\n+}\n+\n+/*\n+ * Check for reachability of a remote-tracking\n+ * ref in the reflog entries of its local ref.\n+ */\n+static void check_if_includes_upstream(struct ref *remote)\n+{\n+\tstruct ref *local = get_local_ref(remote->name);\n+\tif (!local)\n+\t\treturn;\n+\n+\tif (is_reachable_in_reflog(local->name, remote) <= 0)\n+\t\tremote->unreachable = 1;\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n \t\t      struct ref *ref)\n@@ -2282,8 +2439,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\tref->expect_old_sha1 = 1;\n \t\tif (!entry->use_tracking)\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n-\t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\t\telse if (remote_tracking(remote, ref->name,\n+\t\t\t\t\t &ref->old_oid_expect,\n+\t\t\t\t\t &ref->tracking_ref))\n \t\t\toidclr(&ref->old_oid_expect);\n+\t\telse\n+\t\t\tref->check_reachable = cas->use_force_if_includes;\n \t\treturn;\n \t}\n \n@@ -2292,8 +2453,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\treturn;\n \n \tref->expect_old_sha1 = 1;\n-\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\tif (remote_tracking(remote, ref->name,\n+\t\t\t    &ref->old_oid_expect,\n+\t\t\t    &ref->tracking_ref))\n \t\toidclr(&ref->old_oid_expect);\n+\telse\n+\t\tref->check_reachable = cas->use_force_if_includes;\n }\n \n void apply_push_cas(struct push_cas_option *cas,\n@@ -2301,6 +2466,15 @@ void apply_push_cas(struct push_cas_option *cas,\n \t\t    struct ref *remote_refs)\n {\n \tstruct ref *ref;\n-\tfor (ref = remote_refs; ref; ref = ref->next)\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n \t\tapply_cas(cas, remote, ref);\n+\n+\t\t/*\n+\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n+\t\t * mode, and if \"--force-if-includes\" was specified, run\n+\t\t * the check.\n+\t\t */\n+\t\tif (ref->check_reachable)\n+\t\t\tcheck_if_includes_upstream(ref);\n+\t}\n }\ndiff --git a/remote.h b/remote.h\nindex eb62a47044..2d5391d281 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -107,12 +107,20 @@ struct ref {\n \tstruct object_id new_oid;\n \tstruct object_id old_oid_expect; /* used by expect-old */\n \tchar *symref;\n+\tchar *tracking_ref;\n \tunsigned int\n \t\tforce:1,\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\t/* Need to check if local reflog reaches the remote tip. */\n+\t\tcheck_reachable:1,\n+\t\t/*\n+\t\t * Store the result of the check enabled by \"check_reachable\";\n+\t\t * implies the local reflog does not reach the remote tip.\n+\t\t */\n+\t\tunreachable:1;\n \n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -142,6 +150,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n \t\tREF_STATUS_REJECT_SHALLOW,\n+\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT,\n@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);\n \n struct push_cas_option {\n \tunsigned use_tracking_for_rest:1;\n+\tunsigned use_force_if_includes:1;\n \tstruct push_cas {\n \t\tstruct object_id expect;\n \t\tunsigned use_tracking:1;\ndiff --git a/send-pack.c b/send-pack.c\nindex c9698070fc..eb4a44270b 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\ndiff --git a/transport-helper.c b/transport-helper.c\nindex b573b6c730..6157de30c7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"remote ref updated since checkout\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\n \t\t\t\tstring_list_clear(&cas_options, 0);\ndiff --git a/transport.c b/transport.c\nindex ffe2115845..65fcd22b20 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,\n \t\t\t\t \"stale info\",\n \t\t\t\t report, porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"remote ref updated since checkout\",\n+\t\t\t\t report, porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n \n \t\tstrbuf_reset(&buf);\n-- \n2.28.0\n\n"},{"id":"406500","messageId":"20200927141747.78047-3-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200927141747.78047-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v8 2/3] push: parse and set flag for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-27T14:17:46Z","receivedAt":"2020-09-27T14:18:42Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"The previous commit added the necessary machinery to implement the\n\"--force-if-includes\" protection, when \"--force-with-lease\" is used\nwithout giving exact object the remote still ought to have. Surface\nthe feature by adding a command line option and a configuration\nvariable to enable it.\n\n - Add a flag: \"TRANSPORT_PUSH_FORCE_IF_INCLUDES\" to indicate that the\n   new option was passed from the command line of via configuration\n   settings; update command line and configuration parsers to set the\n   new flag accordingly.\n\n - Introduce a new configuration option \"push.useForceIfIncludes\", which\n   is equivalent to setting \"--force-if-includes\" in the command line.\n\n - Update \"remote-curl\" to recognize and pass this option to \"send-pack\"\n   when enabled.\n\n - Update \"advise\" to catch the reject reason \"REJECT_REF_NEEDS_UPDATE\",\n   set when the ref status is \"REF_STATUS_REJECT_REMOTE_UPDATED\" and\n   (optionally) print a help message when the push fails.\n\n - The new option is a \"no-op\" in the following scenarios:\n    * When used without \"--force-with-lease\".\n    * When used with \"--force-with-lease\", and if the expected commit\n      on the remote side is specified as an argument.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n advice.c            |  3 +++\n advice.h            |  2 ++\n builtin/push.c      | 27 +++++++++++++++++++++++++++\n builtin/send-pack.c |  7 +++++++\n remote-curl.c       | 14 +++++++++++++-\n transport-helper.c  |  5 +++++\n transport.c         |  2 ++\n transport.h         | 15 ++++++++++-----\n 8 files changed, 69 insertions(+), 6 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex f0a3d32d20..164742305f 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;\n int advice_push_fetch_first = 1;\n int advice_push_needs_force = 1;\n int advice_push_unqualified_ref_name = 1;\n+int advice_push_ref_needs_update = 1;\n int advice_status_hints = 1;\n int advice_status_u_option = 1;\n int advice_status_ahead_behind_warning = 1;\n@@ -72,6 +73,7 @@ static struct {\n \t{ \"pushFetchFirst\", &advice_push_fetch_first },\n \t{ \"pushNeedsForce\", &advice_push_needs_force },\n \t{ \"pushUnqualifiedRefName\", &advice_push_unqualified_ref_name },\n+\t{ \"pushRefNeedsUpdate\", &advice_push_ref_needs_update },\n \t{ \"statusHints\", &advice_status_hints },\n \t{ \"statusUoption\", &advice_status_u_option },\n \t{ \"statusAheadBehindWarning\", &advice_status_ahead_behind_warning },\n@@ -116,6 +118,7 @@ static struct {\n \t[ADVICE_PUSH_ALREADY_EXISTS]\t\t\t= { \"pushAlreadyExists\", 1 },\n \t[ADVICE_PUSH_FETCH_FIRST]\t\t\t= { \"pushFetchFirst\", 1 },\n \t[ADVICE_PUSH_NEEDS_FORCE]\t\t\t= { \"pushNeedsForce\", 1 },\n+\t[ADVICE_PUSH_REF_NEEDS_UPDATE]\t\t\t= { \"pushRefNeedsUpdate\", 1 },\n \n \t/* make this an alias for backward compatibility */\n \t[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]\t\t= { \"pushNonFastForward\", 1 },\ndiff --git a/advice.h b/advice.h\nindex 16f2c11642..bc2432980a 100644\n--- a/advice.h\n+++ b/advice.h\n@@ -11,6 +11,7 @@ extern int advice_push_already_exists;\n extern int advice_push_fetch_first;\n extern int advice_push_needs_force;\n extern int advice_push_unqualified_ref_name;\n+extern int advice_push_ref_needs_update;\n extern int advice_status_hints;\n extern int advice_status_u_option;\n extern int advice_status_ahead_behind_warning;\n@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;\n \tADVICE_PUSH_UNQUALIFIED_REF_NAME,\n \tADVICE_PUSH_UPDATE_REJECTED_ALIAS,\n \tADVICE_PUSH_UPDATE_REJECTED,\n+\tADVICE_PUSH_REF_NEEDS_UPDATE,\n \tADVICE_RESET_QUIET_WARNING,\n \tADVICE_RESOLVE_CONFLICT,\n \tADVICE_RM_HINTS,\ndiff --git a/builtin/push.c b/builtin/push.c\nindex 0eeb2c8dd5..908b557edb 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =\n \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n \t   \"without using the '--force' option.\\n\");\n \n+static const char message_advice_ref_needs_update[] =\n+\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n+\t   \"branch has been updated since the last checkout. You may want\\n\"\n+\t   \"to integrate those changes locally (e.g., 'git pull ...')\\n\"\n+\t   \"before forcing an update.\\n\");\n+\n static void advise_pull_before_push(void)\n {\n \tif (!advice_push_non_ff_current || !advice_push_update_rejected)\n@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)\n \tadvise(_(message_advice_ref_needs_force));\n }\n \n+static void advise_ref_needs_update(void)\n+{\n+\tif (!advice_push_ref_needs_update || !advice_push_update_rejected)\n+\t\treturn;\n+\tadvise(_(message_advice_ref_needs_update));\n+}\n+\n static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\t\t     int flags)\n {\n@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\tadvise_ref_fetch_first();\n \t} else if (reject_reasons & REJECT_NEEDS_FORCE) {\n \t\tadvise_ref_needs_force();\n+\t} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {\n+\t\tadvise_ref_needs_update();\n \t}\n \n \treturn 1;\n@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\tif (!v)\n \t\t\treturn config_error_nonbool(k);\n \t\treturn color_parse(v, push_colors[slot]);\n+\t} else if (!strcmp(k, \"push.useforceifincludes\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\t*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\telse\n+\t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\treturn 0;\n \t}\n \n \treturn git_default_config(k, v, NULL);\n@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t\t       N_(\"require old value of ref to be at this value\"),\n \t\t\t       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),\n+\t\tOPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,\n+\t\t\tN_(\"require remote updates to be integrated locally\"),\n+\t\t\tTRANSPORT_PUSH_FORCE_IF_INCLUDES),\n \t\tOPT_CALLBACK(0, \"recurse-submodules\", &recurse_submodules, \"(check|on-demand|no)\",\n \t\t\t     N_(\"control recursive pushing of submodules\"), option_parse_recurse_submodules),\n \t\tOPT_BOOL_F( 0 , \"thin\", &thin, N_(\"use thin pack\"), PARSE_OPT_NOCOMPLETE),\n@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \tif ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))\n \t\tdie(_(\"--all and --mirror are incompatible\"));\n \n+\tif (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))\n+\t\tcas.use_force_if_includes = 1;\n+\n \tfor_each_string_list_item(item, push_options)\n \t\tif (strchr(item->string, '\\n'))\n \t\t\tdie(_(\"push options must not have new line characters\"));\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 516cba7336..a284ada051 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tint force_if_includes = 0;\n \tstruct packet_reader reader;\n \n \tstruct option options[] = {\n@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t  N_(\"require old value of ref to be at this value\"),\n \t\t  PARSE_OPT_OPTARG, parseopt_push_cas_option),\n+\t\tOPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,\n+\t\t\t N_(\"require remote updates to be integrated locally\")),\n \t\tOPT_END()\n \t};\n \n@@ -304,6 +307,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tif (!is_empty_cas(&cas))\n \t\tapply_push_cas(&cas, remote, remote_refs);\n \n+\tif (!is_empty_cas(&cas) && force_if_includes)\n+\t\tcas.use_force_if_includes = 1;\n+\n+\n \tset_ref_status_for_push(remote_refs, args.send_mirror,\n \t\targs.force_update);\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 32cc4a0c55..0290b04891 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -44,7 +44,8 @@ struct options {\n \t\tfrom_promisor : 1,\n \n \t\tatomic : 1,\n-\t\tobject_format : 1;\n+\t\tobject_format : 1,\n+\t\tforce_if_includes : 1;\n \tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)\n \t\tstring_list_append(&cas_options, val.buf);\n \t\tstrbuf_release(&val);\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.force_if_includes = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.force_if_includes = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n \t} else if (!strcmp(name, \"cloning\")) {\n \t\tif (!strcmp(value, \"true\"))\n \t\t\toptions.cloning = 1;\n@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)\n \t\tstrvec_push(&args, cas_option->string);\n \tstrvec_push(&args, url.buf);\n \n+\tif (options.force_if_includes)\n+\t\tstrvec_push(&args, \"--force-if-includes\");\n+\n \tstrvec_push(&args, \"--stdin\");\n \tfor (i = 0; i < nr_spec; i++)\n \t\tpacket_buf_write(&preamble, \"%s\\n\", specs[i]);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 6157de30c7..5f6e0b3bd8 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,\n \t\tif (set_helper_option(transport, TRANS_OPT_ATOMIC, \"true\") != 0)\n \t\t\tdie(_(\"helper %s does not support --atomic\"), name);\n \n+\tif (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)\n+\t\tif (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, \"true\") != 0)\n+\t\t\tdie(_(\"helper %s does not support --%s\"),\n+\t\t\t    name, TRANS_OPT_FORCE_IF_INCLUDES);\n+\n \tif (flags & TRANSPORT_PUSH_OPTIONS) {\n \t\tstruct string_list_item *item;\n \t\tfor_each_string_list_item(item, transport->push_options)\ndiff --git a/transport.c b/transport.c\nindex 65fcd22b20..47da955e4f 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t\t*reject_reasons |= REJECT_FETCH_FIRST;\n \t\t} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {\n \t\t\t*reject_reasons |= REJECT_NEEDS_FORCE;\n+\t\t} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {\n+\t\t\t*reject_reasons |= REJECT_REF_NEEDS_UPDATE;\n \t\t}\n \t}\n \tfree(head);\ndiff --git a/transport.h b/transport.h\nindex ca409ea1e4..24558c027d 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -136,6 +136,7 @@ struct transport {\n #define TRANSPORT_PUSH_ATOMIC\t\t\t(1<<13)\n #define TRANSPORT_PUSH_OPTIONS\t\t\t(1<<14)\n #define TRANSPORT_RECURSE_SUBMODULES_ONLY\t(1<<15)\n+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES\t(1<<16)\n \n int transport_summary_width(const struct ref *refs);\n \n@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);\n /* Request atomic (all-or-nothing) updates when pushing */\n #define TRANS_OPT_ATOMIC \"atomic\"\n \n+/* Require remote changes to be integrated locally. */\n+#define TRANS_OPT_FORCE_IF_INCLUDES \"force-if-includes\"\n+\n /**\n  * Returns 0 if the option was used, non-zero otherwise. Prints a\n  * message to stderr if the option is not used.\n@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,\n void transport_set_verbosity(struct transport *transport, int verbosity,\n \tint force_progress);\n \n-#define REJECT_NON_FF_HEAD     0x01\n-#define REJECT_NON_FF_OTHER    0x02\n-#define REJECT_ALREADY_EXISTS  0x04\n-#define REJECT_FETCH_FIRST     0x08\n-#define REJECT_NEEDS_FORCE     0x10\n+#define REJECT_NON_FF_HEAD      0x01\n+#define REJECT_NON_FF_OTHER     0x02\n+#define REJECT_ALREADY_EXISTS   0x04\n+#define REJECT_FETCH_FIRST      0x08\n+#define REJECT_NEEDS_FORCE      0x10\n+#define REJECT_REF_NEEDS_UPDATE 0x20\n \n int transport_push(struct repository *repo,\n \t\t   struct transport *connection,\n-- \n2.28.0\n\n"},{"id":"406501","messageId":"20200927141747.78047-4-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200927141747.78047-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v8 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-27T14:17:47Z","receivedAt":"2020-09-27T14:18:46Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Update test cases for the new option, and document its usage\nand update related references.\n\nUpdate test cases for the new option, and document its usage\nand update related references.\n\n - t/t5533-push-cas.sh:\n   Update test cases for \"compare-and-swap\" when used along with\n   \"--force-if-includes\" helps mitigate overwrites when remote\n   refs are updated in the background; allows forced updates when\n   changes from remote are integrated locally.\n\n - Documentation:\n   Add reference for the new option, configuration setting\n   (\"push.useForceIfIncludes\") and advise messages.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/advice.txt |   9 +-\n Documentation/config/push.txt   |   6 ++\n Documentation/git-push.txt      |  26 +++++-\n t/t5533-push-cas.sh             | 140 ++++++++++++++++++++++++++++++++\n 4 files changed, 177 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\nindex bdd37c3eaa..acbd0c09aa 100644\n--- a/Documentation/config/advice.txt\n+++ b/Documentation/config/advice.txt\n@@ -10,9 +10,8 @@ advice.*::\n \t\tthat the check is disabled.\n \tpushUpdateRejected::\n \t\tSet this variable to 'false' if you want to disable\n-\t\t'pushNonFFCurrent',\n-\t\t'pushNonFFMatching', 'pushAlreadyExists',\n-\t\t'pushFetchFirst', and 'pushNeedsForce'\n+\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n+\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n \t\tsimultaneously.\n \tpushNonFFCurrent::\n \t\tAdvice shown when linkgit:git-push[1] fails due to a\n@@ -41,6 +40,10 @@ advice.*::\n \t\twe can still suggest that the user push to either\n \t\trefs/heads/* or refs/tags/* based on the type of the\n \t\tsource object.\n+\tpushRefNeedsUpdate::\n+\t\tShown when linkgit:git-push[1] rejects a forced update of\n+\t\ta branch when its remote-tracking ref has updates that we\n+\t\tdo not have locally.\n \tstatusAheadBehind::\n \t\tShown when linkgit:git-status[1] computes the ahead/behind\n \t\tcounts for a local ref compared to its remote tracking ref,\ndiff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\nindex f5e5b38c68..21b256e0a4 100644\n--- a/Documentation/config/push.txt\n+++ b/Documentation/config/push.txt\n@@ -114,3 +114,9 @@ push.recurseSubmodules::\n \tspecifying '--recurse-submodules=check|on-demand|no'.\n \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n \tset (in which case a 'true' value means 'on-demand').\n+\n+push.useForceIfIncludes::\n+\tIf set to \"true\", it is equivalent to specifying\n+\t`--force-if-includes` as an option to linkgit:git-push[1]\n+\tin the command line. Adding `--no-force-if-includes` at the\n+\ttime of push overrides this configuration setting.\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..fb3a220386 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]\n \t   [-u | --set-upstream] [-o <string> | --push-option=<string>]\n \t   [--[no-]signed|--signed=(true|false|if-asked)]\n-\t   [--force-with-lease[=<refname>[:<expect>]]]\n+\t   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]\n \t   [--no-verify] [<repository> [<refspec>...]]\n \n DESCRIPTION\n@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, specifying `--force-if-includes` an an ancillary option\n+along with `--force-with-lease[=<refname>]` (i.e., without saying what\n+exact commit the ref on the remote side must be pointing at, or which\n+refs on the remote side are being protected) at the time of \"push\" will\n+verify if updates from the remote-tracking refs that may have been\n+implicitly updated in the background are integrated locally before\n+allowing a forced update.\n \n -f::\n --force::\n@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n origin +master` to force a push to the `master` branch). See the\n `<refspec>...` section above for details.\n \n+--[no-]force-if-includes::\n+\tForce an update only if the tip of the remote-tracking ref\n+\thas been integrated locally.\n++\n+This option enables a check that verifies if the tip of the\n+remote-tracking ref is reachable from one of the \"reflog\" entries of\n+the local branch based in it for a rewrite. The check ensures that any\n+updates from the remote have been incorporated locally by rejecting the\n+forced update if that is not the case.\n++\n+If the option is passed without specifying `--force-with-lease`, or\n+specified along with `--force-with-lease=<refname>:<expect>`, it is\n+a \"no-op\".\n++\n+Specifying `--no-force-if-includes` disables this behavior.\n+\n --repo=<repository>::\n \tThis option is equivalent to the <repository> argument. If both\n \tare specified, the command-line argument takes precedence.\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..a36b371aeb 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -13,6 +13,46 @@ setup_srcdst_basic () {\n \t)\n }\n \n+# For tests with \"--force-if-includes\".\n+setup_src_dup_dst () {\n+\trm -fr src dup dst &&\n+\tgit init --bare dst &&\n+\tgit clone --no-local dst src &&\n+\tgit clone --no-local dst dup\n+\t(\n+\t\tcd src &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\ttest_commit C &&\n+\t\tgit push\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit fetch &&\n+\t\tgit merge origin/master &&\n+\t\tgit switch -c branch master~2 &&\n+\t\ttest_commit D &&\n+\t\ttest_commit E &&\n+\t\tgit push --all\n+\t) &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch master &&\n+\t\tgit fetch --all &&\n+\t\tgit branch branch --track origin/branch &&\n+\t\tgit rebase origin/master\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit switch master &&\n+\t\ttest_commit F &&\n+\t\ttest_commit G &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit push origin --all\n+\t)\n+}\n+\n test_expect_success setup '\n \t# create template repository\n \ttest_commit A &&\n@@ -256,4 +296,104 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n \n+test_expect_success 'background updates to remote can be mitigated with \"--force-if-includes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-with-lease --force-if-includes --all\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success 'background updates to remote can be mitigated with \"push.useForceIfIncludes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\tgit config --local push.useForceIfIncludes true &&\n+\t\ttest_must_fail git push --force-with-lease=master origin master\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\ttest_cmp expect.master actual.master\n+'\n+\n+test_expect_success '\"--force-if-includes\" should be disabled for --force-with-lease=\"<refname>:<expect>\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/master >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tremote_head=\"$(git rev-parse refs/remotes/origin/master)\" &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes --force-with-lease=\"master:$remote_head\" 2>err &&\n+\t\tgrep \"stale info\" err\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/master >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\")' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\", local rebase)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit rebase --onto HEAD~4 HEAD~1 &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow deletes ' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\tgit pull --rebase origin branch &&\n+\t\tgit push --force-if-includes --force-with-lease=branch origin :branch\n+\t)\n+'\n+\n test_done\n-- \n2.28.0\n\n"},{"id":"406547","messageId":"xmqqtuvhn6yx.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200927141747.78047-1-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v8 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-28T17:31:34Z","receivedAt":"2020-09-28T17:31:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Add a new option: \"--force-if-includes\" to \"git-push\" where forced\n> updates are allowed only if the tip of the remote-tracking ref has\n> been integrated locally, by verifying if the tip of the remote-tracking\n> ref -- on which a local branch has based on -- is reachable from at\n> least one of the \"reflog\" entries of the branch about to be updated\n> by force on the remote.\n\nhttps://travis-ci.org/github/git/git/jobs/730962458 is a build of\n'seen' with this topic, and the same 'seen' without this topic is\nhttps://travis-ci.org/github/git/git/builds/730857608 that passes\nall the jobs.  It is curious why one particular job fails while\nothers in the same build is OK.  I've seen this pattern for this\ntopic in the past few days.\n\nThe failure in t5533-push-cas.sh is sort-of understandable as the\ntopic directly touches the area of the code the failing test\nexercises, but the failure in t3701 is totally unexpected.\n\nYou can go down to the bottom of the page and click on the\nci/print-test-failures.sh line to see which test piece fails.\n\nAnything rings a bell?\n\nThanks.\n\n\n"},{"id":"406548","messageId":"20200928174619.GB24813@szeder.dev","threadId":"54190","inReplyTo":"xmqqtuvhn6yx.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v8 0/3] push: add \"--[no-]force-if-includes\"","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-09-28T17:46:19Z","receivedAt":"2020-09-28T17:46:26Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Sep 28, 2020 at 10:31:34AM -0700, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> > Add a new option: \"--force-if-includes\" to \"git-push\" where forced\n> > updates are allowed only if the tip of the remote-tracking ref has\n> > been integrated locally, by verifying if the tip of the remote-tracking\n> > ref -- on which a local branch has based on -- is reachable from at\n> > least one of the \"reflog\" entries of the branch about to be updated\n> > by force on the remote.\n> \n> https://travis-ci.org/github/git/git/jobs/730962458 is a build of\n> 'seen' with this topic, and the same 'seen' without this topic is\n> https://travis-ci.org/github/git/git/builds/730857608 that passes\n> all the jobs.  It is curious why one particular job fails while\n> others in the same build is OK.\n\nThat build runs the test suite with a bunch of GIT_TEST_* knobs\nenabled, and the last two tests added in this series fail when run as:\n\n  GIT_TEST_COMMIT_GRAPH=1 ./t5533-push-cas.sh\n\n> The failure in t5533-push-cas.sh is sort-of understandable as the\n> topic directly touches the area of the code the failing test\n> exercises, but the failure in t3701 is totally unexpected.\n\nThat's not a failure, but the fix of a known breakage: we expect\nfailure from the scripted 'git add -i' in two tests, but the builtin\n'git add -i' fixes those issues, and those two tests succeed with\nGIT_TEST_ADD_I_USE_BUILTIN=1.\n\n"},{"id":"406559","messageId":"20200928193400.GA88208@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"20200928174619.GB24813@szeder.dev","subject":"Re: [PATCH v8 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-28T19:34:00Z","receivedAt":"2020-09-28T19:34:13Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hello,\n\nOn 09/28/2020 19:46, SZEDER Gábor wrote:\n> On Mon, Sep 28, 2020 at 10:31:34AM -0700, Junio C Hamano wrote:\n> > Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> > \n> > > Add a new option: \"--force-if-includes\" to \"git-push\" where forced\n> > > updates are allowed only if the tip of the remote-tracking ref has\n> > > been integrated locally, by verifying if the tip of the remote-tracking\n> > > ref -- on which a local branch has based on -- is reachable from at\n> > > least one of the \"reflog\" entries of the branch about to be updated\n> > > by force on the remote.\n> > \n> > https://travis-ci.org/github/git/git/jobs/730962458 is a build of\n> > 'seen' with this topic, and the same 'seen' without this topic is\n> > https://travis-ci.org/github/git/git/builds/730857608 that passes\n> > all the jobs.  It is curious why one particular job fails while\n> > others in the same build is OK.\n> \n> That build runs the test suite with a bunch of GIT_TEST_* knobs\n> enabled, and the last two tests added in this series fail when run as:\n> \n>   GIT_TEST_COMMIT_GRAPH=1 ./t5533-push-cas.sh\n \nThanks for the heads-up. It turns out that \"in_merge_bases_many()\"\nreturns different results depending on \"GIT_TEST_COMMIT_GRAPH\".\nInitially I thought that it might be related to batching the entries,\nbut that is not the case.\n\nOne of the tests that is failing is:\n  cd src &&\n  git switch branch &&\n  test_commit I &&\n  git switch master &&\n  test_commit J &&\n  git pull --rebase origin master &&\n  git push --force-if-includes --force-with-lease=\"master\"\n\nHere, we are testing to check if forced updates are allowed after\nthe remote changes have been incorporated locally, which is true\nin this case and should pass.\n\n\"in_merge_bases_many()\" used in the check as follows:\n\n  for (chunk = list.items; chunk < list.items + list.nr; chunk += size) {\n\t  size = list.items + list.nr - chunk;\n\t  if (MERGE_BASES_BATCH_SIZE < size)\n\t        size = MERGE_BASES_BATCH_SIZE;\n\n\t  if ((ret = in_merge_bases_many(commit, size, chunk)))\n\t        break;\n  }\n\nIn \"repo_in_merge_bases_many()\" [1], the following condition evaluates\nto true when \"GIT_TEST_COMMIT_GRAPH\" is 1.\n\n\tgeneration = commit_graph_generation(commit);\n\tif (generation > min_generation)\n\t\treturn ret;\n\nUnfortunately, I am unfamiliar with the code, and not sure why this\nhappens; I remember Junio mention [2] something about generation\nnumbers could it be related to that?\n\nA possible \"workaround\" is to use \"in_merge_bases()\" for each of the\ncommits we collect in the list, and the tests seem to pass with\n\"GIT_TEST_COMMIT_GRAPH\" being set; but I wonder if that's the right\nway to fix this.\n\n[1] https://git.kernel.org/pub/scm/git/git.git/tree/commit-reach.c#n319\n[2] https://public-inbox.org/git/xmqqft7djzz0.fsf@gitster.c.googlers.com/\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"406564","messageId":"xmqq4knhn0he.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200928193400.GA88208@mail.clickyotomy.dev","subject":"Re: [PATCH v8 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-28T19:51:41Z","receivedAt":"2020-09-28T19:51:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n>> That build runs the test suite with a bunch of GIT_TEST_* knobs\n>> enabled, and the last two tests added in this series fail when run as:\n>> \n>>   GIT_TEST_COMMIT_GRAPH=1 ./t5533-push-cas.sh\n>  \n> Thanks for the heads-up. It turns out that \"in_merge_bases_many()\"\n> returns different results depending on \"GIT_TEST_COMMIT_GRAPH\".\n> Initially I thought that it might be related to batching the entries,\n> but that is not the case.\n>\n> One of the tests that is failing is:\n>   cd src &&\n>   git switch branch &&\n>   test_commit I &&\n>   git switch master &&\n>   test_commit J &&\n>   git pull --rebase origin master &&\n>   git push --force-if-includes --force-with-lease=\"master\"\n>\n> Here, we are testing to check if forced updates are allowed after\n> the remote changes have been incorporated locally, which is true\n> in this case and should pass.\n>\n> \"in_merge_bases_many()\" used in the check as follows:\n> ...\n> Unfortunately, I am unfamiliar with the code, and not sure why this\n> happens; I remember Junio mention [2] something about generation\n> numbers could it be related to that?\n\nNow it's time to summon the commit-graph folks.  I think we should\nassume that it a bug in the code with commit-graph if it produces a\nresult that is different from the code without, until we prove\notherwise (e.g. in a history with clock-skew, traditional traversal\nof A..B could give a wrong result where commit-graph may produce a\ncorrect result.  I however think the topology-based merge-base\ncomputation does not suffer from the same issue).\n\nThanks.  \n"},{"id":"406565","messageId":"xmqqzh59llhr.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20200928174619.GB24813@szeder.dev","subject":"Re: [PATCH v8 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-28T20:00:48Z","receivedAt":"2020-09-28T20:00:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor <szeder.dev@gmail.com> writes:\n\n> That build runs the test suite with a bunch of GIT_TEST_* knobs\n> enabled,...\n\nAhh, OK, I knew there was one \"funny\" job but did not realize\nlinux-gcc was that one.  It is understandable why that particular\njob behaves differently from others ;-)\n\nThanks.\n"},{"id":"406678","messageId":"1377ee27-3f71-a12f-41f6-613e5a00695a@iee.email","threadId":"54190","inReplyTo":"20200927141747.78047-4-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v8 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2020-09-30T12:54:39Z","receivedAt":"2020-09-30T12:54:42Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi, spelling nit.\n\nOn 27/09/2020 15:17, Srinidhi Kaushik wrote:\n> Update test cases for the new option, and document its usage\n> and update related references.\n>\n> Update test cases for the new option, and document its usage\n> and update related references.\n>\n>  - t/t5533-push-cas.sh:\n>    Update test cases for \"compare-and-swap\" when used along with\n>    \"--force-if-includes\" helps mitigate overwrites when remote\n>    refs are updated in the background; allows forced updates when\n>    changes from remote are integrated locally.\n>\n>  - Documentation:\n>    Add reference for the new option, configuration setting\n>    (\"push.useForceIfIncludes\") and advise messages.\n>\n> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n> ---\n>  Documentation/config/advice.txt |   9 +-\n>  Documentation/config/push.txt   |   6 ++\n>  Documentation/git-push.txt      |  26 +++++-\n>  t/t5533-push-cas.sh             | 140 ++++++++++++++++++++++++++++++++\n>  4 files changed, 177 insertions(+), 4 deletions(-)\n>\n> diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\n> index bdd37c3eaa..acbd0c09aa 100644\n> --- a/Documentation/config/advice.txt\n> +++ b/Documentation/config/advice.txt\n> @@ -10,9 +10,8 @@ advice.*::\n>  \t\tthat the check is disabled.\n>  \tpushUpdateRejected::\n>  \t\tSet this variable to 'false' if you want to disable\n> -\t\t'pushNonFFCurrent',\n> -\t\t'pushNonFFMatching', 'pushAlreadyExists',\n> -\t\t'pushFetchFirst', and 'pushNeedsForce'\n> +\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n> +\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n>  \t\tsimultaneously.\n>  \tpushNonFFCurrent::\n>  \t\tAdvice shown when linkgit:git-push[1] fails due to a\n> @@ -41,6 +40,10 @@ advice.*::\n>  \t\twe can still suggest that the user push to either\n>  \t\trefs/heads/* or refs/tags/* based on the type of the\n>  \t\tsource object.\n> +\tpushRefNeedsUpdate::\n> +\t\tShown when linkgit:git-push[1] rejects a forced update of\n> +\t\ta branch when its remote-tracking ref has updates that we\n> +\t\tdo not have locally.\n>  \tstatusAheadBehind::\n>  \t\tShown when linkgit:git-status[1] computes the ahead/behind\n>  \t\tcounts for a local ref compared to its remote tracking ref,\n> diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\n> index f5e5b38c68..21b256e0a4 100644\n> --- a/Documentation/config/push.txt\n> +++ b/Documentation/config/push.txt\n> @@ -114,3 +114,9 @@ push.recurseSubmodules::\n>  \tspecifying '--recurse-submodules=check|on-demand|no'.\n>  \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n>  \tset (in which case a 'true' value means 'on-demand').\n> +\n> +push.useForceIfIncludes::\n> +\tIf set to \"true\", it is equivalent to specifying\n> +\t`--force-if-includes` as an option to linkgit:git-push[1]\n> +\tin the command line. Adding `--no-force-if-includes` at the\n> +\ttime of push overrides this configuration setting.\n> diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\n> index 3b8053447e..fb3a220386 100644\n> --- a/Documentation/git-push.txt\n> +++ b/Documentation/git-push.txt\n> @@ -13,7 +13,7 @@ SYNOPSIS\n>  \t   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]\n>  \t   [-u | --set-upstream] [-o <string> | --push-option=<string>]\n>  \t   [--[no-]signed|--signed=(true|false|if-asked)]\n> -\t   [--force-with-lease[=<refname>[:<expect>]]]\n> +\t   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]\n>  \t   [--no-verify] [<repository> [<refspec>...]]\n>  \n>  DESCRIPTION\n> @@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally\n>  force push changes to `master` if the remote version is still at\n>  `base`, regardless of what your local `remotes/origin/master` has been\n>  updated to in the background.\n> ++\n> +Alternatively, specifying `--force-if-includes` an an ancillary option\n\ns/ an an / as an /\n\n> +along with `--force-with-lease[=<refname>]` (i.e., without saying what\n> +exact commit the ref on the remote side must be pointing at, or which\n> +refs on the remote side are being protected) at the time of \"push\" will\n> +verify if updates from the remote-tracking refs that may have been\n> +implicitly updated in the background are integrated locally before\n> +allowing a forced update.\n>  \n>  -f::\n>  --force::\n> @@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n>  origin +master` to force a push to the `master` branch). See the\n>  `<refspec>...` section above for details.\n>  \n> +--[no-]force-if-includes::\n> +\tForce an update only if the tip of the remote-tracking ref\n> +\thas been integrated locally.\n> ++\n> +This option enables a check that verifies if the tip of the\n> +remote-tracking ref is reachable from one of the \"reflog\" entries of\n> +the local branch based in it for a rewrite. The check ensures that any\n> +updates from the remote have been incorporated locally by rejecting the\n> +forced update if that is not the case.\n> ++\n> +If the option is passed without specifying `--force-with-lease`, or\n> +specified along with `--force-with-lease=<refname>:<expect>`, it is\n> +a \"no-op\".\n> ++\n> +Specifying `--no-force-if-includes` disables this behavior.\n> +\n>  --repo=<repository>::\n>  \tThis option is equivalent to the <repository> argument. If both\n>  \tare specified, the command-line argument takes precedence.\n> diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\n> index 0b0eb1d025..a36b371aeb 100755\n> --- a/t/t5533-push-cas.sh\n> +++ b/t/t5533-push-cas.sh\n> @@ -13,6 +13,46 @@ setup_srcdst_basic () {\n>  \t)\n>  }\n>  \n> +# For tests with \"--force-if-includes\".\n> +setup_src_dup_dst () {\n> +\trm -fr src dup dst &&\n> +\tgit init --bare dst &&\n> +\tgit clone --no-local dst src &&\n> +\tgit clone --no-local dst dup\n> +\t(\n> +\t\tcd src &&\n> +\t\ttest_commit A &&\n> +\t\ttest_commit B &&\n> +\t\ttest_commit C &&\n> +\t\tgit push\n> +\t) &&\n> +\t(\n> +\t\tcd dup &&\n> +\t\tgit fetch &&\n> +\t\tgit merge origin/master &&\n> +\t\tgit switch -c branch master~2 &&\n> +\t\ttest_commit D &&\n> +\t\ttest_commit E &&\n> +\t\tgit push --all\n> +\t) &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch master &&\n> +\t\tgit fetch --all &&\n> +\t\tgit branch branch --track origin/branch &&\n> +\t\tgit rebase origin/master\n> +\t) &&\n> +\t(\n> +\t\tcd dup &&\n> +\t\tgit switch master &&\n> +\t\ttest_commit F &&\n> +\t\ttest_commit G &&\n> +\t\tgit switch branch &&\n> +\t\ttest_commit H &&\n> +\t\tgit push origin --all\n> +\t)\n> +}\n> +\n>  test_expect_success setup '\n>  \t# create template repository\n>  \ttest_commit A &&\n> @@ -256,4 +296,104 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n>  \t)\n>  '\n>  \n> +test_expect_success 'background updates to remote can be mitigated with \"--force-if-includes\"' '\n> +\tsetup_src_dup_dst &&\n> +\ttest_when_finished \"rm -fr dst src dup\" &&\n> +\tgit ls-remote dst refs/heads/master >expect.master &&\n> +\tgit ls-remote dst refs/heads/master >expect.branch &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch branch &&\n> +\t\ttest_commit I &&\n> +\t\tgit switch master &&\n> +\t\ttest_commit J &&\n> +\t\tgit fetch --all &&\n> +\t\ttest_must_fail git push --force-with-lease --force-if-includes --all\n> +\t) &&\n> +\tgit ls-remote dst refs/heads/master >actual.master &&\n> +\tgit ls-remote dst refs/heads/master >actual.branch &&\n> +\ttest_cmp expect.master actual.master &&\n> +\ttest_cmp expect.branch actual.branch\n> +'\n> +\n> +test_expect_success 'background updates to remote can be mitigated with \"push.useForceIfIncludes\"' '\n> +\tsetup_src_dup_dst &&\n> +\ttest_when_finished \"rm -fr dst src dup\" &&\n> +\tgit ls-remote dst refs/heads/master >expect.master &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch branch &&\n> +\t\ttest_commit I &&\n> +\t\tgit switch master &&\n> +\t\ttest_commit J &&\n> +\t\tgit fetch --all &&\n> +\t\tgit config --local push.useForceIfIncludes true &&\n> +\t\ttest_must_fail git push --force-with-lease=master origin master\n> +\t) &&\n> +\tgit ls-remote dst refs/heads/master >actual.master &&\n> +\ttest_cmp expect.master actual.master\n> +'\n> +\n> +test_expect_success '\"--force-if-includes\" should be disabled for --force-with-lease=\"<refname>:<expect>\"' '\n> +\tsetup_src_dup_dst &&\n> +\ttest_when_finished \"rm -fr dst src dup\" &&\n> +\tgit ls-remote dst refs/heads/master >expect.master &&\n> +\tgit ls-remote dst refs/heads/master >expect.branch &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch branch &&\n> +\t\ttest_commit I &&\n> +\t\tgit switch master &&\n> +\t\ttest_commit J &&\n> +\t\tremote_head=\"$(git rev-parse refs/remotes/origin/master)\" &&\n> +\t\tgit fetch --all &&\n> +\t\ttest_must_fail git push --force-if-includes --force-with-lease=\"master:$remote_head\" 2>err &&\n> +\t\tgrep \"stale info\" err\n> +\t) &&\n> +\tgit ls-remote dst refs/heads/master >actual.master &&\n> +\tgit ls-remote dst refs/heads/master >actual.branch &&\n> +\ttest_cmp expect.master actual.master &&\n> +\ttest_cmp expect.branch actual.branch\n> +'\n> +\n> +test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\")' '\n> +\tsetup_src_dup_dst &&\n> +\ttest_when_finished \"rm -fr dst src dup\" &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch branch &&\n> +\t\ttest_commit I &&\n> +\t\tgit switch master &&\n> +\t\ttest_commit J &&\n> +\t\tgit pull --rebase origin master &&\n> +\t\tgit push --force-if-includes --force-with-lease=\"master\"\n> +\t)\n> +'\n> +\n> +test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\", local rebase)' '\n> +\tsetup_src_dup_dst &&\n> +\ttest_when_finished \"rm -fr dst src dup\" &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch branch &&\n> +\t\ttest_commit I &&\n> +\t\tgit switch master &&\n> +\t\ttest_commit J &&\n> +\t\tgit pull --rebase origin master &&\n> +\t\tgit rebase --onto HEAD~4 HEAD~1 &&\n> +\t\tgit push --force-if-includes --force-with-lease=\"master\"\n> +\t)\n> +'\n> +\n> +test_expect_success '\"--force-if-includes\" should allow deletes ' '\n> +\tsetup_src_dup_dst &&\n> +\ttest_when_finished \"rm -fr dst src dup\" &&\n> +\t(\n> +\t\tcd src &&\n> +\t\tgit switch branch &&\n> +\t\tgit pull --rebase origin branch &&\n> +\t\tgit push --force-if-includes --force-with-lease=branch origin :branch\n> +\t)\n> +'\n> +\n>  test_done\n\n"},{"id":"406682","messageId":"20200930142724.GA24694@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"1377ee27-3f71-a12f-41f6-613e5a00695a@iee.email","subject":"Re: [PATCH v8 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-09-30T14:27:24Z","receivedAt":"2020-09-30T14:27:31Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Philip,\n\nOn 09/30/2020 13:54, Philip Oakley wrote:\n> Hi, spelling nit.\n> \n> > On 27/09/2020 15:17, Srinidhi Kaushik wrote:\n> > [...]\n> > +Alternatively, specifying `--force-if-includes` an an ancillary option\n> \n> s/ an an / as an /\n\nThanks! I will fix this in the next set.\n-- \nSrinidhi Kaushik\n"},{"id":"406754","messageId":"20201001082118.19441-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20200927141747.78047-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v9 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-01T08:21:15Z","receivedAt":"2020-10-01T08:21:29Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new option: \"--force-if-includes\" to \"git-push\" where forced\nupdates are allowed only if the tip of the remote-tracking ref has\nbeen integrated locally, by verifying if the tip of the remote-tracking\nref -- on which a local branch has based on -- is reachable from at\nleast one of the \"reflog\" entries of the branch about to be updated\nby force on the remote.\n\nThis option can be used with \"--force-with-lease\" with setups where\nthe remote-tracking refs of the repository are implicitly updated in\nthe background to help prevent unintended remote overwrites.\n\nIf a local branch is based on a remote ref for a rewrite, and if that\nremote-tracking ref is updated by a push from another repository after\nit has been checked out locally, force updating that branch to remote\nwith \"--force-with-lease[=<refname>[:<expect>]]\" without specifying\nthe \"<expect>\" value, can cause the update that happened in-between\nthe checkout and forced push to be lost.\n\nChanges since v8:\n  - Disable \"commit-graph\" when \"in_merge_bases_many()\" is called\n    for this check, because it returns different results depending\n    on whether \"commit-graph\" is enabled [1].\n\n  - Rename the commit list name, remove redundant comments and fix\n    some typos.\n\nSrinidhi Kaushik (3):\n  push: add reflog check for \"--force-if-includes\"\n  push: parse and set flag for \"--force-if-includes\"\n  t, doc: update tests, reference for \"--force-if-includes\"\n\n Documentation/config/advice.txt |   9 +-\n Documentation/config/push.txt   |   6 +\n Documentation/git-push.txt      |  26 ++++-\n advice.c                        |   3 +\n advice.h                        |   2 +\n builtin/push.c                  |  27 +++++\n builtin/send-pack.c             |  12 ++\n remote-curl.c                   |  14 ++-\n remote.c                        | 198 ++++++++++++++++++++++++++++++--\n remote.h                        |  12 +-\n send-pack.c                     |   1 +\n t/t5533-push-cas.sh             | 137 ++++++++++++++++++++++\n transport-helper.c              |  10 ++\n transport.c                     |   8 ++\n transport.h                     |  15 ++-\n 15 files changed, 462 insertions(+), 18 deletions(-)\n\nbase-commit: 306ee63a703ad67c54ba1209dc11dd9ea500dc1f\n[1] https://public-inbox.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com\n\n--\n2.28.0\n"},{"id":"406755","messageId":"20201001082118.19441-2-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20201001082118.19441-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-01T08:21:16Z","receivedAt":"2020-10-01T08:21:33Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a check to verify if the remote-tracking ref of the local branch\nis reachable from one of its \"reflog\" entries.\n\nThe check iterates through the local ref's reflog to see if there\nis an entry for the remote-tracking ref and collecting any commits\nthat are seen, into a list; the iteration stops if an entry in the\nreflog matches the remote ref or if the entry timestamp is older\nthe latest entry of the remote ref's \"reflog\". If there wasn't an\nentry found for the remote ref, \"in_merge_bases_many()\" is called\nto check if it is reachable from the list of collected commits.\n\nWhen a local branch that is based on a remote ref, has been rewound\nand is to be force pushed on the remote, \"--force-if-includes\" runs\na check that ensures any updates to the remote-tracking ref that may\nhave happened (by push from another repository) in-between the time\nof the last update to the local branch (via \"git-pull\", for instance)\nand right before the time of push, have been integrated locally\nbefore allowing a forced update.\n\nIf the new option is passed without specifying \"--force-with-lease\",\nor specified along with \"--force-with-lease=<refname>:<expect>\" it\nis a \"no-op\".\n\nCalls to \"in_merge_bases_many()\" return different results depending\non whether the \"commit-graph\" feature is enabled or not -- it is\ntemporarily disabled when the check runs [1].\n\n[1] https://lore.kernel.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n builtin/send-pack.c |   5 ++\n remote.c            | 198 ++++++++++++++++++++++++++++++++++++++++++--\n remote.h            |  12 ++-\n send-pack.c         |   1 +\n transport-helper.c  |   5 ++\n transport.c         |   6 ++\n 6 files changed, 219 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 7af148d733..516cba7336 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n \n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"remote ref updated since checkout\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\ndiff --git a/remote.c b/remote.c\nindex eafc14cbe7..98a578f5dc 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t * with the remote-tracking branch to find the value\n \t\t * to expect, but we did not have such a tracking\n \t\t * branch.\n+\t\t *\n+\t\t * If the tip of the remote-tracking ref is unreachable\n+\t\t * from any reflog entry of its local ref indicating a\n+\t\t * possible update since checkout; reject the push.\n \t\t */\n \t\tif (ref->expect_old_sha1) {\n \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n+\t\t\telse if (ref->check_reachable && ref->unreachable)\n+\t\t\t\treject_reason =\n+\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n \t\t\telse\n-\t\t\t\t/* If the ref isn't stale then force the update. */\n+\t\t\t\t/*\n+\t\t\t\t * If the ref isn't stale, and is reachable\n+\t\t\t\t * from from one of the reflog entries of\n+\t\t\t\t * the local branch, force the update.\n+\t\t\t\t */\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n \n@@ -2251,12 +2262,13 @@ int is_empty_cas(const struct push_cas_option *cas)\n \n /*\n  * Look at remote.fetch refspec and see if we have a remote\n- * tracking branch for the refname there.  Fill its current\n- * value in sha1[].\n+ * tracking branch for the refname there. Fill the name of\n+ * the remote-tracking branch in *dst_refname, and the name\n+ * of the commit object at its tip in oid[].\n  * If we cannot do so, return negative to signal an error.\n  */\n static int remote_tracking(struct remote *remote, const char *refname,\n-\t\t\t   struct object_id *oid)\n+\t\t\t   struct object_id *oid, char **dst_refname)\n {\n \tchar *dst;\n \n@@ -2265,9 +2277,164 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \t\treturn -1; /* no tracking ref for refname at remote */\n \tif (read_ref(dst, oid))\n \t\treturn -1; /* we know what the tracking ref is but we cannot read it */\n+\n+\t*dst_refname = dst;\n \treturn 0;\n }\n \n+/*\n+ * The struct \"reflog_commit_list\" and related helper functions\n+ * for list manipulation are used for collecting commits into a\n+ * list during reflog traversals in \"check_and_collect_until()\".\n+ */\n+struct reflog_commit_list {\n+\tstruct commit **item;\n+\tsize_t nr, alloc;\n+};\n+\n+/* Append a commit to the list. */\n+static void append_commit(struct reflog_commit_list *list,\n+\t\t\t  struct commit *commit)\n+{\n+\tALLOC_GROW(list->item, list->nr + 1, list->alloc);\n+\tlist->item[list->nr++] = commit;\n+}\n+\n+/* Free and reset the list. */\n+static void free_reflog_commit_list(struct reflog_commit_list *list)\n+{\n+\tFREE_AND_NULL(list->item);\n+\tlist->nr = list->alloc = 0;\n+}\n+\n+struct check_and_collect_until_cb_data {\n+\tstruct commit *remote_commit;\n+\tstruct reflog_commit_list *local_commits;\n+\ttimestamp_t remote_reflog_timestamp;\n+};\n+\n+/* Get the timestamp of the latest entry. */\n+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,\n+\t\t       const char *ident, timestamp_t timestamp,\n+\t\t       int tz, const char *message, void *cb_data)\n+{\n+\ttimestamp_t *ts = cb_data;\n+\t*ts = timestamp;\n+\treturn 1;\n+}\n+\n+static int check_and_collect_until(struct object_id *o_oid,\n+\t\t\t\t   struct object_id *n_oid,\n+\t\t\t\t   const char *ident, timestamp_t timestamp,\n+\t\t\t\t   int tz, const char *message, void *cb_data)\n+{\n+\tstruct commit *commit;\n+\tstruct check_and_collect_until_cb_data *cb = cb_data;\n+\n+\t/* An entry was found. */\n+\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n+\t\treturn 1;\n+\n+\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n+\t\tappend_commit(cb->local_commits, commit);\n+\n+\t/*\n+\t * If the reflog entry timestamp is older than the remote ref's\n+\t * latest reflog entry, there is no need to check or collect\n+\t * entries older than this one.\n+\t */\n+\tif (timestamp < cb->remote_reflog_timestamp)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+#define MERGE_BASES_BATCH_SIZE 8\n+\n+/*\n+ * Iterate through the reflog of the local ref to check if there is an entry\n+ * for the given remote-tracking ref; runs until the timestamp of an entry is\n+ * older than latest timestamp of remote-tracking ref's reflog. Any commits\n+ * are that seen along the way are collected into a list to check if the\n+ * remote-tracking ref is reachable from any of them.\n+ */\n+static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n+{\n+\ttimestamp_t date;\n+\tstruct commit *commit;\n+\tstruct commit **chunk;\n+\tstruct check_and_collect_until_cb_data cb;\n+\tstruct reflog_commit_list list = { NULL, 0, 0 };\n+\tsize_t size = 0;\n+\tint ret = 0;\n+\n+\tcommit = lookup_commit_reference(the_repository, &remote->old_oid);\n+\tif (!commit)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Get the timestamp from the latest entry\n+\t * of the remote-tracking ref's reflog.\n+\t */\n+\tfor_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);\n+\n+\tcb.remote_commit = commit;\n+\tcb.local_commits = &list;\n+\tcb.remote_reflog_timestamp = date;\n+\tret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);\n+\n+\t/* We found an entry in the reflog. */\n+\tif (ret > 0)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Check if the remote commit is reachable from any\n+\t * of the commits in the collected list, in batches.\n+\t */\n+\tfor (chunk = list.item; chunk < list.item + list.nr; chunk += size) {\n+\t\tsize = list.item + list.nr - chunk;\n+\t\tif (MERGE_BASES_BATCH_SIZE < size)\n+\t\t\tsize = MERGE_BASES_BATCH_SIZE;\n+\n+\t\tif ((ret = in_merge_bases_many(commit, size, chunk)))\n+\t\t\tbreak;\n+\t}\n+\n+cleanup_return:\n+\tfree_reflog_commit_list(&list);\n+\treturn ret;\n+}\n+\n+/* Toggle the \"commit-graph\" feature; return the previously set state. */\n+static int toggle_commit_graph(struct repository *repo, int disable) {\n+\tint prev = repo->commit_graph_disabled;\n+\trepo->commit_graph_disabled = disable;\n+\treturn prev;\n+}\n+\n+/*\n+ * Check for reachability of a remote-tracking\n+ * ref in the reflog entries of its local ref.\n+ */\n+static void check_if_includes_upstream(struct ref *remote)\n+{\n+\tint prev;\n+\tstruct ref *local = get_local_ref(remote->name);\n+\tif (!local)\n+\t\treturn;\n+\n+\t/*\n+\t * TODO: Remove \"toggle_commit_graph()\" calls around the check.\n+\t * Depending on whether \"commit-graph\" enabled or not,\n+\t * \"in_merge_bases_many()\" returns different results;\n+\t * disable it temporarily when the check runs.\n+\t */\n+\tprev = toggle_commit_graph(the_repository, 1);\n+\tif (is_reachable_in_reflog(local->name, remote) <= 0)\n+\t\tremote->unreachable = 1;\n+\ttoggle_commit_graph(the_repository, prev);\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n \t\t      struct ref *ref)\n@@ -2282,8 +2449,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\tref->expect_old_sha1 = 1;\n \t\tif (!entry->use_tracking)\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n-\t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\t\telse if (remote_tracking(remote, ref->name,\n+\t\t\t\t\t &ref->old_oid_expect,\n+\t\t\t\t\t &ref->tracking_ref))\n \t\t\toidclr(&ref->old_oid_expect);\n+\t\telse\n+\t\t\tref->check_reachable = cas->use_force_if_includes;\n \t\treturn;\n \t}\n \n@@ -2292,8 +2463,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\treturn;\n \n \tref->expect_old_sha1 = 1;\n-\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\tif (remote_tracking(remote, ref->name,\n+\t\t\t    &ref->old_oid_expect,\n+\t\t\t    &ref->tracking_ref))\n \t\toidclr(&ref->old_oid_expect);\n+\telse\n+\t\tref->check_reachable = cas->use_force_if_includes;\n }\n \n void apply_push_cas(struct push_cas_option *cas,\n@@ -2301,6 +2476,15 @@ void apply_push_cas(struct push_cas_option *cas,\n \t\t    struct ref *remote_refs)\n {\n \tstruct ref *ref;\n-\tfor (ref = remote_refs; ref; ref = ref->next)\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n \t\tapply_cas(cas, remote, ref);\n+\n+\t\t/*\n+\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n+\t\t * mode, and if \"--force-if-includes\" was specified, run\n+\t\t * the check.\n+\t\t */\n+\t\tif (ref->check_reachable)\n+\t\t\tcheck_if_includes_upstream(ref);\n+\t}\n }\ndiff --git a/remote.h b/remote.h\nindex eb62a47044..2d5391d281 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -107,12 +107,20 @@ struct ref {\n \tstruct object_id new_oid;\n \tstruct object_id old_oid_expect; /* used by expect-old */\n \tchar *symref;\n+\tchar *tracking_ref;\n \tunsigned int\n \t\tforce:1,\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\t/* Need to check if local reflog reaches the remote tip. */\n+\t\tcheck_reachable:1,\n+\t\t/*\n+\t\t * Store the result of the check enabled by \"check_reachable\";\n+\t\t * implies the local reflog does not reach the remote tip.\n+\t\t */\n+\t\tunreachable:1;\n \n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -142,6 +150,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n \t\tREF_STATUS_REJECT_SHALLOW,\n+\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT,\n@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);\n \n struct push_cas_option {\n \tunsigned use_tracking_for_rest:1;\n+\tunsigned use_force_if_includes:1;\n \tstruct push_cas {\n \t\tstruct object_id expect;\n \t\tunsigned use_tracking:1;\ndiff --git a/send-pack.c b/send-pack.c\nindex c9698070fc..eb4a44270b 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\ndiff --git a/transport-helper.c b/transport-helper.c\nindex b573b6c730..6157de30c7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"remote ref updated since checkout\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\n \t\t\t\tstring_list_clear(&cas_options, 0);\ndiff --git a/transport.c b/transport.c\nindex ffe2115845..65fcd22b20 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,\n \t\t\t\t \"stale info\",\n \t\t\t\t report, porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"remote ref updated since checkout\",\n+\t\t\t\t report, porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n \n \t\tstrbuf_reset(&buf);\n-- \n2.28.0\n\n"},{"id":"406756","messageId":"20201001082118.19441-3-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20201001082118.19441-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v9 2/3] push: parse and set flag for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-01T08:21:17Z","receivedAt":"2020-10-01T08:21:38Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"The previous commit added the necessary machinery to implement the\n\"--force-if-includes\" protection, when \"--force-with-lease\" is used\nwithout giving exact object the remote still ought to have. Surface\nthe feature by adding a command line option and a configuration\nvariable to enable it.\n\n - Add a flag: \"TRANSPORT_PUSH_FORCE_IF_INCLUDES\" to indicate that the\n   new option was passed from the command line of via configuration\n   settings; update command line and configuration parsers to set the\n   new flag accordingly.\n\n - Introduce a new configuration option \"push.useForceIfIncludes\", which\n   is equivalent to setting \"--force-if-includes\" in the command line.\n\n - Update \"remote-curl\" to recognize and pass this option to \"send-pack\"\n   when enabled.\n\n - Update \"advise\" to catch the reject reason \"REJECT_REF_NEEDS_UPDATE\",\n   set when the ref status is \"REF_STATUS_REJECT_REMOTE_UPDATED\" and\n   (optionally) print a help message when the push fails.\n\n - The new option is a \"no-op\" in the following scenarios:\n    * When used without \"--force-with-lease\".\n    * When used with \"--force-with-lease\", and if the expected commit\n      on the remote side is specified as an argument.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n advice.c            |  3 +++\n advice.h            |  2 ++\n builtin/push.c      | 27 +++++++++++++++++++++++++++\n builtin/send-pack.c |  7 +++++++\n remote-curl.c       | 14 +++++++++++++-\n transport-helper.c  |  5 +++++\n transport.c         |  2 ++\n transport.h         | 15 ++++++++++-----\n 8 files changed, 69 insertions(+), 6 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex f0a3d32d20..164742305f 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;\n int advice_push_fetch_first = 1;\n int advice_push_needs_force = 1;\n int advice_push_unqualified_ref_name = 1;\n+int advice_push_ref_needs_update = 1;\n int advice_status_hints = 1;\n int advice_status_u_option = 1;\n int advice_status_ahead_behind_warning = 1;\n@@ -72,6 +73,7 @@ static struct {\n \t{ \"pushFetchFirst\", &advice_push_fetch_first },\n \t{ \"pushNeedsForce\", &advice_push_needs_force },\n \t{ \"pushUnqualifiedRefName\", &advice_push_unqualified_ref_name },\n+\t{ \"pushRefNeedsUpdate\", &advice_push_ref_needs_update },\n \t{ \"statusHints\", &advice_status_hints },\n \t{ \"statusUoption\", &advice_status_u_option },\n \t{ \"statusAheadBehindWarning\", &advice_status_ahead_behind_warning },\n@@ -116,6 +118,7 @@ static struct {\n \t[ADVICE_PUSH_ALREADY_EXISTS]\t\t\t= { \"pushAlreadyExists\", 1 },\n \t[ADVICE_PUSH_FETCH_FIRST]\t\t\t= { \"pushFetchFirst\", 1 },\n \t[ADVICE_PUSH_NEEDS_FORCE]\t\t\t= { \"pushNeedsForce\", 1 },\n+\t[ADVICE_PUSH_REF_NEEDS_UPDATE]\t\t\t= { \"pushRefNeedsUpdate\", 1 },\n \n \t/* make this an alias for backward compatibility */\n \t[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]\t\t= { \"pushNonFastForward\", 1 },\ndiff --git a/advice.h b/advice.h\nindex 16f2c11642..bc2432980a 100644\n--- a/advice.h\n+++ b/advice.h\n@@ -11,6 +11,7 @@ extern int advice_push_already_exists;\n extern int advice_push_fetch_first;\n extern int advice_push_needs_force;\n extern int advice_push_unqualified_ref_name;\n+extern int advice_push_ref_needs_update;\n extern int advice_status_hints;\n extern int advice_status_u_option;\n extern int advice_status_ahead_behind_warning;\n@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;\n \tADVICE_PUSH_UNQUALIFIED_REF_NAME,\n \tADVICE_PUSH_UPDATE_REJECTED_ALIAS,\n \tADVICE_PUSH_UPDATE_REJECTED,\n+\tADVICE_PUSH_REF_NEEDS_UPDATE,\n \tADVICE_RESET_QUIET_WARNING,\n \tADVICE_RESOLVE_CONFLICT,\n \tADVICE_RM_HINTS,\ndiff --git a/builtin/push.c b/builtin/push.c\nindex 0eeb2c8dd5..908b557edb 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =\n \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n \t   \"without using the '--force' option.\\n\");\n \n+static const char message_advice_ref_needs_update[] =\n+\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n+\t   \"branch has been updated since the last checkout. You may want\\n\"\n+\t   \"to integrate those changes locally (e.g., 'git pull ...')\\n\"\n+\t   \"before forcing an update.\\n\");\n+\n static void advise_pull_before_push(void)\n {\n \tif (!advice_push_non_ff_current || !advice_push_update_rejected)\n@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)\n \tadvise(_(message_advice_ref_needs_force));\n }\n \n+static void advise_ref_needs_update(void)\n+{\n+\tif (!advice_push_ref_needs_update || !advice_push_update_rejected)\n+\t\treturn;\n+\tadvise(_(message_advice_ref_needs_update));\n+}\n+\n static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\t\t     int flags)\n {\n@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\tadvise_ref_fetch_first();\n \t} else if (reject_reasons & REJECT_NEEDS_FORCE) {\n \t\tadvise_ref_needs_force();\n+\t} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {\n+\t\tadvise_ref_needs_update();\n \t}\n \n \treturn 1;\n@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\tif (!v)\n \t\t\treturn config_error_nonbool(k);\n \t\treturn color_parse(v, push_colors[slot]);\n+\t} else if (!strcmp(k, \"push.useforceifincludes\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\t*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\telse\n+\t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\treturn 0;\n \t}\n \n \treturn git_default_config(k, v, NULL);\n@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t\t       N_(\"require old value of ref to be at this value\"),\n \t\t\t       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),\n+\t\tOPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,\n+\t\t\tN_(\"require remote updates to be integrated locally\"),\n+\t\t\tTRANSPORT_PUSH_FORCE_IF_INCLUDES),\n \t\tOPT_CALLBACK(0, \"recurse-submodules\", &recurse_submodules, \"(check|on-demand|no)\",\n \t\t\t     N_(\"control recursive pushing of submodules\"), option_parse_recurse_submodules),\n \t\tOPT_BOOL_F( 0 , \"thin\", &thin, N_(\"use thin pack\"), PARSE_OPT_NOCOMPLETE),\n@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \tif ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))\n \t\tdie(_(\"--all and --mirror are incompatible\"));\n \n+\tif (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))\n+\t\tcas.use_force_if_includes = 1;\n+\n \tfor_each_string_list_item(item, push_options)\n \t\tif (strchr(item->string, '\\n'))\n \t\t\tdie(_(\"push options must not have new line characters\"));\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 516cba7336..a284ada051 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tint force_if_includes = 0;\n \tstruct packet_reader reader;\n \n \tstruct option options[] = {\n@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t  N_(\"require old value of ref to be at this value\"),\n \t\t  PARSE_OPT_OPTARG, parseopt_push_cas_option),\n+\t\tOPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,\n+\t\t\t N_(\"require remote updates to be integrated locally\")),\n \t\tOPT_END()\n \t};\n \n@@ -304,6 +307,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tif (!is_empty_cas(&cas))\n \t\tapply_push_cas(&cas, remote, remote_refs);\n \n+\tif (!is_empty_cas(&cas) && force_if_includes)\n+\t\tcas.use_force_if_includes = 1;\n+\n+\n \tset_ref_status_for_push(remote_refs, args.send_mirror,\n \t\targs.force_update);\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 32cc4a0c55..0290b04891 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -44,7 +44,8 @@ struct options {\n \t\tfrom_promisor : 1,\n \n \t\tatomic : 1,\n-\t\tobject_format : 1;\n+\t\tobject_format : 1,\n+\t\tforce_if_includes : 1;\n \tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)\n \t\tstring_list_append(&cas_options, val.buf);\n \t\tstrbuf_release(&val);\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.force_if_includes = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.force_if_includes = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n \t} else if (!strcmp(name, \"cloning\")) {\n \t\tif (!strcmp(value, \"true\"))\n \t\t\toptions.cloning = 1;\n@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)\n \t\tstrvec_push(&args, cas_option->string);\n \tstrvec_push(&args, url.buf);\n \n+\tif (options.force_if_includes)\n+\t\tstrvec_push(&args, \"--force-if-includes\");\n+\n \tstrvec_push(&args, \"--stdin\");\n \tfor (i = 0; i < nr_spec; i++)\n \t\tpacket_buf_write(&preamble, \"%s\\n\", specs[i]);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 6157de30c7..5f6e0b3bd8 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,\n \t\tif (set_helper_option(transport, TRANS_OPT_ATOMIC, \"true\") != 0)\n \t\t\tdie(_(\"helper %s does not support --atomic\"), name);\n \n+\tif (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)\n+\t\tif (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, \"true\") != 0)\n+\t\t\tdie(_(\"helper %s does not support --%s\"),\n+\t\t\t    name, TRANS_OPT_FORCE_IF_INCLUDES);\n+\n \tif (flags & TRANSPORT_PUSH_OPTIONS) {\n \t\tstruct string_list_item *item;\n \t\tfor_each_string_list_item(item, transport->push_options)\ndiff --git a/transport.c b/transport.c\nindex 65fcd22b20..47da955e4f 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t\t*reject_reasons |= REJECT_FETCH_FIRST;\n \t\t} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {\n \t\t\t*reject_reasons |= REJECT_NEEDS_FORCE;\n+\t\t} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {\n+\t\t\t*reject_reasons |= REJECT_REF_NEEDS_UPDATE;\n \t\t}\n \t}\n \tfree(head);\ndiff --git a/transport.h b/transport.h\nindex ca409ea1e4..24558c027d 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -136,6 +136,7 @@ struct transport {\n #define TRANSPORT_PUSH_ATOMIC\t\t\t(1<<13)\n #define TRANSPORT_PUSH_OPTIONS\t\t\t(1<<14)\n #define TRANSPORT_RECURSE_SUBMODULES_ONLY\t(1<<15)\n+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES\t(1<<16)\n \n int transport_summary_width(const struct ref *refs);\n \n@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);\n /* Request atomic (all-or-nothing) updates when pushing */\n #define TRANS_OPT_ATOMIC \"atomic\"\n \n+/* Require remote changes to be integrated locally. */\n+#define TRANS_OPT_FORCE_IF_INCLUDES \"force-if-includes\"\n+\n /**\n  * Returns 0 if the option was used, non-zero otherwise. Prints a\n  * message to stderr if the option is not used.\n@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,\n void transport_set_verbosity(struct transport *transport, int verbosity,\n \tint force_progress);\n \n-#define REJECT_NON_FF_HEAD     0x01\n-#define REJECT_NON_FF_OTHER    0x02\n-#define REJECT_ALREADY_EXISTS  0x04\n-#define REJECT_FETCH_FIRST     0x08\n-#define REJECT_NEEDS_FORCE     0x10\n+#define REJECT_NON_FF_HEAD      0x01\n+#define REJECT_NON_FF_OTHER     0x02\n+#define REJECT_ALREADY_EXISTS   0x04\n+#define REJECT_FETCH_FIRST      0x08\n+#define REJECT_NEEDS_FORCE      0x10\n+#define REJECT_REF_NEEDS_UPDATE 0x20\n \n int transport_push(struct repository *repo,\n \t\t   struct transport *connection,\n-- \n2.28.0\n\n"},{"id":"406757","messageId":"20201001082118.19441-4-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20201001082118.19441-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v9 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-01T08:21:18Z","receivedAt":"2020-10-01T08:21:39Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Update test cases for the new option, and document its usage\nand update related references.\n\nUpdate test cases for the new option, and document its usage\nand update related references.\n\n - t/t5533-push-cas.sh:\n   Update test cases for \"compare-and-swap\" when used along with\n   \"--force-if-includes\" helps mitigate overwrites when remote\n   refs are updated in the background; allows forced updates when\n   changes from remote are integrated locally.\n\n - Documentation:\n   Add reference for the new option, configuration setting\n   (\"push.useForceIfIncludes\") and advise messages.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/advice.txt |   9 ++-\n Documentation/config/push.txt   |   6 ++\n Documentation/git-push.txt      |  26 +++++-\n t/t5533-push-cas.sh             | 137 ++++++++++++++++++++++++++++++++\n 4 files changed, 174 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\nindex bdd37c3eaa..acbd0c09aa 100644\n--- a/Documentation/config/advice.txt\n+++ b/Documentation/config/advice.txt\n@@ -10,9 +10,8 @@ advice.*::\n \t\tthat the check is disabled.\n \tpushUpdateRejected::\n \t\tSet this variable to 'false' if you want to disable\n-\t\t'pushNonFFCurrent',\n-\t\t'pushNonFFMatching', 'pushAlreadyExists',\n-\t\t'pushFetchFirst', and 'pushNeedsForce'\n+\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n+\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n \t\tsimultaneously.\n \tpushNonFFCurrent::\n \t\tAdvice shown when linkgit:git-push[1] fails due to a\n@@ -41,6 +40,10 @@ advice.*::\n \t\twe can still suggest that the user push to either\n \t\trefs/heads/* or refs/tags/* based on the type of the\n \t\tsource object.\n+\tpushRefNeedsUpdate::\n+\t\tShown when linkgit:git-push[1] rejects a forced update of\n+\t\ta branch when its remote-tracking ref has updates that we\n+\t\tdo not have locally.\n \tstatusAheadBehind::\n \t\tShown when linkgit:git-status[1] computes the ahead/behind\n \t\tcounts for a local ref compared to its remote tracking ref,\ndiff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\nindex f5e5b38c68..21b256e0a4 100644\n--- a/Documentation/config/push.txt\n+++ b/Documentation/config/push.txt\n@@ -114,3 +114,9 @@ push.recurseSubmodules::\n \tspecifying '--recurse-submodules=check|on-demand|no'.\n \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n \tset (in which case a 'true' value means 'on-demand').\n+\n+push.useForceIfIncludes::\n+\tIf set to \"true\", it is equivalent to specifying\n+\t`--force-if-includes` as an option to linkgit:git-push[1]\n+\tin the command line. Adding `--no-force-if-includes` at the\n+\ttime of push overrides this configuration setting.\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..ab103c82cf 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]\n \t   [-u | --set-upstream] [-o <string> | --push-option=<string>]\n \t   [--[no-]signed|--signed=(true|false|if-asked)]\n-\t   [--force-with-lease[=<refname>[:<expect>]]]\n+\t   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]\n \t   [--no-verify] [<repository> [<refspec>...]]\n \n DESCRIPTION\n@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, specifying `--force-if-includes` as an ancillary option\n+along with `--force-with-lease[=<refname>]` (i.e., without saying what\n+exact commit the ref on the remote side must be pointing at, or which\n+refs on the remote side are being protected) at the time of \"push\" will\n+verify if updates from the remote-tracking refs that may have been\n+implicitly updated in the background are integrated locally before\n+allowing a forced update.\n \n -f::\n --force::\n@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n origin +master` to force a push to the `master` branch). See the\n `<refspec>...` section above for details.\n \n+--[no-]force-if-includes::\n+\tForce an update only if the tip of the remote-tracking ref\n+\thas been integrated locally.\n++\n+This option enables a check that verifies if the tip of the\n+remote-tracking ref is reachable from one of the \"reflog\" entries of\n+the local branch based in it for a rewrite. The check ensures that any\n+updates from the remote have been incorporated locally by rejecting the\n+forced update if that is not the case.\n++\n+If the option is passed without specifying `--force-with-lease`, or\n+specified along with `--force-with-lease=<refname>:<expect>`, it is\n+a \"no-op\".\n++\n+Specifying `--no-force-if-includes` disables this behavior.\n+\n --repo=<repository>::\n \tThis option is equivalent to the <repository> argument. If both\n \tare specified, the command-line argument takes precedence.\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..7813e8470e 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -13,6 +13,46 @@ setup_srcdst_basic () {\n \t)\n }\n \n+# For tests with \"--force-if-includes\".\n+setup_src_dup_dst () {\n+\trm -fr src dup dst &&\n+\tgit init --bare dst &&\n+\tgit clone --no-local dst src &&\n+\tgit clone --no-local dst dup\n+\t(\n+\t\tcd src &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\ttest_commit C &&\n+\t\tgit push origin\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit fetch &&\n+\t\tgit merge origin/master &&\n+\t\tgit switch -c branch master~2 &&\n+\t\ttest_commit D &&\n+\t\ttest_commit E &&\n+\t\tgit push origin --all\n+\t) &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch master &&\n+\t\tgit fetch --all &&\n+\t\tgit branch branch --track origin/branch &&\n+\t\tgit rebase origin/master\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit switch master &&\n+\t\ttest_commit F &&\n+\t\ttest_commit G &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit push origin --all\n+\t)\n+}\n+\n test_expect_success setup '\n \t# create template repository\n \ttest_commit A &&\n@@ -256,4 +296,101 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n \n+test_expect_success 'background updates to remote can be mitigated with \"--force-if-includes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/branch >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-with-lease --force-if-includes --all\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/branch >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success 'background updates to remote can be mitigated with \"push.useForceIfIncludes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\tgit config --local push.useForceIfIncludes true &&\n+\t\ttest_must_fail git push --force-with-lease=master origin master\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\ttest_cmp expect.master actual.master\n+'\n+\n+test_expect_success '\"--force-if-includes\" should be disabled for --force-with-lease=\"<refname>:<expect>\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tremote_head=\"$(git rev-parse refs/remotes/origin/master)\" &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes --force-with-lease=\"master:$remote_head\" 2>err &&\n+\t\tgrep \"stale info\" err\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\ttest_cmp expect.master actual.master\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\")' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\", local rebase)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit rebase --onto HEAD~4 HEAD~1 &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow deletes' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\tgit pull --rebase origin branch &&\n+\t\tgit push --force-if-includes --force-with-lease=\"branch\" origin :branch\n+\t)\n+'\n+\n test_done\n-- \n2.28.0\n\n"},{"id":"406763","messageId":"xmqqlfgqc5kq.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20201001082118.19441-1-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v9 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-01T15:46:13Z","receivedAt":"2020-10-01T15:46:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Changes since v8:\n>   - Disable \"commit-graph\" when \"in_merge_bases_many()\" is called\n>     for this check, because it returns different results depending\n>     on whether \"commit-graph\" is enabled [1].\n\nIs that a wise move, though?  If the \"different results\" is\nexpected, then it is a different story, but I would think it is a\nbug in commit-graph codepath if it produces a result different from\nwhat the callers expect, and disabling from the caller's end would\nmean that we lose one opportunity to help commit-graph folks to go\nand fix their bugs, no?\n\nOther than that, I think the topic is in quite a good shape.  Thanks\nfor working on polishing it.\n\n"},{"id":"406784","messageId":"xmqq4knddg5v.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"xmqqlfgqc5kq.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v9 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-01T17:12:12Z","receivedAt":"2020-10-01T17:13:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n>\n>> Changes since v8:\n>>   - Disable \"commit-graph\" when \"in_merge_bases_many()\" is called\n>>     for this check, because it returns different results depending\n>>     on whether \"commit-graph\" is enabled [1].\n>\n> Is that a wise move, though?  If the \"different results\" is\n> expected, then it is a different story, but I would think it is a\n> bug in commit-graph codepath if it produces a result different from\n> what the callers expect, and disabling from the caller's end would\n> mean that we lose one opportunity to help commit-graph folks to go\n> and fix their bugs, no?\n>\n> Other than that, I think the topic is in quite a good shape.  Thanks\n> for working on polishing it.\n\nIn other words, how about doing it like so.\n\nIn an ideal world, folks who know more about commit-graph than we do\nwill find what's broken in in_merge_bases_many() when commit-graph\nis in use, before I finish lecturing against hiding a breakage under\nthe rug.  Let's see if another call for help helps ;-)\n\n remote.c | 15 ++++++++++++++-\n 1 file changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git i/remote.c w/remote.c\nindex 98a578f5dc..361b8f1c0e 100644\n--- i/remote.c\n+++ w/remote.c\n@@ -2408,7 +2408,20 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n /* Toggle the \"commit-graph\" feature; return the previously set state. */\n static int toggle_commit_graph(struct repository *repo, int disable) {\n \tint prev = repo->commit_graph_disabled;\n-\trepo->commit_graph_disabled = disable;\n+\tstatic int should_toggle = -1;\n+\n+\tif (should_toggle < 0) {\n+\t\t/*\n+\t\t * The in_merge_bases_many() seems to misbehave when\n+\t\t * the commit-graph feature is in use.  Disable it for\n+\t\t * normal users, but keep it enabled when specifically\n+\t\t * testing the feature.\n+\t\t */\n+\t\tshould_toggle = !git_env_bool(\"GIT_TEST_COMMIT_GRAPH\", 0);\n+\t}\n+\n+\tif (should_toggle)\n+\t\trepo->commit_graph_disabled = disable;\n \treturn prev;\n }\n \n"},{"id":"406785","messageId":"20201001175443.GA28444@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqq4knddg5v.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v9 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-01T17:54:43Z","receivedAt":"2020-10-01T17:54:56Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hello,\n\nOn 10/01/2020 10:12, Junio C Hamano wrote:\n> Junio C Hamano <gitster@pobox.com> writes:\n> \n> > Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> >\n> >> Changes since v8:\n> >>   - Disable \"commit-graph\" when \"in_merge_bases_many()\" is called\n> >>     for this check, because it returns different results depending\n> >>     on whether \"commit-graph\" is enabled [1].\n> >\n> > Is that a wise move, though?  If the \"different results\" is\n> > expected, then it is a different story, but I would think it is a\n> > bug in commit-graph codepath if it produces a result different from\n> > what the callers expect, and disabling from the caller's end would\n> > mean that we lose one opportunity to help commit-graph folks to go\n> > and fix their bugs, no?\n\nI didn't want want to cause a delay with this patch. Since the new\noption was seemingly working without it, I decided to disable it here\nand added a \"TODO\" in the comments to remove \"toggle_commit_graph()\"\nin the future. We can definitely put this on hold and wait until the\nwith and without \"commit-graph\" result disparities are clarified.\n\n> > Other than that, I think the topic is in quite a good shape.  Thanks\n> > for working on polishing it.\n\nThanks, I appreciate it!\n \n> In other words, how about doing it like so.\n> \n> In an ideal world, folks who know more about commit-graph than we do\n> will find what's broken in in_merge_bases_many() when commit-graph\n> is in use, before I finish lecturing against hiding a breakage under\n> the rug.  Let's see if another call for help helps ;-)\n\n:)\n \n>  remote.c | 15 ++++++++++++++-\n>  1 file changed, 14 insertions(+), 1 deletion(-)\n> \n> diff --git i/remote.c w/remote.c\n> index 98a578f5dc..361b8f1c0e 100644\n> --- i/remote.c\n> +++ w/remote.c\n> @@ -2408,7 +2408,20 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n>  /* Toggle the \"commit-graph\" feature; return the previously set state. */\n>  static int toggle_commit_graph(struct repository *repo, int disable) {\n>  \tint prev = repo->commit_graph_disabled;\n> -\trepo->commit_graph_disabled = disable;\n> +\tstatic int should_toggle = -1;\n> +\n> +\tif (should_toggle < 0) {\n> +\t\t/*\n> +\t\t * The in_merge_bases_many() seems to misbehave when\n> +\t\t * the commit-graph feature is in use.  Disable it for\n> +\t\t * normal users, but keep it enabled when specifically\n> +\t\t * testing the feature.\n> +\t\t */\n> +\t\tshould_toggle = !git_env_bool(\"GIT_TEST_COMMIT_GRAPH\", 0);\n> +\t}\n> +\n> +\tif (should_toggle)\n> +\t\trepo->commit_graph_disabled = disable;\n>  \treturn prev;\n>  }\n>  \n\nAh, this will keep a record of the behavior in the tests; nice,\nwill update in the next set.\n\nI looked around a little bit trying to understand what was happening\nhere. As mentioned before [1], \"repo_in_merge_bases_many()\" returns\nearly because of this:\n\n  for (i = 0; i < nr_reference; i++) {\n\t  if (repo_parse_commit(r, reference[i]))\n\t\t  return ret;\n\n\t  generation = commit_graph_generation(reference[i]);\n\t  if (generation < min_generation)\n\t\t  min_generation = generation;\n\t  fprintf(stderr,\n\t\t  \"[%s]: (local): generation: %u, min_generation: %u\\n\",\n\t\t  oid_to_hex(&reference[i]->object.oid),\n\t\t  generation,\n\t\t  min_generation);\n  }\n\n  generation = commit_graph_generation(commit);\n  fprintf(stderr, \"[%s]: (remote) generation: %u, min_generation: %u\\n\",\n\t  oid_to_hex(&commit->object.oid), generation, min_generation);\n  if (generation > min_generation) {\n\t  return ret;\n  }\n\n\nI made some changes locally to display the generation numbers of all\nthe commits that were collected during the \"reflog\" traversal.\n\nIn the beginning we get the minimum generation number of the list of\nthe commits in \"reference[]\" to see if the \"commit\" is reachable from\none of the items in \"reference[]\". Since in this case, the generation\nnumber of \"commit\" is higher than minimum amongst the members of\n\"reference\", does it mean it cannot be reached?\n\nWhen \"GIT_TEST_COMMIT_GRAPH\" is turned off, all of the commits in\n\"reference\" as well as \"commit\" have \"GENERATION_NUMBER_INFINITY\",\nand the path moves forward to \"paint_down_to_common()\" and correctly\nidentifies the reachability.\n\nI ran the test again, but this time with running \"git-commit-graph\"\nright before pushing:\n\n  (a) git commit-graph write --reachable, and the commit's generation\n      number was \"GENERATION_NUMBER_INFINITY\".\n\n  (b) git-show-ref -s | git commit-graph write --stdin-commits, and\n      the commit's generation number was 5.\n\nand since generation number of \"commit\" was always higher than the\nminimum -- it returns that it is not reachable from any of \"reference\".\n\nOne of the failing tests in t5533 was modified (for debugging), and\nthe following shows the difference in behavior of\n\"repo_in_merge_bases_many()\". The test checks if the remote tip is\nreachable from the reflog entries of the local branch after\n\"git pull --rebase\" is run.\n\n  $ git log --graph --oneline # (before \"pull --rebase\")\n   * be2465f J\n   * 157d38b C\n   * f9a2dac B\n   * 112d1ac A\n\n  $ git log --graph --oneline # (after \"pull --rebase\")\n   * 7c16010 J\n   * b995a30 G\n   * 00b6961 F\n   * 157d38b C\n   * f9a2dac B\n   * 112d1ac A\n\n  $ git reflog master\n  7c16010 master@{0}: pull --rebase origin master (finish): refs/heads/master onto b995a30227dd14b34b6f7728201aefac8cc12296\n  be2465f master@{1}: commit: J\n  157d38b master@{2}: commit: C\n  f9a2dac master@{3}: commit: B\n  112d1ac master@{4}: commit (initial): A\n\n  - (a) is run.\n    $ git push --force-if-includes --force-with-lease=master\n    [7c16010bad84d8b53873875c2e242890920360f2]: (local):  generation: 4294967295, min_generation: 4294967295\n    [be2465f6a155acb8f5eb9ee876bad730e0f656cf]: (local):  generation: 4, min_generation: 4\n    [157d38b4112d457e6645c7c4e9a486e6189be435]: (local):  generation: 3, min_generation: 3\n    [f9a2dac17e4f8cafaa9655d40cb86c53094da8d6]: (local):  generation: 2, min_generation: 2\n    [112d1ac551b908f10b995d7e41456f4cd8f071c5]: (local):  generation: 1, min_generation: 1\n    [b995a30227dd14b34b6f7728201aefac8cc12296]: (remote): generation: 4294967295, min_generation: 1\n    [...] fail.\n\n  - \"git fetch --all\" and (b) is run.\n    $ git push --force-if-includes --force-with-lease=master\n    [7c16010bad84d8b53873875c2e242890920360f2]: (local):  generation: 4294967295, min_generation: 4294967295\n    [be2465f6a155acb8f5eb9ee876bad730e0f656cf]: (local):  generation: 4, min_generation: 4\n    [157d38b4112d457e6645c7c4e9a486e6189be435]: (local):  generation: 3, min_generation: 3\n    [f9a2dac17e4f8cafaa9655d40cb86c53094da8d6]: (local):  generation: 2, min_generation: 2\n    [112d1ac551b908f10b995d7e41456f4cd8f071c5]: (local):  generation: 1, min_generation: 1\n    [b995a30227dd14b34b6f7728201aefac8cc12296]: (remote): generation: 5, min_generation: 1\n    [...] fail.\n\n  - neither (a), nor (b) are run, and \"core.commitGraph\" is disabled.\n    $ git push --force-if-includes --force-with-lease=master\n    [7c16010bad84d8b53873875c2e242890920360f2]: (local):  generation: 4294967295, min_generation: 4294967295\n    [be2465f6a155acb8f5eb9ee876bad730e0f656cf]: (local):  generation: 4294967295, min_generation: 4294967295\n    [157d38b4112d457e6645c7c4e9a486e6189be435]: (local):  generation: 4294967295, min_generation: 4294967295\n    [f9a2dac17e4f8cafaa9655d40cb86c53094da8d6]: (local):  generation: 4294967295, min_generation: 4294967295\n    [112d1ac551b908f10b995d7e41456f4cd8f071c5]: (local):  generation: 4294967295, min_generation: 4294967295\n    [b995a30227dd14b34b6f7728201aefac8cc12296]: (remote): generation: 4294967295, min_generation: 4294967295\n    [...] pass.\n\nIt looks like G (b995a30) is reachable from J (7c16010), but\n\"repo_in_merge_bases_many()\" disagrees when \"GIT_TEST_COMMIT_GRAPH\"\nis enabled. I hope this information helps a little to understand\nwhy this happens and whether it is a bug or not.\n\n[1] https://public-inbox.org/git/20200928193400.GA88208@mail.clickyotomy.dev\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"406787","messageId":"xmqqtuvdbxvu.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20201001175443.GA28444@mail.clickyotomy.dev","subject":"Re: [PATCH v9 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-01T18:32:21Z","receivedAt":"2020-10-01T18:32:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Ah, this will keep a record of the behavior in the tests; nice,\n> will update in the next set.\n\nI actually think this kind of \"sweep the breakage under the rug\"\nbelongs not in the code that uses the (suspected broken) API but\nin the code that is suspected to be broken, i.e. somewhere in\ncommit.c that is involved in in_merge_bases_many() codepath.\n\nSo if you are going to reroll, I would rather see you drop this part\nentirely, and add a similar \"this code is broken so we won't run in\nthe real user's environment\" to protect all code that call\nin_merge_bases_many() from the breakage, not just your new one.\n\n> One of the failing tests in t5533 was modified (for debugging), and\n> the following shows the difference in behavior of\n> \"repo_in_merge_bases_many()\". The test checks if the remote tip is\n> reachable from the reflog entries of the local branch after\n> \"git pull --rebase\" is run.\n> ...\n> It looks like G (b995a30) is reachable from J (7c16010), but\n> \"repo_in_merge_bases_many()\" disagrees when \"GIT_TEST_COMMIT_GRAPH\"\n> is enabled. I hope this information helps a little to understand\n> why this happens and whether it is a bug or not.\n\nWonderful.  With this good start, perhaps we may not have to discuss\ndisabling the commit-graph code in your code or in commit.c for all\ncallers ;-)\n\n"},{"id":"406817","messageId":"nycvar.QRO.7.76.6.2010021550170.50@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"20201001082118.19441-2-shrinidhi.kaushik@gmail.com","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-02T13:52:43Z","receivedAt":"2020-10-02T13:52:54Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Srinidhi,\n\n\nOn Thu, 1 Oct 2020, Srinidhi Kaushik wrote:\n\n> Add a check to verify if the remote-tracking ref of the local branch\n> is reachable from one of its \"reflog\" entries.\n>\n> The check iterates through the local ref's reflog to see if there\n> is an entry for the remote-tracking ref and collecting any commits\n> that are seen, into a list; the iteration stops if an entry in the\n> reflog matches the remote ref or if the entry timestamp is older\n> the latest entry of the remote ref's \"reflog\". If there wasn't an\n> entry found for the remote ref, \"in_merge_bases_many()\" is called\n> to check if it is reachable from the list of collected commits.\n>\n> When a local branch that is based on a remote ref, has been rewound\n> and is to be force pushed on the remote, \"--force-if-includes\" runs\n> a check that ensures any updates to the remote-tracking ref that may\n> have happened (by push from another repository) in-between the time\n> of the last update to the local branch (via \"git-pull\", for instance)\n> and right before the time of push, have been integrated locally\n> before allowing a forced update.\n>\n> If the new option is passed without specifying \"--force-with-lease\",\n> or specified along with \"--force-with-lease=<refname>:<expect>\" it\n> is a \"no-op\".\n>\n> Calls to \"in_merge_bases_many()\" return different results depending\n> on whether the \"commit-graph\" feature is enabled or not -- it is\n> temporarily disabled when the check runs [1].\n>\n> [1] https://lore.kernel.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com\n\nI can verify that the multiple calls to `in_merge_bases_many()` lead to a\nproblem, and I intend to debug this further, but it is the wrong function\nto call to begin with.\n\nWith these two patches, the tests pass for me, and they also reduce the\ncomplexity quite a bit (Junio, could I ask you to put them on top of\nsk/force-if-includes?):\n\n-- snipsnap --\nFrom 0e7bd31c4cb0ae08ad772ac230eea2dd7a884886 Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin <johannes.schindelin@gmx.de>\nDate: Fri, 2 Oct 2020 15:33:05 +0200\nSubject: [PATCH 1/2] fixup??? push: add reflog check for \"--force-if-includes\"\n\nThis follows the pattern used elsewhere.\n\nMaybe we should also rename this to `commit_array`? It is not a linked\nlist, after all.\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n remote.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/remote.c b/remote.c\nindex 37533cafc44..2c6c63aa906 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -2441,6 +2441,7 @@ struct reflog_commit_list {\n \tstruct commit **item;\n \tsize_t nr, alloc;\n };\n+#define REFLOG_COMMIT_LIST_INIT { NULL, 0, 0 }\n\n /* Append a commit to the list. */\n static void append_commit(struct reflog_commit_list *list,\n@@ -2514,7 +2515,7 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n \tstruct commit *commit;\n \tstruct commit **chunk;\n \tstruct check_and_collect_until_cb_data cb;\n-\tstruct reflog_commit_list list = { NULL, 0, 0 };\n+\tstruct reflog_commit_list list = REFLOG_COMMIT_LIST_INIT;\n \tsize_t size = 0;\n \tint ret = 0;\n\n--\n2.28.0.windows.1.18.g5300e52e185\n\n\nFrom 10ea5640015f4bc7144e8e5b025e31294329c600 Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin <johannes.schindelin@gmx.de>\nDate: Fri, 2 Oct 2020 15:35:58 +0200\nSubject: [PATCH 2/2] fixup??? push: add reflog check for \"--force-if-includes\"\n\nWe should not call `in_merge_bases_many()` repeatedly: there is a much\nbetter API for that: `get_reachable_subset()`.\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n remote.c | 43 ++++---------------------------------------\n 1 file changed, 4 insertions(+), 39 deletions(-)\n\ndiff --git a/remote.c b/remote.c\nindex 2c6c63aa906..881415921e2 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -2513,10 +2513,9 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n {\n \ttimestamp_t date;\n \tstruct commit *commit;\n-\tstruct commit **chunk;\n \tstruct check_and_collect_until_cb_data cb;\n \tstruct reflog_commit_list list = REFLOG_COMMIT_LIST_INIT;\n-\tsize_t size = 0;\n+\tstruct commit_list *reachable;\n \tint ret = 0;\n\n \tcommit = lookup_commit_reference(the_repository, &remote->old_oid);\n@@ -2542,61 +2541,27 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n \t * Check if the remote commit is reachable from any\n \t * of the commits in the collected list, in batches.\n \t */\n-\tfor (chunk = list.item; chunk < list.item + list.nr; chunk += size) {\n-\t\tsize = list.item + list.nr - chunk;\n-\t\tif (MERGE_BASES_BATCH_SIZE < size)\n-\t\t\tsize = MERGE_BASES_BATCH_SIZE;\n-\n-\t\tif ((ret = in_merge_bases_many(commit, size, chunk)))\n-\t\t\tbreak;\n-\t}\n+\treachable = get_reachable_subset(list.item, list.nr, &commit, 1, 0);\n+\tret = !!reachable;\n+\tfree_commit_list(reachable);\n\n cleanup_return:\n \tfree_reflog_commit_list(&list);\n \treturn ret;\n }\n\n-/* Toggle the \"commit-graph\" feature; return the previously set state. */\n-static int toggle_commit_graph(struct repository *repo, int disable) {\n-\tint prev = repo->commit_graph_disabled;\n-\tstatic int should_toggle = -1;\n-\n-\tif (should_toggle < 0) {\n-\t\t/*\n-\t\t * The in_merge_bases_many() seems to misbehave when\n-\t\t * the commit-graph feature is in use.  Disable it for\n-\t\t * normal users, but keep it enabled when specifically\n-\t\t * testing the feature.\n-\t\t */\n-\t\tshould_toggle = !git_env_bool(\"GIT_TEST_COMMIT_GRAPH\", 0);\n-\t}\n-\n-\tif (should_toggle)\n-\t\trepo->commit_graph_disabled = disable;\n-\treturn prev;\n-}\n-\n /*\n  * Check for reachability of a remote-tracking\n  * ref in the reflog entries of its local ref.\n  */\n static void check_if_includes_upstream(struct ref *remote)\n {\n-\tint prev;\n \tstruct ref *local = get_local_ref(remote->name);\n \tif (!local)\n \t\treturn;\n\n-\t/*\n-\t * TODO: Remove \"toggle_commit_graph()\" calls around the check.\n-\t * Depending on whether \"commit-graph\" enabled or not,\n-\t * \"in_merge_bases_many()\" returns different results;\n-\t * disable it temporarily when the check runs.\n-\t */\n-\tprev = toggle_commit_graph(the_repository, 1);\n \tif (is_reachable_in_reflog(local->name, remote) <= 0)\n \t\tremote->unreachable = 1;\n-\ttoggle_commit_graph(the_repository, prev);\n }\n\n static void apply_cas(struct push_cas_option *cas,\n--\n2.28.0.windows.1.18.g5300e52e185\n\n"},{"id":"406820","messageId":"nycvar.QRO.7.76.6.2010021642210.50@tvgsbejvaqbjf.bet","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2010021550170.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-02T14:50:01Z","receivedAt":"2020-10-02T14:50:21Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 2 Oct 2020, Johannes Schindelin wrote:\n\n>\n> On Thu, 1 Oct 2020, Srinidhi Kaushik wrote:\n>\n> > Add a check to verify if the remote-tracking ref of the local branch\n> > is reachable from one of its \"reflog\" entries.\n> >\n> > The check iterates through the local ref's reflog to see if there\n> > is an entry for the remote-tracking ref and collecting any commits\n> > that are seen, into a list; the iteration stops if an entry in the\n> > reflog matches the remote ref or if the entry timestamp is older\n> > the latest entry of the remote ref's \"reflog\". If there wasn't an\n> > entry found for the remote ref, \"in_merge_bases_many()\" is called\n> > to check if it is reachable from the list of collected commits.\n> >\n> > When a local branch that is based on a remote ref, has been rewound\n> > and is to be force pushed on the remote, \"--force-if-includes\" runs\n> > a check that ensures any updates to the remote-tracking ref that may\n> > have happened (by push from another repository) in-between the time\n> > of the last update to the local branch (via \"git-pull\", for instance)\n> > and right before the time of push, have been integrated locally\n> > before allowing a forced update.\n> >\n> > If the new option is passed without specifying \"--force-with-lease\",\n> > or specified along with \"--force-with-lease=<refname>:<expect>\" it\n> > is a \"no-op\".\n> >\n> > Calls to \"in_merge_bases_many()\" return different results depending\n> > on whether the \"commit-graph\" feature is enabled or not -- it is\n> > temporarily disabled when the check runs [1].\n> >\n> > [1] https://lore.kernel.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com\n>\n> I can verify that the multiple calls to `in_merge_bases_many()` lead to a\n> problem, and I intend to debug this further, but it is the wrong function\n> to call to begin with.\n\nIt was actually Stolee who figured this out: the shortcut at the beginning\nof `in_merge_bases_many()` which tries to exit early when the generation\nnumber of the `commit` indicates that it cannot be reached from the\n`reference` commits (because their generation numbers are smaller) has a\nbug in the logic. Obviously, the generation numbers are only used when\ncommit-graph is used, therefore things broke only in the `linux-gcc` job.\n\nStolee will send out a patch shortly.\n\nHaving said that, the change I suggested (to use `get_reachable_subset()`\ninstead of repeated `in_merge_bases_many()`) is _still_ the right thing to\ndo: we are not actually interested in the merge bases at all, but in\nreachability, and in the future there might be more efficient ways to\ndetermine that than painting down all the way to merge bases.\n\nThanks,\nDscho\n\n"},{"id":"406822","messageId":"20201002150710.GA54370@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2010021550170.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-02T15:07:10Z","receivedAt":"2020-10-02T15:07:27Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Johannes,\n\nOn 10/02/2020 15:52, Johannes Schindelin wrote:\n> Hi Srinidhi,\n> \n> \n> On Thu, 1 Oct 2020, Srinidhi Kaushik wrote:\n> \n> > Add a check to verify if the remote-tracking ref of the local branch\n> > is reachable from one of its \"reflog\" entries.\n> >\n> > The check iterates through the local ref's reflog to see if there\n> > is an entry for the remote-tracking ref and collecting any commits\n> > that are seen, into a list; the iteration stops if an entry in the\n> > reflog matches the remote ref or if the entry timestamp is older\n> > the latest entry of the remote ref's \"reflog\". If there wasn't an\n> > entry found for the remote ref, \"in_merge_bases_many()\" is called\n> > to check if it is reachable from the list of collected commits.\n> >\n> > When a local branch that is based on a remote ref, has been rewound\n> > and is to be force pushed on the remote, \"--force-if-includes\" runs\n> > a check that ensures any updates to the remote-tracking ref that may\n> > have happened (by push from another repository) in-between the time\n> > of the last update to the local branch (via \"git-pull\", for instance)\n> > and right before the time of push, have been integrated locally\n> > before allowing a forced update.\n> >\n> > If the new option is passed without specifying \"--force-with-lease\",\n> > or specified along with \"--force-with-lease=<refname>:<expect>\" it\n> > is a \"no-op\".\n> >\n> > Calls to \"in_merge_bases_many()\" return different results depending\n> > on whether the \"commit-graph\" feature is enabled or not -- it is\n> > temporarily disabled when the check runs [1].\n> >\n> > [1] https://lore.kernel.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com\n> \n> I can verify that the multiple calls to `in_merge_bases_many()` lead to a\n> problem, and I intend to debug this further, but it is the wrong function\n> to call to begin with.\n> \n> With these two patches, the tests pass for me, and they also reduce the\n> complexity quite a bit (Junio, could I ask you to put them on top of\n> sk/force-if-includes?):\n\nThanks for looking into this. :)\n \n> -- snipsnap --\n> From 0e7bd31c4cb0ae08ad772ac230eea2dd7a884886 Mon Sep 17 00:00:00 2001\n> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> Date: Fri, 2 Oct 2020 15:33:05 +0200\n> Subject: [PATCH 1/2] fixup??? push: add reflog check for \"--force-if-includes\"\n> \n> This follows the pattern used elsewhere.\n> \n> Maybe we should also rename this to `commit_array`? It is not a linked\n> list, after all.\n\nMakes sense. I'll change it.\n \n> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n> ---\n>  remote.c | 3 ++-\n>  1 file changed, 2 insertions(+), 1 deletion(-)\n> \n> diff --git a/remote.c b/remote.c\n> index 37533cafc44..2c6c63aa906 100644\n> --- a/remote.c\n> +++ b/remote.c\n> @@ -2441,6 +2441,7 @@ struct reflog_commit_list {\n>  \tstruct commit **item;\n>  \tsize_t nr, alloc;\n>  };\n> +#define REFLOG_COMMIT_LIST_INIT { NULL, 0, 0 }\n> \n>  /* Append a commit to the list. */\n>  static void append_commit(struct reflog_commit_list *list,\n> @@ -2514,7 +2515,7 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n>  \tstruct commit *commit;\n>  \tstruct commit **chunk;\n>  \tstruct check_and_collect_until_cb_data cb;\n> -\tstruct reflog_commit_list list = { NULL, 0, 0 };\n> +\tstruct reflog_commit_list list = REFLOG_COMMIT_LIST_INIT;\n>  \tsize_t size = 0;\n>  \tint ret = 0;\n> \n> --\n> 2.28.0.windows.1.18.g5300e52e185\n> \n> \n> From 10ea5640015f4bc7144e8e5b025e31294329c600 Mon Sep 17 00:00:00 2001\n> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> Date: Fri, 2 Oct 2020 15:35:58 +0200\n> Subject: [PATCH 2/2] fixup??? push: add reflog check for \"--force-if-includes\"\n> \n> We should not call `in_merge_bases_many()` repeatedly: there is a much\n> better API for that: `get_reachable_subset()`.\n\nPerfect. I wasn't aware of this.\n\n> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n> ---\n>  remote.c | 43 ++++---------------------------------------\n>  1 file changed, 4 insertions(+), 39 deletions(-)\n> \n> diff --git a/remote.c b/remote.c\n> index 2c6c63aa906..881415921e2 100644\n> --- a/remote.c\n> +++ b/remote.c\n> @@ -2513,10 +2513,9 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n>  {\n>  \ttimestamp_t date;\n>  \tstruct commit *commit;\n> -\tstruct commit **chunk;\n>  \tstruct check_and_collect_until_cb_data cb;\n>  \tstruct reflog_commit_list list = REFLOG_COMMIT_LIST_INIT;\n> -\tsize_t size = 0;\n> +\tstruct commit_list *reachable;\n>  \tint ret = 0;\n> \n>  \tcommit = lookup_commit_reference(the_repository, &remote->old_oid);\n> @@ -2542,61 +2541,27 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n>  \t * Check if the remote commit is reachable from any\n>  \t * of the commits in the collected list, in batches.\n>  \t */\n> -\tfor (chunk = list.item; chunk < list.item + list.nr; chunk += size) {\n> -\t\tsize = list.item + list.nr - chunk;\n> -\t\tif (MERGE_BASES_BATCH_SIZE < size)\n> -\t\t\tsize = MERGE_BASES_BATCH_SIZE;\n> -\n> -\t\tif ((ret = in_merge_bases_many(commit, size, chunk)))\n> -\t\t\tbreak;\n> -\t}\n> +\treachable = get_reachable_subset(list.item, list.nr, &commit, 1, 0);\n> +\tret = !!reachable;\n> +\tfree_commit_list(reachable);\n> \n>  cleanup_return:\n>  \tfree_reflog_commit_list(&list);\n>  \treturn ret;\n>  }\n> \n> -/* Toggle the \"commit-graph\" feature; return the previously set state. */\n> -static int toggle_commit_graph(struct repository *repo, int disable) {\n> -\tint prev = repo->commit_graph_disabled;\n> -\tstatic int should_toggle = -1;\n> -\n> -\tif (should_toggle < 0) {\n> -\t\t/*\n> -\t\t * The in_merge_bases_many() seems to misbehave when\n> -\t\t * the commit-graph feature is in use.  Disable it for\n> -\t\t * normal users, but keep it enabled when specifically\n> -\t\t * testing the feature.\n> -\t\t */\n> -\t\tshould_toggle = !git_env_bool(\"GIT_TEST_COMMIT_GRAPH\", 0);\n> -\t}\n> -\n> -\tif (should_toggle)\n> -\t\trepo->commit_graph_disabled = disable;\n> -\treturn prev;\n> -}\n> -\n\nOK. The tests are passing with or without \"GIT_TEST_COMMIT_GRAPH\"\nby switching to \"get_reachable_subset()\" we don't have to toggle\nte feature during the check.\n\n>  /*\n>   * Check for reachability of a remote-tracking\n>   * ref in the reflog entries of its local ref.\n>   */\n>  static void check_if_includes_upstream(struct ref *remote)\n>  {\n> -\tint prev;\n>  \tstruct ref *local = get_local_ref(remote->name);\n>  \tif (!local)\n>  \t\treturn;\n> \n> -\t/*\n> -\t * TODO: Remove \"toggle_commit_graph()\" calls around the check.\n> -\t * Depending on whether \"commit-graph\" enabled or not,\n> -\t * \"in_merge_bases_many()\" returns different results;\n> -\t * disable it temporarily when the check runs.\n> -\t */\n> -\tprev = toggle_commit_graph(the_repository, 1);\n>  \tif (is_reachable_in_reflog(local->name, remote) <= 0)\n>  \t\tremote->unreachable = 1;\n> -\ttoggle_commit_graph(the_repository, prev);\n>  }\n> \n>  static void apply_cas(struct push_cas_option *cas,\n> --\n> 2.28.0.windows.1.18.g5300e52e185\n> \n\nAgain, thank you so much working on this! If you'd like, I can go ahead\nand apply these patches and rename \"reflog_commit_list\" to \"commit_array\"\nin the next series (v10).\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"406824","messageId":"xmqqy2koa97t.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2010021642210.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-02T16:22:46Z","receivedAt":"2020-10-02T16:22:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Having said that, the change I suggested (to use `get_reachable_subset()`\n> instead of repeated `in_merge_bases_many()`) is _still_ the right thing to\n> do: we are not actually interested in the merge bases at all, but in\n> reachability, and in the future there might be more efficient ways to\n> determine that than painting down all the way to merge bases.\n\nI agree with you that the age-old implementation has an obvious room\nfor optimization.  I think I already pointed out a #leftoverbit that\nwe can invent a version of paint_down_to_common() that can\nshort-circuit and return immediately after one side (the \"commit\"\nside) gets painted, so that in_merge_bases_many() can stop\nimmediately after finding out that the answer is \"true\".\n\nThe function is *not* about computing the merge base across the\ncommits on the \"reference\" side but finding out if \"commit\" is\nreachable from any in the \"reference\" side, so (1) it has a wrong\nname and more importantly (2) it wants to do something quite similar\nto get_reachable_subset(), but it is much less ambitious.\n\nget_reachable_subset() is capable of doing a lot more.  Unlike the\nolder in_merge_bases_many() that allowed only one commit on the\ncandidate for an ancestor side, it can throw a set and ask \"which\nones among these are reachable from the other set\".\n\nSo from the \"semantics\" point of view, get_reachable_subset() is\noverkill and less suitable than in_merge_bases_many() for this\nparticular application.  We know we have only one candidate, and we\nwant to ask \"is this reachable, or not?\" a single bit question.  In\nany case, they should yield the right answer from correctness point\nof view ;-)\n\nHaving said that.\n\nI do not think in the longer term we should keep both.  Clearly the\nget_reachable_subset() function can handle more general cases, so it\nwould make a lot of sense to make in_merge_bases_many() into a thin\nwrapper that feeds just a single commit array on one side to be\nfiltered while feeding the \"reference\" commits to the other side, as\nlong as we can demonstrate that the result is just as correct as,\nand it is not slower than, the current implementation.  That may be\na bit larger than a typical #leftoverbit but would be a good clean-up\nproject.\n\n"},{"id":"406825","messageId":"xmqqtuvca91a.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"nycvar.QRO.7.76.6.2010021550170.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-02T16:26:41Z","receivedAt":"2020-10-02T16:26:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> I can verify that the multiple calls to `in_merge_bases_many()` lead to a\n> problem ...\n\nIs it because it forgets to clear the marks?  I do not think so.\nBesides the sample history demonstrated by the failing test were so\nsmall that it didn't need multiple calls, IIRC.  There was a trivial\nbug in the function when commit-graph was enabled and there is no\nreason to avoid calling the function multiple times, right?\n\nI just wanted to make sure that a \"in-merge-bases-many cannot be\ncalled twice\" myth does not get etched in the archive.\n\nThanks for solving the puzzle with Derrick, by the way.  Very much\nappreciated.  I was wondering if my CC'ing commit-graph folks were\nsomehow not reaching the intended recipients.\n\n\n"},{"id":"406828","messageId":"xmqqft6wa8dc.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20201002150710.GA54370@mail.clickyotomy.dev","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-02T16:41:03Z","receivedAt":"2020-10-02T16:41:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n>> We should not call `in_merge_bases_many()` repeatedly: there is a much\n>> better API for that: `get_reachable_subset()`.\n>\n> Perfect. I wasn't aware of this.\n\nThis is possibly a piece of misinformation.  in_merge_bases_many()\nis designed to be callable more than once.  get_reachable_subset()\nmay be an overkill as we only are interested in a single \"is this\none an ancestor of any of these?\", not \"which ones among these are\nancestors of the other set?\".\n\n> OK. The tests are passing with or without \"GIT_TEST_COMMIT_GRAPH\"\n> by switching to \"get_reachable_subset()\" we don't have to toggle\n> te feature during the check.\n\nCorrect.  Once the \"see if this one is reachable from any of these\"\nis fixed (either by correcting the broken in_merge_bases_many() or\nusing get_reachable_subset()), we can get rid of this hack.\n\n> Again, thank you so much working on this! If you'd like, I can go ahead\n> and apply these patches and rename \"reflog_commit_list\" to \"commit_array\"\n> in the next series (v10).\n\nI like the s/list/array/ change, but I do not think switching to\nget_reachable_subset() and having to receive a commit list only to\nfree the list is warranted.\n\nDerrick sent a fix to in_merge_bases_many() in the near-by thread.\n\nThanks.\n"},{"id":"406829","messageId":"xmqqblhka7wx.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20201001175443.GA28444@mail.clickyotomy.dev","subject":"Re: [PATCH v9 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-02T16:50:54Z","receivedAt":"2020-10-02T16:50:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> I didn't want want to cause a delay with this patch. Since the new\n> option was seemingly working without it,...\n\nIt is a good example to help other new contributors to understand an\nimportant point in how the development in common works, so let me\nsay this.\n\nI did very much wanted to keep the bug exposed at least to the test\nsuite.  Since the broken helper were designed to be used in many\nother places in the code, and we had a simple reproduction recipe in\nthis topic, using it as an opening to help debug and fix bugs in the\nbroken helper had higher priority than adding the \"--force-if-includes\"\nfeature.\n\nWe help the contributors who have been involved in the broken helper\nby delaying this topic a bit and leaving the reproduction readily\navailable to them, so that they help us who are working on a piece\nof code that wants to see the broken helper fixed.  \n\nThat way everybody benefits.\n\nIt's not like a corporate development where your interest lies in\nshipping your piece regardless of the work done by other teams,\nwhere it might serve you better by using the second best tool for\nthe task, to avoid the tool that ought to be best but does not work\nwell *and* you do not want to help the team that manages that best\ntool, even if helping them may benefit the whole organization.\n\nSo, let's play well together.  Yield a bit to help others and let\nothers also help you.\n\nThanks.\n"},{"id":"406833","messageId":"20201002193906.GA78209@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqft6wa8dc.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-02T19:39:06Z","receivedAt":"2020-10-02T19:39:16Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Junio,\n\nOn 10/02/2020 09:41, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> >> We should not call `in_merge_bases_many()` repeatedly: there is a much\n> >> better API for that: `get_reachable_subset()`.\n> >\n> > Perfect. I wasn't aware of this.\n> \n> This is possibly a piece of misinformation.  in_merge_bases_many()\n> is designed to be callable more than once.  get_reachable_subset()\n> may be an overkill as we only are interested in a single \"is this\n> one an ancestor of any of these?\", not \"which ones among these are\n> ancestors of the other set?\".\n\nNoted; even though \"get_reachable_subset()\" and \"in_merge_bases_many()\"\n(after the commit-graph fix) return the same result, I suppose the\nlatter was designed for this specific use-case.\n\n> > OK. The tests are passing with or without \"GIT_TEST_COMMIT_GRAPH\"\n> > by switching to \"get_reachable_subset()\" we don't have to toggle\n> > te feature during the check.\n> \n> Correct.  Once the \"see if this one is reachable from any of these\"\n> is fixed (either by correcting the broken in_merge_bases_many() or\n> using get_reachable_subset()), we can get rid of this hack.\n\nOK. Shall I update the next set by reverting the \"disable commit-graph\"\nchange, s/list/array/ and leaving the rest as is -- if we decide to go\nforward with \"in_merge_bases_many()\", that is?\n\n> > Again, thank you so much working on this! If you'd like, I can go ahead\n> > and apply these patches and rename \"reflog_commit_list\" to \"commit_array\"\n> > in the next series (v10).\n> \n> I like the s/list/array/ change, but I do not think switching to\n> get_reachable_subset() and having to receive a commit list only to\n> free the list is warranted.\n> \n> Derrick sent a fix to in_merge_bases_many() in the near-by thread.\n\nNice! Will take a look.\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"406834","messageId":"20201002194244.GB78209@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqblhka7wx.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v9 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-02T19:42:44Z","receivedAt":"2020-10-02T19:42:55Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hi Junio,\n\nOn 10/02/2020 09:50, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> \n> > I didn't want want to cause a delay with this patch. Since the new\n> > option was seemingly working without it,...\n> \n> It is a good example to help other new contributors to understand an\n> important point in how the development in common works, so let me\n> say this.\n> \n> I did very much wanted to keep the bug exposed at least to the test\n> suite.  Since the broken helper were designed to be used in many\n> other places in the code, and we had a simple reproduction recipe in\n> this topic, using it as an opening to help debug and fix bugs in the\n> broken helper had higher priority than adding the \"--force-if-includes\"\n> feature.\n> \n> We help the contributors who have been involved in the broken helper\n> by delaying this topic a bit and leaving the reproduction readily\n> available to them, so that they help us who are working on a piece\n> of code that wants to see the broken helper fixed.  \n> \n> That way everybody benefits.\n> \n> It's not like a corporate development where your interest lies in\n> shipping your piece regardless of the work done by other teams,\n> where it might serve you better by using the second best tool for\n> the task, to avoid the tool that ought to be best but does not work\n> well *and* you do not want to help the team that manages that best\n> tool, even if helping them may benefit the whole organization.\n> \n> So, let's play well together.  Yield a bit to help others and let\n> others also help you.\n> \n> Thanks.\n\nThank you for pointing this out. You're right; I should not have\nrushed to disabling the feature because it wasn't working with my\npatch instead of waiting for the issue to be investigated. This is\nvaluable advice, and I will keep this in mind when making future\ncontributions.\n\nThanks.\n-- \nSrinidhi Kaushik\n"},{"id":"406840","messageId":"xmqqmu148jxu.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20201002193906.GA78209@mail.clickyotomy.dev","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-02T20:14:05Z","receivedAt":"2020-10-02T20:14:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Hi Junio,\n>\n> On 10/02/2020 09:41, Junio C Hamano wrote:\n>> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n>> \n>> >> We should not call `in_merge_bases_many()` repeatedly: there is a much\n>> >> better API for that: `get_reachable_subset()`.\n>> >\n>> > Perfect. I wasn't aware of this.\n>> \n>> This is possibly a piece of misinformation.  in_merge_bases_many()\n>> is designed to be callable more than once.  get_reachable_subset()\n>> may be an overkill as we only are interested in a single \"is this\n>> one an ancestor of any of these?\", not \"which ones among these are\n>> ancestors of the other set?\".\n>\n> Noted; even though \"get_reachable_subset()\" and \"in_merge_bases_many()\"\n> (after the commit-graph fix) return the same result, I suppose the\n> latter was designed for this specific use-case.\n\nYes, in_merge_bases_many() was invented first in 4c4b27e8 (commit.c:\nadd in_merge_bases_many(), 2013-03-04) for this exact use case.  For\nuse cases where callers have multiple \"these may be ancestors\"\ncandidates, instead of having to iterate over them and calling\nin_merge_bases_many() multiple times, get_reachable_subset() was\nadded much later at fcb2c076 (commit-reach: implement\nget_reachable_subset, 2018-11-02).\n\n> OK. Shall I update the next set by reverting the \"disable commit-graph\"\n> change, s/list/array/ and leaving the rest as is -- if we decide to go\n> forward with \"in_merge_bases_many()\", that is?\n\nYes, that would be the ideal endgame.  What I pushed out to 'seen'\nhas the removal of \"disable\" bit as a SQUASH??? commit at the tip,\nbut not s/list/array renaming.\n\nThanks.\n"},{"id":"406842","messageId":"20201002205830.GD78209@mail.clickyotomy.dev","threadId":"54190","inReplyTo":"xmqqmu148jxu.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-02T20:58:30Z","receivedAt":"2020-10-02T20:58:38Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Hello,\n\nOn 10/02/2020 13:14, Junio C Hamano wrote:\n> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n> > Noted; even though \"get_reachable_subset()\" and \"in_merge_bases_many()\"\n> > (after the commit-graph fix) return the same result, I suppose the\n> > latter was designed for this specific use-case.\n> \n> Yes, in_merge_bases_many() was invented first in 4c4b27e8 (commit.c:\n> add in_merge_bases_many(), 2013-03-04) for this exact use case.  For\n> use cases where callers have multiple \"these may be ancestors\"\n> candidates, instead of having to iterate over them and calling\n> in_merge_bases_many() multiple times, get_reachable_subset() was\n> added much later at fcb2c076 (commit-reach: implement\n> get_reachable_subset, 2018-11-02).\n\nGot it. Thanks for the detailed explanation and reference.\n \n> > OK. Shall I update the next set by reverting the \"disable commit-graph\"\n> > change, s/list/array/ and leaving the rest as is -- if we decide to go\n> > forward with \"in_merge_bases_many()\", that is?\n> \n> Yes, that would be the ideal endgame.  What I pushed out to 'seen'\n> has the removal of \"disable\" bit as a SQUASH??? commit at the tip,\n> but not s/list/array renaming.\n> \n> Thanks.\n\nAlright, I will add those changes in the next set. Also, I saw in the\nother thread that you tested commit-graph fix on this series and the\ntests are passing -- thanks for checking.\n-- \nSrinidhi Kaushik\n"},{"id":"406845","messageId":"xmqqd0208g5a.fsf@gitster.c.googlers.com","threadId":"54190","inReplyTo":"20201002205830.GD78209@mail.clickyotomy.dev","subject":"Re: [PATCH v9 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-02T21:36:01Z","receivedAt":"2020-10-02T21:36:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:\n\n> Alright, I will add those changes in the next set. Also, I saw in the\n> other thread that you tested commit-graph fix on this series and the\n> tests are passing -- thanks for checking.\n\nWhat I pushed out on 'seen':\n\n - starts building sk/force-if-includes topic on 6c430a647cb99 (at\n   the tip of master sometime ago)\n\n - merges Derrick's fix ds/in-merge-bases-many-optim-bug topic\n\n - and then applies the previous three patches on top\n\n - and finally have the SQUASH??? fix-up.\n\nRebuilding your series on top of this commit\n\n    aed0800ca6 Merge branch 'ds/in-merge-bases-many-optim-bug' into sk/force-if-includes\n\nwould be the most appropriate.\n\nThanks.\n\n\n"},{"id":"406861","messageId":"20201003121046.60604-1-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20201001082118.19441-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v10 0/3] push: add \"--[no-]force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-03T12:10:43Z","receivedAt":"2020-10-03T12:10:56Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a new option: \"--force-if-includes\" to \"git-push\" where forced\nupdates are allowed only if the tip of the remote-tracking ref has\nbeen integrated locally, by verifying if the tip of the remote-tracking\nref -- on which a local branch has based on -- is reachable from at\nleast one of the \"reflog\" entries of the branch about to be updated\nby force on the remote.\n\nThis option can be used with \"--force-with-lease\" with setups where\nthe remote-tracking refs of the repository are implicitly updated in\nthe background to help prevent unintended remote overwrites.\n\nIf a local branch is based on a remote ref for a rewrite, and if that\nremote-tracking ref is updated by a push from another repository after\nit has been checked out locally, force updating that branch to remote\nwith \"--force-with-lease[=<refname>[:<expect>]]\" without specifying\nthe \"<expect>\" value, can cause the update that happened in-between\nthe checkout and forced push to be lost.\n\nChanges since v9:\n  - Remove \"toggle_commit_graph()\" because \"in_merge_bases_many()\"\n    has been fixed [1].\n\n  - Rename \"reflog_commit_list\" to \"reflog_commit_array\", and related\n    comments.\n\nSrinidhi Kaushik (3):\n  push: add reflog check for \"--force-if-includes\"\n  push: parse and set flag for \"--force-if-includes\"\n  t, doc: update tests, reference for \"--force-if-includes\"\n\n Documentation/config/advice.txt |   9 +-\n Documentation/config/push.txt   |   6 ++\n Documentation/git-push.txt      |  26 ++++-\n advice.c                        |   3 +\n advice.h                        |   2 +\n builtin/push.c                  |  27 +++++\n builtin/send-pack.c             |  11 ++\n remote-curl.c                   |  14 ++-\n remote.c                        | 184 ++++++++++++++++++++++++++++++--\n remote.h                        |  12 ++-\n send-pack.c                     |   1 +\n t/t5533-push-cas.sh             | 137 ++++++++++++++++++++++++\n transport-helper.c              |  10 ++\n transport.c                     |   8 ++\n transport.h                     |  15 ++-\n 15 files changed, 447 insertions(+), 18 deletions(-)\n\nbase-commit: aed0800ca6 (Merge branch 'ds/in-merge-bases-many-optim-bug'\n\t\t\t into sk/force-if-includes, 2020-10-02)\n\n[1] https://public-inbox.org/git/pull.739.git.1601650736489.gitgitgadget@gmail.com\n--\n2.28.0\n"},{"id":"406862","messageId":"20201003121046.60604-3-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20201003121046.60604-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v10 2/3] push: parse and set flag for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-03T12:10:45Z","receivedAt":"2020-10-03T12:11:02Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"The previous commit added the necessary machinery to implement the\n\"--force-if-includes\" protection, when \"--force-with-lease\" is used\nwithout giving exact object the remote still ought to have. Surface\nthe feature by adding a command line option and a configuration\nvariable to enable it.\n\n - Add a flag: \"TRANSPORT_PUSH_FORCE_IF_INCLUDES\" to indicate that the\n   new option was passed from the command line of via configuration\n   settings; update command line and configuration parsers to set the\n   new flag accordingly.\n\n - Introduce a new configuration option \"push.useForceIfIncludes\", which\n   is equivalent to setting \"--force-if-includes\" in the command line.\n\n - Update \"remote-curl\" to recognize and pass this option to \"send-pack\"\n   when enabled.\n\n - Update \"advise\" to catch the reject reason \"REJECT_REF_NEEDS_UPDATE\",\n   set when the ref status is \"REF_STATUS_REJECT_REMOTE_UPDATED\" and\n   (optionally) print a help message when the push fails.\n\n - The new option is a \"no-op\" in the following scenarios:\n    * When used without \"--force-with-lease\".\n    * When used with \"--force-with-lease\", and if the expected commit\n      on the remote side is specified as an argument.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n advice.c            |  3 +++\n advice.h            |  2 ++\n builtin/push.c      | 27 +++++++++++++++++++++++++++\n builtin/send-pack.c |  6 ++++++\n remote-curl.c       | 14 +++++++++++++-\n transport-helper.c  |  5 +++++\n transport.c         |  2 ++\n transport.h         | 15 ++++++++++-----\n 8 files changed, 68 insertions(+), 6 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex f0a3d32d20..164742305f 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;\n int advice_push_fetch_first = 1;\n int advice_push_needs_force = 1;\n int advice_push_unqualified_ref_name = 1;\n+int advice_push_ref_needs_update = 1;\n int advice_status_hints = 1;\n int advice_status_u_option = 1;\n int advice_status_ahead_behind_warning = 1;\n@@ -72,6 +73,7 @@ static struct {\n \t{ \"pushFetchFirst\", &advice_push_fetch_first },\n \t{ \"pushNeedsForce\", &advice_push_needs_force },\n \t{ \"pushUnqualifiedRefName\", &advice_push_unqualified_ref_name },\n+\t{ \"pushRefNeedsUpdate\", &advice_push_ref_needs_update },\n \t{ \"statusHints\", &advice_status_hints },\n \t{ \"statusUoption\", &advice_status_u_option },\n \t{ \"statusAheadBehindWarning\", &advice_status_ahead_behind_warning },\n@@ -116,6 +118,7 @@ static struct {\n \t[ADVICE_PUSH_ALREADY_EXISTS]\t\t\t= { \"pushAlreadyExists\", 1 },\n \t[ADVICE_PUSH_FETCH_FIRST]\t\t\t= { \"pushFetchFirst\", 1 },\n \t[ADVICE_PUSH_NEEDS_FORCE]\t\t\t= { \"pushNeedsForce\", 1 },\n+\t[ADVICE_PUSH_REF_NEEDS_UPDATE]\t\t\t= { \"pushRefNeedsUpdate\", 1 },\n \n \t/* make this an alias for backward compatibility */\n \t[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]\t\t= { \"pushNonFastForward\", 1 },\ndiff --git a/advice.h b/advice.h\nindex 16f2c11642..bc2432980a 100644\n--- a/advice.h\n+++ b/advice.h\n@@ -11,6 +11,7 @@ extern int advice_push_already_exists;\n extern int advice_push_fetch_first;\n extern int advice_push_needs_force;\n extern int advice_push_unqualified_ref_name;\n+extern int advice_push_ref_needs_update;\n extern int advice_status_hints;\n extern int advice_status_u_option;\n extern int advice_status_ahead_behind_warning;\n@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;\n \tADVICE_PUSH_UNQUALIFIED_REF_NAME,\n \tADVICE_PUSH_UPDATE_REJECTED_ALIAS,\n \tADVICE_PUSH_UPDATE_REJECTED,\n+\tADVICE_PUSH_REF_NEEDS_UPDATE,\n \tADVICE_RESET_QUIET_WARNING,\n \tADVICE_RESOLVE_CONFLICT,\n \tADVICE_RM_HINTS,\ndiff --git a/builtin/push.c b/builtin/push.c\nindex 0eeb2c8dd5..908b557edb 100644\n--- a/builtin/push.c\n+++ b/builtin/push.c\n@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =\n \t   \"or update a remote ref to make it point at a non-commit object,\\n\"\n \t   \"without using the '--force' option.\\n\");\n \n+static const char message_advice_ref_needs_update[] =\n+\tN_(\"Updates were rejected because the tip of the remote-tracking\\n\"\n+\t   \"branch has been updated since the last checkout. You may want\\n\"\n+\t   \"to integrate those changes locally (e.g., 'git pull ...')\\n\"\n+\t   \"before forcing an update.\\n\");\n+\n static void advise_pull_before_push(void)\n {\n \tif (!advice_push_non_ff_current || !advice_push_update_rejected)\n@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)\n \tadvise(_(message_advice_ref_needs_force));\n }\n \n+static void advise_ref_needs_update(void)\n+{\n+\tif (!advice_push_ref_needs_update || !advice_push_update_rejected)\n+\t\treturn;\n+\tadvise(_(message_advice_ref_needs_update));\n+}\n+\n static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\t\t     int flags)\n {\n@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,\n \t\tadvise_ref_fetch_first();\n \t} else if (reject_reasons & REJECT_NEEDS_FORCE) {\n \t\tadvise_ref_needs_force();\n+\t} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {\n+\t\tadvise_ref_needs_update();\n \t}\n \n \treturn 1;\n@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)\n \t\tif (!v)\n \t\t\treturn config_error_nonbool(k);\n \t\treturn color_parse(v, push_colors[slot]);\n+\t} else if (!strcmp(k, \"push.useforceifincludes\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\t*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\telse\n+\t\t\t*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;\n+\t\treturn 0;\n \t}\n \n \treturn git_default_config(k, v, NULL);\n@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t\t       N_(\"require old value of ref to be at this value\"),\n \t\t\t       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),\n+\t\tOPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,\n+\t\t\tN_(\"require remote updates to be integrated locally\"),\n+\t\t\tTRANSPORT_PUSH_FORCE_IF_INCLUDES),\n \t\tOPT_CALLBACK(0, \"recurse-submodules\", &recurse_submodules, \"(check|on-demand|no)\",\n \t\t\t     N_(\"control recursive pushing of submodules\"), option_parse_recurse_submodules),\n \t\tOPT_BOOL_F( 0 , \"thin\", &thin, N_(\"use thin pack\"), PARSE_OPT_NOCOMPLETE),\n@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)\n \tif ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))\n \t\tdie(_(\"--all and --mirror are incompatible\"));\n \n+\tif (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))\n+\t\tcas.use_force_if_includes = 1;\n+\n \tfor_each_string_list_item(item, push_options)\n \t\tif (strchr(item->string, '\\n'))\n \t\t\tdie(_(\"push options must not have new line characters\"));\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 516cba7336..a7e01667b0 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tint force_if_includes = 0;\n \tstruct packet_reader reader;\n \n \tstruct option options[] = {\n@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_(\"<refname>:<expect>\"),\n \t\t  N_(\"require old value of ref to be at this value\"),\n \t\t  PARSE_OPT_OPTARG, parseopt_push_cas_option),\n+\t\tOPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,\n+\t\t\t N_(\"require remote updates to be integrated locally\")),\n \t\tOPT_END()\n \t};\n \n@@ -304,6 +307,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tif (!is_empty_cas(&cas))\n \t\tapply_push_cas(&cas, remote, remote_refs);\n \n+\tif (!is_empty_cas(&cas) && force_if_includes)\n+\t\tcas.use_force_if_includes = 1;\n+\n \tset_ref_status_for_push(remote_refs, args.send_mirror,\n \t\targs.force_update);\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 32cc4a0c55..0290b04891 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -44,7 +44,8 @@ struct options {\n \t\tfrom_promisor : 1,\n \n \t\tatomic : 1,\n-\t\tobject_format : 1;\n+\t\tobject_format : 1,\n+\t\tforce_if_includes : 1;\n \tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)\n \t\tstring_list_append(&cas_options, val.buf);\n \t\tstrbuf_release(&val);\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.force_if_includes = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.force_if_includes = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n \t} else if (!strcmp(name, \"cloning\")) {\n \t\tif (!strcmp(value, \"true\"))\n \t\t\toptions.cloning = 1;\n@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)\n \t\tstrvec_push(&args, cas_option->string);\n \tstrvec_push(&args, url.buf);\n \n+\tif (options.force_if_includes)\n+\t\tstrvec_push(&args, \"--force-if-includes\");\n+\n \tstrvec_push(&args, \"--stdin\");\n \tfor (i = 0; i < nr_spec; i++)\n \t\tpacket_buf_write(&preamble, \"%s\\n\", specs[i]);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 6157de30c7..5f6e0b3bd8 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,\n \t\tif (set_helper_option(transport, TRANS_OPT_ATOMIC, \"true\") != 0)\n \t\t\tdie(_(\"helper %s does not support --atomic\"), name);\n \n+\tif (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)\n+\t\tif (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, \"true\") != 0)\n+\t\t\tdie(_(\"helper %s does not support --%s\"),\n+\t\t\t    name, TRANS_OPT_FORCE_IF_INCLUDES);\n+\n \tif (flags & TRANSPORT_PUSH_OPTIONS) {\n \t\tstruct string_list_item *item;\n \t\tfor_each_string_list_item(item, transport->push_options)\ndiff --git a/transport.c b/transport.c\nindex 65fcd22b20..47da955e4f 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t\t*reject_reasons |= REJECT_FETCH_FIRST;\n \t\t} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {\n \t\t\t*reject_reasons |= REJECT_NEEDS_FORCE;\n+\t\t} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {\n+\t\t\t*reject_reasons |= REJECT_REF_NEEDS_UPDATE;\n \t\t}\n \t}\n \tfree(head);\ndiff --git a/transport.h b/transport.h\nindex ca409ea1e4..24558c027d 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -136,6 +136,7 @@ struct transport {\n #define TRANSPORT_PUSH_ATOMIC\t\t\t(1<<13)\n #define TRANSPORT_PUSH_OPTIONS\t\t\t(1<<14)\n #define TRANSPORT_RECURSE_SUBMODULES_ONLY\t(1<<15)\n+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES\t(1<<16)\n \n int transport_summary_width(const struct ref *refs);\n \n@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);\n /* Request atomic (all-or-nothing) updates when pushing */\n #define TRANS_OPT_ATOMIC \"atomic\"\n \n+/* Require remote changes to be integrated locally. */\n+#define TRANS_OPT_FORCE_IF_INCLUDES \"force-if-includes\"\n+\n /**\n  * Returns 0 if the option was used, non-zero otherwise. Prints a\n  * message to stderr if the option is not used.\n@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,\n void transport_set_verbosity(struct transport *transport, int verbosity,\n \tint force_progress);\n \n-#define REJECT_NON_FF_HEAD     0x01\n-#define REJECT_NON_FF_OTHER    0x02\n-#define REJECT_ALREADY_EXISTS  0x04\n-#define REJECT_FETCH_FIRST     0x08\n-#define REJECT_NEEDS_FORCE     0x10\n+#define REJECT_NON_FF_HEAD      0x01\n+#define REJECT_NON_FF_OTHER     0x02\n+#define REJECT_ALREADY_EXISTS   0x04\n+#define REJECT_FETCH_FIRST      0x08\n+#define REJECT_NEEDS_FORCE      0x10\n+#define REJECT_REF_NEEDS_UPDATE 0x20\n \n int transport_push(struct repository *repo,\n \t\t   struct transport *connection,\n-- \n2.28.0\n\n"},{"id":"406863","messageId":"20201003121046.60604-2-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20201003121046.60604-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v10 1/3] push: add reflog check for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-03T12:10:44Z","receivedAt":"2020-10-03T12:11:02Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Add a check to verify if the remote-tracking ref of the local branch\nis reachable from one of its \"reflog\" entries.\n\nThe check iterates through the local ref's reflog to see if there\nis an entry for the remote-tracking ref and collecting any commits\nthat are seen, into a list; the iteration stops if an entry in the\nreflog matches the remote ref or if the entry timestamp is older\nthe latest entry of the remote ref's \"reflog\". If there wasn't an\nentry found for the remote ref, \"in_merge_bases_many()\" is called\nto check if it is reachable from the list of collected commits.\n\nWhen a local branch that is based on a remote ref, has been rewound\nand is to be force pushed on the remote, \"--force-if-includes\" runs\na check that ensures any updates to the remote-tracking ref that may\nhave happened (by push from another repository) in-between the time\nof the last update to the local branch (via \"git-pull\", for instance)\nand right before the time of push, have been integrated locally\nbefore allowing a forced update.\n\nIf the new option is passed without specifying \"--force-with-lease\",\nor specified along with \"--force-with-lease=<refname>:<expect>\" it\nis a \"no-op\".\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n builtin/send-pack.c |   5 ++\n remote.c            | 184 ++++++++++++++++++++++++++++++++++++++++++--\n remote.h            |  12 ++-\n send-pack.c         |   1 +\n transport-helper.c  |   5 ++\n transport.c         |   6 ++\n 6 files changed, 205 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 7af148d733..516cba7336 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)\n \t\t\tmsg = \"stale info\";\n \t\t\tbreak;\n \n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"remote ref updated since checkout\";\n+\t\t\tbreak;\n+\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n \t\t\tres = \"error\";\n \t\t\tmsg = \"already exists\";\ndiff --git a/remote.c b/remote.c\nindex eafc14cbe7..10e0be9097 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,\n \t\t * with the remote-tracking branch to find the value\n \t\t * to expect, but we did not have such a tracking\n \t\t * branch.\n+\t\t *\n+\t\t * If the tip of the remote-tracking ref is unreachable\n+\t\t * from any reflog entry of its local ref indicating a\n+\t\t * possible update since checkout; reject the push.\n \t\t */\n \t\tif (ref->expect_old_sha1) {\n \t\t\tif (!oideq(&ref->old_oid, &ref->old_oid_expect))\n \t\t\t\treject_reason = REF_STATUS_REJECT_STALE;\n+\t\t\telse if (ref->check_reachable && ref->unreachable)\n+\t\t\t\treject_reason =\n+\t\t\t\t\tREF_STATUS_REJECT_REMOTE_UPDATED;\n \t\t\telse\n-\t\t\t\t/* If the ref isn't stale then force the update. */\n+\t\t\t\t/*\n+\t\t\t\t * If the ref isn't stale, and is reachable\n+\t\t\t\t * from from one of the reflog entries of\n+\t\t\t\t * the local branch, force the update.\n+\t\t\t\t */\n \t\t\t\tforce_ref_update = 1;\n \t\t}\n \n@@ -2251,12 +2262,13 @@ int is_empty_cas(const struct push_cas_option *cas)\n \n /*\n  * Look at remote.fetch refspec and see if we have a remote\n- * tracking branch for the refname there.  Fill its current\n- * value in sha1[].\n+ * tracking branch for the refname there. Fill the name of\n+ * the remote-tracking branch in *dst_refname, and the name\n+ * of the commit object at its tip in oid[].\n  * If we cannot do so, return negative to signal an error.\n  */\n static int remote_tracking(struct remote *remote, const char *refname,\n-\t\t\t   struct object_id *oid)\n+\t\t\t   struct object_id *oid, char **dst_refname)\n {\n \tchar *dst;\n \n@@ -2265,9 +2277,150 @@ static int remote_tracking(struct remote *remote, const char *refname,\n \t\treturn -1; /* no tracking ref for refname at remote */\n \tif (read_ref(dst, oid))\n \t\treturn -1; /* we know what the tracking ref is but we cannot read it */\n+\n+\t*dst_refname = dst;\n \treturn 0;\n }\n \n+/*\n+ * The struct \"reflog_commit_array\" and related helper functions\n+ * are used for collecting commits into an array during reflog\n+ * traversals in \"check_and_collect_until()\".\n+ */\n+struct reflog_commit_array {\n+\tstruct commit **item;\n+\tsize_t nr, alloc;\n+};\n+\n+#define REFLOG_COMMIT_ARRAY_INIT { NULL, 0, 0 }\n+\n+/* Append a commit to the array. */\n+static void append_commit(struct reflog_commit_array *arr,\n+\t\t\t  struct commit *commit)\n+{\n+\tALLOC_GROW(arr->item, arr->nr + 1, arr->alloc);\n+\tarr->item[arr->nr++] = commit;\n+}\n+\n+/* Free and reset the array. */\n+static void free_commit_array(struct reflog_commit_array *arr)\n+{\n+\tFREE_AND_NULL(arr->item);\n+\tarr->nr = arr->alloc = 0;\n+}\n+\n+struct check_and_collect_until_cb_data {\n+\tstruct commit *remote_commit;\n+\tstruct reflog_commit_array *local_commits;\n+\ttimestamp_t remote_reflog_timestamp;\n+};\n+\n+/* Get the timestamp of the latest entry. */\n+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,\n+\t\t       const char *ident, timestamp_t timestamp,\n+\t\t       int tz, const char *message, void *cb_data)\n+{\n+\ttimestamp_t *ts = cb_data;\n+\t*ts = timestamp;\n+\treturn 1;\n+}\n+\n+static int check_and_collect_until(struct object_id *o_oid,\n+\t\t\t\t   struct object_id *n_oid,\n+\t\t\t\t   const char *ident, timestamp_t timestamp,\n+\t\t\t\t   int tz, const char *message, void *cb_data)\n+{\n+\tstruct commit *commit;\n+\tstruct check_and_collect_until_cb_data *cb = cb_data;\n+\n+\t/* An entry was found. */\n+\tif (oideq(n_oid, &cb->remote_commit->object.oid))\n+\t\treturn 1;\n+\n+\tif ((commit = lookup_commit_reference(the_repository, n_oid)))\n+\t\tappend_commit(cb->local_commits, commit);\n+\n+\t/*\n+\t * If the reflog entry timestamp is older than the remote ref's\n+\t * latest reflog entry, there is no need to check or collect\n+\t * entries older than this one.\n+\t */\n+\tif (timestamp < cb->remote_reflog_timestamp)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+#define MERGE_BASES_BATCH_SIZE 8\n+\n+/*\n+ * Iterate through the reflog of the local ref to check if there is an entry\n+ * for the given remote-tracking ref; runs until the timestamp of an entry is\n+ * older than latest timestamp of remote-tracking ref's reflog. Any commits\n+ * are that seen along the way are collected into an array to check if the\n+ * remote-tracking ref is reachable from any of them.\n+ */\n+static int is_reachable_in_reflog(const char *local, const struct ref *remote)\n+{\n+\ttimestamp_t date;\n+\tstruct commit *commit;\n+\tstruct commit **chunk;\n+\tstruct check_and_collect_until_cb_data cb;\n+\tstruct reflog_commit_array arr = REFLOG_COMMIT_ARRAY_INIT;\n+\tsize_t size = 0;\n+\tint ret = 0;\n+\n+\tcommit = lookup_commit_reference(the_repository, &remote->old_oid);\n+\tif (!commit)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Get the timestamp from the latest entry\n+\t * of the remote-tracking ref's reflog.\n+\t */\n+\tfor_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);\n+\n+\tcb.remote_commit = commit;\n+\tcb.local_commits = &arr;\n+\tcb.remote_reflog_timestamp = date;\n+\tret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);\n+\n+\t/* We found an entry in the reflog. */\n+\tif (ret > 0)\n+\t\tgoto cleanup_return;\n+\n+\t/*\n+\t * Check if the remote commit is reachable from any\n+\t * of the commits in the collected array, in batches.\n+\t */\n+\tfor (chunk = arr.item; chunk < arr.item + arr.nr; chunk += size) {\n+\t\tsize = arr.item + arr.nr - chunk;\n+\t\tif (MERGE_BASES_BATCH_SIZE < size)\n+\t\t\tsize = MERGE_BASES_BATCH_SIZE;\n+\n+\t\tif ((ret = in_merge_bases_many(commit, size, chunk)))\n+\t\t\tbreak;\n+\t}\n+\n+cleanup_return:\n+\tfree_commit_array(&arr);\n+\treturn ret;\n+}\n+\n+/*\n+ * Check for reachability of a remote-tracking\n+ * ref in the reflog entries of its local ref.\n+ */\n+static void check_if_includes_upstream(struct ref *remote)\n+{\n+\tstruct ref *local = get_local_ref(remote->name);\n+\tif (!local)\n+\t\treturn;\n+\n+\tif (is_reachable_in_reflog(local->name, remote) <= 0)\n+\t\tremote->unreachable = 1;\n+}\n+\n static void apply_cas(struct push_cas_option *cas,\n \t\t      struct remote *remote,\n \t\t      struct ref *ref)\n@@ -2282,8 +2435,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\tref->expect_old_sha1 = 1;\n \t\tif (!entry->use_tracking)\n \t\t\toidcpy(&ref->old_oid_expect, &entry->expect);\n-\t\telse if (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\t\telse if (remote_tracking(remote, ref->name,\n+\t\t\t\t\t &ref->old_oid_expect,\n+\t\t\t\t\t &ref->tracking_ref))\n \t\t\toidclr(&ref->old_oid_expect);\n+\t\telse\n+\t\t\tref->check_reachable = cas->use_force_if_includes;\n \t\treturn;\n \t}\n \n@@ -2292,8 +2449,12 @@ static void apply_cas(struct push_cas_option *cas,\n \t\treturn;\n \n \tref->expect_old_sha1 = 1;\n-\tif (remote_tracking(remote, ref->name, &ref->old_oid_expect))\n+\tif (remote_tracking(remote, ref->name,\n+\t\t\t    &ref->old_oid_expect,\n+\t\t\t    &ref->tracking_ref))\n \t\toidclr(&ref->old_oid_expect);\n+\telse\n+\t\tref->check_reachable = cas->use_force_if_includes;\n }\n \n void apply_push_cas(struct push_cas_option *cas,\n@@ -2301,6 +2462,15 @@ void apply_push_cas(struct push_cas_option *cas,\n \t\t    struct ref *remote_refs)\n {\n \tstruct ref *ref;\n-\tfor (ref = remote_refs; ref; ref = ref->next)\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n \t\tapply_cas(cas, remote, ref);\n+\n+\t\t/*\n+\t\t * If \"compare-and-swap\" is in \"use_tracking[_for_rest]\"\n+\t\t * mode, and if \"--force-if-includes\" was specified, run\n+\t\t * the check.\n+\t\t */\n+\t\tif (ref->check_reachable)\n+\t\t\tcheck_if_includes_upstream(ref);\n+\t}\n }\ndiff --git a/remote.h b/remote.h\nindex eb62a47044..2d5391d281 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -107,12 +107,20 @@ struct ref {\n \tstruct object_id new_oid;\n \tstruct object_id old_oid_expect; /* used by expect-old */\n \tchar *symref;\n+\tchar *tracking_ref;\n \tunsigned int\n \t\tforce:1,\n \t\tforced_update:1,\n \t\texpect_old_sha1:1,\n \t\texact_oid:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\t/* Need to check if local reflog reaches the remote tip. */\n+\t\tcheck_reachable:1,\n+\t\t/*\n+\t\t * Store the result of the check enabled by \"check_reachable\";\n+\t\t * implies the local reflog does not reach the remote tip.\n+\t\t */\n+\t\tunreachable:1;\n \n \tenum {\n \t\tREF_NOT_MATCHED = 0, /* initial value */\n@@ -142,6 +150,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n \t\tREF_STATUS_REJECT_SHALLOW,\n+\t\tREF_STATUS_REJECT_REMOTE_UPDATED,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT,\n@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);\n \n struct push_cas_option {\n \tunsigned use_tracking_for_rest:1;\n+\tunsigned use_force_if_includes:1;\n \tstruct push_cas {\n \t\tstruct object_id expect;\n \t\tunsigned use_tracking:1;\ndiff --git a/send-pack.c b/send-pack.c\nindex 2d2f9997ac..b4d4a10a71 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar\n \tcase REF_STATUS_REJECT_FETCH_FIRST:\n \tcase REF_STATUS_REJECT_NEEDS_FORCE:\n \tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \tcase REF_STATUS_REJECT_NODELETE:\n \t\treturn CHECK_REF_STATUS_REJECTED;\n \tcase REF_STATUS_UPTODATE:\ndiff --git a/transport-helper.c b/transport-helper.c\nindex b573b6c730..6157de30c7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,\n \t\t\tstatus = REF_STATUS_REJECT_STALE;\n \t\t\tFREE_AND_NULL(msg);\n \t\t}\n+\t\telse if (!strcmp(msg, \"remote ref updated since checkout\")) {\n+\t\t\tstatus = REF_STATUS_REJECT_REMOTE_UPDATED;\n+\t\t\tFREE_AND_NULL(msg);\n+\t\t}\n \t\telse if (!strcmp(msg, \"forced update\")) {\n \t\t\tforced = 1;\n \t\t\tFREE_AND_NULL(msg);\n@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,\n \t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n \t\tcase REF_STATUS_REJECT_STALE:\n \t\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\t\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n \t\t\tif (atomic) {\n \t\t\t\treject_atomic_push(remote_refs, mirror);\n \t\t\t\tstring_list_clear(&cas_options, 0);\ndiff --git a/transport.c b/transport.c\nindex ffe2115845..65fcd22b20 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,\n \t\t\t\t \"stale info\",\n \t\t\t\t report, porcelain, summary_width);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t \"remote ref updated since checkout\",\n+\t\t\t\t report, porcelain, summary_width);\n+\t\tbreak;\n \tcase REF_STATUS_REJECT_SHALLOW:\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t \"new shallow roots not allowed\",\n@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,\n \t\tif (!r->peer_ref) continue;\n \t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n \t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n \t\tif (r->status == REF_STATUS_UPTODATE) continue;\n \n \t\tstrbuf_reset(&buf);\n-- \n2.28.0\n\n"},{"id":"406864","messageId":"20201003121046.60604-4-shrinidhi.kaushik@gmail.com","threadId":"54190","inReplyTo":"20201003121046.60604-1-shrinidhi.kaushik@gmail.com","subject":"[PATCH v10 3/3] t, doc: update tests, reference for \"--force-if-includes\"","fromName":"Srinidhi Kaushik","fromEmail":"shrinidhi.kaushik@gmail.com","sentAt":"2020-10-03T12:10:46Z","receivedAt":"2020-10-03T12:11:03Z","isPatch":true,"sender":{"key":"shrinidhi.kaushik@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3019537?v=4"},"body":"Update test cases for the new option, and document its usage\nand update related references.\n\nUpdate test cases for the new option, and document its usage\nand update related references.\n\n - t/t5533-push-cas.sh:\n   Update test cases for \"compare-and-swap\" when used along with\n   \"--force-if-includes\" helps mitigate overwrites when remote\n   refs are updated in the background; allows forced updates when\n   changes from remote are integrated locally.\n\n - Documentation:\n   Add reference for the new option, configuration setting\n   (\"push.useForceIfIncludes\") and advise messages.\n\nSigned-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>\n---\n Documentation/config/advice.txt |   9 ++-\n Documentation/config/push.txt   |   6 ++\n Documentation/git-push.txt      |  26 +++++-\n t/t5533-push-cas.sh             | 137 ++++++++++++++++++++++++++++++++\n 4 files changed, 174 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt\nindex bdd37c3eaa..acbd0c09aa 100644\n--- a/Documentation/config/advice.txt\n+++ b/Documentation/config/advice.txt\n@@ -10,9 +10,8 @@ advice.*::\n \t\tthat the check is disabled.\n \tpushUpdateRejected::\n \t\tSet this variable to 'false' if you want to disable\n-\t\t'pushNonFFCurrent',\n-\t\t'pushNonFFMatching', 'pushAlreadyExists',\n-\t\t'pushFetchFirst', and 'pushNeedsForce'\n+\t\t'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',\n+\t\t'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'\n \t\tsimultaneously.\n \tpushNonFFCurrent::\n \t\tAdvice shown when linkgit:git-push[1] fails due to a\n@@ -41,6 +40,10 @@ advice.*::\n \t\twe can still suggest that the user push to either\n \t\trefs/heads/* or refs/tags/* based on the type of the\n \t\tsource object.\n+\tpushRefNeedsUpdate::\n+\t\tShown when linkgit:git-push[1] rejects a forced update of\n+\t\ta branch when its remote-tracking ref has updates that we\n+\t\tdo not have locally.\n \tstatusAheadBehind::\n \t\tShown when linkgit:git-status[1] computes the ahead/behind\n \t\tcounts for a local ref compared to its remote tracking ref,\ndiff --git a/Documentation/config/push.txt b/Documentation/config/push.txt\nindex f5e5b38c68..21b256e0a4 100644\n--- a/Documentation/config/push.txt\n+++ b/Documentation/config/push.txt\n@@ -114,3 +114,9 @@ push.recurseSubmodules::\n \tspecifying '--recurse-submodules=check|on-demand|no'.\n \tIf not set, 'no' is used by default, unless 'submodule.recurse' is\n \tset (in which case a 'true' value means 'on-demand').\n+\n+push.useForceIfIncludes::\n+\tIf set to \"true\", it is equivalent to specifying\n+\t`--force-if-includes` as an option to linkgit:git-push[1]\n+\tin the command line. Adding `--no-force-if-includes` at the\n+\ttime of push overrides this configuration setting.\ndiff --git a/Documentation/git-push.txt b/Documentation/git-push.txt\nindex 3b8053447e..ab103c82cf 100644\n--- a/Documentation/git-push.txt\n+++ b/Documentation/git-push.txt\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]\n \t   [-u | --set-upstream] [-o <string> | --push-option=<string>]\n \t   [--[no-]signed|--signed=(true|false|if-asked)]\n-\t   [--force-with-lease[=<refname>[:<expect>]]]\n+\t   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]\n \t   [--no-verify] [<repository> [<refspec>...]]\n \n DESCRIPTION\n@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally\n force push changes to `master` if the remote version is still at\n `base`, regardless of what your local `remotes/origin/master` has been\n updated to in the background.\n++\n+Alternatively, specifying `--force-if-includes` as an ancillary option\n+along with `--force-with-lease[=<refname>]` (i.e., without saying what\n+exact commit the ref on the remote side must be pointing at, or which\n+refs on the remote side are being protected) at the time of \"push\" will\n+verify if updates from the remote-tracking refs that may have been\n+implicitly updated in the background are integrated locally before\n+allowing a forced update.\n \n -f::\n --force::\n@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push\n origin +master` to force a push to the `master` branch). See the\n `<refspec>...` section above for details.\n \n+--[no-]force-if-includes::\n+\tForce an update only if the tip of the remote-tracking ref\n+\thas been integrated locally.\n++\n+This option enables a check that verifies if the tip of the\n+remote-tracking ref is reachable from one of the \"reflog\" entries of\n+the local branch based in it for a rewrite. The check ensures that any\n+updates from the remote have been incorporated locally by rejecting the\n+forced update if that is not the case.\n++\n+If the option is passed without specifying `--force-with-lease`, or\n+specified along with `--force-with-lease=<refname>:<expect>`, it is\n+a \"no-op\".\n++\n+Specifying `--no-force-if-includes` disables this behavior.\n+\n --repo=<repository>::\n \tThis option is equivalent to the <repository> argument. If both\n \tare specified, the command-line argument takes precedence.\ndiff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh\nindex 0b0eb1d025..7813e8470e 100755\n--- a/t/t5533-push-cas.sh\n+++ b/t/t5533-push-cas.sh\n@@ -13,6 +13,46 @@ setup_srcdst_basic () {\n \t)\n }\n \n+# For tests with \"--force-if-includes\".\n+setup_src_dup_dst () {\n+\trm -fr src dup dst &&\n+\tgit init --bare dst &&\n+\tgit clone --no-local dst src &&\n+\tgit clone --no-local dst dup\n+\t(\n+\t\tcd src &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\ttest_commit C &&\n+\t\tgit push origin\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit fetch &&\n+\t\tgit merge origin/master &&\n+\t\tgit switch -c branch master~2 &&\n+\t\ttest_commit D &&\n+\t\ttest_commit E &&\n+\t\tgit push origin --all\n+\t) &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch master &&\n+\t\tgit fetch --all &&\n+\t\tgit branch branch --track origin/branch &&\n+\t\tgit rebase origin/master\n+\t) &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit switch master &&\n+\t\ttest_commit F &&\n+\t\ttest_commit G &&\n+\t\tgit switch branch &&\n+\t\ttest_commit H &&\n+\t\tgit push origin --all\n+\t)\n+}\n+\n test_expect_success setup '\n \t# create template repository\n \ttest_commit A &&\n@@ -256,4 +296,101 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up\n \t)\n '\n \n+test_expect_success 'background updates to remote can be mitigated with \"--force-if-includes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\tgit ls-remote dst refs/heads/branch >expect.branch &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-with-lease --force-if-includes --all\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\tgit ls-remote dst refs/heads/branch >actual.branch &&\n+\ttest_cmp expect.master actual.master &&\n+\ttest_cmp expect.branch actual.branch\n+'\n+\n+test_expect_success 'background updates to remote can be mitigated with \"push.useForceIfIncludes\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit fetch --all &&\n+\t\tgit config --local push.useForceIfIncludes true &&\n+\t\ttest_must_fail git push --force-with-lease=master origin master\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\ttest_cmp expect.master actual.master\n+'\n+\n+test_expect_success '\"--force-if-includes\" should be disabled for --force-with-lease=\"<refname>:<expect>\"' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\tgit ls-remote dst refs/heads/master >expect.master &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tremote_head=\"$(git rev-parse refs/remotes/origin/master)\" &&\n+\t\tgit fetch --all &&\n+\t\ttest_must_fail git push --force-if-includes --force-with-lease=\"master:$remote_head\" 2>err &&\n+\t\tgrep \"stale info\" err\n+\t) &&\n+\tgit ls-remote dst refs/heads/master >actual.master &&\n+\ttest_cmp expect.master actual.master\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\")' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow forced update after a rebase (\"pull --rebase\", local rebase)' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\ttest_commit I &&\n+\t\tgit switch master &&\n+\t\ttest_commit J &&\n+\t\tgit pull --rebase origin master &&\n+\t\tgit rebase --onto HEAD~4 HEAD~1 &&\n+\t\tgit push --force-if-includes --force-with-lease=\"master\"\n+\t)\n+'\n+\n+test_expect_success '\"--force-if-includes\" should allow deletes' '\n+\tsetup_src_dup_dst &&\n+\ttest_when_finished \"rm -fr dst src dup\" &&\n+\t(\n+\t\tcd src &&\n+\t\tgit switch branch &&\n+\t\tgit pull --rebase origin branch &&\n+\t\tgit push --force-if-includes --force-with-lease=\"branch\" origin :branch\n+\t)\n+'\n+\n test_done\n-- \n2.28.0\n\n"}]}