{"thread":{"id":"62615","subject":"[PATCH] bundle: remove unneeded code","startedAt":"2024-12-09T10:41:57Z","lastAt":"2024-12-10T02:11:09Z","messageCount":4,"participants":["Toon Claes","karthik nayak","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"508853","messageId":"20241209-fix-bundle-create-race-v1-1-e6513bdcbf8a@iotcl.com","threadId":"62615","inReplyTo":null,"subject":"[PATCH] bundle: remove unneeded code","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-12-09T10:41:29Z","receivedAt":"2024-12-09T10:41:57Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"The changes in commit c06793a4ed (allow git-bundle to create bottomless\nbundle, 2007-08-08) ensure annotated tags are properly preserved when\ncreating a bundle using a revision range operation.\n\nAt the time the range notation would peel the ends to their\ncorresponding commit, meaning ref v2.0 would point to the v2.0^0 commit.\nSo the above workaround was introduced. This code looks up the ref\nbefore it's written to the bundle, and if the ref doesn't point to the\nobject we expect (for tags this would be a tag object), we skip the ref\nfrom the bundle. Instead, when the ref is a tag that's the positive end\nof the range (e.g. v2.0 from the range \"v1.0..v2.0\"), then that ref is\nwritten to the bundle instead.\n\nLater, in 895c5ba3c1 (revision: do not peel tags used in range notation,\n2013-09-19), the behavior of parsing ranges was changed and the problem\nwas fixed at the cause. But the workaround in bundle.c was not reverted.\n\nNow it seems this workaround can cause a race condition. git-bundle(1)\nuses setup_revisions() to parse the input into `struct rev_info`. Later,\nin write_bundle_refs(), it uses this info to write refs to the bundle.\nAs mentioned at this point each ref is looked up again and checked\nwhether it points to the object we expect. If not, the ref is not\nwritten to the bundle. But, when creating a bundle in a heavy traffic\nrepository (a repo with many references, and frequent ref updates) it's\npossible a branch ref was updated between setup_revisions() and\nwrite_bundle_refs() and thus the extra check causes the ref to be\nskipped.\n\nThe workaround was originally added to deal with tags, but the code path\nalso gets hit by non-tag refs, causing this race condition. Because it's\nno longer needed, remove it and fix the possible race condition.\n\nSigned-off-by: Toon Claes <toon@iotcl.com>\n---\nEarlier I reported[1] a race condition bug could occur if a ref is\nupdated while a git-bundle(1) creation process is running. I've\ndemonstrated it's possible to reproduce this by using a debugger.\n\n> To reproduce, I've been running git-bundle(1) with\n> `create my.bndl --all --ignore-missing` in a debugger. I've set a\n> breakpoint at bundle.c:515[1] where setup_revisions() is called. After\n> stepping over this line I see in the debugger `revs.pending` is\n> populated.\n>\n>     (gdb) p *revs.pending.objects\n>     $6 = {item = 0x7a2fb0, name = 0x78d7e0 \"refs/heads/master\", path = 0x0, mode = 12288}\n>     (gdb) p *revs.pending.objects.item\n>     $7 = {parsed = 1, type = 1, flags = 0, oid = {hash = \"R\\026\\370\\365\\304\\b\\236\\302\\234\\344\\232\\372\\024t4\\302>\\017\\001c\\000\\000\\000\\000sS\\344\\367\\377\\177\\000\", algo = 1}}\n>\n> The hash value is the binary representation of\n> `5216f8f5c4089ec29ce49afa147434c23e0f0163`, the current HEAD of\n> `master`. At this point I've updated `master` in another terminal\n> window:\n>\n>     git commit --allow-empty -m\"dummy\"\n>\n> Then in the debugger I continue the process to create the bundle. The\n> resulting bundle seems to be missing `refs/heads/master`.\n\nI started digging into this bug again. And I've discovered the changes\nmade by c06793a4ed (allow git-bundle to create bottomless bundle,\n2007-08-08) cause some trouble. Those changes look up the refs for a\nsecond time, and if they mismatch, they might not get written to the\nbundle output file.\n\nIt seems the workaround is not needed anymore since\n895c5ba3c1 (revision: do not peel tags used in range notation,\n2013-09-19) and thus I'm removing it while adding some tests.\nUnfortunately, I was not able to implement a test that hits the race\ncondition bug.\n\n[1]: https://lore.kernel.org/git/87eddlpx5k.fsf@iotcl.com/\n---\n bundle.c               | 30 ------------------------------\n t/t6020-bundle-misc.sh | 34 ++++++++++++++++++++++++++++++++++\n 2 files changed, 34 insertions(+), 30 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 4773b51eb1df8057466c87f48445c49bc1f594ee..dfb5b7a5ec6b98e00078359afe991bac55cae739 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -420,36 +420,6 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \t\t\t\te->name);\n \t\t\tgoto skip_write_ref;\n \t\t}\n-\t\t/*\n-\t\t * If you run \"git bundle create bndl v1.0..v2.0\", the\n-\t\t * name of the positive ref is \"v2.0\" but that is the\n-\t\t * commit that is referenced by the tag, and not the tag\n-\t\t * itself.\n-\t\t */\n-\t\tif (!oideq(&oid, &e->item->oid)) {\n-\t\t\t/*\n-\t\t\t * Is this the positive end of a range expressed\n-\t\t\t * in terms of a tag (e.g. v2.0 from the range\n-\t\t\t * \"v1.0..v2.0\")?\n-\t\t\t */\n-\t\t\tstruct commit *one = lookup_commit_reference(revs->repo, &oid);\n-\t\t\tstruct object *obj;\n-\n-\t\t\tif (e->item == &(one->object)) {\n-\t\t\t\t/*\n-\t\t\t\t * Need to include e->name as an\n-\t\t\t\t * independent ref to the pack-objects\n-\t\t\t\t * input, so that the tag is included\n-\t\t\t\t * in the output; otherwise we would\n-\t\t\t\t * end up triggering \"empty bundle\"\n-\t\t\t\t * error.\n-\t\t\t\t */\n-\t\t\t\tobj = parse_object_or_die(&oid, e->name);\n-\t\t\t\tobj->flags |= SHOWN;\n-\t\t\t\tadd_pending_object(revs, obj, e->name);\n-\t\t\t}\n-\t\t\tgoto skip_write_ref;\n-\t\t}\n \n \t\tref_count++;\n \t\twrite_or_die(bundle_fd, oid_to_hex(&e->item->oid), the_hash_algo->hexsz);\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 5d444bfe201a330527e86dde7229721fc386fc93..f398a59424dcd025ce616cadcd7eece9be5301a3 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -504,6 +504,40 @@ test_expect_success 'unfiltered bundle with --objects' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'bottomless bundle upto tag' '\n+\tgit bundle create v2.bdl \\\n+\t\tv2 &&\n+\n+\tgit bundle verify v2.bdl |\n+\t\tmake_user_friendly_and_stable_output >actual &&\n+\n+\tformat_and_save_expect <<-EOF &&\n+\tThe bundle contains this ref:\n+\t<TAG-2> refs/tags/v2\n+\tThe bundle records a complete history.\n+\t$HASH_MESSAGE\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'bundle between two tags' '\n+\tgit bundle create v1-v2.bdl \\\n+\t\tv1..v2 &&\n+\n+\tgit bundle verify v1-v2.bdl |\n+\t\tmake_user_friendly_and_stable_output >actual &&\n+\n+\tformat_and_save_expect <<-EOF &&\n+\tThe bundle contains this ref:\n+\t<TAG-2> refs/tags/v2\n+\tThe bundle requires these 2 refs:\n+\t<COMMIT-E> Z\n+\t<COMMIT-B> Z\n+\t$HASH_MESSAGE\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n for filter in \"blob:none\" \"tree:0\" \"tree:1\" \"blob:limit=100\"\n do\n \ttest_expect_success \"filtered bundle: $filter\" '\n\n---\nbase-commit: e66fd72e972df760a53c3d6da023c17adfc426d6\nchange-id: 20241204-fix-bundle-create-race-d0bdca1bf999\n\nBest regards,\n-- \nToon Claes <toon@iotcl.com>\n\n"},{"id":"508866","messageId":"CAOLa=ZQpBf-ZE8BmXRUXJ247p44Enj0vrJv2KHVDXtGzY6SU8w@mail.gmail.com","threadId":"62615","inReplyTo":"20241209-fix-bundle-create-race-v1-1-e6513bdcbf8a@iotcl.com","subject":"Re: [PATCH] bundle: remove unneeded code","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-09T13:09:20Z","receivedAt":"2024-12-09T13:09:21Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Toon Claes <toon@iotcl.com> writes:\n\n> The changes in commit c06793a4ed (allow git-bundle to create bottomless\n> bundle, 2007-08-08) ensure annotated tags are properly preserved when\n> creating a bundle using a revision range operation.\n>\n> At the time the range notation would peel the ends to their\n> corresponding commit, meaning ref v2.0 would point to the v2.0^0 commit.\n> So the above workaround was introduced. This code looks up the ref\n> before it's written to the bundle, and if the ref doesn't point to the\n> object we expect (for tags this would be a tag object), we skip the ref\n> from the bundle. Instead, when the ref is a tag that's the positive end\n> of the range (e.g. v2.0 from the range \"v1.0..v2.0\"), then that ref is\n> written to the bundle instead.\n>\n> Later, in 895c5ba3c1 (revision: do not peel tags used in range notation,\n> 2013-09-19), the behavior of parsing ranges was changed and the problem\n> was fixed at the cause. But the workaround in bundle.c was not reverted.\n>\n\nInteresting to read the progression in these changes. Good digging.\n\n> Now it seems this workaround can cause a race condition. git-bundle(1)\n> uses setup_revisions() to parse the input into `struct rev_info`. Later,\n> in write_bundle_refs(), it uses this info to write refs to the bundle.\n> As mentioned at this point each ref is looked up again and checked\n> whether it points to the object we expect. If not, the ref is not\n> written to the bundle. But, when creating a bundle in a heavy traffic\n> repository (a repo with many references, and frequent ref updates) it's\n> possible a branch ref was updated between setup_revisions() and\n> write_bundle_refs() and thus the extra check causes the ref to be\n> skipped.\n>\n\nThis makes sense, once the input is parsed in `setup_revisions()`,\nthose'd be the values we want to use. Checking for values again is a\ndefinite race condition.\n\n> The workaround was originally added to deal with tags, but the code path\n> also gets hit by non-tag refs, causing this race condition. Because it's\n> no longer needed, remove it and fix the possible race condition.\n\nNice, simple fix.\n\n[snip]\n\n> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n> index 5d444bfe201a330527e86dde7229721fc386fc93..f398a59424dcd025ce616cadcd7eece9be5301a3 100755\n> --- a/t/t6020-bundle-misc.sh\n> +++ b/t/t6020-bundle-misc.sh\n> @@ -504,6 +504,40 @@ test_expect_success 'unfiltered bundle with --objects' '\n>  \ttest_cmp expect actual\n>  '\n>\n> +test_expect_success 'bottomless bundle upto tag' '\n> +\tgit bundle create v2.bdl \\\n> +\t\tv2 &&\n> +\n> +\tgit bundle verify v2.bdl |\n> +\t\tmake_user_friendly_and_stable_output >actual &&\n> +\n> +\tformat_and_save_expect <<-EOF &&\n> +\tThe bundle contains this ref:\n> +\t<TAG-2> refs/tags/v2\n> +\tThe bundle records a complete history.\n> +\t$HASH_MESSAGE\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'bundle between two tags' '\n> +\tgit bundle create v1-v2.bdl \\\n> +\t\tv1..v2 &&\n> +\n> +\tgit bundle verify v1-v2.bdl |\n> +\t\tmake_user_friendly_and_stable_output >actual &&\n> +\n> +\tformat_and_save_expect <<-EOF &&\n> +\tThe bundle contains this ref:\n> +\t<TAG-2> refs/tags/v2\n> +\tThe bundle requires these 2 refs:\n> +\t<COMMIT-E> Z\n> +\t<COMMIT-B> Z\n> +\t$HASH_MESSAGE\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n> +\n\nShouldn't we add a test for an annotated tag and verify that the tag\nobject is also included in the bundle?\n\nThanks\n\nKarthik\n"},{"id":"508897","messageId":"xmqqfrmwmlzh.fsf@gitster.g","threadId":"62615","inReplyTo":"CAOLa=ZQpBf-ZE8BmXRUXJ247p44Enj0vrJv2KHVDXtGzY6SU8w@mail.gmail.com","subject":"Re: [PATCH] bundle: remove unneeded code","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-10T00:15:46Z","receivedAt":"2024-12-10T00:15:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"karthik nayak <karthik.188@gmail.com> writes:\n\n>> The workaround was originally added to deal with tags, but the code path\n>> also gets hit by non-tag refs, causing this race condition. Because it's\n>> no longer needed, remove it and fix the possible race condition.\n>\n> Nice, simple fix.\n>\n> [snip]\n> ...\n> Shouldn't we add a test for an annotated tag and verify that the tag\n> object is also included in the bundle?\n\nI agree that it is a good idea, especially because 895c5ba3c1\n(revision: do not peel tags used in range notation, 2013-09-19)\nnaturally does not come with a test for \"git bundle\".\n\nThanks.\n"},{"id":"508900","messageId":"xmqqzfl4l22t.fsf@gitster.g","threadId":"62615","inReplyTo":"20241209-fix-bundle-create-race-v1-1-e6513bdcbf8a@iotcl.com","subject":"Re: [PATCH] bundle: remove unneeded code","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-10T02:11:06Z","receivedAt":"2024-12-10T02:11:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Toon Claes <toon@iotcl.com> writes:\n\n> The changes in commit c06793a4ed (allow git-bundle to create bottomless\n> bundle, 2007-08-08) ensure annotated tags are properly preserved when\n> creating a bundle using a revision range operation.\n>\n> At the time the range notation would peel the ends to their\n> corresponding commit, meaning ref v2.0 would point to the v2.0^0 commit.\n> So the above workaround was introduced. This code looks up the ref\n> before it's written to the bundle, and if the ref doesn't point to the\n> object we expect (for tags this would be a tag object), we skip the ref\n> from the bundle. Instead, when the ref is a tag that's the positive end\n> of the range (e.g. v2.0 from the range \"v1.0..v2.0\"), then that ref is\n> written to the bundle instead.\n>\n> Later, in 895c5ba3c1 (revision: do not peel tags used in range notation,\n> 2013-09-19), the behavior of parsing ranges was changed and the problem\n> was fixed at the cause. But the workaround in bundle.c was not reverted.\n>\n> Now it seems this workaround can cause a race condition. git-bundle(1)\n> uses setup_revisions() to parse the input into `struct rev_info`. Later,\n> in write_bundle_refs(), it uses this info to write refs to the bundle.\n> As mentioned at this point each ref is looked up again and checked\n> whether it points to the object we expect. If not, the ref is not\n> written to the bundle. But, when creating a bundle in a heavy traffic\n> repository (a repo with many references, and frequent ref updates) it's\n> possible a branch ref was updated between setup_revisions() and\n> write_bundle_refs() and thus the extra check causes the ref to be\n> skipped.\n>\n> The workaround was originally added to deal with tags, but the code path\n> also gets hit by non-tag refs, causing this race condition. Because it's\n> no longer needed, remove it and fix the possible race condition.\n\nIt is always a pleasure to read a patch based on the idea to\ndirectly target a nicely analyzed \"root cause\".\n\nThanks.\n"}]}