{"thread":{"id":"52451","subject":"Parallel fetch and commit graph writing results in locking failure (even on linux)","startedAt":"2019-12-13T20:40:59Z","lastAt":"2019-12-15T19:44:19Z","messageCount":8,"participants":["Thomas Braun","Derrick Stolee","Jeff King","SZEDER Gábor","Ralf Thielow"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"388145","messageId":"492636883.190386.1576264842701@ox.hosteurope.de","threadId":"52451","inReplyTo":null,"subject":"Parallel fetch and commit graph writing results in locking failure (even on linux)","fromName":"Thomas Braun","fromEmail":"thomas.braun@virtuell-zuhause.de","sentAt":"2019-12-13T19:20:42Z","receivedAt":"2019-12-13T20:40:59Z","isPatch":false,"sender":{"key":"thomas.braun@virtuell-zuhause.de","avatar":"https://avatars.githubusercontent.com/u/1185677?v=4"},"body":"Hi,\n\non git version da72936f (Git 2.24, 2019-11-04) and debian stretch I currently get every now and then the following error during fetching\n\n$git fetch --all --jobs 12\nFordere an von origin\nFordere an von XXXX\nFordere an von YYYY\nFordere an von ZZZZ\nFordere an von EEEE\nVon github.com:tango-controls/cppTango\n 37cc52f8..4550a743 tango-9-lts -> origin/tango-9-lts\nCommit-Graph Generierungsnummern berechnen: 100% (14/14), Fertig.\nCommit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\nfatal: Konnte '/home/firma/devel/cppTango/.git/objects/info/commit-graphs/commit-graph-chain.lock' nicht erstellen: Die Datei existiert bereits.\n\nEin anderer Git-Prozess scheint in diesem Repository ausgeführt\nzu werden, zum Beispiel ein noch offener Editor von 'git commit'.\nBitte stellen Sie sicher, dass alle Prozesse beendet wurden und\nversuchen Sie es erneut. Falls es immer noch fehlschlägt, könnte\nein früherer Git-Prozess in diesem Repository abgestürzt sein:\nLöschen Sie die Datei manuell um fortzufahren.\nKonnte 'myFork' nicht anfordern (Exit-Code: 128)\nCommit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\nCommit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\nCommit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\n\n(Sorry for the german text, this is not easily reproducible.)\nIt complains that it could not create the lock file as it already exists.\n\nI've set the following possible relevant settings:\n\n[core]\n commitGraph = true\n\n[fetch]\n prune = true\n writeCommitGraph = true\n\n[protocol]\n version = 2\n\nAnything obvious I'm doing wrong?\n\nThanks,\nThomas\n\nPS: The error is also present on latest git for windows.\n"},{"id":"388147","messageId":"bdb9201f-b77f-ab3c-251f-d902c76fa9bc@gmail.com","threadId":"52451","inReplyTo":"492636883.190386.1576264842701@ox.hosteurope.de","subject":"Re: Parallel fetch and commit graph writing results in locking failure (even on linux)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-12-13T19:35:47Z","receivedAt":"2019-12-13T20:41:01Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/13/2019 2:20 PM, Thomas Braun wrote:\n> Hi,\n> \n> on git version da72936f (Git 2.24, 2019-11-04) and debian stretch I currently get every now and then the following error during fetching\n> \n> $git fetch --all --jobs 12\n> Fordere an von origin\n> Fordere an von XXXX\n> Fordere an von YYYY\n> Fordere an von ZZZZ\n> Fordere an von EEEE\n> Von github.com:tango-controls/cppTango\n> 37cc52f8..4550a743 tango-9-lts -> origin/tango-9-lts\n> Commit-Graph Generierungsnummern berechnen: 100% (14/14), Fertig.\n> Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\n> fatal: Konnte '/home/firma/devel/cppTango/.git/objects/info/commit-graphs/commit-graph-chain.lock' nicht erstellen: Die Datei existiert bereits.\n> \n> Ein anderer Git-Prozess scheint in diesem Repository ausgeführt\n> zu werden, zum Beispiel ein noch offener Editor von 'git commit'.\n> Bitte stellen Sie sicher, dass alle Prozesse beendet wurden und\n> versuchen Sie es erneut. Falls es immer noch fehlschlägt, könnte\n> ein früherer Git-Prozess in diesem Repository abgestürzt sein:\n> Löschen Sie die Datei manuell um fortzufahren.\n> Konnte 'myFork' nicht anfordern (Exit-Code: 128)\n> Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\n> Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\n> Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\n> \n> (Sorry for the german text, this is not easily reproducible.)\n> It complains that it could not create the lock file as it already exists.\n> \n> I've set the following possible relevant settings:\n> \n> [core]\n> commitGraph = true\n> \n> [fetch]\n> prune = true\n> writeCommitGraph = true\n> \n> [protocol]\n> version = 2\n> \n> Anything obvious I'm doing wrong?\n\nI don't think so. I think you just found a bug where the\nfetch.writeCommitGraph logic doesn't work with parallel fetch\njobs (only one can write at a time).\n\nI believe the fix would be to write the commit-graph after\nall of the jobs have completed, which should mean we need to\nmove the call to write_commit_graph_reachable() somewhere else\ninside builtin/fetch.c.\n\nI'll take a look now.\n\nThanks,\n-Stolee\n\n"},{"id":"388152","messageId":"20191213195215.GA862734@coredump.intra.peff.net","threadId":"52451","inReplyTo":"bdb9201f-b77f-ab3c-251f-d902c76fa9bc@gmail.com","subject":"Re: Parallel fetch and commit graph writing results in locking failure (even on linux)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-12-13T19:52:15Z","receivedAt":"2019-12-13T20:41:09Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 13, 2019 at 02:35:47PM -0500, Derrick Stolee wrote:\n\n> I don't think so. I think you just found a bug where the\n> fetch.writeCommitGraph logic doesn't work with parallel fetch\n> jobs (only one can write at a time).\n> \n> I believe the fix would be to write the commit-graph after\n> all of the jobs have completed, which should mean we need to\n> move the call to write_commit_graph_reachable() somewhere else\n> inside builtin/fetch.c.\n\nThis should be fixed in master by bcb06e204c (Merge branch\n'js/fetch-multi-lockfix', 2019-12-01), I think.\n\n\n-Peff\n"},{"id":"388157","messageId":"b7d5a758-9753-bb8b-f66e-6435fb19046b@gmail.com","threadId":"52451","inReplyTo":"20191213195215.GA862734@coredump.intra.peff.net","subject":"Re: Parallel fetch and commit graph writing results in locking failure (even on linux)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-12-13T19:58:57Z","receivedAt":"2019-12-13T20:41:14Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/13/2019 2:52 PM, Jeff King wrote:\n> On Fri, Dec 13, 2019 at 02:35:47PM -0500, Derrick Stolee wrote:\n> \n>> I don't think so. I think you just found a bug where the\n>> fetch.writeCommitGraph logic doesn't work with parallel fetch\n>> jobs (only one can write at a time).\n>>\n>> I believe the fix would be to write the commit-graph after\n>> all of the jobs have completed, which should mean we need to\n>> move the call to write_commit_graph_reachable() somewhere else\n>> inside builtin/fetch.c.\n> \n> This should be fixed in master by bcb06e204c (Merge branch\n> 'js/fetch-multi-lockfix', 2019-12-01), I think.\n\nThanks, Peff. That exactly looks like the right fix. The\nactual commit is 7d8e72b9 (\"fetch: avoid locking issues between\nfetch.jobs/fetch.writeCommitGraph\" 2019-11-03). I had forgotten\nthat this was already fixed.\n\nHere is the diff, for reference:\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 8d27f8abb7..20bcda09c4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1602,7 +1602,8 @@ static int fetch_multiple(struct string_list *list, int max_children)\n return errcode;\n }\n \n- argv_array_pushl(&argv, \"fetch\", \"--append\", \"--no-auto-gc\", NULL);\n+ argv_array_pushl(&argv, \"fetch\", \"--append\", \"--no-auto-gc\",\n+ \"--no-write-commit-graph\", NULL);\n add_options_to_argv(&argv);\n \n if (max_children != 1 && list->nr != 1) {\n\n-Stolee\n"},{"id":"388160","messageId":"20191213201522.GM6527@szeder.dev","threadId":"52451","inReplyTo":"492636883.190386.1576264842701@ox.hosteurope.de","subject":"Re: Parallel fetch and commit graph writing results in locking failure (even on linux)","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-12-13T20:15:22Z","receivedAt":"2019-12-13T20:41:19Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Dec 13, 2019 at 08:20:42PM +0100, Thomas Braun wrote:\n> Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\n> \n> (Sorry for the german text, this is not easily reproducible.)\n\nIt's unrelated to your issue, and I'm not a native German or English\nspeaker... but that translation doesn't look quite right to me.\n\nThe original is:\n\n _(\"Computing commit graph generation numbers\"),\n\nwhere the word \"generation\" isn't used in the sense of \"creation\"\n(erzeugen, generieren), but rather as in \"generation gap\" or \"my\nparents are one generation older than me\". So I don't think that\n\"Generierung...\" is the right word to use here, but perhaps\n\"Generationsnummer\".\n\nBut who am I to argue about German software ranslation?! :)\n\n"},{"id":"388230","messageId":"264571040.270538.1576430345179@ox.hosteurope.de","threadId":"52451","inReplyTo":"b7d5a758-9753-bb8b-f66e-6435fb19046b@gmail.com","subject":"Re: Parallel fetch and commit graph writing results in locking failure (even on linux)","fromName":"Thomas Braun","fromEmail":"thomas.braun@virtuell-zuhause.de","sentAt":"2019-12-15T17:19:05Z","receivedAt":"2019-12-15T17:19:09Z","isPatch":false,"sender":{"key":"thomas.braun@virtuell-zuhause.de","avatar":"https://avatars.githubusercontent.com/u/1185677?v=4"},"body":"> Derrick Stolee hat am 13. Dezember 2019 um 20:58 geschrieben:\n> \n> \n> On 12/13/2019 2:52 PM, Jeff King wrote:\n> > On Fri, Dec 13, 2019 at 02:35:47PM -0500, Derrick Stolee wrote:\n> > \n> >> I don't think so. I think you just found a bug where the\n> >> fetch.writeCommitGraph logic doesn't work with parallel fetch\n> >> jobs (only one can write at a time).\n> >>\n> >> I believe the fix would be to write the commit-graph after\n> >> all of the jobs have completed, which should mean we need to\n> >> move the call to write_commit_graph_reachable() somewhere else\n> >> inside builtin/fetch.c.\n> > \n> > This should be fixed in master by bcb06e204c (Merge branch\n> > 'js/fetch-multi-lockfix', 2019-12-01), I think.\n> \n> Thanks, Peff. That exactly looks like the right fix. The\n> actual commit is 7d8e72b9 (\"fetch: avoid locking issues between\n> fetch.jobs/fetch.writeCommitGraph\" 2019-11-03). I had forgotten\n> that this was already fixed.\n\nThanks Stolee and Peff, I'll update to a version with that fix. And in the unlikely case this does not resolve my issue, report back.\n \n> Here is the diff, for reference:\n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 8d27f8abb7..20bcda09c4 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -1602,7 +1602,8 @@ static int fetch_multiple(struct string_list *list, int max_children)\n> return errcode;\n> }\n> \n> - argv_array_pushl(&argv, \"fetch\", \"--append\", \"--no-auto-gc\", NULL);\n> + argv_array_pushl(&argv, \"fetch\", \"--append\", \"--no-auto-gc\",\n> + \"--no-write-commit-graph\", NULL);\n> add_options_to_argv(&argv);\n> \n> if (max_children != 1 && list->nr != 1) {\n> \n> -Stolee\n>\n"},{"id":"388232","messageId":"2099929548.270607.1576431348524@ox.hosteurope.de","threadId":"52451","inReplyTo":"20191213201522.GM6527@szeder.dev","subject":"german language fix: Generierung vs. Generation [was: Re: Parallel fetch and commit graph writing results in locking failure (even on linux)]","fromName":"Thomas Braun","fromEmail":"thomas.braun@virtuell-zuhause.de","sentAt":"2019-12-15T17:35:48Z","receivedAt":"2019-12-15T17:35:51Z","isPatch":false,"sender":{"key":"thomas.braun@virtuell-zuhause.de","avatar":"https://avatars.githubusercontent.com/u/1185677?v=4"},"body":"\n> SZEDER Gábor hat am 13. Dezember 2019 um 21:15 geschrieben:\n> \n> \n> On Fri, Dec 13, 2019 at 08:20:42PM +0100, Thomas Braun wrote:\n> > Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.\n> > \n> > (Sorry for the german text, this is not easily reproducible.)\n> \n> It's unrelated to your issue, and I'm not a native German or English\n> speaker... but that translation doesn't look quite right to me.\n> \n> The original is:\n> \n> _(\"Computing commit graph generation numbers\"),\n> \n> where the word \"generation\" isn't used in the sense of \"creation\"\n> (erzeugen, generieren), but rather as in \"generation gap\" or \"my\n> parents are one generation older than me\". So I don't think that\n> \"Generierung...\" is the right word to use here, but perhaps\n> \"Generationsnummer\".\n> \n> But who am I to argue about German software ranslation?! :)\n\nYour hunch is right I would say (not that it matters much but I'm a native speaker).\n\nSo how about the following patch (CC'ing the original translator)\n\ncommit 724bf68da28b75e04a9fda34b08e95259ade9ec3 (HEAD -> master)\nAuthor: Thomas Braun \nDate: Sun Dec 15 18:28:42 2019 +0100\n\n lang de: Reword generation numbers\n \n The english term generation is here not used in the sense of \"to\n generate\" but in the sense of \"generations of beings\".\n \n This corrects the initial translation from cf4c0c25 (l10n: update German\n translation, 2018-12-06).\n \n Fixed-by: \"SZEDER Gábor\" \n\ndiff --git a/po/de.po b/po/de.po\nindex 066326a687..773e361f6f 100644\n--- a/po/de.po\n+++ b/po/de.po\n@@ -1535,7 +1535,7 @@ msgstr \"Lösche Commit-Markierungen in Commit-Graph\"\n \n #: commit-graph.c:1104\n msgid \"Computing commit graph generation numbers\"\n-msgstr \"Commit-Graph Generierungsnummern berechnen\"\n+msgstr \"Commit-Graph Generationsnummern berechnen\"\n \n #: commit-graph.c:1179\n #, c-format\n"},{"id":"388235","messageId":"20191215194412.7549-1-ralf.thielow@gmail.com","threadId":"52451","inReplyTo":"2099929548.270607.1576431348524@ox.hosteurope.de","subject":"[PATCH] l10n: de.po: Reword generation numbers","fromName":"Ralf Thielow","fromEmail":"ralf.thielow@gmail.com","sentAt":"2019-12-15T19:44:12Z","receivedAt":"2019-12-15T19:44:19Z","isPatch":true,"sender":{"key":"ralf.thielow@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1275832?v=4"},"body":"From: Thomas Braun \n\nThe english term generation is here not used in the sense of \"to\ngenerate\" but in the sense of \"generations of beings\".\n\nThis corrects the initial translation from cf4c0c25 (l10n: update German\ntranslation, 2018-12-06).\n\nFixed-by: SZEDER Gábor \nSigned-off-by: Ralf Thielow \n---\nThank you. The change looks good to me. I'll send a pull request\nwith this patch to the German translation repo at Github.\n\n po/de.po | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/po/de.po b/po/de.po\nindex 066326a687..773e361f6f 100644\n--- a/po/de.po\n+++ b/po/de.po\n@@ -1535,7 +1535,7 @@ msgstr \"Lösche Commit-Markierungen in Commit-Graph\"\n \n #: commit-graph.c:1104\n msgid \"Computing commit graph generation numbers\"\n-msgstr \"Commit-Graph Generierungsnummern berechnen\"\n+msgstr \"Commit-Graph Generationsnummern berechnen\"\n \n #: commit-graph.c:1179\n #, c-format\n-- \n2.24.1.735.g03f4e72817\n\n"}]}