{"thread":{"id":"53771","subject":"Commit graph chains with no corresponding files?","startedAt":"2020-06-29T22:07:59Z","lastAt":"2021-02-27T02:50:45Z","messageCount":6,"participants":["Jonathan Tan","Derrick Stolee","Bryan Turner"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"400782","messageId":"20200629220744.1054093-1-jonathantanmy@google.com","threadId":"53771","inReplyTo":null,"subject":"Commit graph chains with no corresponding files?","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-06-29T22:07:43Z","receivedAt":"2020-06-29T22:07:59Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"At $DAYJOB, a few people have reported \"warning: unable to find all\ncommit-graph files\" warnings. Their commit-graph-chain files have a few\nlines, but they only have one commit graph file with very few commits. I\nsuspected something happening during fetch, because (as far as I know) a\nfetch may cause an incremental commit graph to be written, but I ran a\nfetch on a large repository myself and didn't run into this problem.\n\nHas anyone ran into this problem before, and know how to reproduce?\n"},{"id":"400791","messageId":"92b218ca-f2b4-db9d-9f9c-1d071ea4fa97@gmail.com","threadId":"53771","inReplyTo":"20200629220744.1054093-1-jonathantanmy@google.com","subject":"Re: Commit graph chains with no corresponding files?","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-30T01:51:39Z","receivedAt":"2020-06-30T01:51:43Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/29/2020 6:07 PM, Jonathan Tan wrote:\n> At $DAYJOB, a few people have reported \"warning: unable to find all\n> commit-graph files\" warnings. Their commit-graph-chain files have a few\n> lines, but they only have one commit graph file with very few commits. I\n> suspected something happening during fetch, because (as far as I know) a\n> fetch may cause an incremental commit graph to be written, but I ran a\n> fetch on a large repository myself and didn't run into this problem.\n> \n> Has anyone ran into this problem before, and know how to reproduce?\n\nThe incremental commit-graph code deletes any commit-graph files\nthat do not appear in the chain. I believe this is done by comparing\nthe contents of the \".git/objects/info/commit-graphs/\" directory to\nthe contents of the chain file.\n\nThese appear to be case-sensitive, full-path comparisons.\n\nIt is _possible_ that something like a case switch or a symlink\ncould be causing a problem here. That's where I would look on\nthe affected systems.\n\nLikely the full-path comparison in expire_commit_graphs() should\nbe dropped in favor of local filename comparisons. A case-\nsensitive match is less likely to be important here since Git\nis writing the paths itself and should get the proper case back\nfrom the directory listing.\n\nThanks,\n-Stolee\n\n"},{"id":"401668","messageId":"20200716225725.4143039-1-jonathantanmy@google.com","threadId":"53771","inReplyTo":"92b218ca-f2b4-db9d-9f9c-1d071ea4fa97@gmail.com","subject":"[FYI] commit-graph: trace expiry of commit graph links","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-07-16T22:57:25Z","receivedAt":"2020-07-16T23:09:51Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"When an obsolete link in the commit graph chain is deleted, and trace2\nis enabled, trace a message that it is deleted, along with the list of\nlinks before and after the current chain refresh.\n\nThe messages are emitted using trace2_data_string() and not\ntrace2_printf() because the latter is not implemented for some trace\ntargets.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\nThis patch is just for informational purposes for people who have the\nsame problem and/or want to help diagnose.\n\nI've tested that this produces the expected trace2 messages using:\n\n  git commit-graph write --reachable --split=replace\n\n> It is _possible_ that something like a case switch or a symlink\n> could be causing a problem here. That's where I would look on\n> the affected systems.\n\nIndeed a symlink is present - the affected repositories are using the\ngit-repo [1] local mirror feature, which causes .git/objects (among\nother things) to be a symlink - but I couldn't figure out how this\nsymlink would cause problems. In particular, looking at the code, all\nrelevant filenames seem to be constructed from ctx->odb->path, so no\nmatter which names were used to get to the object directory, all names\nare built from those names, and this seems to be true in practice as\nwell.\n\nSo I've added some trace2 messages (in this patch), and let's see if I\ncan figure out what's going on. I'll write back if I find something.\n\n[1] https://gerrit.googlesource.com/git-repo\n---\n commit-graph.c | 33 ++++++++++++++++++++++++++++++++-\n 1 file changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 328ab06fd4..b5bd2ac6de 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -2009,6 +2009,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tstruct dirent *de;\n \tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n+\tint commit_graph_deleted = 0;\n \n \tif (ctx->split_opts && ctx->split_opts->expire_time)\n \t\texpire_time = ctx->split_opts->expire_time;\n@@ -2050,8 +2051,38 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \t\t\t}\n \t\t}\n \n-\t\tif (!found)\n+\t\tif (!found) {\n+\t\t\tif (trace2_is_enabled()) {\n+\t\t\t\tstruct strbuf message = STRBUF_INIT;\n+\n+\t\t\t\tstrbuf_addf(&message, \"Deleting '%s' because it is not in [\", path.buf);\n+\t\t\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\t\t\tif (i != 0)\n+\t\t\t\t\t\tstrbuf_addstr(&message, \", \");\n+\t\t\t\t\tstrbuf_addf(&message, \"'%s'\", ctx->commit_graph_filenames_after[i]);\n+\t\t\t\t}\n+\t\t\t\tstrbuf_addstr(&message, \"]\");\n+\t\t\t\ttrace2_data_string(\"commit-graph\", the_repository, \"graph-deletion\", message.buf);\n+\t\t\t\tstrbuf_release(&message);\n+\t\t\t\tcommit_graph_deleted = 1;\n+\t\t\t}\n \t\t\tunlink(path.buf);\n+\t\t}\n+\t}\n+\n+\tif (commit_graph_deleted) {\n+\t\tstruct strbuf message = STRBUF_INIT;\n+\t\tuint32_t i;\n+\n+\t\tstrbuf_addstr(&message, \"The commit graphs before were [\");\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_before; i++) {\n+\t\t\tif (i != 0)\n+\t\t\t\tstrbuf_addstr(&message, \", \");\n+\t\t\tstrbuf_addf(&message, \"'%s'\", ctx->commit_graph_filenames_before[i]);\n+\t\t}\n+\t\tstrbuf_addstr(&message, \"]\");\n+\t\ttrace2_data_string(\"commit-graph\", the_repository, \"graph-before\", message.buf);\n+\t\tstrbuf_release(&message);\n \t}\n \n out:\n-- \n2.28.0.rc0.105.gf9edc3c819-goog\n\n"},{"id":"417784","messageId":"CAGyf7-G_OdS_0o7j64HA79n9Qv13SxciQSG+gfY7Qj8kNRQS5Q@mail.gmail.com","threadId":"53771","inReplyTo":"92b218ca-f2b4-db9d-9f9c-1d071ea4fa97@gmail.com","subject":"Re: Commit graph chains with no corresponding files?","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2021-02-25T04:54:05Z","receivedAt":"2021-02-25T04:55:14Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Mon, Jun 29, 2020 at 6:51 PM Derrick Stolee <stolee@gmail.com> wrote:\n>\n> On 6/29/2020 6:07 PM, Jonathan Tan wrote:\n> > At $DAYJOB, a few people have reported \"warning: unable to find all\n> > commit-graph files\" warnings. Their commit-graph-chain files have a few\n> > lines, but they only have one commit graph file with very few commits. I\n> > suspected something happening during fetch, because (as far as I know) a\n> > fetch may cause an incremental commit graph to be written, but I ran a\n> > fetch on a large repository myself and didn't run into this problem.\n> >\n> > Has anyone ran into this problem before, and know how to reproduce?\n\nI don't have any specific reproduction steps, but we've just run into\nour first case of this on Git 2.29. I ended up kicking off a full `git\ncommit-graph write` to fix it. That displayed the same warning, but\ncommands run after it no longer do. Prior to writing the new graph, I\nhad this:\n$ ls\ncommit-graph-chain  graph-88f5fe6e0c659e3742e556982263813d528ead81.graph\n\nAfterward, the `objects/info/commits-graphs` directory still exists\nbut is empty, and there is now an `objects/commit-graph` that didn't\nexist before. `git commit-graph verify` seems happy with the state of\nthings.\n\n>\n> The incremental commit-graph code deletes any commit-graph files\n> that do not appear in the chain. I believe this is done by comparing\n> the contents of the \".git/objects/info/commit-graphs/\" directory to\n> the contents of the chain file.\n>\n> These appear to be case-sensitive, full-path comparisons.\n>\n> It is _possible_ that something like a case switch or a symlink\n> could be causing a problem here. That's where I would look on\n> the affected systems.\n\nAre commit graphs potentially problematic in repositories that are\nborrowing objects from other repositories via alternates? Have there\nbeen important changes to commit graphs since 2.29?\n\n>\n> Likely the full-path comparison in expire_commit_graphs() should\n> be dropped in favor of local filename comparisons. A case-\n> sensitive match is less likely to be important here since Git\n> is writing the paths itself and should get the proper case back\n> from the directory listing.\n>\n> Thanks,\n> -Stolee\n>\n\n-b\n"},{"id":"417818","messageId":"fc8a2c0f-24b7-5884-b669-bb9700f3ba84@gmail.com","threadId":"53771","inReplyTo":"CAGyf7-G_OdS_0o7j64HA79n9Qv13SxciQSG+gfY7Qj8kNRQS5Q@mail.gmail.com","subject":"Re: Commit graph chains with no corresponding files?","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-25T14:20:37Z","receivedAt":"2021-02-25T14:21:22Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/24/2021 11:54 PM, Bryan Turner wrote:\n> On Mon, Jun 29, 2020 at 6:51 PM Derrick Stolee <stolee@gmail.com> wrote:\n>>\n>> On 6/29/2020 6:07 PM, Jonathan Tan wrote:\n>>> At $DAYJOB, a few people have reported \"warning: unable to find all\n>>> commit-graph files\" warnings. Their commit-graph-chain files have a few\n>>> lines, but they only have one commit graph file with very few commits. I\n>>> suspected something happening during fetch, because (as far as I know) a\n>>> fetch may cause an incremental commit graph to be written, but I ran a\n>>> fetch on a large repository myself and didn't run into this problem.\n>>>\n>>> Has anyone ran into this problem before, and know how to reproduce?\n> \n> I don't have any specific reproduction steps, but we've just run into\n> our first case of this on Git 2.29. I ended up kicking off a full `git\n> commit-graph write` to fix it. That displayed the same warning, but\n> commands run after it no longer do. Prior to writing the new graph, I\n> had this:\n> $ ls\n> commit-graph-chain  graph-88f5fe6e0c659e3742e556982263813d528ead81.graph\n\nThe contents of the 'commit-graph-chain' file are critical to diagnosing\nthe problems here. Likely it had multiple lines.\n\n> Afterward, the `objects/info/commits-graphs` directory still exists\n> but is empty, and there is now an `objects/commit-graph` that didn't\n> exist before. `git commit-graph verify` seems happy with the state of\n> things.\n\nYes, a full rewrite without \"--split\" will get you to this state.\n\n>> The incremental commit-graph code deletes any commit-graph files\n>> that do not appear in the chain. I believe this is done by comparing\n>> the contents of the \".git/objects/info/commit-graphs/\" directory to\n>> the contents of the chain file.\n>>\n>> These appear to be case-sensitive, full-path comparisons.\n>>\n>> It is _possible_ that something like a case switch or a symlink\n>> could be causing a problem here. That's where I would look on\n>> the affected systems.\n> \n> Are commit graphs potentially problematic in repositories that are\n> borrowing objects from other repositories via alternates?\n\nThis was definitely part of the design, with the intention of\nworking with a common base in the alternate. However, if the\nalternate collapses layers, then the repo that is borrowing\nfrom that alternate may have a broken chain.\n\nIt is likely a better setup to have the alternate keep a\ncommit-graph file and leave the dependent repos clear of a\ncommit-graph. _Or_ the dependent repos should use a full\ncommit-graph instead of a chain.\n\nIf you have a better idea for how to make this work, then there\nis room for improvement.\n\nFor example, if we ensure during the commit-graph write that\nall layers of the chain are within our local repo, then these\ndependency issues go away without breaking any old Git versions\nthat are reading the data.\n\n> Have there\n> been important changes to commit graphs since 2.29?\n\nNot in the area of commit-graph chains.\n\nThanks,\n-Stolee\n"},{"id":"417936","messageId":"CAGyf7-EzPrX8D5pnZKOGtiUHb6AZJs7Un4u39tOrcHkFi6gi-w@mail.gmail.com","threadId":"53771","inReplyTo":"fc8a2c0f-24b7-5884-b669-bb9700f3ba84@gmail.com","subject":"Re: Commit graph chains with no corresponding files?","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2021-02-27T02:49:33Z","receivedAt":"2021-02-27T02:50:45Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Thu, Feb 25, 2021 at 6:20 AM Derrick Stolee <stolee@gmail.com> wrote:\n>\n> On 2/24/2021 11:54 PM, Bryan Turner wrote:\n> > On Mon, Jun 29, 2020 at 6:51 PM Derrick Stolee <stolee@gmail.com> wrote:\n> >>\n> >> On 6/29/2020 6:07 PM, Jonathan Tan wrote:\n> >>> At $DAYJOB, a few people have reported \"warning: unable to find all\n> >>> commit-graph files\" warnings. Their commit-graph-chain files have a few\n> >>> lines, but they only have one commit graph file with very few commits. I\n> >>> suspected something happening during fetch, because (as far as I know) a\n> >>> fetch may cause an incremental commit graph to be written, but I ran a\n> >>> fetch on a large repository myself and didn't run into this problem.\n> >>>\n> >>> Has anyone ran into this problem before, and know how to reproduce?\n> >\n> > I don't have any specific reproduction steps, but we've just run into\n> > our first case of this on Git 2.29. I ended up kicking off a full `git\n> > commit-graph write` to fix it. That displayed the same warning, but\n> > commands run after it no longer do. Prior to writing the new graph, I\n> > had this:\n> > $ ls\n> > commit-graph-chain  graph-88f5fe6e0c659e3742e556982263813d528ead81.graph\n>\n> The contents of the 'commit-graph-chain' file are critical to diagnosing\n> the problems here. Likely it had multiple lines.\n>\n> > Afterward, the `objects/info/commits-graphs` directory still exists\n> > but is empty, and there is now an `objects/commit-graph` that didn't\n> > exist before. `git commit-graph verify` seems happy with the state of\n> > things.\n>\n> Yes, a full rewrite without \"--split\" will get you to this state.\n>\n> >> The incremental commit-graph code deletes any commit-graph files\n> >> that do not appear in the chain. I believe this is done by comparing\n> >> the contents of the \".git/objects/info/commit-graphs/\" directory to\n> >> the contents of the chain file.\n> >>\n> >> These appear to be case-sensitive, full-path comparisons.\n> >>\n> >> It is _possible_ that something like a case switch or a symlink\n> >> could be causing a problem here. That's where I would look on\n> >> the affected systems.\n> >\n> > Are commit graphs potentially problematic in repositories that are\n> > borrowing objects from other repositories via alternates?\n>\n> This was definitely part of the design, with the intention of\n> working with a common base in the alternate. However, if the\n> alternate collapses layers, then the repo that is borrowing\n> from that alternate may have a broken chain.\n\nThanks for the analysis, Derrick. This seems like a likely culprit for\nhow the repository got into this state, because it is a fork (of a\nfork) and does use a series of alternates.\n\n>\n> It is likely a better setup to have the alternate keep a\n> commit-graph file and leave the dependent repos clear of a\n> commit-graph. _Or_ the dependent repos should use a full\n> commit-graph instead of a chain.\n\nSkipping writing the commit graph in forks seems like a reasonable\nplace for us to start, given the way it currently works, but always\nwriting full graphs may be another option. If the fork is able to\nborrow the commit graph from its origin across the alternate, though,\nthen that implies there may not be a lot of value in writing commit\ngraphs in the forks (since they're likely to share the majority of\ntheir refs with their origin).\n\n>\n> If you have a better idea for how to make this work, then there\n> is room for improvement.\n>\n> For example, if we ensure during the commit-graph write that\n> all layers of the chain are within our local repo, then these\n> dependency issues go away without breaking any old Git versions\n> that are reading the data.\n\nNaively, this was the way I assumed it already worked--which is why I\nwas writing commit graphs in forks in the first place.\n\n>\n> > Have there\n> > been important changes to commit graphs since 2.29?\n>\n> Not in the area of commit-graph chains.\n\nThanks again!\n-b\n\n>\n> Thanks,\n> -Stolee\n"}]}