{"thread":{"id":"28779","subject":"Repository data loss in fast-export with a merge of a deleted submodule","startedAt":"2011-10-27T19:27:49Z","lastAt":"2011-11-30T07:15:19Z","messageCount":5,"participants":["Joshua Jensen","Jens Lehmann","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"178376","messageId":"4EA9B0B5.6060005@workspacewhiz.com","threadId":"28779","inReplyTo":null,"subject":"Repository data loss in fast-export with a merge of a deleted submodule","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2011-10-27T19:27:49Z","receivedAt":"2011-10-27T19:27:49Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":"Hello.\n\nWe had a submodule that we deleted and then added back into the \nrepository at the same location as the former submodule.  When running \nfast-export, the newly 'added' files for the merge commit are listed and \nthen are followed with a:\n\nM ... path/to/submodule/file\nD path/to/submodule\n\nOn fast-import, the resultant repository becomes corrupt due to the \nDelete instruction above occurring AFTER the file adds/modifications.  \nThe new repository does not match the old repository where the \nfast-export was performed.\n\nI have included a repro script below.  I have not been able to test this \non Git 1.7.7.1, but I have tested on Git 1.7.7 (msysGit version).\n\nPlease compare the differences between the generated main.fe and \nnewmain.fe files.  newmain.fe has data loss.\n\nI am not familiar with the fast-export code.  Can anyone help out?\n\nThanks.\n\nJosh\n\n---------\n\nrm -rf main brokenmain sub main.fenewmain.fe\n\n# Create the submodule.\nmkdir sub\ncd sub\ngit init\necho file > file\ngit add file\ngit commit -m file\ncd ..\n\n# Create the main repository.\nmkdir main\ncd main\ngit init\n\n# Add the submodule.\ngit submodule add ../sub sub\ngit commit -m \"Add submodule\"\n\n# Remove the submodule.\nrm -rf sub\ngit rm sub .gitmodules\ngit commit -m \"Remove submodule\"\n\n# Add sub/file to the master branch.\nmkdir sub\necho file > sub/file\ngit add sub/file\ngit commit -m \"Add sub/file\"\nif [ -f sub/file ]; then\n     echo \"main: master branch: sub/file exists.\"\nfi\n\n# Delete the submodule directory manually, because we know that the \nincoming merge will need it gone.\ngit checkout -B will-be-broken HEAD^^\nrm -rf sub\ngit merge --no-ff master\n\n# sub/file exists within the 'will-be-broken' branch.\nif [ -f sub/file ]; then\n     echo \"main: will-be-broken branch: sub/file exists.\"\nfi\n\n# Export out the main repository.\ngit fast-export --all > ../main.fe\n\n# Create the brokenmain repository.\ncd ..\nmkdir brokenmain\ncd brokenmain\ngit init\n\n# Import in everything from the main repository.\ngit fast-import < ../main.fe\n\n# sub/file exists within the master branch.\ngit checkout master\nif [ -f sub/file ]; then\n     echo \"brokenmain: master branch: sub/file exists.\"\nfi\n\n# sub/file SHOULD exist within the 'will-be-broken' branch but doesn't.\ngit checkout will-be-broken\nif [ ! -f sub/file ]; then\n     echo \"brokenmain: will-be-broken branch: sub/file SHOULD exist but \ndoesn't.\"\nfi\n\n# Export out the brokenmain repository.\ngit fast-export --all > ../brokenmain.fe\n"},{"id":"178777","messageId":"4EB2BBB5.3030908@workspacewhiz.com","threadId":"28779","inReplyTo":"4EA9B0B5.6060005@workspacewhiz.com","subject":"Re: Repository data loss in fast-export with a merge of a deleted submodule","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2011-11-03T16:05:09Z","receivedAt":"2011-11-03T16:05:09Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":"----- Original Message -----\nFrom: Joshua Jensen\nDate: 10/27/2011 1:27 PM\n> We had a submodule that we deleted and then added back into the \n> repository at the same location as the former submodule.  When running \n> fast-export, the newly 'added' files for the merge commit are listed \n> and then are followed with a:\n>\n> M ... path/to/submodule/file\n> D path/to/submodule\n>\n> On fast-import, the resultant repository becomes corrupt due to the \n> Delete instruction above occurring AFTER the file adds/modifications.  \n> The new repository does not match the old repository where the \n> fast-export was performed.\n>\n> I am not familiar with the fast-export code.  Can anyone help out?\nOkay, I looked into this further, and I came up with a patch that works \nfor me.  Nevertheless, I do not understand exactly what is going on \nhere, so I would like to defer to someone else's patch to fix the issue.\n\n-Josh\n\n\n---\nbuiltin/fast-export.c |    8 ++++++++\n1 files changed, 8 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 9836e6b..1abc470 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -161,6 +161,14 @@ static int depth_first(const void *a_, const void *b_)\n                name_a = a->one ? a->one->path : a->two->path;\n                name_b = b->one ? b->one->path : b->two->path;\n+             /*\n+             * Move 'D'elete entries first.\n+             */\n+             if (a->status == 'D')\n+                             return -1;\n+             else if (b->status == 'D')\n+                             return 1;\n+\n                len_a = strlen(name_a);\n                len_b = strlen(name_b);\n                len = (len_a < len_b) ? len_a : len_b;\n-- \n"},{"id":"179432","messageId":"4EC12E8B.3050909@workspacewhiz.com","threadId":"28779","inReplyTo":"4EB2BBB5.3030908@workspacewhiz.com","subject":"Re: Repository data loss in fast-export with a merge of a deleted submodule","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2011-11-14T15:06:51Z","receivedAt":"2011-11-14T15:06:51Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":"----- Original Message -----\nFrom: Joshua Jensen\nDate: 11/3/2011 10:05 AM\n> ----- Original Message -----\n> From: Joshua Jensen\n> Date: 10/27/2011 1:27 PM\n>> We had a submodule that we deleted and then added back into the \n>> repository at the same location as the former submodule.  When \n>> running fast-export, the newly 'added' files for the merge commit are \n>> listed and then are followed with a:\n>>\n>> M ... path/to/submodule/file\n>> D path/to/submodule\n>>\n>> On fast-import, the resultant repository becomes corrupt due to the \n>> Delete instruction above occurring AFTER the file \n>> adds/modifications.  The new repository does not match the old \n>> repository where the fast-export was performed.\n>>\n>> I am not familiar with the fast-export code.  Can anyone help out?\n> Okay, I looked into this further, and I came up with a patch that \n> works for me.  Nevertheless, I do not understand exactly what is going \n> on here, so I would like to defer to someone else's patch to fix the \n> issue.\n>\nHi.\n\n__This is a genuine data loss problem in Git.__\n\nI'm confused at the lack of response to this.  I first posted about the \nissue **2-1/2 weeks ago**, and there have been no responses  Does no one \ncare?\n\nIn case no one received the messages, you can find them at [1] and [2].\n\n-Josh\n\n[1] http://www.spinics.net/lists/git/msg168295.html\n[2] http://www.spinics.net/lists/git/msg168691.html\n"},{"id":"179436","messageId":"4EC17146.3000202@web.de","threadId":"28779","inReplyTo":"4EC12E8B.3050909@workspacewhiz.com","subject":"Re: Repository data loss in fast-export with a merge of a deleted submodule","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2011-11-14T19:51:34Z","receivedAt":"2011-11-14T19:51:34Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 14.11.2011 16:06, schrieb Joshua Jensen:\n> ----- Original Message -----\n> From: Joshua Jensen\n> Date: 11/3/2011 10:05 AM\n>> ----- Original Message -----\n>> From: Joshua Jensen\n>> Date: 10/27/2011 1:27 PM\n>>> We had a submodule that we deleted and then added back into the repository at the same location as the former submodule.  When running fast-export, the newly 'added' files for the merge commit are listed and then are followed with a:\n>>>\n>>> M ... path/to/submodule/file\n>>> D path/to/submodule\n>>>\n>>> On fast-import, the resultant repository becomes corrupt due to the Delete instruction above occurring AFTER the file adds/modifications.  The new repository does not match the old repository where the fast-export was performed.\n>>>\n>>> I am not familiar with the fast-export code.  Can anyone help out?\n>> Okay, I looked into this further, and I came up with a patch that works for me.  Nevertheless, I do not understand exactly what is going on here, so I would like to defer to someone else's patch to fix the issue.\n>>\n> Hi.\n> \n> __This is a genuine data loss problem in Git.__\n> \n> I'm confused at the lack of response to this.  I first posted about the issue **2-1/2 weeks ago**, and there have been no responses  Does no one care?\n\nMaybe no one cares, people didn't read the message (or forgot about it)\nor they are too busy ... thanks for prodding us again.\n\nWhile I'm interested in this issue because submodules are affected, I'm\nvery short on Git time these days and can't investigate this issue\nfurther (and I have no clue about export/import either). I added the last\ntwo people who touched depth_first() in builtin/fast-export.c to the CC,\nmaybe they can tell us more about your patch to solve this issue (found\nin [2]).\n\n> In case no one received the messages, you can find them at [1] and [2].\n> \n> -Josh\n> \n> [1] http://www.spinics.net/lists/git/msg168295.html\n> [2] http://www.spinics.net/lists/git/msg168691.html\n"},{"id":"180141","messageId":"20111130071519.GF5317@sigill.intra.peff.net","threadId":"28779","inReplyTo":"4EC12E8B.3050909@workspacewhiz.com","subject":"Re: Repository data loss in fast-export with a merge of a deleted submodule","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-30T07:15:19Z","receivedAt":"2011-11-30T07:15:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 14, 2011 at 08:06:51AM -0700, Joshua Jensen wrote:\n\n> __This is a genuine data loss problem in Git.__\n> \n> I'm confused at the lack of response to this.  I first posted about\n> the issue **2-1/2 weeks ago**, and there have been no responses  Does\n> no one care?\n\nStill not much response.\n\nI think the keywords \"submodule\" and \"fast-export\" in the subject line\nhit a lot of people's do-not-care filters.\n\nI read your original two messages. It does seem like a simple ordering\nproblem from your description. I suspect you would get more response to\nactually post your patch with a commit message explaining the problem,\nand an accompanying test. And then at the very least, one outcome could\nbe Junio picking up the patch. :)\n\nI think you have all of those components spread across your messages,\nand just need to polish them and put them in one place.\n\nRegarding your patch itself, your explanation make sense to me and the\ngoal of your patch looks reasonable. Bearing in mind that I know\nvirtually nothing about the innards fast-import/fast-export.\n\nBut for the patch text itself:\n\n> @@ -161,6 +161,14 @@ static int depth_first(const void *a_, const void *b_)\n>                name_a = a->one ? a->one->path : a->two->path;\n>                name_b = b->one ? b->one->path : b->two->path;\n> +             /*\n> +             * Move 'D'elete entries first.\n> +             */\n> +             if (a->status == 'D')\n> +                             return -1;\n> +             else if (b->status == 'D')\n> +                             return 1;\n> +\n>                len_a = strlen(name_a);\n>                len_b = strlen(name_b);\n>                len = (len_a < len_b) ? len_a : len_b;\n\nIf you have multiple deleted entries, doesn't this leave them in a\nrandom order at the beginning of the list? Does that matter? If they are\nboth 'D', should they be compared as usual? I.e.:\n\n  if (a->status != b->status) {\n          if (a->status == 'D')\n                  return -1;\n          if (b->status == 'D')\n                  return 1;\n  }\n\n  /* and now we do the rest of the function as usual... */\n\n-Peff\n"}]}