{"thread":{"id":"13910","subject":"git-fast-export bug, commits emmitted in incorrect order causing parent data to be lost from commits turning essentially linear repo into \"islands\"","startedAt":"2008-06-12T10:21:40Z","lastAt":"2008-06-12T14:02:10Z","messageCount":7,"participants":["Yves Orton","Johannes Sixt","Michael J Gruber","Philippe Bruhat (BooK)"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"79582","messageId":"1213266100.6940.207.camel@gemini","threadId":"13910","inReplyTo":null,"subject":"git-fast-export bug, commits emmitted in incorrect order causing parent data to be lost from commits turning essentially linear repo into \"islands\"","fromName":"Yves Orton","fromEmail":"yves.orton@booking.com","sentAt":"2008-06-12T10:21:40Z","receivedAt":"2008-06-12T10:21:40Z","isPatch":false,"sender":{"key":"yves.orton@booking.com","avatar":null},"body":"Hi,\n\nIve been working with git-fast-export a bit recently and Ive hit a bug\nthat is causing some trouble. \n\nEssentially it seems that one of our repos git-fast-export fails to emit\nthe proper 'from' information for several commits in the repo. These\ncommits are emitted first without parent data even though their parents\nARE emitted later.\n\nThe code responsible for skipping the parent info is in\n\nbuiltin-fast-export.c around line 402:\n\n        for (i = 0, p = commit->parents; p; p = p->next) {\n                int mark = get_object_mark(&p->item->object);\n                if (!mark)\n                        continue;\n                if (i == 0)\n                        printf(\"from :%d\\n\", mark);\n                else\n                        printf(\"merge :%d\\n\", mark);\n                i++;\n        }\n\nIf i modify this loop to warn when skipping a parent I get a warning for\neach of the \"broken\" commits. Apparently because they are emitted before\ntheir parents the parents have no \"mark\" assigned to them (via\ndecoration) and thus are skipped in this emit process. This would make\nsense for emitting a limited number of patches, but makes no sense when\nthe --all option is used. Ive tried to investigate further but i got\nlost in a twisty maze of routines in revision.c, which apparently is\nresponsible for building a list of items to emit in the correct order. \n\nHowever i think it is notable that both gitk and git log seem quite able\nto deal with things properly, thus i find it a bit strange that\nfast-export would get it wrong.\n\nUnfortunately I have no idea how to create a minimal repo that\nillustrates this problem. \n\nIm currently on git version 1.5.6.rc2.29.g3ba9 (latest version from last\nnight), however this problem shows itself on 1.5.4.3 as well, as well as\nan earlier version whose exact number i no longer know. \n\nOther evidence that might be useful\n\n\tgit log --pretty=format:\"%H:%P\" \n\nshows that every commit but one (the root) has parents. And gitk renders\nthe original repo fine. The repo can be cloned and etc, without trouble.\nThe problem seems to be strictly related to fast-export.\n\nIm not on list so please cc me on any replies. \n\nThanks a lot!\nYves\n"},{"id":"79587","messageId":"48510E2E.6090508@viscovery.net","threadId":"13910","inReplyTo":"1213266100.6940.207.camel@gemini","subject":"Re: git-fast-export bug, commits emmitted in incorrect order causing parent data to be lost from commits turning essentially linear repo into \"islands\"","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-06-12T11:53:18Z","receivedAt":"2008-06-12T11:53:18Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Yves Orton schrieb:\n> Hi,\n> \n> Ive been working with git-fast-export a bit recently and Ive hit a bug\n> that is causing some trouble. \n> \n> Essentially it seems that one of our repos git-fast-export fails to emit\n> the proper 'from' information for several commits in the repo. These\n> commits are emitted first without parent data even though their parents\n> ARE emitted later.\n\nDoes it make a difference if you pass --topo-order to git fast-export?\n(But I don't know for certain that this is even legal.)\n\n-- Hannes\n"},{"id":"79589","messageId":"1213272285.6940.222.camel@gemini","threadId":"13910","inReplyTo":"48510E2E.6090508@viscovery.net","subject":"Re: git-fast-export bug, commits emmitted in incorrect order causing parent data to be lost from commits turning essentially linear repo into \"islands\"","fromName":"Yves Orton","fromEmail":"yves.orton@booking.com","sentAt":"2008-06-12T12:04:45Z","receivedAt":"2008-06-12T12:04:45Z","isPatch":false,"sender":{"key":"yves.orton@booking.com","avatar":null},"body":"On Thu, 2008-06-12 at 13:53 +0200, Johannes Sixt wrote:\n> Yves Orton schrieb:\n> > Hi,\n> > \n> > Ive been working with git-fast-export a bit recently and Ive hit a bug\n> > that is causing some trouble. \n> > \n> > Essentially it seems that one of our repos git-fast-export fails to emit\n> > the proper 'from' information for several commits in the repo. These\n> > commits are emitted first without parent data even though their parents\n> > ARE emitted later.\n> \n> Does it make a difference if you pass --topo-order to git fast-export?\n> (But I don't know for certain that this is even legal.)\n\nYes it does make a difference. A big difference. That would be the\nworkaround I really needed.  At least currently thats the way it looks,\ni havent thoroughly tested the result yet but it certainly looks right.\n\nPerhaps this should be enabled by default to avoid the problem i\nencountered? At least until whatever the cause of the root problem is\nidentified and fixed.\n\nThanks a lot. ++ to you.\n\nCheers,\nyves\n"},{"id":"79591","messageId":"1213272962.6940.231.camel@gemini","threadId":"13910","inReplyTo":"1213272285.6940.222.camel@gemini","subject":"Re: git-fast-export bug, commits emmitted in incorrect order causing parent data to be lost from commits turning essentially linear repo into \"islands\"","fromName":"Yves Orton","fromEmail":"yves.orton@booking.com","sentAt":"2008-06-12T12:16:02Z","receivedAt":"2008-06-12T12:16:02Z","isPatch":false,"sender":{"key":"yves.orton@booking.com","avatar":null},"body":"On Thu, 2008-06-12 at 14:04 +0200, Yves Orton wrote:\n> On Thu, 2008-06-12 at 13:53 +0200, Johannes Sixt wrote:\n> > Yves Orton schrieb:\n> > > Hi,\n> > > \n> > > Ive been working with git-fast-export a bit recently and Ive hit a bug\n> > > that is causing some trouble. \n> > > \n> > > Essentially it seems that one of our repos git-fast-export fails to emit\n> > > the proper 'from' information for several commits in the repo. These\n> > > commits are emitted first without parent data even though their parents\n> > > ARE emitted later.\n> > \n> > Does it make a difference if you pass --topo-order to git fast-export?\n> > (But I don't know for certain that this is even legal.)\n> \n> Yes it does make a difference. A big difference. That would be the\n> workaround I really needed.  At least currently thats the way it looks,\n> i havent thoroughly tested the result yet but it certainly looks right.\n> \n> Perhaps this should be enabled by default to avoid the problem i\n> encountered? At least until whatever the cause of the root problem is\n> identified and fixed.\n> \n> Thanks a lot. ++ to you.\n\nI should add that with this switch enabled the output order is correct,\nHOWEVER the mark number of the first commit is unchanged from the\noriginal. However the parent relationships are correctly restored and\nthe resulting repo has the correct SHA1 stamps. So it looks like the\noriginal traversal order is wrong somehow, and that --topo-order fixes\nit up after the fact.\n\nBut for my immediate needs this is the solution I needed. Again many\nthanks.\n\nBTW, for the record this was needed because we are trying to merge\nmultiple git repos into a single new git repo with each original repo\nmapped into a subdirectory of the new repo, and with commit trees merged\nin more or less the correct order (by date applied more or less). IOW we\ndont want to have multiple \"root commits\" that are later merged. \n\nWe want a more or less linear repo as the result. This bug with\nfast-export was the main showstopper in our efforts.  However, I can\nimagine that this is a problem that many people will want to solve. It\nwould be nice if there was an easier way to do it that what we currently\nare doing (merging and munging multiple fast-export streams into a\nsingle fast-import process). While at this point its probably academic\nany suggestions as to the Best Way to do this would be very much\nwelcome.\n\nCheers and thanks to all you git developers for a great tool!\nYves\n"},{"id":"79594","messageId":"48511A85.9020200@viscovery.net","threadId":"13910","inReplyTo":"1213272962.6940.231.camel@gemini","subject":"Re: git-fast-export bug, commits emmitted in incorrect order causing parent data to be lost from commits turning essentially linear repo into \"islands\"","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-06-12T12:45:57Z","receivedAt":"2008-06-12T12:45:57Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Yves Orton schrieb:\n> BTW, for the record this was needed because we are trying to merge\n> multiple git repos into a single new git repo with each original repo\n> mapped into a subdirectory of the new repo, and with commit trees merged\n> in more or less the correct order (by date applied more or less). IOW we\n> dont want to have multiple \"root commits\" that are later merged. \n\nTry --date-order instead. It might work better for your task, and it still\noffers a topologically correct order.\n\n-- Hannes\n"},{"id":"79595","messageId":"g2r66q$d3j$1@ger.gmane.org","threadId":"13910","inReplyTo":"1213272962.6940.231.camel@gemini","subject":"Re: git-fast-export bug, commits emmitted in incorrect order causing parent data to be lost from commits turning essentially linear repo into \"islands\"","fromName":"Michael J Gruber","fromEmail":"michaeljgruber+gmane@fastmail.fm","sentAt":"2008-06-12T12:52:40Z","receivedAt":"2008-06-12T12:52:40Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Yves Orton venit, vidit, dixit 12.06.2008 14:16:\n> We want a more or less linear repo as the result. This bug with\n> fast-export was the main showstopper in our efforts.  However, I can\n> imagine that this is a problem that many people will want to solve. It\n> would be nice if there was an easier way to do it that what we currently\n> are doing (merging and munging multiple fast-export streams into a\n> single fast-import process). While at this point its probably academic\n> any suggestions as to the Best Way to do this would be very much\n> welcome.\n\nI've done something like this, \"stitching\" the history of different \nrepos together in order to produce one repo, with each of the \nconstituents in a subdir. What I did was an adaption of\n\nhttp://www.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html\n\nbut as a multistep version:\n\n1. Create an empty repo\n2. Add your to-be-stitched repos as remotes, say A B C\n3. Create an empty commit\n4. \"git merge -s ours --no-commit a b c\", where a b c are the root \ncommits of A B C\n5. \"git read-tree --prefix=dir-A/ -u a\" and analogously for b c\n6. \"git commit\", use the common commit message of those commits\n\nNote that git refuses the merge (4.) into an empty (headless) repo, \nwhich is why you need 3. There may be smarter ways.\nIf you don't care about recording the commits as (octopus) merges you \ncan skip 3. and 4. (4. just records merge info in the index).\n\nThen, repeat:\n3'. remove dir-A etc. (I think I used git-rm, I'm sorry I can't recall).\n4. as above (if you want to record as merge)\n5. as above\n6. as above\n\nIf not all of A B C appear in every step then make sure to remove only \nthe ones (in 3'.) which you'll update in 5. You have to remove the dir \nbecause read-tree wants it like that.\n\nI used this for stitching 5 or 6 repos with a short history together, so \nI repeated these steps manually rather than scripting it; all I needed \nwas a list of SHA1s listing which commits from A B C etc. corresponded \nto the same \"step\" in the combined repo.\n\nCheers\nMichael\n"},{"id":"79596","messageId":"20080612140210.GE3830@plop","threadId":"13910","inReplyTo":"g2r66q$d3j$1@ger.gmane.org","subject":"Re: git-fast-export bug, commits emmitted in incorrect order causing parent data to be lost from commits turning essentially linear repo into \"islands\"","fromName":"Philippe Bruhat (BooK)","fromEmail":"philippe.bruhat@free.fr","sentAt":"2008-06-12T14:02:10Z","receivedAt":"2008-06-12T14:02:10Z","isPatch":false,"sender":{"key":"philippe.bruhat@free.fr","avatar":"https://gravatar.com/avatar/5e8b60cb2f5f2cd2671543f2f53dcb8f529490f98492a333f96636622eb3adef?d=mp&s=160"},"body":"On Thu, Jun 12, 2008 at 02:52:40PM +0200, Michael J Gruber wrote:\n> Yves Orton venit, vidit, dixit 12.06.2008 14:16:\n>> We want a more or less linear repo as the result. This bug with\n>> fast-export was the main showstopper in our efforts.  However, I can\n>> imagine that this is a problem that many people will want to solve. It\n>> would be nice if there was an easier way to do it that what we currently\n>> are doing (merging and munging multiple fast-export streams into a\n>> single fast-import process). While at this point its probably academic\n>> any suggestions as to the Best Way to do this would be very much\n>> welcome.\n>\n> I've done something like this, \"stitching\" the history of different  \n> repos together in order to produce one repo, with each of the  \n> constituents in a subdir. What I did was an adaption of\n>\n> http://www.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html\n>\n> but as a multistep version:\n\nWhat we did with Yves was a script doing the following:\n- run git fast-export --all (and --topo-order now) on all the repositories\n  we wanted to merge and read blocks from them\n- pass through all non-commit blocks (munging paths to put the content of\n  each repo in its own directory and renumbering marks to avoid clashes)\n- keep a list of the next commit sent by fast-export for each repo\n- select the oldest commit, and send it through, after stitching in the\n  right place (the point being to determine the \"right place\")\n\nActually, what we are trying to do is produce a single DAG from 2 or\nmore DAGs, while making sure that each \"internal DAG\" is the same.\n(I'm pretty sure this is all trivial stuff for graph mathematicians)\n\nImagine we merged repositories A, B and C in a new repo D, if we replace\nall nodes from D coming from B and C by vertexes, we will end up with\nthe original A graph.\n\nWe defined the \"right place\" as so: when having selected the next commit\nto add to our new graph, each of its new parents is defined by \"the last\nalien child of the original parent\" (or the original parent itself).\n\nFor example, if our new repository being built looks like:\n\n --A7--A8--B4--B5--A9--B7\n            \\\n             --B6\n\nIn this case, A9 was originally attached to A8, but to avoid unnecessary\nbranching in the new repo, we didn't attach it to A8, but to B5 (last\nalien child of A8, descending the tree in a leftmost manner).\n\nNo A node will ever be attached to B6. The next A node originally\nattached to A8 will be attached to B5 again, and one originally attached\nto A9 will be attached to B7. Like this:\n\n                 --A10\n                /\n --A7--A8--B4--B5--A9--B7--A11\n            \\\n             --B6\n\nNow, if we remove all B nodes, we get this:\n\n         --A10\n        /\n --A7--A8--A9--A11\n\nwhich is the original A graph.\n\nFinding the \"last alien child\" works fine with merges, too.\n\nOf course, some commits from A might end up on an unrelated branch of B,\nbut all B branches are irrelevant to A anyway! :-)\n\n-- \n Philippe Bruhat (BooK)\n\n People are all unique- but some are more unique than others.\n                                    (Moral from Groo The Wanderer #22 (Epic))\n"}]}