{"thread":{"id":"16465","subject":"git fast-export | git fast-import doesn't work","startedAt":"2008-11-25T16:44:28Z","lastAt":"2009-05-03T19:06:36Z","messageCount":25,"participants":["Ondrej Certik","Michael J Gruber","Peter Baumann","Miklos Vajna","david@lang.hm","Johannes Schindelin","Johannes Sixt","Andreas Ericsson","Alexander Gavrilov"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"96480","messageId":"85b5c3130811250844u498fbb97m9d1aef6e1397b8c7@mail.gmail.com","threadId":"16465","inReplyTo":null,"subject":"git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2008-11-25T16:44:28Z","receivedAt":"2008-11-25T16:44:28Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"Hi,\n\nI would like to export our whole git repository to patches, and then\nreconstruct it again from scratch. Following the man page of \"git\nfast-export\":\n\n$ git clone git://git.sympy.org/sympy-full-history-20081023.git\n$ cd sympy-full-history-20081023\n$ git fast-export --all --export-marks=marks > patches\n$ cd ..\n$ mkdir sympy-new\n$ cd sympy-new\n$ git init\n$ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches\ngit-fast-import statistics:\n---------------------------------------------------------------------\nAlloc'd objects: 25000\nTotal objects: 21355 ( 144 duplicates )\n blobs : 8009 ( 0 duplicates 4529 deltas)\n trees : 10627 ( 144 duplicates 9189 deltas)\n commits: 2719 ( 0 duplicates 0 deltas)\n tags : 0 ( 0 duplicates 0 deltas)\nTotal branches: 21 ( 26 loads )\n marks: 1048576 ( 10728 unique )\n atoms: 726\nMemory total: 2880 KiB\n pools: 2098 KiB\n objects: 781 KiB\n---------------------------------------------------------------------\npack_report: getpagesize() = 4096\npack_report: core.packedGitWindowSize = 33554432\npack_report: core.packedGitLimit = 268435456\npack_report: pack_used_ctr = 40706\npack_report: pack_mmap_calls = 2791\npack_report: pack_open_windows = 1 / 2\npack_report: pack_mapped = 26177739 / 35513414\n---------------------------------------------------------------------\n\n\n\nHowever, the repository is very different to the original one. It\ncontains only 191 patches:\n\n$ git log --pretty=oneline | wc -l\n191\n\nand it only contains couple files. Compare this with the original repository:\n\n$ git log --pretty=oneline | wc -l\n2719\n\nWhat am I doing wrong? Is there some other way to do it? I also tried\n\"git format-patch\" and \"git am\" and that almost works, only it changes\nhashes. Is there some way to tell \"git am\" to preserve the hash?\n\nThanks,\nOndrej\n"},{"id":"96482","messageId":"492C367D.3030209@drmicha.warpmail.net","threadId":"16465","inReplyTo":"85b5c3130811250844u498fbb97m9d1aef6e1397b8c7@mail.gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2008-11-25T17:31:41Z","receivedAt":"2008-11-25T17:31:41Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Ondrej Certik venit, vidit, dixit 25.11.2008 17:44:\n> Hi,\n> \n> I would like to export our whole git repository to patches, and then\n> reconstruct it again from scratch. Following the man page of \"git\n> fast-export\":\n> \n> $ git clone git://git.sympy.org/sympy-full-history-20081023.git\n> $ cd sympy-full-history-20081023\n> $ git fast-export --all --export-marks=marks > patches\n> $ cd ..\n> $ mkdir sympy-new\n> $ cd sympy-new\n> $ git init\n> $ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches\n> git-fast-import statistics:\n> ---------------------------------------------------------------------\n> Alloc'd objects: 25000\n> Total objects: 21355 ( 144 duplicates )\n> blobs : 8009 ( 0 duplicates 4529 deltas)\n> trees : 10627 ( 144 duplicates 9189 deltas)\n> commits: 2719 ( 0 duplicates 0 deltas)\n> tags : 0 ( 0 duplicates 0 deltas)\n> Total branches: 21 ( 26 loads )\n> marks: 1048576 ( 10728 unique )\n> atoms: 726\n> Memory total: 2880 KiB\n> pools: 2098 KiB\n> objects: 781 KiB\n> ---------------------------------------------------------------------\n> pack_report: getpagesize() = 4096\n> pack_report: core.packedGitWindowSize = 33554432\n> pack_report: core.packedGitLimit = 268435456\n> pack_report: pack_used_ctr = 40706\n> pack_report: pack_mmap_calls = 2791\n> pack_report: pack_open_windows = 1 / 2\n> pack_report: pack_mapped = 26177739 / 35513414\n> ---------------------------------------------------------------------\n> \n> \n> \n> However, the repository is very different to the original one. It\n> contains only 191 patches:\n> \n> $ git log --pretty=oneline | wc -l\n> 191\n> \n> and it only contains couple files. Compare this with the original repository:\n> \n> $ git log --pretty=oneline | wc -l\n> 2719\n\nI get the same stats (with the dups) but a perfect rev count, when I use\ngit log --all. The reason is that the history in the imported repo is\ndisconnected at various places (at tagging commits)! Your command counts\nonly the revs backwards to the first \"disconnection\".\n\nSo, the real issue is: Why has the result these cuts in the history?\nI don't know, I just noticed that turning on rename and copy detection\nmakes git-fast-import crash, which shouldn't happen either. Something's\nnot right here. CC'ing the authors of im- and export.\n\nBTW: Maybe you can accomplish what you want with different means? Why\nexport|import directly to git?\n\nMichael\n--\ngit 1.6.0.4.608.ga9645\n"},{"id":"96483","messageId":"20081125173447.GB13935@m62s10.vlinux.de","threadId":"16465","inReplyTo":"85b5c3130811250844u498fbb97m9d1aef6e1397b8c7@mail.gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Peter Baumann","fromEmail":"waste.manager@gmx.de","sentAt":"2008-11-25T17:34:47Z","receivedAt":"2008-11-25T17:34:47Z","isPatch":false,"sender":{"key":"waste.manager@gmx.de","avatar":null},"body":"On Tue, Nov 25, 2008 at 05:44:28PM +0100, Ondrej Certik wrote:\n> Hi,\n> \n> I would like to export our whole git repository to patches, and then\n> reconstruct it again from scratch. Following the man page of \"git\n> fast-export\":\n> \n\nPerhabs you are looking for git filter-branch, because it seems you want\nto change the history in some way (e.g. remove a wrongly committed file)?\n\nNevertheless, I expect your shown commands below to procude the same\nrepo again, so you might be on something ...\n\n-Peter\n\n\n> $ git clone git://git.sympy.org/sympy-full-history-20081023.git\n> $ cd sympy-full-history-20081023\n> $ git fast-export --all --export-marks=marks > patches\n> $ cd ..\n> $ mkdir sympy-new\n> $ cd sympy-new\n> $ git init\n> $ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches\n> git-fast-import statistics:\n> ---------------------------------------------------------------------\n> Alloc'd objects: 25000\n> Total objects: 21355 ( 144 duplicates )\n> blobs : 8009 ( 0 duplicates 4529 deltas)\n> trees : 10627 ( 144 duplicates 9189 deltas)\n> commits: 2719 ( 0 duplicates 0 deltas)\n> tags : 0 ( 0 duplicates 0 deltas)\n> Total branches: 21 ( 26 loads )\n> marks: 1048576 ( 10728 unique )\n> atoms: 726\n> Memory total: 2880 KiB\n> pools: 2098 KiB\n> objects: 781 KiB\n> ---------------------------------------------------------------------\n> pack_report: getpagesize() = 4096\n> pack_report: core.packedGitWindowSize = 33554432\n> pack_report: core.packedGitLimit = 268435456\n> pack_report: pack_used_ctr = 40706\n> pack_report: pack_mmap_calls = 2791\n> pack_report: pack_open_windows = 1 / 2\n> pack_report: pack_mapped = 26177739 / 35513414\n> ---------------------------------------------------------------------\n> \n> \n> \n> However, the repository is very different to the original one. It\n> contains only 191 patches:\n> \n> $ git log --pretty=oneline | wc -l\n> 191\n> \n> and it only contains couple files. Compare this with the original repository:\n> \n> $ git log --pretty=oneline | wc -l\n> 2719\n> \n> What am I doing wrong? Is there some other way to do it? I also tried\n> \"git format-patch\" and \"git am\" and that almost works, only it changes\n> hashes. Is there some way to tell \"git am\" to preserve the hash?\n> \n> Thanks,\n> Ondrej\n"},{"id":"96492","messageId":"20081125204108.GF4746@genesis.frugalware.org","threadId":"16465","inReplyTo":"492C367D.3030209@drmicha.warpmail.net","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2008-11-25T20:41:08Z","receivedAt":"2008-11-25T20:41:08Z","isPatch":false,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber wrote:\n> I don't know, I just noticed that turning on rename and copy detection\n> makes git-fast-import crash, which shouldn't happen either. Something's\n> not right here. CC'ing the authors of im- and export.\n\nCould you please write a testcase that reproduces your problem?\n\n> Why export|import directly to git?\n\nI guess he did not know about filter-branch. :)\n"},{"id":"96504","messageId":"85b5c3130811251539n6cb175b4p185d37385bf43d1e@mail.gmail.com","threadId":"16465","inReplyTo":"20081125204108.GF4746@genesis.frugalware.org","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2008-11-25T23:39:44Z","receivedAt":"2008-11-25T23:39:44Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"On Tue, Nov 25, 2008 at 9:41 PM, Miklos Vajna wrote:\n> On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber wrote:\n>> I don't know, I just noticed that turning on rename and copy detection\n>> makes git-fast-import crash, which shouldn't happen either. Something's\n>> not right here. CC'ing the authors of im- and export.\n>\n> Could you please write a testcase that reproduces your problem?\n>\n>> Why export|import directly to git?\n>\n> I guess he did not know about filter-branch. :)\n\nI know about filter-branch (but I am not sure it can do what I want).\nI made a mistake of not explaining what I want, instead I suggested (a\npossibly wrong) solution. I want to export the whole git repository as\na set of human readable patches, that can be assembled back into a git\nrepository (with the same hashes as the original one) if needed. The\nreason I want that is that if we later decide to switch to another\nVCS, we have all the information to reproduce the repository. Another\nreason is to be sure that we know all the sources that are needed to\nconstruct the repository, e.g. that there are no binary blobs\n(possibly containing malicious code). Another reason I want that is to\nbe able to rewrite the history, in particular, we have one Mercurial\nrepository with some old history and another Mercurial history with a\nnewer history and I just want to concatenate them together into one\ngit repository.\n\nIn each case I know several workarounds, but if there is a way to just\nconvert the whole git repository into a set of patches and (and be\nable to convert everything back including the same hashes), then it'd\nbe awesome.\n\nSee also this thread why people want this (and I assumed git can do\nthis from this thread):\n\nhttp://groups.google.com/group/sage-devel/browse_thread/thread/7b116d902ee20d9c/\n\nThanks,\nOndrej\n"},{"id":"96505","messageId":"85b5c3130811251544u5423c69p87c11241a2685f2f@mail.gmail.com","threadId":"16465","inReplyTo":"492C367D.3030209@drmicha.warpmail.net","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2008-11-25T23:44:00Z","receivedAt":"2008-11-25T23:44:00Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"On Tue, Nov 25, 2008 at 6:31 PM, Michael J Gruber\n wrote:\n> Ondrej Certik venit, vidit, dixit 25.11.2008 17:44:\n>> Hi,\n>>\n>> I would like to export our whole git repository to patches, and then\n>> reconstruct it again from scratch. Following the man page of \"git\n>> fast-export\":\n>>\n>> $ git clone git://git.sympy.org/sympy-full-history-20081023.git\n>> $ cd sympy-full-history-20081023\n>> $ git fast-export --all --export-marks=marks > patches\n>> $ cd ..\n>> $ mkdir sympy-new\n>> $ cd sympy-new\n>> $ git init\n>> $ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches\n>> git-fast-import statistics:\n>> ---------------------------------------------------------------------\n>> Alloc'd objects: 25000\n>> Total objects: 21355 ( 144 duplicates )\n>> blobs : 8009 ( 0 duplicates 4529 deltas)\n>> trees : 10627 ( 144 duplicates 9189 deltas)\n>> commits: 2719 ( 0 duplicates 0 deltas)\n>> tags : 0 ( 0 duplicates 0 deltas)\n>> Total branches: 21 ( 26 loads )\n>> marks: 1048576 ( 10728 unique )\n>> atoms: 726\n>> Memory total: 2880 KiB\n>> pools: 2098 KiB\n>> objects: 781 KiB\n>> ---------------------------------------------------------------------\n>> pack_report: getpagesize() = 4096\n>> pack_report: core.packedGitWindowSize = 33554432\n>> pack_report: core.packedGitLimit = 268435456\n>> pack_report: pack_used_ctr = 40706\n>> pack_report: pack_mmap_calls = 2791\n>> pack_report: pack_open_windows = 1 / 2\n>> pack_report: pack_mapped = 26177739 / 35513414\n>> ---------------------------------------------------------------------\n>>\n>>\n>>\n>> However, the repository is very different to the original one. It\n>> contains only 191 patches:\n>>\n>> $ git log --pretty=oneline | wc -l\n>> 191\n>>\n>> and it only contains couple files. Compare this with the original repository:\n>>\n>> $ git log --pretty=oneline | wc -l\n>> 2719\n>\n> I get the same stats (with the dups) but a perfect rev count, when I use\n> git log --all. The reason is that the history in the imported repo is\n> disconnected at various places (at tagging commits)! Your command counts\n> only the revs backwards to the first \"disconnection\".\n\nYou are right! I didn't know about \"git log --all\".\n\n>\n> So, the real issue is: Why has the result these cuts in the history?\n\nYes, I would like to know this too. E.g. if it is a problem with our\nrepository, or a problem in git, or whether it is just not supposed to\nwork.\n\n> I don't know, I just noticed that turning on rename and copy detection\n> makes git-fast-import crash, which shouldn't happen either. Something's\n> not right here. CC'ing the authors of im- and export.\n>\n> BTW: Maybe you can accomplish what you want with different means? Why\n> export|import directly to git?\n\nI just answered this in my other email. Basically there are\nworkarounds, but I would feel safe if I can (correctly) reconstruct\nthe whole git repository from a human readable patches.\n\nOndrej\n"},{"id":"96507","messageId":"alpine.DEB.1.10.0811251548280.20120@asgard.lang.hm","threadId":"16465","inReplyTo":"85b5c3130811251539n6cb175b4p185d37385bf43d1e@mail.gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"","fromEmail":"david@lang.hm","sentAt":"2008-11-25T23:53:16Z","receivedAt":"2008-11-25T23:53:16Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Wed, 26 Nov 2008, Ondrej Certik wrote:\n\n> On Tue, Nov 25, 2008 at 9:41 PM, Miklos Vajna wrote:\n>> On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber wrote:\n>>> I don't know, I just noticed that turning on rename and copy detection\n>>> makes git-fast-import crash, which shouldn't happen either. Something's\n>>> not right here. CC'ing the authors of im- and export.\n>>\n>> Could you please write a testcase that reproduces your problem?\n>>\n>>> Why export|import directly to git?\n>>\n>> I guess he did not know about filter-branch. :)\n>\n> I know about filter-branch (but I am not sure it can do what I want).\n> I made a mistake of not explaining what I want, instead I suggested (a\n> possibly wrong) solution. I want to export the whole git repository as\n> a set of human readable patches, that can be assembled back into a git\n> repository (with the same hashes as the original one)\n\nthe same hashes on the file is easy, the same hashes on the commits is \nextremely hard. which is it that you are looking for.\n\nthat being said, the test of being able to do a export|import is a good \none to test that the export format and import format actually match.\n\nDavid Lang\n\n> if needed. The\n> reason I want that is that if we later decide to switch to another\n> VCS, we have all the information to reproduce the repository. Another\n> reason is to be sure that we know all the sources that are needed to\n> construct the repository, e.g. that there are no binary blobs\n> (possibly containing malicious code). Another reason I want that is to\n> be able to rewrite the history, in particular, we have one Mercurial\n> repository with some old history and another Mercurial history with a\n> newer history and I just want to concatenate them together into one\n> git repository.\n>\n> In each case I know several workarounds, but if there is a way to just\n> convert the whole git repository into a set of patches and (and be\n> able to convert everything back including the same hashes), then it'd\n> be awesome.\n>\n> See also this thread why people want this (and I assumed git can do\n> this from this thread):\n>\n> http://groups.google.com/group/sage-devel/browse_thread/thread/7b116d902ee20d9c/\n>\n> Thanks,\n> Ondrej\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"96508","messageId":"alpine.DEB.1.00.0811260113140.30769@pacific.mpi-cbg.de","threadId":"16465","inReplyTo":"85b5c3130811250844u498fbb97m9d1aef6e1397b8c7@mail.gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-11-26T00:14:37Z","receivedAt":"2008-11-26T00:14:37Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 25 Nov 2008, Ondrej Certik wrote:\n\n> I would like to export our whole git repository to patches, and then \n> reconstruct it again from scratch. Following the man page of \"git \n> fast-export\":\n>\n> [...] \n> \n> However, the repository is very different to the original one. It \n> contains only 191 patches:\n\nCan you try again with a Git version that contains the commit \n2075ffb5(fast-export: use an unsorted string list for extra_refs)?\n\nCiao,\nDscho\n"},{"id":"96527","messageId":"85b5c3130811260135g4646bf72iaf57f599fdd21a0c@mail.gmail.com","threadId":"16465","inReplyTo":"alpine.DEB.1.00.0811260113140.30769@pacific.mpi-cbg.de","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2008-11-26T09:35:37Z","receivedAt":"2008-11-26T09:35:37Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin\n wrote:\n> Hi,\n>\n> On Tue, 25 Nov 2008, Ondrej Certik wrote:\n>\n>> I would like to export our whole git repository to patches, and then\n>> reconstruct it again from scratch. Following the man page of \"git\n>> fast-export\":\n>>\n>> [...]\n>>\n>> However, the repository is very different to the original one. It\n>> contains only 191 patches:\n>\n> Can you try again with a Git version that contains the commit\n> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?\n\nI tried the next branch:\n\n$ git --version\ngit version 1.6.0.4.1060.g9433b\n\nthat contains the 2075ffb5 patch. I haven't observed any change ---\nthe \"git log\" still only shows 191 commits (git log --all shows\neverything).\n\nOndrej\n"},{"id":"96532","messageId":"85b5c3130811260218s7529914eyb56a05ec1ca34b8f@mail.gmail.com","threadId":"16465","inReplyTo":"85b5c3130811260135g4646bf72iaf57f599fdd21a0c@mail.gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2008-11-26T10:18:35Z","receivedAt":"2008-11-26T10:18:35Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik wrote:\n> On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin\n> wrote:\n>> Hi,\n>>\n>> On Tue, 25 Nov 2008, Ondrej Certik wrote:\n>>\n>>> I would like to export our whole git repository to patches, and then\n>>> reconstruct it again from scratch. Following the man page of \"git\n>>> fast-export\":\n>>>\n>>> [...]\n>>>\n>>> However, the repository is very different to the original one. It\n>>> contains only 191 patches:\n>>\n>> Can you try again with a Git version that contains the commit\n>> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?\n>\n> I tried the next branch:\n>\n> $ git --version\n> git version 1.6.0.4.1060.g9433b\n>\n> that contains the 2075ffb5 patch. I haven't observed any change ---\n> the \"git log\" still only shows 191 commits (git log --all shows\n> everything).\n\nI deleted all tags and then fast-exported and imported, now all the\ncommits show in \"git log\", however, the patches are wrongly connected.\nBasically, both repositories are identical (including hashes) up to\nthis commit:\n\nd717177d4 (fixed downloads instructions in the README and a typo)\n\nHowever, the original repo (sympy-full-history-20081023) contains 3\nchildren at this commit:\n\nParent: fecac34251934e98a05631440d3ce151585f2391 (David added to credits)\nChild: 03ccb60798d62f94ac9d2ec9472dc7333f67b420 (Allow to specify\nline width in 2D plotting.)\nChild: 203124d834488781db5429d941eeb60e396990c8 (credits improvements)\nChild: 77146885f1b7aa49184f27c2297488c3d1201106 (Speed \"import sympy\"\nup as in the last release.)\n\nbut the newly created repository only 2:\n\nParent: fecac34251934e98a05631440d3ce151585f2391 (David added to credits)\nChild: 203124d834488781db5429d941eeb60e396990c8 (credits improvements)\nChild: 77146885f1b7aa49184f27c2297488c3d1201106 (Speed \"import sympy\"\nup as in the last release.)\n\n\nAnd from that point on, the hashes mishmatch and sometimes the commits\nare just wrongly connected (e.g. for example d2dc6b3's parent is\n0adafe3, but 0adafe3 was committed half a year later after\nd2dc6b3...), so it's a mess. Also the checkouted files are not\ncomplete.\n\n\nNow, if you look at the \"patches\" file, to which I saved the results\nof \"git fast-export\", you can find that the commit d717177d4 (fixed\ndownloads instructions in the README and a typo) has the mark :6540,\nand if you search for this mark in the patches file, you can only find\n2 children:\n\ncommit refs/heads/master\nmark :6542\nauthor Ondrej Certik 1198803347 +0100\ncommitter Ondrej Certik 1198803347 +0100\ndata 21\ncredits improvements\nfrom :6540\nM 100644 :6541 README\n\n\nand:\n\ncommit refs/heads/master\nmark :6551\nauthor Ondrej Certik 1198951670 +0100\ncommitter Ondrej Certik 1198951670 +0100\ndata 48\nSpeed \"import sympy\" up as in the last release.\nfrom :6540\nM 100644 :6550 sympy/printing/preview.py\n\n\nhowever, the third child doesn't contain the \"from :6540\":\n\ncommit refs/heads/master\nmark :24\nauthor Ondrej Certik 1198798384 +0100\ncommitter Ondrej Certik 1198798384 +0100\ndata 101\nAllow to specify line width in 2D plotting.\n\n\nSo imho that's a bug in git fast-export. What do you think?\n\nOndrej\n"},{"id":"96533","messageId":"492D227C.8020805@drmicha.warpmail.net","threadId":"16465","inReplyTo":"alpine.DEB.1.00.0811260113140.30769@pacific.mpi-cbg.de","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2008-11-26T10:18:36Z","receivedAt":"2008-11-26T10:18:36Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:\n> Hi,\n> \n> On Tue, 25 Nov 2008, Ondrej Certik wrote:\n> \n>> I would like to export our whole git repository to patches, and then \n>> reconstruct it again from scratch. Following the man page of \"git \n>> fast-export\":\n>>\n>> [...] \n>>\n>> However, the repository is very different to the original one. It \n>> contains only 191 patches:\n> \n> Can you try again with a Git version that contains the commit \n> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?\n\nWith that commit cherry-picked onto today's master I get the same effect:\n\nA) git fast-import crashes with dump generated by git fast-export -M -C\nB) git fast-import produces disconnected DAG with dump generated by git\nfast-export\n\nWith git 1.5.4, A) is good but B) is still bad.\nBisecting gives me (after remembering to put make in the run script,\nuhm...)...\n\n...\n\n...tadah:\n\n\nae7c5dcef92d46cfc8987fde2c264614fe475bd1 is first bad commit\ncommit ae7c5dcef92d46cfc8987fde2c264614fe475bd1\nAuthor: Alexander Gavrilov \nDate: Sun Jul 27 00:52:54 2008 +0400\n\n Support copy and rename detection in fast-export.\n\nOh well :(\n\nMichael\n"},{"id":"96541","messageId":"alpine.DEB.1.00.0811261344120.30769@pacific.mpi-cbg.de","threadId":"16465","inReplyTo":"492D227C.8020805@drmicha.warpmail.net","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-11-26T12:46:39Z","receivedAt":"2008-11-26T12:46:39Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ondrej & Michael,\n\nOn Wed, 26 Nov 2008, Michael J Gruber wrote:\n\n> Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:\n> \n> > Can you try again with a Git version that contains the commit \n> > 2075ffb5(fast-export: use an unsorted string list for extra_refs)?\n\nOkay, so both of your use cases seem to be real bugs in fast-export. May \nI respectfully submit a request for a test script (as patch to \nt/t9301-fast-export.sh) which is as short as possible and shows the \nrespective bugs?\n\nMy Git time budget these days is almost negative, so I will not be able to \nwork on these issues without having a small and concise example.\n\nThanks so much,\nDscho\n"},{"id":"96542","messageId":"85b5c3130811260503k445b0debr2f681f9766cfc361@mail.gmail.com","threadId":"16465","inReplyTo":"alpine.DEB.1.00.0811261344120.30769@pacific.mpi-cbg.de","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2008-11-26T13:03:30Z","receivedAt":"2008-11-26T13:03:30Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"On Wed, Nov 26, 2008 at 1:46 PM, Johannes Schindelin\n wrote:\n> Hi Ondrej & Michael,\n>\n> On Wed, 26 Nov 2008, Michael J Gruber wrote:\n>\n>> Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:\n>>\n>> > Can you try again with a Git version that contains the commit\n>> > 2075ffb5(fast-export: use an unsorted string list for extra_refs)?\n>\n> Okay, so both of your use cases seem to be real bugs in fast-export. May\n> I respectfully submit a request for a test script (as patch to\n> t/t9301-fast-export.sh) which is as short as possible and shows the\n> respective bugs?\n>\n> My Git time budget these days is almost negative, so I will not be able to\n> work on these issues without having a small and concise example.\n\nI'll do my best and try to create a simple example. I'll report back\nif I have something.\n\nOndrej\n"},{"id":"96551","messageId":"492D6CC3.2050408@drmicha.warpmail.net","threadId":"16465","inReplyTo":"85b5c3130811260218s7529914eyb56a05ec1ca34b8f@mail.gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2008-11-26T15:35:31Z","receivedAt":"2008-11-26T15:35:31Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Ondrej Certik venit, vidit, dixit 26.11.2008 11:18:\n> On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik wrote:\n>> On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin\n>> wrote:\n>>> Hi,\n>>>\n>>> On Tue, 25 Nov 2008, Ondrej Certik wrote:\n>>>\n>>>> I would like to export our whole git repository to patches, and then\n>>>> reconstruct it again from scratch. Following the man page of \"git\n>>>> fast-export\":\n>>>>\n>>>> [...]\n>>>>\n>>>> However, the repository is very different to the original one. It\n>>>> contains only 191 patches:\n>>> Can you try again with a Git version that contains the commit\n>>> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?\n>> I tried the next branch:\n>>\n>> $ git --version\n>> git version 1.6.0.4.1060.g9433b\n>>\n>> that contains the 2075ffb5 patch. I haven't observed any change ---\n>> the \"git log\" still only shows 191 commits (git log --all shows\n>> everything).\n> \n> I deleted all tags and then fast-exported and imported, now all the\n> commits show in \"git log\", however, the patches are wrongly connected.\n> Basically, both repositories are identical (including hashes) up to\n> this commit:\n> \n> d717177d4 (fixed downloads instructions in the README and a typo)\n> \n> However, the original repo (sympy-full-history-20081023) contains 3\n> children at this commit:\n\nThere's some nice 3 way branching and double 2 way merging going on. I\ncut out the interesting part of the graph, making d717177d4 and\n6e869485f parentless. The resulting mini DAG is reproduced correctly by\nexport|import, even with -M -C.\n\nMichael\n"},{"id":"96552","messageId":"85b5c3130811260750y2e24436ye2426ccfc2f66071@mail.gmail.com","threadId":"16465","inReplyTo":"492D6CC3.2050408@drmicha.warpmail.net","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2008-11-26T15:50:14Z","receivedAt":"2008-11-26T15:50:14Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"On Wed, Nov 26, 2008 at 4:35 PM, Michael J Gruber\n wrote:\n> Ondrej Certik venit, vidit, dixit 26.11.2008 11:18:\n>> On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik wrote:\n>>> On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin\n>>> wrote:\n>>>> Hi,\n>>>>\n>>>> On Tue, 25 Nov 2008, Ondrej Certik wrote:\n>>>>\n>>>>> I would like to export our whole git repository to patches, and then\n>>>>> reconstruct it again from scratch. Following the man page of \"git\n>>>>> fast-export\":\n>>>>>\n>>>>> [...]\n>>>>>\n>>>>> However, the repository is very different to the original one. It\n>>>>> contains only 191 patches:\n>>>> Can you try again with a Git version that contains the commit\n>>>> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?\n>>> I tried the next branch:\n>>>\n>>> $ git --version\n>>> git version 1.6.0.4.1060.g9433b\n>>>\n>>> that contains the 2075ffb5 patch. I haven't observed any change ---\n>>> the \"git log\" still only shows 191 commits (git log --all shows\n>>> everything).\n>>\n>> I deleted all tags and then fast-exported and imported, now all the\n>> commits show in \"git log\", however, the patches are wrongly connected.\n>> Basically, both repositories are identical (including hashes) up to\n>> this commit:\n>>\n>> d717177d4 (fixed downloads instructions in the README and a typo)\n>>\n>> However, the original repo (sympy-full-history-20081023) contains 3\n>> children at this commit:\n>\n> There's some nice 3 way branching and double 2 way merging going on. I\n> cut out the interesting part of the graph, making d717177d4 and\n> 6e869485f parentless. The resulting mini DAG is reproduced correctly by\n> export|import, even with -M -C.\n\nI am also trying to make the example simpler. I tried to squash the\nfirst uninteresting ~1500 commits into one, but \"git rebase -i\"\nuterrly fails after squashing about 600 commits. Still investigating.\n\nOndrej\n"},{"id":"96556","messageId":"492D7AEF.5030908@viscovery.net","threadId":"16465","inReplyTo":"85b5c3130811260750y2e24436ye2426ccfc2f66071@mail.gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-11-26T16:35:59Z","receivedAt":"2008-11-26T16:35:59Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Ondrej Certik schrieb:\n> I am also trying to make the example simpler. I tried to squash the\n> first uninteresting ~1500 commits into one, but \"git rebase -i\"\n> uterrly fails after squashing about 600 commits. Still investigating.\n\nDon't use rebase. Set a graft and rewrite the history:\n\n $ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \\\n\t\t.git/info/grafts\n\nAssuming \"first 1500\" means the \"most recent 1500\" commits. But you get\nthe idea. You can truncate history as well by omitting the second SHA1.\nIt's very convenient to keep gitk open and File->Reload after each graft\nthat you set.\n\nWhen you're done with setting grafts:\n\n $ git filter-branch -f --tag-name-filter cat -- --all\n\n(You are doing this on a copy of your repository, don't you?)\n\n-- Hannes\n"},{"id":"96555","messageId":"alpine.DEB.1.00.0811261739110.30769@pacific.mpi-cbg.de","threadId":"16465","inReplyTo":"85b5c3130811260750y2e24436ye2426ccfc2f66071@mail.gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-11-26T16:40:58Z","receivedAt":"2008-11-26T16:40:58Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 26 Nov 2008, Ondrej Certik wrote:\n\n> I am also trying to make the example simpler. I tried to squash the \n> first uninteresting ~1500 commits into one, but \"git rebase -i\" uterrly \n> fails after squashing about 600 commits. Still investigating.\n\n1500... wow.\n\nThe best idea would probably be to just \"edit\" the first, delete the rest \nof the 1500, and then 'git read-tree -u -m \"' on \nthe command line (when git rebase stops after the \"edit\" command).\n\nrebase -i was _never_ meant for such _massive_ interactions; that's just \ntoo much for a shell script.\n\nCiao,\nDscho\n"},{"id":"96557","messageId":"492D7CF5.1020202@drmicha.warpmail.net","threadId":"16465","inReplyTo":"alpine.DEB.1.00.0811261739110.30769@pacific.mpi-cbg.de","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2008-11-26T16:44:37Z","receivedAt":"2008-11-26T16:44:37Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Johannes Schindelin venit, vidit, dixit 26.11.2008 17:40:\n> Hi,\n> \n> On Wed, 26 Nov 2008, Ondrej Certik wrote:\n> \n>> I am also trying to make the example simpler. I tried to squash the \n>> first uninteresting ~1500 commits into one, but \"git rebase -i\" uterrly \n>> fails after squashing about 600 commits. Still investigating.\n> \n> 1500... wow.\n> \n> The best idea would probably be to just \"edit\" the first, delete the rest \n> of the 1500, and then 'git read-tree -u -m \"' on \n> the command line (when git rebase stops after the \"edit\" command).\n> \n> rebase -i was _never_ meant for such _massive_ interactions; that's just \n> too much for a shell script.\n\nOr chop the DAG with grafts.\n\nRemoving the tags one by one I noticed that for several of them, removal\nof the tag increases the number of commits on the connected DAG\ncomponent containing master (in the ex/imported repo), and that one\nreaches the correct number with still a few tags left in there. Yet, the\ntopology is wrong in several places; I think all of them can be\nattributed to missing parent info (which even creates new roots in some\nplaces).\n\nLooking at the source I suspect that fast-export fails to denote\nparenthood in the case of yet unmarked parents (last for-loop of\nhandle_commit() in builtin_fast_export.c). But I don't really know that\ncode at all.\n\nMichael\n\nP.S.: That git repo itself is a product of\nhg-fast-export|git-fast-import, right?\n"},{"id":"96559","messageId":"alpine.DEB.1.00.0811261804550.30769@pacific.mpi-cbg.de","threadId":"16465","inReplyTo":"492D7CF5.1020202@drmicha.warpmail.net","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-11-26T17:08:54Z","receivedAt":"2008-11-26T17:08:54Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 26 Nov 2008, Michael J Gruber wrote:\n\n> Looking at the source I suspect that fast-export fails to denote \n> parenthood in the case of yet unmarked parents (last for-loop of \n> handle_commit() in builtin_fast_export.c). But I don't really know that \n> code at all.\n\nI strongly doubt so. Noticed the use of has_unshown_parent(commit) in \nboth cases before calling handle_commit()?\n\nIn any case, here is a script that I wrote _long_ time ago, to be able to \nreconstruct history from the output of \"git rev-list --all --parents\". \nMaybe this helps you in reconstructing something that is handled \nincorrectly by fast-export | fast-import, but is lighter than a full-blown \nrepository.\n\n-- snip --\n#!/bin/sh\n\n# Given the output of git-rev-list, this reconstructs the DAG of the \nhistory\n\ni=0\ntac | while read rev parents; do\n let i=$i+1\n echo $i > a1\n git add a1\n tree=$(git write-tree)\n parents=\"$(for parent in $parents\n do\n echo -n \"-p $(git rev-parse sp-$parent) \"\n done)\"\n commit=$(echo \"$rev $i\" | git commit-tree $tree $parents)\n git tag sp-$rev $commit\ndone\n-- snap --\n\nCiao,\nDscho\n\n--\n\n |\n\nCeci n'est pas une pipe\n"},{"id":"96561","messageId":"85b5c3130811260921s474bc724hb74b54e21e8be912@mail.gmail.com","threadId":"16465","inReplyTo":"alpine.DEB.1.00.0811261739110.30769@pacific.mpi-cbg.de","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2008-11-26T17:21:32Z","receivedAt":"2008-11-26T17:21:32Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"On Wed, Nov 26, 2008 at 5:40 PM, Johannes Schindelin\n wrote:\n> Hi,\n>\n> On Wed, 26 Nov 2008, Ondrej Certik wrote:\n>\n>> I am also trying to make the example simpler. I tried to squash the\n>> first uninteresting ~1500 commits into one, but \"git rebase -i\" uterrly\n>> fails after squashing about 600 commits. Still investigating.\n>\n> 1500... wow.\n>\n> The best idea would probably be to just \"edit\" the first, delete the rest\n> of the 1500, and then 'git read-tree -u -m \"' on\n> the command line (when git rebase stops after the \"edit\" command).\n\nThat worked, thanks! My original repo:\n\nA -- B -- ... --- D --- E --- ...\n\nwhere E and the rest of the commits (there are branches and merges in\nthere) are the ones that I need to preserve, but all the commits\nbetween B and D can be squashed (~1500 of them). So I created a\nbranch:\n\nA -- B -- ... --- D\n\nthen squashed the commits using the technique you described above, so\nnow I have:\n\nA -- BD --\n\nand now I would like to append \"E -- ...\" to it -- is there any way to\ndo that? I tried rebase, but that destroys all the branches and merges\nand those are necessary to reproduce the fast-export bug.\n\n>\n> rebase -i was _never_ meant for such _massive_ interactions; that's just\n> too much for a shell script.\n\nIn fact, I think it would work, but there is probably another bug,\nthat I am hitting, maybe due to whitespace problems --- in the\noriginal repository, the patches are linear, but when I create a\nbranch before the failing patch and then cherry-pick it (that should\nwork), it fails and creates conflicts. The same behavior is with git\nrebase. So I'll investigate more and report it in a separate thread,\nas it is not related to fast-export.\n\nOndrej\n"},{"id":"96597","messageId":"85b5c3130811261805n5628ee7agcff3d6ed7dea10bb@mail.gmail.com","threadId":"16465","inReplyTo":"492D7AEF.5030908@viscovery.net","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2008-11-27T02:05:16Z","receivedAt":"2008-11-27T02:05:16Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"Hi Johannes!\n\nOn Wed, Nov 26, 2008 at 5:35 PM, Johannes Sixt wrote:\n> Ondrej Certik schrieb:\n>> I am also trying to make the example simpler. I tried to squash the\n>> first uninteresting ~1500 commits into one, but \"git rebase -i\"\n>> uterrly fails after squashing about 600 commits. Still investigating.\n>\n> Don't use rebase. Set a graft and rewrite the history:\n>\n> $ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \\\n> .git/info/grafts\n>\n> Assuming \"first 1500\" means the \"most recent 1500\" commits. But you get\n> the idea. You can truncate history as well by omitting the second SHA1.\n> It's very convenient to keep gitk open and File->Reload after each graft\n> that you set.\n>\n> When you're done with setting grafts:\n>\n> $ git filter-branch -f --tag-name-filter cat -- --all\n\nIndeed, this seems to be working robustly. Thanks!\n\n>\n> (You are doing this on a copy of your repository, don't you?)\n\nYes.\n\n\nI spent the whole today trying to isolate the bug, but so far I\nhaven't succeeded. Unfortunately, I need to work on other things now,\nso I am postponing this to some later time. The repository that\nreproduces it will stay online, so anyone feel free to produce a nice\nand simple (failing) test for the bug.\n\nThanks,\nOndrej\n"},{"id":"96609","messageId":"492E57D6.3010703@op5.se","threadId":"16465","inReplyTo":"85b5c3130811260921s474bc724hb74b54e21e8be912@mail.gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2008-11-27T08:18:30Z","receivedAt":"2008-11-27T08:18:30Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Ondrej Certik wrote:\n> On Wed, Nov 26, 2008 at 5:40 PM, Johannes Schindelin\n> wrote:\n>> Hi,\n>>\n>> On Wed, 26 Nov 2008, Ondrej Certik wrote:\n>>\n>>> I am also trying to make the example simpler. I tried to squash the\n>>> first uninteresting ~1500 commits into one, but \"git rebase -i\" uterrly\n>>> fails after squashing about 600 commits. Still investigating.\n>> 1500... wow.\n>>\n>> The best idea would probably be to just \"edit\" the first, delete the rest\n>> of the 1500, and then 'git read-tree -u -m \"' on\n>> the command line (when git rebase stops after the \"edit\" command).\n> \n> That worked, thanks! My original repo:\n> \n> A -- B -- ... --- D --- E --- ...\n> \n> where E and the rest of the commits (there are branches and merges in\n> there) are the ones that I need to preserve, but all the commits\n> between B and D can be squashed (~1500 of them). So I created a\n> branch:\n> \n> A -- B -- ... --- D\n> \n> then squashed the commits using the technique you described above, so\n> now I have:\n> \n> A -- BD --\n> \n> and now I would like to append \"E -- ...\" to it -- is there any way to\n> do that? I tried rebase, but that destroys all the branches and merges\n> and those are necessary to reproduce the fast-export bug.\n> \n\n git rebase -p\n\nIf your git is old, you'll need\n\n git rebase -i -p\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"97301","messageId":"200812071425.52908.angavrilov@gmail.com","threadId":"16465","inReplyTo":"alpine.DEB.1.00.0811261804550.30769@pacific.mpi-cbg.de","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Alexander Gavrilov","fromEmail":"angavrilov@gmail.com","sentAt":"2008-12-07T11:25:52Z","receivedAt":"2008-12-07T11:25:52Z","isPatch":false,"sender":{"key":"angavrilov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42666?v=4"},"body":"On Wednesday 26 November 2008 20:08:54 Johannes Schindelin wrote:\n> On Wed, 26 Nov 2008, Michael J Gruber wrote:\n> > Looking at the source I suspect that fast-export fails to denote \n> > parenthood in the case of yet unmarked parents (last for-loop of \n> > handle_commit() in builtin_fast_export.c). But I don't really know that \n> > code at all.\n> \n> I strongly doubt so. Noticed the use of has_unshown_parent(commit) in \n> both cases before calling handle_commit()?\n> \n> In any case, here is a script that I wrote _long_ time ago, to be able to \n> reconstruct history from the output of \"git rev-list --all --parents\". \n> Maybe this helps you in reconstructing something that is handled \n> incorrectly by fast-export | fast-import, but is lighter than a full-blown \n> repository.\n\nToday I had time to investigate this problem, and found:\n\n1) The root of the problem is that fast-export really wants to walk\n revisions in topological order, but actually receives them in date\n order. While it is usually a good guess at topology, this repository\n contains some children that are older than their parent commits,\n e.g. see dd22c7d51a4debf18a3b2e35c61a1fec0175e4e0\n\n2) It tries to fix minor deviations by checking the SHOWN flag.\n However, it still breaks in two ways:\n\n a) SHOWN is apparently set by simply walking the commits, so\n if the parent was earlier encountered on a different branch of\n the DAG, it will be handled as already shown. Basically, this\n check is only good to determine if we reached the end of the\n chain, and should start to backtrack.\n\n b) If I modify the code to use a completely separate flag, it still\n doesn't work, because the commits are placed on the stack in\n the wrong order, so handle_tail gets stuck, and fails to unwind\n it completely.\n\nSo, apparently, the only way to fix it is to require topological order:\n\n\ndiff --git a/builtin-fast-export.c b/builtin-fast-export.c\nindex 7d5d57a..d9261fa 100644\n--- a/builtin-fast-export.c\n+++ b/builtin-fast-export.c\n@@ -490,6 +490,7 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n git_config(git_default_config, NULL);\n\n init_revisions(&revs, prefix);\n+ revs.topo_order = 1; /* force topological ordering */\n argc = setup_revisions(argc, argv, &revs, NULL);\n argc = parse_options(argc, argv, options, fast_export_usage, 0);\n if (argc > 1)\n\n\n(It is also possible to specify --topo-order on the command line)\n\nAs for the failure to import the output of fast-export with copy detection,\nit is a natural consequence of a messed up order of commits, because\ncopy and move commands depend on the original file being there.\n\nThe attachment contains a (proper) export of a simplified sample of the\nstructure around dd22c7d51a4d, which clearly reproduces the problem\nbecause all of the timestamps were made identical as a side effect of\nsimplification. By crafting timestamps it is probably possible to minimize\nit even further.\n\nAlexander\n\n\nblob\nmark :1\ndata 5\n2696\n\nreset refs/heads/test\ncommit refs/heads/test\nmark :2\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\ne4644a29ab6cb1993fc0dab55c320b7f2076e17b 2696\nM 100644 :1 foo/a1\n\nblob\nmark :3\ndata 5\n2697\n\ncommit refs/heads/test\nmark :4\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\nd558c3feba34d69e8accc990267d60c3a11644c1 2697\nfrom :2\nM 100644 :3 foo/a1\n\nblob\nmark :5\ndata 5\n2698\n\ncommit refs/heads/test\nmark :6\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n2e24b35039e9253a9089162f40113e28069cce56 2698\nfrom :4\nM 100644 :5 foo/a1\n\nblob\nmark :7\ndata 5\n2699\n\ncommit refs/heads/test\nmark :8\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n0f78bea330e3a3b32f9204dfbdb9ca466ef78962 2699\nfrom :6\nM 100644 :7 foo/a1\n\nblob\nmark :9\ndata 5\n2700\n\ncommit refs/heads/test\nmark :10\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n681ea67a40527f192bf279ea6346004120c64702 2700\nfrom :8\nM 100644 :9 foo/a1\n\nblob\nmark :11\ndata 5\n2701\n\ncommit refs/heads/test\nmark :12\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n7faf0e5fb3c060385859caead8692d37c9700e55 2701\nfrom :10\nM 100644 :11 foo/a1\n\nblob\nmark :13\ndata 5\n2702\n\ncommit refs/heads/test\nmark :14\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\nd6ea85387203d59a9cddda6746bd3266598dc91d 2702\nfrom :12\nM 100644 :13 foo/a1\n\nblob\nmark :15\ndata 5\n2703\n\ncommit refs/heads/test\nmark :16\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\nfa22fab1d6d4cc192e288696fcb8947ebfd9e4a1 2703\nfrom :14\nM 100644 :15 foo/a1\n\nblob\nmark :17\ndata 5\n2704\n\ncommit refs/heads/test\nmark :18\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\ndd22c7d51a4debf18a3b2e35c61a1fec0175e4e0 2704\nfrom :16\nM 100644 :17 foo/a1\n\nblob\nmark :19\ndata 5\n2705\n\ncommit refs/heads/test\nmark :20\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n84e2addc90ccf17d1cfcf657c241cf6cda5d46c1 2705\nfrom :18\nM 100644 :19 foo/a1\n\nblob\nmark :21\ndata 5\n2706\n\ncommit refs/heads/test\nmark :22\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n6584b0f9e720d3035739a33e5a24d3b4e3129127 2706\nfrom :20\nM 100644 :21 foo/a1\n\nblob\nmark :23\ndata 5\n2707\n\ncommit refs/heads/test\nmark :24\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n75e46ad7079a9841332731e4701b911b73b5be23 2707\nfrom :22\nM 100644 :23 foo/a1\n\nblob\nmark :25\ndata 5\n2708\n\ncommit refs/heads/test\nmark :26\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n52d087c1e3aba5c1d158e6fb35fea9c40602ac11 2708\nfrom :24\nM 100644 :25 foo/a1\n\nblob\nmark :27\ndata 5\n2709\n\ncommit refs/heads/test\nmark :28\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n384679f01372baa92ce6e43475b44e2eb607ce67 2709\nfrom :26\nM 100644 :27 foo/a1\n\nblob\nmark :29\ndata 5\n2710\n\ncommit refs/heads/test\nmark :30\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n98def8453c487d7f54ff01d92268a0aa2c68f872 2710\nfrom :28\nM 100644 :29 foo/a1\n\nblob\nmark :31\ndata 5\n2711\n\ncommit refs/heads/test\nmark :32\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n529c626be804ea6201d53856f9d4a30d62b75248 2711\nfrom :30\nM 100644 :31 foo/a1\n\nblob\nmark :33\ndata 5\n2712\n\ncommit refs/heads/test\nmark :34\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\nc75ea627851e93d6fcbc3d45db1aff00832a403e 2712\nfrom :32\nmerge :10\nM 100644 :33 foo/a1\n\nblob\nmark :35\ndata 5\n2713\n\ncommit refs/heads/test\nmark :36\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n8c436c18ab0922c71fa2a111632d2fc9e1783342 2713\nfrom :34\nM 100644 :35 foo/a1\n\nblob\nmark :37\ndata 5\n2714\n\ncommit refs/heads/test\nmark :38\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\nc71f0c84d5b812445f1857e23c8737eeb886ae3e 2714\nfrom :36\nmerge :12\nM 100644 :37 foo/a1\n\nblob\nmark :39\ndata 5\n2715\n\ncommit refs/heads/test\nmark :40\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n1086e78f66e10ca873e75d245442f8001f7893cf 2715\nfrom :38\nmerge :14\nM 100644 :39 foo/a1\n\nblob\nmark :41\ndata 5\n2716\n\ncommit refs/heads/test\nmark :42\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n8c3e2e17eb6892069c32696784795c737fb703dd 2716\nfrom :40\nmerge :16\nM 100644 :41 foo/a1\n\nblob\nmark :43\ndata 5\n2717\n\ncommit refs/heads/test\nmark :44\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\nc9d47e2b0166eb0d3ff8af2cb65e315b96d02af8 2717\nfrom :42\nM 100644 :43 foo/a1\n\nblob\nmark :45\ndata 5\n2718\n\ncommit refs/heads/test\nmark :46\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n537dfcc8c4d15b6a1774e8b37e2ac808972c0cb5 2718\nfrom :44\nM 100644 :45 foo/a1\n\nblob\nmark :47\ndata 5\n2719\n\ncommit refs/heads/test\nmark :48\nauthor Alexander Gavrilov 1228639325 +0300\ncommitter Alexander Gavrilov 1228639325 +0300\ndata 46\n594b654469e74362bf7e77872e5bc802a03fbf01 2719\nfrom :46\nM 100644 :47 foo/a1\n\n"},{"id":"97371","messageId":"alpine.DEB.1.00.0812081911550.7516@eeepc-johanness","threadId":"16465","inReplyTo":"200812071425.52908.angavrilov@gmail.com","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-12-08T18:13:33Z","receivedAt":"2008-12-08T18:13:33Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 7 Dec 2008, Alexander Gavrilov wrote:\n\n> On Wednesday 26 November 2008 20:08:54 Johannes Schindelin wrote:\n> > On Wed, 26 Nov 2008, Michael J Gruber wrote:\n> > > Looking at the source I suspect that fast-export fails to denote \n> > > parenthood in the case of yet unmarked parents (last for-loop of \n> > > handle_commit() in builtin_fast_export.c). But I don't really know \n> > > that code at all.\n> > \n> > I strongly doubt so. Noticed the use of has_unshown_parent(commit) in \n> > both cases before calling handle_commit()?\n> > \n> > In any case, here is a script that I wrote _long_ time ago, to be able \n> > to reconstruct history from the output of \"git rev-list --all \n> > --parents\". Maybe this helps you in reconstructing something that is \n> > handled incorrectly by fast-export | fast-import, but is lighter than \n> > a full-blown repository.\n> \n> Today I had time to investigate this problem, and found:\n> \n> 1) The root of the problem is that fast-export really wants to walk\n> revisions in topological order, but actually receives them in date\n> order.\n\nIndeed. Can you submit this patch with a proper commit message, adding a \ntest for the issue by setting a bogus GIT_COMMITTER_DATE explicitly?\n\nThanks,\nDscho\n"},{"id":"112927","messageId":"85b5c3130905031206o43fd37d7y6b12eb0f0ba3d80e@mail.gmail.com","threadId":"16465","inReplyTo":"492D7AEF.5030908@viscovery.net","subject":"Re: git fast-export | git fast-import doesn't work","fromName":"Ondrej Certik","fromEmail":"ondrej@certik.cz","sentAt":"2009-05-03T19:06:36Z","receivedAt":"2009-05-03T19:06:36Z","isPatch":false,"sender":{"key":"ondrej@certik.cz","avatar":"https://gravatar.com/avatar/106d05822138a98292e09d1580dbe8f27e05faecd87f13d5d58d889d0687771c?d=mp&s=160"},"body":"On Wed, Nov 26, 2008 at 9:35 AM, Johannes Sixt wrote:\n> Ondrej Certik schrieb:\n>> I am also trying to make the example simpler. I tried to squash the\n>> first uninteresting ~1500 commits into one, but \"git rebase -i\"\n>> uterrly fails after squashing about 600 commits. Still investigating.\n>\n> Don't use rebase. Set a graft and rewrite the history:\n>\n>   $ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \\\n>                .git/info/grafts\n>\n> Assuming \"first 1500\" means the \"most recent 1500\" commits. But you get\n> the idea. You can truncate history as well by omitting the second SHA1.\n> It's very convenient to keep gitk open and File->Reload after each graft\n> that you set.\n>\n> When you're done with setting grafts:\n>\n>   $ git filter-branch -f --tag-name-filter cat -- --all\n>\n> (You are doing this on a copy of your repository, don't you?)\n\nThanks for the tip with grafts. I tried that on some other repository\nwhen I needed to truncate the history and it works like a charm.\n\nOndrej\n"}]}