git fast-export | git fast-import doesn't work

> Hi,
> 
> I would like to export our whole git repository to patches, and then
> reconstruct it again from scratch. Following the man page of "git
> fast-export":
> 
> $ git clone git://git.sympy.org/sympy-full-history-20081023.git
> $ cd sympy-full-history-20081023
> $ git fast-export --all --export-marks=marks > patches
> $ cd ..
> $ mkdir sympy-new
> $ cd sympy-new
> $ git init
> $ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
> git-fast-import statistics:
> ---------------------------------------------------------------------
> Alloc'd objects:      25000
> Total objects:        21355 (       144 duplicates                  )
>       blobs  :         8009 (         0 duplicates       4529 deltas)
>       trees  :        10627 (       144 duplicates       9189 deltas)
>       commits:         2719 (         0 duplicates          0 deltas)
>       tags   :            0 (         0 duplicates          0 deltas)
> Total branches:          21 (        26 loads     )
>       marks:        1048576 (     10728 unique    )
>       atoms:            726
> Memory total:          2880 KiB
>        pools:          2098 KiB
>      objects:           781 KiB
> ---------------------------------------------------------------------
> pack_report: getpagesize()            =       4096
> pack_report: core.packedGitWindowSize =   33554432
> pack_report: core.packedGitLimit      =  268435456
> pack_report: pack_used_ctr            =      40706
> pack_report: pack_mmap_calls          =       2791
> pack_report: pack_open_windows        =          1 /          2
> pack_report: pack_mapped              =   26177739 /   35513414
> ---------------------------------------------------------------------
> 
> 
> 
> However, the repository is very different to the original one. It
> contains only 191 patches:
> 
> $ git log --pretty=oneline | wc -l
> 191
> 
> and it only contains couple files. Compare this with the original repository:
> 
> $ git log --pretty=oneline | wc -l
> 2719

> On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber <git@drmicha.warpmail.net> wrote:
>> I don't know, I just noticed that turning on rename and copy detection
>> makes git-fast-import crash, which shouldn't happen either. Something's
>> not right here. CC'ing the authors of im- and export.
>
> Could you please write a testcase that reproduces your problem?
>
>> Why export|import directly to git?
>
> I guess he did not know about filter-branch. :)

> On Tue, Nov 25, 2008 at 9:41 PM, Miklos Vajna <vmiklos@frugalware.org> wrote:
>> On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber <git@drmicha.warpmail.net> wrote:
>>> I don't know, I just noticed that turning on rename and copy detection
>>> makes git-fast-import crash, which shouldn't happen either. Something's
>>> not right here. CC'ing the authors of im- and export.
>>
>> Could you please write a testcase that reproduces your problem?
>>
>>> Why export|import directly to git?
>>
>> I guess he did not know about filter-branch. :)
>
> I know about filter-branch (but I am not sure it can do what I want).
> I made a mistake of not explaining what I want, instead I suggested (a
> possibly wrong) solution. I want to export the whole git repository as
> a set of human readable patches, that can be assembled back into a git
> repository (with the same hashes as the original one)

> if needed. The
> reason I want that is that if we later decide to switch to another
> VCS, we have all the information to reproduce the repository. Another
> reason is to be sure that we know all the sources that are needed to
> construct the repository, e.g. that there are no binary blobs
> (possibly containing malicious code). Another reason I want that is to
> be able to rewrite the history, in particular, we have one Mercurial
> repository with some old history and another Mercurial history with a
> newer history and I just want to concatenate them together into one
> git repository.
>
> In each case I know several workarounds, but if there is a way to just
> convert the whole git repository into a set of patches and (and be
> able to convert everything back including the same hashes), then it'd
> be awesome.
>
> See also this thread why people want this (and I assumed git can do
> this from this thread):
>
> http://groups.google.com/group/sage-devel/browse_thread/thread/7b116d902ee20d9c/
>
> Thanks,
> Ondrej
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

> Ondrej Certik venit, vidit, dixit 25.11.2008 17:44:
>> Hi,
>>
>> I would like to export our whole git repository to patches, and then
>> reconstruct it again from scratch. Following the man page of "git
>> fast-export":
>>
>> $ git clone git://git.sympy.org/sympy-full-history-20081023.git
>> $ cd sympy-full-history-20081023
>> $ git fast-export --all --export-marks=marks > patches
>> $ cd ..
>> $ mkdir sympy-new
>> $ cd sympy-new
>> $ git init
>> $ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
>> git-fast-import statistics:
>> ---------------------------------------------------------------------
>> Alloc'd objects:      25000
>> Total objects:        21355 (       144 duplicates                  )
>>       blobs  :         8009 (         0 duplicates       4529 deltas)
>>       trees  :        10627 (       144 duplicates       9189 deltas)
>>       commits:         2719 (         0 duplicates          0 deltas)
>>       tags   :            0 (         0 duplicates          0 deltas)
>> Total branches:          21 (        26 loads     )
>>       marks:        1048576 (     10728 unique    )
>>       atoms:            726
>> Memory total:          2880 KiB
>>        pools:          2098 KiB
>>      objects:           781 KiB
>> ---------------------------------------------------------------------
>> pack_report: getpagesize()            =       4096
>> pack_report: core.packedGitWindowSize =   33554432
>> pack_report: core.packedGitLimit      =  268435456
>> pack_report: pack_used_ctr            =      40706
>> pack_report: pack_mmap_calls          =       2791
>> pack_report: pack_open_windows        =          1 /          2
>> pack_report: pack_mapped              =   26177739 /   35513414
>> ---------------------------------------------------------------------
>>
>>
>>
>> However, the repository is very different to the original one. It
>> contains only 191 patches:
>>
>> $ git log --pretty=oneline | wc -l
>> 191
>>
>> and it only contains couple files. Compare this with the original repository:
>>
>> $ git log --pretty=oneline | wc -l
>> 2719
>
> I get the same stats (with the dups) but a perfect rev count, when I use
> git log --all. The reason is that the history in the imported repo is
> disconnected at various places (at tagging commits)! Your command counts
> only the revs backwards to the first "disconnection".

> I don't know, I just noticed that turning on rename and copy detection
> makes git-fast-import crash, which shouldn't happen either. Something's
> not right here. CC'ing the authors of im- and export.
>
> BTW: Maybe you can accomplish what you want with different means? Why
> export|import directly to git?

> Hi,
> 
> I would like to export our whole git repository to patches, and then
> reconstruct it again from scratch. Following the man page of "git
> fast-export":
> 

> $ git clone git://git.sympy.org/sympy-full-history-20081023.git
> $ cd sympy-full-history-20081023
> $ git fast-export --all --export-marks=marks > patches
> $ cd ..
> $ mkdir sympy-new
> $ cd sympy-new
> $ git init
> $ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
> git-fast-import statistics:
> ---------------------------------------------------------------------
> Alloc'd objects:      25000
> Total objects:        21355 (       144 duplicates                  )
>       blobs  :         8009 (         0 duplicates       4529 deltas)
>       trees  :        10627 (       144 duplicates       9189 deltas)
>       commits:         2719 (         0 duplicates          0 deltas)
>       tags   :            0 (         0 duplicates          0 deltas)
> Total branches:          21 (        26 loads     )
>       marks:        1048576 (     10728 unique    )
>       atoms:            726
> Memory total:          2880 KiB
>        pools:          2098 KiB
>      objects:           781 KiB
> ---------------------------------------------------------------------
> pack_report: getpagesize()            =       4096
> pack_report: core.packedGitWindowSize =   33554432
> pack_report: core.packedGitLimit      =  268435456
> pack_report: pack_used_ctr            =      40706
> pack_report: pack_mmap_calls          =       2791
> pack_report: pack_open_windows        =          1 /          2
> pack_report: pack_mapped              =   26177739 /   35513414
> ---------------------------------------------------------------------
> 
> 
> 
> However, the repository is very different to the original one. It
> contains only 191 patches:
> 
> $ git log --pretty=oneline | wc -l
> 191
> 
> and it only contains couple files. Compare this with the original repository:
> 
> $ git log --pretty=oneline | wc -l
> 2719
> 
> What am I doing wrong? Is there some other way to do it? I also tried
> "git format-patch" and "git am" and that almost works, only it changes
> hashes. Is there some way to tell "git am" to preserve the hash?
> 
> Thanks,
> Ondrej

> I would like to export our whole git repository to patches, and then 
> reconstruct it again from scratch. Following the man page of "git 
> fast-export":
>
> [...] 
> 
> However, the repository is very different to the original one. It 
> contains only 191 patches:

> Hi,
>
> On Tue, 25 Nov 2008, Ondrej Certik wrote:
>
>> I would like to export our whole git repository to patches, and then
>> reconstruct it again from scratch. Following the man page of "git
>> fast-export":
>>
>> [...]
>>
>> However, the repository is very different to the original one. It
>> contains only 191 patches:
>
> Can you try again with a Git version that contains the commit
> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?

> On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>> Hi,
>>
>> On Tue, 25 Nov 2008, Ondrej Certik wrote:
>>
>>> I would like to export our whole git repository to patches, and then
>>> reconstruct it again from scratch. Following the man page of "git
>>> fast-export":
>>>
>>> [...]
>>>
>>> However, the repository is very different to the original one. It
>>> contains only 191 patches:
>>
>> Can you try again with a Git version that contains the commit
>> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?
>
> I tried the next branch:
>
> $ git --version
> git version 1.6.0.4.1060.g9433b
>
> that contains the 2075ffb5 patch. I haven't observed any change ---
> the "git log" still only shows 191 commits (git log --all shows
> everything).

> On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik <ondrej@certik.cz> wrote:
>> On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
>> <Johannes.Schindelin@gmx.de> wrote:
>>> Hi,
>>>
>>> On Tue, 25 Nov 2008, Ondrej Certik wrote:
>>>
>>>> I would like to export our whole git repository to patches, and then
>>>> reconstruct it again from scratch. Following the man page of "git
>>>> fast-export":
>>>>
>>>> [...]
>>>>
>>>> However, the repository is very different to the original one. It
>>>> contains only 191 patches:
>>> Can you try again with a Git version that contains the commit
>>> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?
>> I tried the next branch:
>>
>> $ git --version
>> git version 1.6.0.4.1060.g9433b
>>
>> that contains the 2075ffb5 patch. I haven't observed any change ---
>> the "git log" still only shows 191 commits (git log --all shows
>> everything).
> 
> I deleted all tags and then fast-exported and imported, now all the
> commits show in "git log", however, the patches are wrongly connected.
> Basically, both repositories are identical (including hashes) up to
> this commit:
> 
> d717177d4  (fixed downloads instructions in the README and a typo)
> 
> However, the original repo (sympy-full-history-20081023) contains 3
> children at this commit:

> Ondrej Certik venit, vidit, dixit 26.11.2008 11:18:
>> On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik <ondrej@certik.cz> wrote:
>>> On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
>>> <Johannes.Schindelin@gmx.de> wrote:
>>>> Hi,
>>>>
>>>> On Tue, 25 Nov 2008, Ondrej Certik wrote:
>>>>
>>>>> I would like to export our whole git repository to patches, and then
>>>>> reconstruct it again from scratch. Following the man page of "git
>>>>> fast-export":
>>>>>
>>>>> [...]
>>>>>
>>>>> However, the repository is very different to the original one. It
>>>>> contains only 191 patches:
>>>> Can you try again with a Git version that contains the commit
>>>> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?
>>> I tried the next branch:
>>>
>>> $ git --version
>>> git version 1.6.0.4.1060.g9433b
>>>
>>> that contains the 2075ffb5 patch. I haven't observed any change ---
>>> the "git log" still only shows 191 commits (git log --all shows
>>> everything).
>>
>> I deleted all tags and then fast-exported and imported, now all the
>> commits show in "git log", however, the patches are wrongly connected.
>> Basically, both repositories are identical (including hashes) up to
>> this commit:
>>
>> d717177d4  (fixed downloads instructions in the README and a typo)
>>
>> However, the original repo (sympy-full-history-20081023) contains 3
>> children at this commit:
>
> There's some nice 3 way branching and double 2 way merging going on. I
> cut out the interesting part of the graph, making d717177d4 and
> 6e869485f parentless. The resulting mini DAG is reproduced correctly by
> export|import, even with -M -C.

> Ondrej Certik schrieb:
>> I am also trying to make the example simpler. I tried to squash the
>> first uninteresting ~1500 commits into one, but "git rebase -i"
>> uterrly fails after squashing about 600 commits. Still investigating.
>
> Don't use rebase. Set a graft and rewrite the history:
>
>   $ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \
>                .git/info/grafts
>
> Assuming "first 1500" means the "most recent 1500" commits. But you get
> the idea. You can truncate history as well by omitting the second SHA1.
> It's very convenient to keep gitk open and File->Reload after each graft
> that you set.
>
> When you're done with setting grafts:
>
>   $ git filter-branch -f --tag-name-filter cat -- --all

> Ondrej Certik schrieb:
>> I am also trying to make the example simpler. I tried to squash the
>> first uninteresting ~1500 commits into one, but "git rebase -i"
>> uterrly fails after squashing about 600 commits. Still investigating.
>
> Don't use rebase. Set a graft and rewrite the history:
>
>   $ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \
>                .git/info/grafts
>
> Assuming "first 1500" means the "most recent 1500" commits. But you get
> the idea. You can truncate history as well by omitting the second SHA1.
> It's very convenient to keep gitk open and File->Reload after each graft
> that you set.
>
> When you're done with setting grafts:
>
>   $ git filter-branch -f --tag-name-filter cat -- --all
>
> (You are doing this on a copy of your repository, don't you?)

> Hi,
> 
> On Wed, 26 Nov 2008, Ondrej Certik wrote:
> 
>> I am also trying to make the example simpler. I tried to squash the 
>> first uninteresting ~1500 commits into one, but "git rebase -i" uterrly 
>> fails after squashing about 600 commits. Still investigating.
> 
> 1500... wow.
> 
> The best idea would probably be to just "edit" the first, delete the rest 
> of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on 
> the command line (when git rebase stops after the "edit" command).
> 
> rebase -i was _never_ meant for such _massive_ interactions; that's just 
> too much for a shell script.

> On Wed, 26 Nov 2008, Michael J Gruber wrote:
> > Looking at the source I suspect that fast-export fails to denote 
> > parenthood in the case of yet unmarked parents (last for-loop of 
> > handle_commit() in builtin_fast_export.c). But I don't really know that 
> > code at all.
> 
> I strongly doubt so.  Noticed the use of has_unshown_parent(commit) in 
> both cases before calling handle_commit()?
> 
> In any case, here is a script that I wrote _long_ time ago, to be able to 
> reconstruct history from the output of "git rev-list --all --parents".  
> Maybe this helps you in reconstructing something that is handled 
> incorrectly by fast-export | fast-import, but is lighter than a full-blown 
> repository.

> On Wednesday 26 November 2008 20:08:54 Johannes Schindelin wrote:
> > On Wed, 26 Nov 2008, Michael J Gruber wrote:
> > > Looking at the source I suspect that fast-export fails to denote 
> > > parenthood in the case of yet unmarked parents (last for-loop of 
> > > handle_commit() in builtin_fast_export.c). But I don't really know 
> > > that code at all.
> > 
> > I strongly doubt so.  Noticed the use of has_unshown_parent(commit) in 
> > both cases before calling handle_commit()?
> > 
> > In any case, here is a script that I wrote _long_ time ago, to be able 
> > to reconstruct history from the output of "git rev-list --all 
> > --parents".  Maybe this helps you in reconstructing something that is 
> > handled incorrectly by fast-export | fast-import, but is lighter than 
> > a full-blown repository.
> 
> Today I had time to investigate this problem, and found:
> 
> 1) The root of the problem is that fast-export really wants to walk
>     revisions in topological order, but actually receives them in date
>     order.

> Hi,
>
> On Wed, 26 Nov 2008, Ondrej Certik wrote:
>
>> I am also trying to make the example simpler. I tried to squash the
>> first uninteresting ~1500 commits into one, but "git rebase -i" uterrly
>> fails after squashing about 600 commits. Still investigating.
>
> 1500... wow.
>
> The best idea would probably be to just "edit" the first, delete the rest
> of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on
> the command line (when git rebase stops after the "edit" command).

> On Wed, Nov 26, 2008 at 5:40 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>> Hi,
>>
>> On Wed, 26 Nov 2008, Ondrej Certik wrote:
>>
>>> I am also trying to make the example simpler. I tried to squash the
>>> first uninteresting ~1500 commits into one, but "git rebase -i" uterrly
>>> fails after squashing about 600 commits. Still investigating.
>> 1500... wow.
>>
>> The best idea would probably be to just "edit" the first, delete the rest
>> of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on
>> the command line (when git rebase stops after the "edit" command).
> 
> That worked, thanks! My original repo:
> 
> A -- B -- ... --- D --- E --- ...
> 
> where E and the rest of the commits (there are branches and merges in
> there) are the ones that I need to preserve, but all the commits
> between B and D can be squashed (~1500 of them). So I created a
> branch:
> 
> A -- B -- ... --- D
> 
> then squashed the commits using the technique you described above, so
> now I have:
> 
> A -- BD --
> 
> and now I would like to append "E -- ..." to it -- is there any way to
> do that? I tried rebase, but that destroys all the branches and merges
> and those are necessary to reproduce the fast-export bug.
> 

> Hi,
> 
> On Tue, 25 Nov 2008, Ondrej Certik wrote:
> 
>> I would like to export our whole git repository to patches, and then 
>> reconstruct it again from scratch. Following the man page of "git 
>> fast-export":
>>
>> [...] 
>>
>> However, the repository is very different to the original one. It 
>> contains only 191 patches:
> 
> Can you try again with a Git version that contains the commit 
> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?

> Hi Ondrej & Michael,
>
> On Wed, 26 Nov 2008, Michael J Gruber wrote:
>
>> Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:
>>
>> > Can you try again with a Git version that contains the commit
>> > 2075ffb5(fast-export: use an unsorted string list for extra_refs)?
>
> Okay, so both of your use cases seem to be real bugs in fast-export.  May
> I respectfully submit a request for a test script (as patch to
> t/t9301-fast-export.sh)  which is as short as possible and shows the
> respective bugs?
>
> My Git time budget these days is almost negative, so I will not be able to
> work on these issues without having a small and concise example.