{"thread":{"id":"38471","subject":"git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","startedAt":"2015-01-29T22:03:47Z","lastAt":"2015-02-25T20:33:59Z","messageCount":14,"participants":["Valery Yundin","Eric Wong","Nico Schlömer","Kyle J. McKay"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"255418","messageId":"CABNxngNDwf_Cy77OzvMg__kCNoTz5y1a2KKG1vobYjE_m_aLkQ@mail.gmail.com","threadId":"38471","inReplyTo":null,"subject":"git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Valery Yundin","fromEmail":"yuvalery@gmail.com","sentAt":"2015-01-29T22:03:47Z","receivedAt":"2015-01-29T22:03:47Z","isPatch":false,"sender":{"key":"yuvalery@gmail.com","avatar":"https://gravatar.com/avatar/c56698d0b66ab632cd85309da9a3b1794cb912fb9f37e0f0e488d15e3791672a?d=mp&s=160"},"body":"Hi,\n\nLooks like there might be a bug in SVN import code. Here is the\ncommand that fails\n\n> git svn clone --username anonymous svn://powhegbox.mib.infn.it/trunk/POWHEG-BOX\n\nr217 = 2e6cdb1f4604b2256195590fa8275632971eac42 (refs/remotes/git-svn)\n        M       lhefread.f\n        M       Z_plus_jet/Born.f\n        M       Z_plus_jet/powheg.input\n        M       Z_plus_jet/init_processes.f\n        D       JJ/madgraph_3_flavours/born/nexternal.inc\nwrite .git/Git_svn_hash_bl5Cj3: Bad file descriptor\n at /usr/lib/perl5/vendor_perl/5.20.1/x86_64-linux-thread-multi/SVN/Ra.pm\nline 623.\n\nTested on:\ngit-svn version 2.3.0.rc2 (svn 1.8.11) - FAIL\ngit-svn version 2.2.2 (svn 1.8.10) - FAIL\ngit-svn version 1.8.4.5 (svn 1.8.11) - WORKS\n\nPS unfortunately I don't have admin access to SVN repository\n\nWith best regards, Valery Yundin.\n"},{"id":"255420","messageId":"20150129233429.GA7162@dcvr.yhbt.net","threadId":"38471","inReplyTo":"CABNxngNDwf_Cy77OzvMg__kCNoTz5y1a2KKG1vobYjE_m_aLkQ@mail.gmail.com","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2015-01-29T23:34:30Z","receivedAt":"2015-01-29T23:34:30Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Valery Yundin <yuvalery@gmail.com> wrote:\n> Tested on:\n> git-svn version 2.3.0.rc2 (svn 1.8.11) - FAIL\n> git-svn version 2.2.2 (svn 1.8.10) - FAIL\n> git-svn version 1.8.4.5 (svn 1.8.11) - WORKS\n\nThank you for that info.  Do you think you can bisect which\nversions/revisions of git-svn introduced that failure for us?  I don't\nhave much time this part of the year for git-svn, but maybe it's related\nto the performance work we did around fall 2014.\n\nThanks again.\n"},{"id":"255422","messageId":"CABNxngPErFiTzUNK6P90Ug5fVtJSYD9XrGrQzK8+mt2u4g=Xyg@mail.gmail.com","threadId":"38471","inReplyTo":"20150129233429.GA7162@dcvr.yhbt.net","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Valery Yundin","fromEmail":"yuvalery@gmail.com","sentAt":"2015-01-29T23:59:20Z","receivedAt":"2015-01-29T23:59:20Z","isPatch":false,"sender":{"key":"yuvalery@gmail.com","avatar":"https://gravatar.com/avatar/c56698d0b66ab632cd85309da9a3b1794cb912fb9f37e0f0e488d15e3791672a?d=mp&s=160"},"body":"Hi,\n\nHere you go:\ndfa72fdb96befbd790f623bb2909a347176753c2 is the first bad commit\ncommit dfa72fdb96befbd790f623bb2909a347176753c2\nAuthor: Eric Wong <normalperson@yhbt.net>\nDate:   Fri Oct 24 22:53:52 2014 +0000\n\n    git-svn: reload RA every log-window-size\n\n    Despite attempting to use local memory pools everywhere we can,\n    (including our call to SVN::Ra::do_update and all subsequent reporter\n    calls), there does not appear to be a way to force the Git::SVN::Fetcher\n    callbacks to use a pool other than the per-SVN::Ra pool.\n    Git::SVN::Fetcher ends up using the main RA pool which grows\n    monotonically in size for the lifetime of the RA object.\n\n    Thus the only way to free that memory appears to be to destroy and\n    recreate the RA connection for at every --log-window-size interval.\n\n    This reduces memory usage over the course of fetching 10K revisions\n    using a test repository created with the script at the end of this\n    commit message.\n    .........\n\nCheers,\nValery\nWith best regards, Mr. Valery Yundin.\n\n\nOn 30 January 2015 at 00:34, Eric Wong <normalperson@yhbt.net> wrote:\n> Valery Yundin <yuvalery@gmail.com> wrote:\n>> Tested on:\n>> git-svn version 2.3.0.rc2 (svn 1.8.11) - FAIL\n>> git-svn version 2.2.2 (svn 1.8.10) - FAIL\n>> git-svn version 1.8.4.5 (svn 1.8.11) - WORKS\n>\n> Thank you for that info.  Do you think you can bisect which\n> versions/revisions of git-svn introduced that failure for us?  I don't\n> have much time this part of the year for git-svn, but maybe it's related\n> to the performance work we did around fall 2014.\n>\n> Thanks again.\n"},{"id":"255423","messageId":"20150130002247.GA22519@dcvr.yhbt.net","threadId":"38471","inReplyTo":"CABNxngPErFiTzUNK6P90Ug5fVtJSYD9XrGrQzK8+mt2u4g=Xyg@mail.gmail.com","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2015-01-30T00:22:47Z","receivedAt":"2015-01-30T00:22:47Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Valery Yundin <yuvalery@gmail.com> wrote:\n> Hi,\n> \n> Here you go:\n> dfa72fdb96befbd790f623bb2909a347176753c2 is the first bad commit\n\nThank you.  Can you give the following patch a try?\nI have not been able to reproduce the problem on my end.\nIf it doesn't work out, I might be out of ideas for a bit :/\nIncreasing --log-window-size will help you run longer without\nthe error, but that's not ideal as it can also eat memory.\n\n-----------------------8<----------------------\nFrom: Eric Wong <normalperson@yhbt.net>\nSubject: [PATCH] git-svn: destroy all tempfiles when reloading RA\n\nThis may fix the errors some users are seeing with:\n\"write .git/Git_svn_hash_XXXXXX: Bad file descriptor\"\n\nThanks to Valery Yundin for helping bisect the problem introduced in\ncommit dfa72fdb96befbd790f623bb2909a347176753c2\n(git-svn: reload RA every log-window-size)\n\nCc: Valery Yundin <yuvalery@gmail.com>\nSigned-off-by: Eric Wong <normalperson@yhbt.net>\n---\n perl/Git.pm        | 6 ++++++\n perl/Git/SVN/Ra.pm | 1 +\n 2 files changed, 7 insertions(+)\n\ndiff --git a/perl/Git.pm b/perl/Git.pm\nindex b5905ee..698018e 100644\n--- a/perl/Git.pm\n+++ b/perl/Git.pm\n@@ -1347,6 +1347,12 @@ sub temp_path {\n \t$TEMP_FILES{$temp_fd}{fname};\n }\n \n+sub temp_reset_all {\n+\tunlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n+\t%TEMP_FILEMAP = ();\n+\t%TEMP_FILES = ();\n+}\n+\n sub END {\n \tunlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n }\ndiff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm\nindex 622535e..878679d 100644\n--- a/perl/Git/SVN/Ra.pm\n+++ b/perl/Git/SVN/Ra.pm\n@@ -397,6 +397,7 @@ sub gs_fetch_loop_common {\n \t\t$_[0] = undef;\n \t\t$self = undef;\n \t\t$RA = undef;\n+\t\tGit->temp_reset_all;\n \t\t$gpool->clear;\n \t\t$self = Git::SVN::Ra->new($ra_url);\n \t\t$ra_invalid = undef;\n-- \nEW\n"},{"id":"255424","messageId":"CABNxngMVsaSLmzf0ZsaXZqXhD+hOaRmz-uuSzm0ALDFhKjnrXA@mail.gmail.com","threadId":"38471","inReplyTo":"20150130002247.GA22519@dcvr.yhbt.net","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Valery Yundin","fromEmail":"yuvalery@gmail.com","sentAt":"2015-01-30T00:40:31Z","receivedAt":"2015-01-30T00:40:31Z","isPatch":false,"sender":{"key":"yuvalery@gmail.com","avatar":"https://gravatar.com/avatar/c56698d0b66ab632cd85309da9a3b1794cb912fb9f37e0f0e488d15e3791672a?d=mp&s=160"},"body":"Hi,\n\nYour patch seems to fix the problem.\nI tried several times and I can svn clone the whole repository in one\ngo without a crash.\n\nThanks,\nValery\n\nOn 30 January 2015 at 01:22, Eric Wong <normalperson@yhbt.net> wrote:\n> Valery Yundin <yuvalery@gmail.com> wrote:\n>> Hi,\n>>\n>> Here you go:\n>> dfa72fdb96befbd790f623bb2909a347176753c2 is the first bad commit\n>\n> Thank you.  Can you give the following patch a try?\n> I have not been able to reproduce the problem on my end.\n> If it doesn't work out, I might be out of ideas for a bit :/\n> Increasing --log-window-size will help you run longer without\n> the error, but that's not ideal as it can also eat memory.\n>\n> -----------------------8<----------------------\n> From: Eric Wong <normalperson@yhbt.net>\n> Subject: [PATCH] git-svn: destroy all tempfiles when reloading RA\n>\n> This may fix the errors some users are seeing with:\n> \"write .git/Git_svn_hash_XXXXXX: Bad file descriptor\"\n>\n> Thanks to Valery Yundin for helping bisect the problem introduced in\n> commit dfa72fdb96befbd790f623bb2909a347176753c2\n> (git-svn: reload RA every log-window-size)\n>\n> Cc: Valery Yundin <yuvalery@gmail.com>\n> Signed-off-by: Eric Wong <normalperson@yhbt.net>\n> ---\n>  perl/Git.pm        | 6 ++++++\n>  perl/Git/SVN/Ra.pm | 1 +\n>  2 files changed, 7 insertions(+)\n>\n> diff --git a/perl/Git.pm b/perl/Git.pm\n> index b5905ee..698018e 100644\n> --- a/perl/Git.pm\n> +++ b/perl/Git.pm\n> @@ -1347,6 +1347,12 @@ sub temp_path {\n>         $TEMP_FILES{$temp_fd}{fname};\n>  }\n>\n> +sub temp_reset_all {\n> +       unlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n> +       %TEMP_FILEMAP = ();\n> +       %TEMP_FILES = ();\n> +}\n> +\n>  sub END {\n>         unlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n>  }\n> diff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm\n> index 622535e..878679d 100644\n> --- a/perl/Git/SVN/Ra.pm\n> +++ b/perl/Git/SVN/Ra.pm\n> @@ -397,6 +397,7 @@ sub gs_fetch_loop_common {\n>                 $_[0] = undef;\n>                 $self = undef;\n>                 $RA = undef;\n> +               Git->temp_reset_all;\n>                 $gpool->clear;\n>                 $self = Git::SVN::Ra->new($ra_url);\n>                 $ra_invalid = undef;\n> --\n> EW\n"},{"id":"255425","messageId":"20150130013017.GA10713@dcvr.yhbt.net","threadId":"38471","inReplyTo":"CABNxngMVsaSLmzf0ZsaXZqXhD+hOaRmz-uuSzm0ALDFhKjnrXA@mail.gmail.com","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2015-01-30T01:30:17Z","receivedAt":"2015-01-30T01:30:17Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Valery Yundin <yuvalery@gmail.com> wrote:\n> Hi,\n> \n> Your patch seems to fix the problem.\n> I tried several times and I can svn clone the whole repository in one\n> go without a crash.\n\nThanks for the confirmation.  Cc-ing a few other folks who encountered\nthis problem (and Bcc-ing some folks who emailed me privately).\n\nCan the rest of you give this patch a try on your respective platforms\nand confirm the fix?\n\nhttp://article.gmane.org/gmane.comp.version-control.git/263168/raw\n(also: http://mid.gmane.org/20150130002247.GA22519@dcvr.yhbt.net )\n\nJunio: assuming all goes well with testers, can you apply my patch\nto the appropriate maintenance tracks with Tested-by:s?\nI'm going offline in a few hours and don't think I'll be around\nmuch the next week or so.\n\nThanks.\n\n> Thanks,\n> Valery\n> \n> On 30 January 2015 at 01:22, Eric Wong <normalperson@yhbt.net> wrote:\n> > Valery Yundin <yuvalery@gmail.com> wrote:\n> >> Hi,\n> >>\n> >> Here you go:\n> >> dfa72fdb96befbd790f623bb2909a347176753c2 is the first bad commit\n> >\n> > Thank you.  Can you give the following patch a try?\n> > I have not been able to reproduce the problem on my end.\n> > If it doesn't work out, I might be out of ideas for a bit :/\n> > Increasing --log-window-size will help you run longer without\n> > the error, but that's not ideal as it can also eat memory.\n> >\n> > -----------------------8<----------------------\n> > From: Eric Wong <normalperson@yhbt.net>\n> > Subject: [PATCH] git-svn: destroy all tempfiles when reloading RA\n> >\n> > This may fix the errors some users are seeing with:\n> > \"write .git/Git_svn_hash_XXXXXX: Bad file descriptor\"\n> >\n> > Thanks to Valery Yundin for helping bisect the problem introduced in\n> > commit dfa72fdb96befbd790f623bb2909a347176753c2\n> > (git-svn: reload RA every log-window-size)\n> >\n> > Cc: Valery Yundin <yuvalery@gmail.com>\n> > Signed-off-by: Eric Wong <normalperson@yhbt.net>\n> > ---\n> >  perl/Git.pm        | 6 ++++++\n> >  perl/Git/SVN/Ra.pm | 1 +\n> >  2 files changed, 7 insertions(+)\n> >\n> > diff --git a/perl/Git.pm b/perl/Git.pm\n> > index b5905ee..698018e 100644\n> > --- a/perl/Git.pm\n> > +++ b/perl/Git.pm\n> > @@ -1347,6 +1347,12 @@ sub temp_path {\n> >         $TEMP_FILES{$temp_fd}{fname};\n> >  }\n> >\n> > +sub temp_reset_all {\n> > +       unlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n> > +       %TEMP_FILEMAP = ();\n> > +       %TEMP_FILES = ();\n> > +}\n> > +\n> >  sub END {\n> >         unlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n> >  }\n> > diff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm\n> > index 622535e..878679d 100644\n> > --- a/perl/Git/SVN/Ra.pm\n> > +++ b/perl/Git/SVN/Ra.pm\n> > @@ -397,6 +397,7 @@ sub gs_fetch_loop_common {\n> >                 $_[0] = undef;\n> >                 $self = undef;\n> >                 $RA = undef;\n> > +               Git->temp_reset_all;\n> >                 $gpool->clear;\n> >                 $self = Git::SVN::Ra->new($ra_url);\n> >                 $ra_invalid = undef;\n> > --\n> > EW\n> --\n"},{"id":"255459","messageId":"CAK6Z60cGnvPP0ctnyV4tHfMmBGYRidEJzkAG+J2h9BunmtOAsQ@mail.gmail.com","threadId":"38471","inReplyTo":"20150130013017.GA10713@dcvr.yhbt.net","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Nico Schlömer","fromEmail":"nico.schloemer@gmail.com","sentAt":"2015-01-31T12:51:29Z","receivedAt":"2015-01-31T12:51:29Z","isPatch":false,"sender":{"key":"nico.schloemer@gmail.com","avatar":"https://gravatar.com/avatar/b0a6998b0c9bcd0641b4d07789be6f64e21fce87df71a94b952611c7994b9878?d=mp&s=160"},"body":"I tried the patch and I still get\n```\n[...]\nr100 = e2a9b5baa2cebb18591ecb04ff350410d52f36de (refs/remotes/git-svn)\nUnexpected result returned from git cat-file at\n/home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 335.\nFailed to read object 619f9d1d857fb287d06a70c9dac6b8b534d0de6a at\n/home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 336, <GEN16>\nline 757.\n\nerror closing pipe: Bad file descriptor at\n/home/nschloe/libexec/git-core/git-svn line 0.\nerror closing pipe: Bad file descriptor at\n/home/nschloe/libexec/git-core/git-svn line 0.\n```\nwhen\n```\ngit svn clone https://geuz.org/svn/gmsh/trunk\n```\n\nCheers,\nNico\n\nOn Fri, Jan 30, 2015 at 2:30 AM, Eric Wong <normalperson@yhbt.net> wrote:\n> Valery Yundin <yuvalery@gmail.com> wrote:\n>> Hi,\n>>\n>> Your patch seems to fix the problem.\n>> I tried several times and I can svn clone the whole repository in one\n>> go without a crash.\n>\n> Thanks for the confirmation.  Cc-ing a few other folks who encountered\n> this problem (and Bcc-ing some folks who emailed me privately).\n>\n> Can the rest of you give this patch a try on your respective platforms\n> and confirm the fix?\n>\n> http://article.gmane.org/gmane.comp.version-control.git/263168/raw\n> (also: http://mid.gmane.org/20150130002247.GA22519@dcvr.yhbt.net )\n>\n> Junio: assuming all goes well with testers, can you apply my patch\n> to the appropriate maintenance tracks with Tested-by:s?\n> I'm going offline in a few hours and don't think I'll be around\n> much the next week or so.\n>\n> Thanks.\n>\n>> Thanks,\n>> Valery\n>>\n>> On 30 January 2015 at 01:22, Eric Wong <normalperson@yhbt.net> wrote:\n>> > Valery Yundin <yuvalery@gmail.com> wrote:\n>> >> Hi,\n>> >>\n>> >> Here you go:\n>> >> dfa72fdb96befbd790f623bb2909a347176753c2 is the first bad commit\n>> >\n>> > Thank you.  Can you give the following patch a try?\n>> > I have not been able to reproduce the problem on my end.\n>> > If it doesn't work out, I might be out of ideas for a bit :/\n>> > Increasing --log-window-size will help you run longer without\n>> > the error, but that's not ideal as it can also eat memory.\n>> >\n>> > -----------------------8<----------------------\n>> > From: Eric Wong <normalperson@yhbt.net>\n>> > Subject: [PATCH] git-svn: destroy all tempfiles when reloading RA\n>> >\n>> > This may fix the errors some users are seeing with:\n>> > \"write .git/Git_svn_hash_XXXXXX: Bad file descriptor\"\n>> >\n>> > Thanks to Valery Yundin for helping bisect the problem introduced in\n>> > commit dfa72fdb96befbd790f623bb2909a347176753c2\n>> > (git-svn: reload RA every log-window-size)\n>> >\n>> > Cc: Valery Yundin <yuvalery@gmail.com>\n>> > Signed-off-by: Eric Wong <normalperson@yhbt.net>\n>> > ---\n>> >  perl/Git.pm        | 6 ++++++\n>> >  perl/Git/SVN/Ra.pm | 1 +\n>> >  2 files changed, 7 insertions(+)\n>> >\n>> > diff --git a/perl/Git.pm b/perl/Git.pm\n>> > index b5905ee..698018e 100644\n>> > --- a/perl/Git.pm\n>> > +++ b/perl/Git.pm\n>> > @@ -1347,6 +1347,12 @@ sub temp_path {\n>> >         $TEMP_FILES{$temp_fd}{fname};\n>> >  }\n>> >\n>> > +sub temp_reset_all {\n>> > +       unlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n>> > +       %TEMP_FILEMAP = ();\n>> > +       %TEMP_FILES = ();\n>> > +}\n>> > +\n>> >  sub END {\n>> >         unlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n>> >  }\n>> > diff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm\n>> > index 622535e..878679d 100644\n>> > --- a/perl/Git/SVN/Ra.pm\n>> > +++ b/perl/Git/SVN/Ra.pm\n>> > @@ -397,6 +397,7 @@ sub gs_fetch_loop_common {\n>> >                 $_[0] = undef;\n>> >                 $self = undef;\n>> >                 $RA = undef;\n>> > +               Git->temp_reset_all;\n>> >                 $gpool->clear;\n>> >                 $self = Git::SVN::Ra->new($ra_url);\n>> >                 $ra_invalid = undef;\n>> > --\n>> > EW\n>> --\n"},{"id":"255460","messageId":"CABNxngNH20O52QHSivQLvRFSW2hqnVTYuBv0CWn1L53cL+XoGw@mail.gmail.com","threadId":"38471","inReplyTo":"CAK6Z60cGnvPP0ctnyV4tHfMmBGYRidEJzkAG+J2h9BunmtOAsQ@mail.gmail.com","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Valery Yundin","fromEmail":"yuvalery@gmail.com","sentAt":"2015-01-31T14:52:11Z","receivedAt":"2015-01-31T14:52:11Z","isPatch":false,"sender":{"key":"yuvalery@gmail.com","avatar":"https://gravatar.com/avatar/c56698d0b66ab632cd85309da9a3b1794cb912fb9f37e0f0e488d15e3791672a?d=mp&s=160"},"body":"Hi,\n\nIt seems that the same commit dfa72fdb96 is responsible for the error\nin \"git svn clone https://geuz.org/svn/gmsh/trunk\". But unlike in my\ncase, the patch doesn't fix it.\n\nCheers,\nValery\n\n\nOn 31 January 2015 at 13:51, Nico Schlömer <nico.schloemer@gmail.com> wrote:\n> I tried the patch and I still get\n> ```\n> [...]\n> r100 = e2a9b5baa2cebb18591ecb04ff350410d52f36de (refs/remotes/git-svn)\n> Unexpected result returned from git cat-file at\n> /home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 335.\n> Failed to read object 619f9d1d857fb287d06a70c9dac6b8b534d0de6a at\n> /home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 336, <GEN16>\n> line 757.\n>\n> error closing pipe: Bad file descriptor at\n> /home/nschloe/libexec/git-core/git-svn line 0.\n> error closing pipe: Bad file descriptor at\n> /home/nschloe/libexec/git-core/git-svn line 0.\n> ```\n> when\n> ```\n> git svn clone https://geuz.org/svn/gmsh/trunk\n> ```\n>\n> Cheers,\n> Nico\n>\n> On Fri, Jan 30, 2015 at 2:30 AM, Eric Wong <normalperson@yhbt.net> wrote:\n>> Valery Yundin <yuvalery@gmail.com> wrote:\n>>> Hi,\n>>>\n>>> Your patch seems to fix the problem.\n>>> I tried several times and I can svn clone the whole repository in one\n>>> go without a crash.\n>>\n>> Thanks for the confirmation.  Cc-ing a few other folks who encountered\n>> this problem (and Bcc-ing some folks who emailed me privately).\n>>\n>> Can the rest of you give this patch a try on your respective platforms\n>> and confirm the fix?\n>>\n>> http://article.gmane.org/gmane.comp.version-control.git/263168/raw\n>> (also: http://mid.gmane.org/20150130002247.GA22519@dcvr.yhbt.net )\n>>\n>> Junio: assuming all goes well with testers, can you apply my patch\n>> to the appropriate maintenance tracks with Tested-by:s?\n>> I'm going offline in a few hours and don't think I'll be around\n>> much the next week or so.\n>>\n>> Thanks.\n>>\n>>> Thanks,\n>>> Valery\n>>>\n>>> On 30 January 2015 at 01:22, Eric Wong <normalperson@yhbt.net> wrote:\n>>> > Valery Yundin <yuvalery@gmail.com> wrote:\n>>> >> Hi,\n>>> >>\n>>> >> Here you go:\n>>> >> dfa72fdb96befbd790f623bb2909a347176753c2 is the first bad commit\n>>> >\n>>> > Thank you.  Can you give the following patch a try?\n>>> > I have not been able to reproduce the problem on my end.\n>>> > If it doesn't work out, I might be out of ideas for a bit :/\n>>> > Increasing --log-window-size will help you run longer without\n>>> > the error, but that's not ideal as it can also eat memory.\n>>> >\n>>> > -----------------------8<----------------------\n>>> > From: Eric Wong <normalperson@yhbt.net>\n>>> > Subject: [PATCH] git-svn: destroy all tempfiles when reloading RA\n>>> >\n>>> > This may fix the errors some users are seeing with:\n>>> > \"write .git/Git_svn_hash_XXXXXX: Bad file descriptor\"\n>>> >\n>>> > Thanks to Valery Yundin for helping bisect the problem introduced in\n>>> > commit dfa72fdb96befbd790f623bb2909a347176753c2\n>>> > (git-svn: reload RA every log-window-size)\n>>> >\n>>> > Cc: Valery Yundin <yuvalery@gmail.com>\n>>> > Signed-off-by: Eric Wong <normalperson@yhbt.net>\n>>> > ---\n>>> >  perl/Git.pm        | 6 ++++++\n>>> >  perl/Git/SVN/Ra.pm | 1 +\n>>> >  2 files changed, 7 insertions(+)\n>>> >\n>>> > diff --git a/perl/Git.pm b/perl/Git.pm\n>>> > index b5905ee..698018e 100644\n>>> > --- a/perl/Git.pm\n>>> > +++ b/perl/Git.pm\n>>> > @@ -1347,6 +1347,12 @@ sub temp_path {\n>>> >         $TEMP_FILES{$temp_fd}{fname};\n>>> >  }\n>>> >\n>>> > +sub temp_reset_all {\n>>> > +       unlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n>>> > +       %TEMP_FILEMAP = ();\n>>> > +       %TEMP_FILES = ();\n>>> > +}\n>>> > +\n>>> >  sub END {\n>>> >         unlink values %TEMP_FILEMAP if %TEMP_FILEMAP;\n>>> >  }\n>>> > diff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm\n>>> > index 622535e..878679d 100644\n>>> > --- a/perl/Git/SVN/Ra.pm\n>>> > +++ b/perl/Git/SVN/Ra.pm\n>>> > @@ -397,6 +397,7 @@ sub gs_fetch_loop_common {\n>>> >                 $_[0] = undef;\n>>> >                 $self = undef;\n>>> >                 $RA = undef;\n>>> > +               Git->temp_reset_all;\n>>> >                 $gpool->clear;\n>>> >                 $self = Git::SVN::Ra->new($ra_url);\n>>> >                 $ra_invalid = undef;\n>>> > --\n>>> > EW\n>>> --\n"},{"id":"256004","messageId":"20150212191807.GA30062@dcvr.yhbt.net","threadId":"38471","inReplyTo":"CABNxngNH20O52QHSivQLvRFSW2hqnVTYuBv0CWn1L53cL+XoGw@mail.gmail.com","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2015-02-12T19:18:07Z","receivedAt":"2015-02-12T19:18:07Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Valery Yundin <yuvalery@gmail.com> wrote:\n> On 31 January 2015 at 13:51, Nico Schlömer <nico.schloemer@gmail.com> wrote:\n> > I tried the patch and I still get\n> > ```\n> > [...]\n> > r100 = e2a9b5baa2cebb18591ecb04ff350410d52f36de (refs/remotes/git-svn)\n> > Unexpected result returned from git cat-file at\n> > /home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 335.\n> > Failed to read object 619f9d1d857fb287d06a70c9dac6b8b534d0de6a at\n> > /home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 336, <GEN16>\n> > line 757.\n> >\n> > error closing pipe: Bad file descriptor at\n> > /home/nschloe/libexec/git-core/git-svn line 0.\n> > error closing pipe: Bad file descriptor at\n> > /home/nschloe/libexec/git-core/git-svn line 0.\n> > ```\n> > when\n> > ```\n> > git svn clone https://geuz.org/svn/gmsh/trunk\n> \n> It seems that the same commit dfa72fdb96 is responsible for the error\n> in \"git svn clone https://geuz.org/svn/gmsh/trunk\". But unlike in my\n> case, the patch doesn't fix it.\n\n(top-posting corrected)\n\nOdd, I managed to clone that without issues, but I couldn't reproduce\nthis problem with or without the tempfile clearing patch applied.\n\n   git svn clone --username gmsh https://geuz.org/svn/gmsh/trunk\n\nAnybody else?\n"},{"id":"256153","messageId":"CAK6Z60dZu_0kNJF2-eJMgnjYVhxfyps53mcfZT41V1DBZZ+zhw@mail.gmail.com","threadId":"38471","inReplyTo":"20150212191807.GA30062@dcvr.yhbt.net","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Nico Schlömer","fromEmail":"nico.schloemer@gmail.com","sentAt":"2015-02-16T15:10:41Z","receivedAt":"2015-02-16T15:10:41Z","isPatch":false,"sender":{"key":"nico.schloemer@gmail.com","avatar":"https://gravatar.com/avatar/b0a6998b0c9bcd0641b4d07789be6f64e21fce87df71a94b952611c7994b9878?d=mp&s=160"},"body":"I just double-checked and I can only produce this issue on one machine\n(tested on 3). Apparently, this is has nothing to do with Git itself\nthen.\n\nAny ideas of what could be wrong?\n\nCheers,\nNico\n\nOn Thu, Feb 12, 2015 at 8:18 PM, Eric Wong <normalperson@yhbt.net> wrote:\n> Valery Yundin <yuvalery@gmail.com> wrote:\n>> On 31 January 2015 at 13:51, Nico Schlömer <nico.schloemer@gmail.com> wrote:\n>> > I tried the patch and I still get\n>> > ```\n>> > [...]\n>> > r100 = e2a9b5baa2cebb18591ecb04ff350410d52f36de (refs/remotes/git-svn)\n>> > Unexpected result returned from git cat-file at\n>> > /home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 335.\n>> > Failed to read object 619f9d1d857fb287d06a70c9dac6b8b534d0de6a at\n>> > /home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 336, <GEN16>\n>> > line 757.\n>> >\n>> > error closing pipe: Bad file descriptor at\n>> > /home/nschloe/libexec/git-core/git-svn line 0.\n>> > error closing pipe: Bad file descriptor at\n>> > /home/nschloe/libexec/git-core/git-svn line 0.\n>> > ```\n>> > when\n>> > ```\n>> > git svn clone https://geuz.org/svn/gmsh/trunk\n>>\n>> It seems that the same commit dfa72fdb96 is responsible for the error\n>> in \"git svn clone https://geuz.org/svn/gmsh/trunk\". But unlike in my\n>> case, the patch doesn't fix it.\n>\n> (top-posting corrected)\n>\n> Odd, I managed to clone that without issues, but I couldn't reproduce\n> this problem with or without the tempfile clearing patch applied.\n>\n>    git svn clone --username gmsh https://geuz.org/svn/gmsh/trunk\n>\n> Anybody else?\n"},{"id":"256186","messageId":"20150217061707.GA4308@dcvr.yhbt.net","threadId":"38471","inReplyTo":"CAK6Z60dZu_0kNJF2-eJMgnjYVhxfyps53mcfZT41V1DBZZ+zhw@mail.gmail.com","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2015-02-17T06:17:07Z","receivedAt":"2015-02-17T06:17:07Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Nico Schlömer <nico.schloemer@gmail.com> wrote:\n> I just double-checked and I can only produce this issue on one machine\n> (tested on 3). Apparently, this is has nothing to do with Git itself\n> then.\n> \n> Any ideas of what could be wrong?\n\nWhat's different about that one machine?\ne.g. SVN version, 64 vs 32-bit, Perl version, etc. could all be\nfactors (assuming identical git versions).\n\nAlso, any chance git was misinstalled somehow or your PATH was not\npointing to the correct git installation?\n\nThanks\n"},{"id":"256626","messageId":"801181a93d829d68b96c4b61d1ebdc3@74d39fa044aa309eaea14b9f57fe79c","threadId":"38471","inReplyTo":"20150217061707.GA4308@dcvr.yhbt.net","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Kyle J. McKay","fromEmail":"mackyle@gmail.com","sentAt":"2015-02-25T10:19:55Z","receivedAt":"2015-02-25T10:19:55Z","isPatch":false,"sender":{"key":"mackyle@gmail.com","avatar":"https://avatars.githubusercontent.com/u/813346?v=4"},"body":"I believe I have been able to track down this problem and implement a\nfix.  Please report back if this patch fixes the problem for you.\n\n-Kyle\n\n-- 8< --\nSubject: [PATCH] Git::SVN::Fetcher: avoid premature 'svn_hash' temp file closure\n\nSince b19138b (git-svn: Make it incrementally faster by minimizing temp\nfiles, v1.6.0), git-svn has been using the Git.pm temp_acquire and\ntemp_release mechanism to avoid unnecessary temp file churn and provide\na speed boost.\n\nHowever, that change introduced a call to temp_acquire inside the\nGit::SVN::Fetcher::close_file function for an 'svn_hash' temp file.\nBecause an SVN::Pool is active at the time this function is called, if\nthe Git::temp_acquire function ends up actually creating a new\nFileHandle for the temp file (which it will the first time it's called\nwith the name 'svn_hash') that FileHandle will end up in the SVN::Pool\nand should that pool have SVN::Pool::clear called on it that FileHandle\nwill be closed out from under Git::temp_acquire.\n\nSince the only call site to Git::temp_acquire with the name 'svn_hash'\nis inside the close_file function, if an 'svn_hash' temp file is ever\ncreated its FileHandle is guaranteed to be created in the active\nSVN::Pool.\n\nThis has not been a problem in the past because the SVN::Pool was not\nbeing cleared.  However, since dfa72fdb (git-svn: reload RA every\nlog-window-size, v2.2.0) the pool has been getting cleared periodically\nat which point the FileHandle for the 'svn_hash' temp file gets closed.\nAny subsequent calls to Git::temp_acquire for 'svn_hash', however,\nsucceed without creating/opening a new temporary file since it still has\nthe now invalid FileHandle in its cache.  Callers that then attempt to\nuse that FileHandle fail with an error.\n\nWe avoid this problem by making sure the 'svn_hash' temp file is created\nin the same place the 'svn_delta_...' and 'git_blob_...' temp files are\n(and then temp_release'd) so that it can be safely used inside the\nclose_file function without having its FileHandle end up in an SVN::Pool\nthat gets cleared.\n\nSigned-off-by: Kyle J. McKay <mackyle@gmail.com>\n---\n perl/Git/SVN/Fetcher.pm | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/perl/Git/SVN/Fetcher.pm b/perl/Git/SVN/Fetcher.pm\nindex 10edb277..613055a3 100644\n--- a/perl/Git/SVN/Fetcher.pm\n+++ b/perl/Git/SVN/Fetcher.pm\n@@ -322,6 +322,14 @@ sub apply_textdelta {\n \t# (but $base does not,) so dup() it for reading in close_file\n \topen my $dup, '<&', $fh or croak $!;\n \tmy $base = $::_repository->temp_acquire(\"git_blob_${$}_$suffix\");\n+\t# close_file may call temp_acquire on 'svn_hash', but because of the\n+\t# call chain, if the temp_acquire call from close_file ends up being the\n+\t# call that first creates the 'svn_hash' temp file, then the FileHandle\n+\t# that's created as a result will end up in an SVN::Pool that we clear\n+\t# in SVN::Ra::gs_fetch_loop_common.  Avoid that by making sure the\n+\t# 'svn_hash' FileHandle is already created before close_file is called.\n+\tmy $tmp_fh = $::_repository->temp_acquire('svn_hash');\n+\t$::_repository->temp_release($tmp_fh, 1);\n \n \tif ($fb->{blob}) {\n \t\tmy ($base_is_link, $size);\n--\n"},{"id":"256656","messageId":"20150225200833.GA10031@dcvr.yhbt.net","threadId":"38471","inReplyTo":"CAK6Z60fqEkM_tON6tcnwBqJzBCvLB=eVJdyXSnNb7N1iR_DSsw@mail.gmail.com","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2015-02-25T20:08:33Z","receivedAt":"2015-02-25T20:08:33Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Nico Schlömer <nico.schloemer@gmail.com> wrote:\n> Thanks Kyle for the patch! I applied it and ran\n> ```\n> git svn clone https://geuz.org/svn/gmsh/trunk\n> ```\n> Unfortunately, I'm still getting\n> ```\n> [...]\n> r100 = e2a9b5baa2cebb18591ecb04ff350410d52f36de (refs/remotes/git-svn)\n> error closing pipe: Bad file descriptor at\n> /usr/share/perl5/Git/SVN/Fetcher.pm line 335.\n> error closing pipe: Bad file descriptor at\n> /usr/share/perl5/Git/SVN/Fetcher.pm line 335.\n> out pipe went bad at /usr/share/perl5/Git.pm line 955.\n\nThanks for testing, is this with or without my other attempt at\nfixing this problem applied?\n\nhttp://mid.gmane.org/20150130002247.GA22519@dcvr.yhbt.net\n(\"git-svn: destroy all tempfiles when reloading RA\")\n\nAnd can you try testing both with/without that patch if you\nhaven't already?  Thanks again.\n"},{"id":"256659","messageId":"8FAF534A-812E-4FBD-AEE4-907C83CEC007@gmail.com","threadId":"38471","inReplyTo":"CAK6Z60fqEkM_tON6tcnwBqJzBCvLB=eVJdyXSnNb7N1iR_DSsw@mail.gmail.com","subject":"Re: git svn import failure : write .git/Git_svn_hash_BmjclS: Bad file descriptor","fromName":"Kyle J. McKay","fromEmail":"mackyle@gmail.com","sentAt":"2015-02-25T20:33:59Z","receivedAt":"2015-02-25T20:33:59Z","isPatch":false,"sender":{"key":"mackyle@gmail.com","avatar":"https://avatars.githubusercontent.com/u/813346?v=4"},"body":"On Feb 25, 2015, at 07:07, Nico Schlömer wrote:\n\n> Thanks Kyle for the patch! I applied it and ran\n> ```\n> git svn clone https://geuz.org/svn/gmsh/trunk\n> ```\n> Unfortunately, I'm still getting\n> ```\n> [...]\n> r100 = e2a9b5baa2cebb18591ecb04ff350410d52f36de (refs/remotes/git-svn)\n> error closing pipe: Bad file descriptor at /usr/share/perl5/Git/SVN/ \n> Fetcher.pm line 335.\n> error closing pipe: Bad file descriptor at /usr/share/perl5/Git/SVN/ \n> Fetcher.pm line 335.\n> out pipe went bad at /usr/share/perl5/Git.pm line 955.\n> ```\n\nAre you certain you're running with the patch?  I ask because earlier  \nyou sent this:\n\nOn Jan 31, 2015, at 04:51, Nico Schlömer wrote:\n\n> I tried the patch and I still get\n> ```\n> [...]\n> r100 = e2a9b5baa2cebb18591ecb04ff350410d52f36de (refs/remotes/git-svn)\n> Unexpected result returned from git cat-file at\n> /home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 335.\n> Failed to read object 619f9d1d857fb287d06a70c9dac6b8b534d0de6a at\n> /home/nschloe/share/perl/5.18.2/Git/SVN/Fetcher.pm line 336, <GEN16>\n> line 757.\n>\n> error closing pipe: Bad file descriptor at\n> /home/nschloe/libexec/git-core/git-svn line 0.\n> error closing pipe: Bad file descriptor at\n> /home/nschloe/libexec/git-core/git-svn line 0.\n> ```\n> when\n> ```\n> git svn clone https://geuz.org/svn/gmsh/trunk\n> ```\n\nAnd as the patch below applies to Fetcher.pm at line 322 and inserts 8  \nlines, I do not see how you could be getting the same error message at  \nline 335 if the patch was present.  Even if you manually applied it by  \nonly inserting the two lines of code, the line numbers would be at  \nleast 337, not 335 as reported above.  I also notice the path to  \nFetcher.pm is different from your earlier report.\n\nThat said, the fact that it happens right after r100 (which is when  \nSVN::Pool::clear is getting called) suggests there's another file  \nhandle getting caught up in the SVN::Pool somehow.  (When I try to  \nclone gmsh, I got to r4885 before a server disconnection.)\n\nIt could be as simple as the open2 call FileHandle result in later  \nperl versions ends up in the SVN::Pool whereas in earlier Perl and/or  \nsvn versions it does not.\n\nWhat happens if you add this change on top of the Fetcher.pm change:\n\ndiff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm\nindex 622535e2..96888228 100644\n--- a/perl/Git/SVN/Ra.pm\n+++ b/perl/Git/SVN/Ra.pm\n@@ -391,6 +391,7 @@ sub longest_common_path {\n  sub gs_fetch_loop_common {\n  \tmy ($self, $base, $head, $gsv, $globs) = @_;\n  \treturn if ($base > $head);\n+\t$::_repository->_open_cat_blob_if_needed;\n  \tmy $gpool = SVN::Pool->new_default;\n  \tmy $ra_url = $self->url;\n  \tmy $reload_ra = sub {\n\n-Kyle\n\n> Cheers,\n> Nico\n>\n> On Wed, Feb 25, 2015 at 11:20 AM Kyle J. McKay <mackyle@gmail.com>  \n> wrote:\n> I believe I have been able to track down this problem and implement a\n> fix.  Please report back if this patch fixes the problem for you.\n>\n> -Kyle\n>\n> -- 8< --\n> Subject: [PATCH] Git::SVN::Fetcher: avoid premature 'svn_hash' temp  \n> file closure\n>\n> Since b19138b (git-svn: Make it incrementally faster by minimizing  \n> temp\n> files, v1.6.0), git-svn has been using the Git.pm temp_acquire and\n> temp_release mechanism to avoid unnecessary temp file churn and  \n> provide\n> a speed boost.\n>\n> However, that change introduced a call to temp_acquire inside the\n> Git::SVN::Fetcher::close_file function for an 'svn_hash' temp file.\n> Because an SVN::Pool is active at the time this function is called, if\n> the Git::temp_acquire function ends up actually creating a new\n> FileHandle for the temp file (which it will the first time it's called\n> with the name 'svn_hash') that FileHandle will end up in the SVN::Pool\n> and should that pool have SVN::Pool::clear called on it that  \n> FileHandle\n> will be closed out from under Git::temp_acquire.\n>\n> Since the only call site to Git::temp_acquire with the name 'svn_hash'\n> is inside the close_file function, if an 'svn_hash' temp file is ever\n> created its FileHandle is guaranteed to be created in the active\n> SVN::Pool.\n>\n> This has not been a problem in the past because the SVN::Pool was not\n> being cleared.  However, since dfa72fdb (git-svn: reload RA every\n> log-window-size, v2.2.0) the pool has been getting cleared  \n> periodically\n> at which point the FileHandle for the 'svn_hash' temp file gets  \n> closed.\n> Any subsequent calls to Git::temp_acquire for 'svn_hash', however,\n> succeed without creating/opening a new temporary file since it still  \n> has\n> the now invalid FileHandle in its cache.  Callers that then attempt to\n> use that FileHandle fail with an error.\n>\n> We avoid this problem by making sure the 'svn_hash' temp file is  \n> created\n> in the same place the 'svn_delta_...' and 'git_blob_...' temp files  \n> are\n> (and then temp_release'd) so that it can be safely used inside the\n> close_file function without having its FileHandle end up in an  \n> SVN::Pool\n> that gets cleared.\n>\n> Signed-off-by: Kyle J. McKay <mackyle@gmail.com>\n> ---\n>  perl/Git/SVN/Fetcher.pm | 8 ++++++++\n>  1 file changed, 8 insertions(+)\n>\n> diff --git a/perl/Git/SVN/Fetcher.pm b/perl/Git/SVN/Fetcher.pm\n> index 10edb277..613055a3 100644\n> --- a/perl/Git/SVN/Fetcher.pm\n> +++ b/perl/Git/SVN/Fetcher.pm\n> @@ -322,6 +322,14 @@ sub apply_textdelta {\n>         # (but $base does not,) so dup() it for reading in close_file\n>         open my $dup, '<&', $fh or croak $!;\n>         my $base = $::_repository->temp_acquire(\"git_blob_${$}_ \n> $suffix\");\n> +       # close_file may call temp_acquire on 'svn_hash', but  \n> because of the\n> +       # call chain, if the temp_acquire call from close_file ends  \n> up being the\n> +       # call that first creates the 'svn_hash' temp file, then the  \n> FileHandle\n> +       # that's created as a result will end up in an SVN::Pool  \n> that we clear\n> +       # in SVN::Ra::gs_fetch_loop_common.  Avoid that by making  \n> sure the\n> +       # 'svn_hash' FileHandle is already created before close_file  \n> is called.\n> +       my $tmp_fh = $::_repository->temp_acquire('svn_hash');\n> +       $::_repository->temp_release($tmp_fh, 1);\n>\n>         if ($fb->{blob}) {\n>                 my ($base_is_link, $size);\n> --\n"}]}