{"thread":{"id":"39090","subject":"Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","startedAt":"2015-04-16T10:03:59Z","lastAt":"2015-04-23T06:52:24Z","messageCount":13,"participants":["Andreas Mohr","Thomas Braun","Johannes Schindelin","Jeff King","David Miller","rupert thurner"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"259481","messageId":"20150416100359.GA19951@rhlx01.hs-esslingen.de","threadId":"39090","inReplyTo":null,"subject":"Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Andreas Mohr","fromEmail":"andi@lisas.de","sentAt":"2015-04-16T10:03:59Z","receivedAt":"2015-04-16T10:03:59Z","isPatch":false,"sender":{"key":"andi@lisas.de","avatar":null},"body":"Hi all,\n\nover the years I've had the same phenomenon with various versions of msysgit\n(now at 1.9.5.msysgit.0, on Windows 7 64bit), so I'm now sufficiently\nconfident of it being a long-standing, longer-term issue and thus I'm\nreporting it now.\n\nSince I'm doing development in a sufficiently rebase-heavy manner,\nI seem to aggregate a lot of objects.\nThus, when fetching content I'm sufficiently frequently greeted with\na git gc run.\nThis, however, does not work fully reliably:\n\n    Auto packing the repository for optimum performance. You may also\n    run \"git gc\" manually. See \"git help gc\" for more information.\n    Counting objects: 206527, done.\n    Delta compression using up to 4 threads.\n    Compressing objects: 100% (27430/27430), done.\n    Writing objects: 100% (206527/206527), done.\n    Total 206527 (delta 178632), reused 206527 (delta 178632)\n    Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.pack' failed. Should I try again? (y/n) n\n    Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.idx' failed. Should I try again? (y/n) n\n    Checking connectivity: 206527, done.\n\nA workable workaround for this recurring issue\n(such a fetch will fail repeatedly,\nthereby hampering my ability to update properly)\nis to manually do a \"git gc --auto\"\nprior to the fetch (which will then succeed).\n\n\n\n\n-- \n¿umop apisdn upside down?\n(by daniweb.com user Bench)\n"},{"id":"259483","messageId":"552F98AC.5030603@virtuell-zuhause.de","threadId":"39090","inReplyTo":"20150416100359.GA19951@rhlx01.hs-esslingen.de","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Thomas Braun","fromEmail":"thomas.braun@virtuell-zuhause.de","sentAt":"2015-04-16T11:10:36Z","receivedAt":"2015-04-16T11:10:36Z","isPatch":false,"sender":{"key":"thomas.braun@virtuell-zuhause.de","avatar":"https://avatars.githubusercontent.com/u/1185677?v=4"},"body":"Am 16.04.2015 um 12:03 schrieb Andreas Mohr:\n> Hi all,\n> \n> over the years I've had the same phenomenon with various versions of msysgit\n> (now at 1.9.5.msysgit.0, on Windows 7 64bit), so I'm now sufficiently\n> confident of it being a long-standing, longer-term issue and thus I'm\n> reporting it now.\n\n(CC'ing msysgit)\n\nHi Andreas,\n\n> Since I'm doing development in a sufficiently rebase-heavy manner,\n> I seem to aggregate a lot of objects.\n> Thus, when fetching content I'm sufficiently frequently greeted with\n> a git gc run.\n> This, however, does not work fully reliably:\n> \n>     Auto packing the repository for optimum performance. You may also\n>     run \"git gc\" manually. See \"git help gc\" for more information.\n>     Counting objects: 206527, done.\n>     Delta compression using up to 4 threads.\n>     Compressing objects: 100% (27430/27430), done.\n>     Writing objects: 100% (206527/206527), done.\n>     Total 206527 (delta 178632), reused 206527 (delta 178632)\n>     Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.pack' failed. Should I try again? (y/n) n\n>     Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.idx' failed. Should I try again? (y/n) n\n>     Checking connectivity: 206527, done.\n> \n> A workable workaround for this recurring issue\n> (such a fetch will fail repeatedly,\n> thereby hampering my ability to update properly)\n> is to manually do a \"git gc --auto\"\n> prior to the fetch (which will then succeed).\n\nI've never had this issue. The error message from unlinking the file\nmeans that someone is still accessing the file and thus it can not be\ndeleted (due to the implicit file locking on windows).\n\nCan you reproduce the error reliably?\n\nThomas\n\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"Git for Windows\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"259484","messageId":"27f1120c2c5231d8c7add8bdac7e3b21@www.dscho.org","threadId":"39090","inReplyTo":"552F98AC.5030603@virtuell-zuhause.de","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2015-04-16T11:31:02Z","receivedAt":"2015-04-16T11:31:02Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn 2015-04-16 13:10, Thomas Braun wrote:\n> Am 16.04.2015 um 12:03 schrieb Andreas Mohr:\n>>\n>> over the years I've had the same phenomenon with various versions of msysgit\n>> (now at 1.9.5.msysgit.0, on Windows 7 64bit), so I'm now sufficiently\n>> confident of it being a long-standing, longer-term issue and thus I'm\n>> reporting it now.\n> \n> (CC'ing msysgit)\n\nGood idea.\n\n>> Since I'm doing development in a sufficiently rebase-heavy manner,\n>> I seem to aggregate a lot of objects.\n>> Thus, when fetching content I'm sufficiently frequently greeted with\n>> a git gc run.\n>> This, however, does not work fully reliably:\n>>\n>>     Auto packing the repository for optimum performance. You may also\n>>     run \"git gc\" manually. See \"git help gc\" for more information.\n>>     Counting objects: 206527, done.\n>>     Delta compression using up to 4 threads.\n>>     Compressing objects: 100% (27430/27430), done.\n>>     Writing objects: 100% (206527/206527), done.\n>>     Total 206527 (delta 178632), reused 206527 (delta 178632)\n>>     Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.pack' failed. Should I try again? (y/n) n\n>>     Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.idx' failed. Should I try again? (y/n) n\n>>     Checking connectivity: 206527, done.\n>>\n>> A workable workaround for this recurring issue\n>> (such a fetch will fail repeatedly,\n>> thereby hampering my ability to update properly)\n>> is to manually do a \"git gc --auto\"\n>> prior to the fetch (which will then succeed).\n> \n> I've never had this issue. The error message from unlinking the file\n> means that someone is still accessing the file and thus it can not be\n> deleted (due to the implicit file locking on windows).\n\nBest guess is that an antivirus is still accessing it. There is a tool called `WhoUses.exe` in msysGit (I do not remember if I included it into Git for Windows 1.x for end users) which could be used to figure out which process accesses a given file still: https://github.com/msysgit/msysgit/blob/master/mingw/bin/WhoUses.exe (maybe that would help you identify the cause of the problem).\n\nCiao,\nJohannes\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"Git for Windows\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"259485","messageId":"20150416113505.GA30818@rhlx01.hs-esslingen.de","threadId":"39090","inReplyTo":"552F98AC.5030603@virtuell-zuhause.de","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Andreas Mohr","fromEmail":"andi@lisas.de","sentAt":"2015-04-16T11:35:05Z","receivedAt":"2015-04-16T11:35:05Z","isPatch":false,"sender":{"key":"andi@lisas.de","avatar":null},"body":"Hi,\n\nsorry, I had sent the prior mail prematurely (hit wrong key)\nand have been busy working on the resubmission...\n\nOn Thu, Apr 16, 2015 at 01:10:36PM +0200, Thomas Braun wrote:\n> Am 16.04.2015 um 12:03 schrieb Andreas Mohr:\n> > Hi all,\n> > \n> > over the years I've had the same phenomenon with various versions of msysgit\n> > (now at 1.9.5.msysgit.0, on Windows 7 64bit), so I'm now sufficiently\n> > confident of it being a long-standing, longer-term issue and thus I'm\n> > reporting it now.\n\n(I've had experience with this issue as early as 1.7.x, I believe).\n\n\n> (CC'ing msysgit)\n> \n> Hi Andreas,\n> \n> > Since I'm doing development in a sufficiently rebase-heavy manner,\n> > I seem to aggregate a lot of objects.\n> > Thus, when fetching content I'm sufficiently frequently greeted with\n> > a git gc run.\n> > This, however, does not work fully reliably:\n> > \n> >     Auto packing the repository for optimum performance. You may also\n> >     run \"git gc\" manually. See \"git help gc\" for more information.\n> >     Counting objects: 206527, done.\n> >     Delta compression using up to 4 threads.\n> >     Compressing objects: 100% (27430/27430), done.\n> >     Writing objects: 100% (206527/206527), done.\n> >     Total 206527 (delta 178632), reused 206527 (delta 178632)\n> >     Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.pack' failed. Should I try again? (y/n) n\n> >     Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.idx' failed. Should I try again? (y/n) n\n> >     Checking connectivity: 206527, done.\n> > \n> > A workable workaround for this recurring issue\n> > (such a fetch will fail repeatedly,\n> > thereby hampering my ability to update properly)\n> > is to manually do a \"git gc --auto\"\n> > prior to the fetch (which will then succeed).\n> \n> I've never had this issue. The error message from unlinking the file\n> means that someone is still accessing the file and thus it can not be\n> deleted (due to the implicit file locking on windows).\n> \n> Can you reproduce the error reliably?\n\nIt seems to be reproducible pretty reliably,\nat least once git thinks it needs to repack (initiated by a fetch operation, I think),\n*and* then the unlink issue successfully turned up\n(which may happen perhaps every 20 fetches of a *very* rebase-heavy workflow).\n\n\nInterim mail content:\n\nI strongly suspect that git's repacking implementation\n(probably unrelated to msysgit-specific deviations,\nIOW, git *core* handling)\nsimply is buggy\nin that it may keep certain file descriptors open\nat least a certain time (depending on scope of implementation/objects!?)\nbeyond having finished its operation (rename?).\nAs a related note, in an unrelated application of mine\nI also encountered issues on Windows\nwhere renaming of in-use files and further use of these files/names\nthen failed (error code was EACCES I believe).\nIOW, this seems to be an issue specific to\nWindows' \"special\" (and sometimes quirky) filesystem handling\nwhich probably does not turn up on many \"other\" platforms,\nthus a historic existing implementation weakness in git's repack handling\ncould not be nailed down in a sufficiently easy manner.\n\n\n\n\nI think I may have the order wrong, however:\nHandling seems to be:\n- repack needed\n- counting objects\n- compressing\n- writing\n- unlink (delete) of all prior non-repacked objects (which fails)\n\n\nI have to admit that at this point in time I'm actually unsure\nwhich higher-level operation it actually is\nthat gets carried out where eventually a repack *implicitly* gets triggered\n(I've got a shell script here which implements clean branch updating,\nwhere I eventually hit the problem during its daily use).\n\n\nSince a standalone git gc --auto *immediately* appears to work\n(after many repeated attempts of failing full-update),\nthis is a strong hint that (in the failure case)\nit's the *PRIOR* (non-repack) operation\nwhich has kept these objects open beyond its actual operation scope.\n\n\nSuspected implementation sample code:\n\nif (operation_needed)\n{\n  operation_workingset set;\n\n  set.DoStuff();\n\n  if (repack_needed)\n  {\n    repack_handler repack;\n\n     repack.DoStuff();\n  }\n}\n\n[NOTE the very prominent scope issues in this example,\nwhich might be the exact reason for hitting such unlink failures -\nsimply due to having kept file descriptors open within the working set]\n\nI have not had a look at git source though\nto actually determine whether there do exist\nsuch severe operation scope issues\nthat I'm strongly contemplating.\n\nAndreas Mohr\n"},{"id":"259486","messageId":"20150416114235.GB30818@rhlx01.hs-esslingen.de","threadId":"39090","inReplyTo":"27f1120c2c5231d8c7add8bdac7e3b21@www.dscho.org","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Andreas Mohr","fromEmail":"andi@lisas.de","sentAt":"2015-04-16T11:42:35Z","receivedAt":"2015-04-16T11:42:35Z","isPatch":false,"sender":{"key":"andi@lisas.de","avatar":null},"body":"Hi,\n\nOn Thu, Apr 16, 2015 at 01:31:02PM +0200, Johannes Schindelin wrote:\n> Hi,\n> \n> On 2015-04-16 13:10, Thomas Braun wrote:\n> > I've never had this issue. The error message from unlinking the file\n> > means that someone is still accessing the file and thus it can not be\n> > deleted (due to the implicit file locking on windows).\n> \n> Best guess is that an antivirus is still accessing it. There is a tool called `WhoUses.exe` in msysGit (I do not remember if I included it into Git for Windows 1.x for end users) which could be used to figure out which process accesses a given file still: https://github.com/msysgit/msysgit/blob/master/mingw/bin/WhoUses.exe (maybe that would help you identify the cause of the problem).\n\nOh my. Botched mail conversation...\nI tried to f'up on this messy start ASAP, so I even managed to omit this final *pre-existing* part:\n\"\nPlease note that this system is hampered by a crappy virus scanner\ndependency (F-Secure),\nwhich could be the culprit for this issue (e.g. by keeping files busy\nfor longer than expected),\nhowever I really don't think that it takes part in this issue.\n\"\n\nThe reason that I suspect that it's not virus scanner related is:\n- standalone git gc --auto works immediately\n  (hmm but this might also point at the opposite - namely virus scanner\n  still accessing files of a prior operation only in case there *was*\n  a prior operation)\n- file descriptor scope handling issue in git source code is very easily imaginable\n- only a very rebase-heavy workflow of a sufficiently large repo\n  is likely to have this issue turn up in a frequently enough manner,\n  thus it's quite likely that it's not observed (or reported) all too often\n\nThanks,\n\nAndreas Mohr\n\n-- \nGNU/Linux. It's not the software that's free, it's you.\n"},{"id":"259487","messageId":"20150416114846.GC30818@rhlx01.hs-esslingen.de","threadId":"39090","inReplyTo":"20150416114235.GB30818@rhlx01.hs-esslingen.de","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Andreas Mohr","fromEmail":"andi@lisas.de","sentAt":"2015-04-16T11:48:46Z","receivedAt":"2015-04-16T11:48:46Z","isPatch":false,"sender":{"key":"andi@lisas.de","avatar":null},"body":"On Thu, Apr 16, 2015 at 01:42:35PM +0200, Andreas Mohr wrote:\n> Hi,\n> \n> On Thu, Apr 16, 2015 at 01:31:02PM +0200, Johannes Schindelin wrote:\n> > Hi,\n> > \n> > On 2015-04-16 13:10, Thomas Braun wrote:\n> > > I've never had this issue. The error message from unlinking the file\n> > > means that someone is still accessing the file and thus it can not be\n> > > deleted (due to the implicit file locking on windows).\n> > \n> > Best guess is that an antivirus is still accessing it. There is a tool called `WhoUses.exe` in msysGit (I do not remember if I included it into Git for Windows 1.x for end users) which could be used to figure out which process accesses a given file still: https://github.com/msysgit/msysgit/blob/master/mingw/bin/WhoUses.exe (maybe that would help you identify the cause of the problem).\n> \n> Oh my. Botched mail conversation...\n> I tried to f'up on this messy start ASAP, so I even managed to omit this final *pre-existing* part:\n> \"\n> Please note that this system is hampered by a crappy virus scanner\n> dependency (F-Secure),\n> which could be the culprit for this issue (e.g. by keeping files busy\n> for longer than expected),\n> however I really don't think that it takes part in this issue.\n> \"\n> \n> The reason that I suspect that it's not virus scanner related is:\n> - standalone git gc --auto works immediately\n>   (hmm but this might also point at the opposite - namely virus scanner\n>   still accessing files of a prior operation only in case there *was*\n>   a prior operation)\n> - file descriptor scope handling issue in git source code is very easily imaginable\n> - only a very rebase-heavy workflow of a sufficiently large repo\n>   is likely to have this issue turn up in a frequently enough manner,\n>   thus it's quite likely that it's not observed (or reported) all too often\n\nOK, at this point in time it's my turn to actually verify\nthat indeed it's NOT the virus scanner:\n- generate rebase-heavy activity\n- update\n- hit issue\n- unload virus (~ scanner?? I'm unsure on exact terminology to be used ;-)\n- update\n- profit!?\n\n(and possibly have a try at WhoUses.exe there, too - thanks for the hint!)\n\nAndreas Mohr\n"},{"id":"259489","messageId":"20150416123552.GA825@rhlx01.hs-esslingen.de","threadId":"39090","inReplyTo":"20150416114846.GC30818@rhlx01.hs-esslingen.de","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Andreas Mohr","fromEmail":"andi@lisas.de","sentAt":"2015-04-16T12:35:52Z","receivedAt":"2015-04-16T12:35:52Z","isPatch":false,"sender":{"key":"andi@lisas.de","avatar":null},"body":"On Thu, Apr 16, 2015 at 01:48:46PM +0200, Andreas Mohr wrote:\n> OK, at this point in time it's my turn to actually verify\n> that indeed it's NOT the virus scanner:\n> - generate rebase-heavy activity\n> - update\n> - hit issue\n> - unload virus (~ scanner?? I'm unsure on exact terminology to be used ;-)\n> - update\n> - profit!?\n\nDespite trying hard (generating a lot of activity, with different repo projects even)\nI cannot reproduce it in a timely manner,\nthus I'll have to wait until repo state has degraded in a sufficient manner\nfor such a larger repack with that issue to occur again\n(probably a matter of weeks).\nOnce it happens, I will:\n- ensure keeping a copy of the entire (problematic-state) repo, and verify reproducibility of its (copied/preserved) breakage\n- unload virus and do other tests\n- report back\n\nAndreas Mohr\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"Git for Windows\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"259493","messageId":"f046adea6791865f97f6d76b0dae01bc@www.dscho.org","threadId":"39090","inReplyTo":"20150416123552.GA825@rhlx01.hs-esslingen.de","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2015-04-16T13:07:46Z","receivedAt":"2015-04-16T13:07:46Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Andreas,\n\nOn 2015-04-16 14:35, Andreas Mohr wrote:\n> On Thu, Apr 16, 2015 at 01:48:46PM +0200, Andreas Mohr wrote:\n>> OK, at this point in time it's my turn to actually verify\n>> that indeed it's NOT the virus scanner:\n>> - generate rebase-heavy activity\n>> - update\n>> - hit issue\n>> - unload virus (~ scanner?? I'm unsure on exact terminology to be used ;-)\n>> - update\n>> - profit!?\n> \n> Despite trying hard (generating a lot of activity, with different repo\n> projects even)\n> I cannot reproduce it in a timely manner,\n> thus I'll have to wait until repo state has degraded in a sufficient manner\n> for such a larger repack with that issue to occur again\n> (probably a matter of weeks).\n> Once it happens, I will:\n> - ensure keeping a copy of the entire (problematic-state) repo, and\n> verify reproducibility of its (copied/preserved) breakage\n> - unload virus and do other tests\n> - report back\n\nI guess the best way to trigger it is by ensuring that a lot of loose objects are accumulated, e.g. by running\n\n```sh\ni=$(date +%s)\nj=0\nwhile test $j -lt 9999\ndo\n    echo \"test $(($i+$j))\" git hash-object -w --stdin\n    j=$(($j+1))\ndone\n```\n\nCiao,\nJohannes\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"Git for Windows\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"259503","messageId":"20150416152849.GA30137@peff.net","threadId":"39090","inReplyTo":"20150416113505.GA30818@rhlx01.hs-esslingen.de","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-16T15:28:50Z","receivedAt":"2015-04-16T15:28:50Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 16, 2015 at 01:35:05PM +0200, Andreas Mohr wrote:\n\n> I strongly suspect that git's repacking implementation\n> (probably unrelated to msysgit-specific deviations,\n> IOW, git *core* handling)\n> simply is buggy\n> in that it may keep certain file descriptors open\n> at least a certain time (depending on scope of implementation/objects!?)\n> beyond having finished its operation (rename?).\n\nHrm. I do not see anything in builtin/fetch.c that closes the packfile\ndescriptors before running \"gc --auto\". So basically the sequence:\n\n  1. Fetch performs actual fetch. It needs to open packfiles to do\n     commit negotiation with other side (the hard work is done\n     by an index-pack subprocess, but it is likely we have to access\n     _some_ objects).\n\n  2. The packfiles remain open and mmap'd (at least on Linux) in the\n     sha1_file.c:packed_git list.\n\n  3. We spawn \"gc --auto\" and wait for it to finish. While we are\n     waiting, the descriptors are still open, but \"gc --auto\" will not be\n     able to delete any packs.\n\nBut this seems too simple to be the problem, as it would mean that just\nabout any \"gc --auto\" that triggers a full repack would be a problem (so\nanytime you have about 50 packs). But maybe the gc \"autodetach\" behavior\nmeans it works racily.\n\nI was able to set up the situation deterministically by running the\nscript below:\n\n-- >8 --\n#!/bin/sh\n\n# XXX tweak this setting as appropriate\nPATH_TO_GIT_BUILD=$HOME/compile/git\nPATH=$PATH_TO_GIT_BUILD/bin-wrappers:$PATH\nrm -rf parent child\n\n# make a parent/child where the child will have to access\n# a packfile to fulfill another fetch\ngit init parent &&\ngit -C parent commit --allow-empty -m base &&\ngit clone parent child &&\ngit -C parent commit --allow-empty -m extra &&\n\n# we want to make our base pack really big, because otherwise\n# git will open/mmap/close it. So we must exceed core.packedgitlimit\ncd child &&\n$PATH_TO_GIT_BUILD/test-genrandom foo 5000000 >file &&\ngit add file &&\ngit commit -m large file &&\ngit repack -ad &&\ngit config core.packedGitLimit 1M &&\n\n# now make some spare packs to bust the gc.autopacklimit\nfor i in 1 2 3 4 5; do\n\tgit commit --allow-empty -m $i &&\n\tgit repack -d\ndone &&\ngit config gc.autoPackLimit 3 &&\ngit config gc.autoDetach false &&\nGIT_TRACE=1 git fetch\n```\n\nI also instrumented my (v1.9.5) git build like this:\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 025bc3e..fc99e5e 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1174,6 +1174,12 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tlist.strdup_strings = 1;\n \tstring_list_clear(&list, 0);\n \n+\t{\n+\t\tstruct packed_git *p;\n+\t\tfor (p = packed_git; p; p = p->next)\n+\t\t\ttrace_printf(\"pack %s has descriptor %d\\n\",\n+\t\t\t\t     p->pack_name, p->pack_fd);\n+\t}\n \trun_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\n \n \treturn result;\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex bb2314c..e8b29cf 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -105,6 +105,7 @@ static void remove_redundant_pack(const char *dir_name, const char *base_name)\n \tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n \t\tstrbuf_setlen(&buf, plen);\n \t\tstrbuf_addstr(&buf, exts[i]);\n+\t\ttrace_printf(\"unlinking %s\\n\", buf.buf);\n \t\tunlink(buf.buf);\n \t}\n \tstrbuf_release(&buf);\n\nto confirm what was happening (because of course on Linux it is\nperfectly fine to delete the open file). If this does trigger the bug\nfor you, though, it should be obvious even without the trace calls. :)\n\n-Peff\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"Git for Windows\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"259507","messageId":"a710472d7bf37757b7341dd99f8c76f3@www.dscho.org","threadId":"39090","inReplyTo":"20150416152849.GA30137@peff.net","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2015-04-16T15:48:42Z","receivedAt":"2015-04-16T15:48:42Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn 2015-04-16 17:28, Jeff King wrote:\n> On Thu, Apr 16, 2015 at 01:35:05PM +0200, Andreas Mohr wrote:\n> \n>> I strongly suspect that git's repacking implementation\n>> (probably unrelated to msysgit-specific deviations,\n>> IOW, git *core* handling)\n>> simply is buggy\n>> in that it may keep certain file descriptors open\n>> at least a certain time (depending on scope of implementation/objects!?)\n>> beyond having finished its operation (rename?).\n> \n> Hrm. [... detailed analysis, including a Minimal, Complete & Verifiable Example ...]\n\nThank you so much! I will definitely test this (at the moment, I have to recreate my build environment in a different VM than I used so far, that takes quite some time...)\n\nThanks!\nDscho\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"Git for Windows\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"259510","messageId":"20150416.115655.1025728315437891295.davem@davemloft.net","threadId":"39090","inReplyTo":"a710472d7bf37757b7341dd99f8c76f3@www.dscho.org","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"David Miller","fromEmail":"davem@davemloft.net","sentAt":"2015-04-16T15:56:55Z","receivedAt":"2015-04-16T15:56:55Z","isPatch":false,"sender":{"key":"davem@davemloft.net","avatar":null},"body":"\nPlease remove git-owner from the CC: list in future replies, thank\nyou. :-)\n"},{"id":"259540","messageId":"20150416205658.GB20385@rhlx01.hs-esslingen.de","threadId":"39090","inReplyTo":"a710472d7bf37757b7341dd99f8c76f3@www.dscho.org","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"Andreas Mohr","fromEmail":"andi@lisas.de","sentAt":"2015-04-16T20:56:58Z","receivedAt":"2015-04-16T20:56:58Z","isPatch":false,"sender":{"key":"andi@lisas.de","avatar":null},"body":"[git-owner CC dutifully removed]\n\nOn Thu, Apr 16, 2015 at 05:48:42PM +0200, Johannes Schindelin wrote:\n> Hi Peff,\n> \n> On 2015-04-16 17:28, Jeff King wrote:\n> > On Thu, Apr 16, 2015 at 01:35:05PM +0200, Andreas Mohr wrote:\n> > \n> >> I strongly suspect that git's repacking implementation\n> >> (probably unrelated to msysgit-specific deviations,\n> >> IOW, git *core* handling)\n> >> simply is buggy\n> >> in that it may keep certain file descriptors open\n> >> at least a certain time (depending on scope of implementation/objects!?)\n> >> beyond having finished its operation (rename?).\n> > \n> > Hrm. [... detailed analysis, including a Minimal, Complete & Verifiable Example ...]\n> \n> Thank you so much! I will definitely test this (at the moment, I have to recreate my build environment in a different VM than I used so far, that takes quite some time...)\n\nYour hash-object script successfully and with ease\nmanaged to provoke the issue again, thanks a lot!\n(syntax issue though: missed a '|' pipe).\n\nAnd I then did some unload tests (force-unloaded, via End Process Tree) of the virus,\nand the unlink issue persisted\n(but to be truly certain, I would have to rename away\nthe entire virus installation tree).\nNot to mention that it already looks anyway\nlike we seem to be on the way of nailing a genuine git handling bug...\n\nAlso, I have a very hard time remembering that the \"retry unlink?\" EVER\nfinally ended up successful (despite virus file activity surely being a very\ntemporary thing!).\n\nSo much for some \"related\" observations that I can contribute currently\n- I had no time left to actually work on it today\nbut I'll try to do some testing given the very detailed\n(and gratifyingly matching :) analysis of Jeff King (thanks a lot, too!).\n\nAndreas Mohr\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"Git for Windows\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"259885","messageId":"cb23672d-75ba-40b9-abff-a7b2b6c5debb@googlegroups.com","threadId":"39090","inReplyTo":"20150416152849.GA30137@peff.net","subject":"Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues","fromName":"rupert thurner","fromEmail":"rupert.thurner@gmail.com","sentAt":"2015-04-23T06:52:24Z","receivedAt":"2015-04-23T06:52:24Z","isPatch":false,"sender":{"key":"rupert.thurner@gmail.com","avatar":null},"body":"hi,\n\ni made a screenshot a couple of weeks ago (attached), and it seems to match \nyour description.\n\nrupert.\n\n\nOn Thursday, April 16, 2015 at 5:28:54 PM UTC+2, Jeff King wrote:\n>\n> On Thu, Apr 16, 2015 at 01:35:05PM +0200, Andreas Mohr wrote: \n>\n> > I strongly suspect that git's repacking implementation \n> > (probably unrelated to msysgit-specific deviations, \n> > IOW, git *core* handling) \n> > simply is buggy \n> > in that it may keep certain file descriptors open \n> > at least a certain time (depending on scope of implementation/objects!?) \n> > beyond having finished its operation (rename?). \n>\n> Hrm. I do not see anything in builtin/fetch.c that closes the packfile \n> descriptors before running \"gc --auto\". So basically the sequence: \n>\n>   1. Fetch performs actual fetch. It needs to open packfiles to do \n>      commit negotiation with other side (the hard work is done \n>      by an index-pack subprocess, but it is likely we have to access \n>      _some_ objects). \n>\n>   2. The packfiles remain open and mmap'd (at least on Linux) in the \n>      sha1_file.c:packed_git list. \n>\n>   3. We spawn \"gc --auto\" and wait for it to finish. While we are \n>      waiting, the descriptors are still open, but \"gc --auto\" will not be \n>      able to delete any packs. \n>\n> But this seems too simple to be the problem, as it would mean that just \n> about any \"gc --auto\" that triggers a full repack would be a problem (so \n> anytime you have about 50 packs). But maybe the gc \"autodetach\" behavior \n> means it works racily. \n>\n> I was able to set up the situation deterministically by running the \n> script below: \n>\n> -- >8 -- \n> #!/bin/sh \n>\n> # XXX tweak this setting as appropriate \n> PATH_TO_GIT_BUILD=$HOME/compile/git \n> PATH=$PATH_TO_GIT_BUILD/bin-wrappers:$PATH \n> rm -rf parent child \n>\n> # make a parent/child where the child will have to access \n> # a packfile to fulfill another fetch \n> git init parent && \n> git -C parent commit --allow-empty -m base && \n> git clone parent child && \n> git -C parent commit --allow-empty -m extra && \n>\n> # we want to make our base pack really big, because otherwise \n> # git will open/mmap/close it. So we must exceed core.packedgitlimit \n> cd child && \n> $PATH_TO_GIT_BUILD/test-genrandom foo 5000000 >file && \n> git add file && \n> git commit -m large file && \n> git repack -ad && \n> git config core.packedGitLimit 1M && \n>\n> # now make some spare packs to bust the gc.autopacklimit \n> for i in 1 2 3 4 5; do \n>         git commit --allow-empty -m $i && \n>         git repack -d \n> done && \n> git config gc.autoPackLimit 3 && \n> git config gc.autoDetach false && \n> GIT_TRACE=1 git fetch \n> ``` \n>\n> I also instrumented my (v1.9.5) git build like this: \n>\n> diff --git a/builtin/fetch.c b/builtin/fetch.c \n> index 025bc3e..fc99e5e 100644 \n> --- a/builtin/fetch.c \n> +++ b/builtin/fetch.c \n> @@ -1174,6 +1174,12 @@ int cmd_fetch(int argc, const char **argv, const \n> char *prefix) \n>          list.strdup_strings = 1; \n>          string_list_clear(&list, 0); \n>   \n> +        { \n> +                struct packed_git *p; \n> +                for (p = packed_git; p; p = p->next) \n> +                        trace_printf(\"pack %s has descriptor %d\\n\", \n> +                                     p->pack_name, p->pack_fd); \n> +        } \n>          run_command_v_opt(argv_gc_auto, RUN_GIT_CMD); \n>   \n>          return result; \n> diff --git a/builtin/repack.c b/builtin/repack.c \n> index bb2314c..e8b29cf 100644 \n> --- a/builtin/repack.c \n> +++ b/builtin/repack.c \n> @@ -105,6 +105,7 @@ static void remove_redundant_pack(const char \n> *dir_name, const char *base_name) \n>          for (i = 0; i < ARRAY_SIZE(exts); i++) { \n>                  strbuf_setlen(&buf, plen); \n>                  strbuf_addstr(&buf, exts[i]); \n> +                trace_printf(\"unlinking %s\\n\", buf.buf); \n>                  unlink(buf.buf); \n>          } \n>          strbuf_release(&buf); \n>\n> to confirm what was happening (because of course on Linux it is \n> perfectly fine to delete the open file). If this does trigger the bug \n> for you, though, it should be obvious even without the trace calls. :) \n>\n> -Peff \n>\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"Git for Windows\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"}]}