{"thread":{"id":"20753","subject":"git-svn intermittent issues with absent_file","startedAt":"2009-08-27T14:56:25Z","lastAt":"2009-08-29T21:35:24Z","messageCount":5,"participants":["Matthias Andree","Eric Wong","Ealdwulf Wuffinga"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"121866","messageId":"op.uzbs4bsq3myrm3@balu.cs.uni-paderborn.de","threadId":"20753","inReplyTo":null,"subject":"git-svn intermittent issues with absent_file","fromName":"Matthias Andree","fromEmail":"matthias.andree@uni-paderborn.de","sentAt":"2009-08-27T14:56:25Z","receivedAt":"2009-08-27T14:56:25Z","isPatch":false,"sender":{"key":"matthias.andree@uni-paderborn.de","avatar":null},"body":"Greetings,\n\nwe seem to have issues with checking out files from an SVN server via \nhttps://. The problem is hard to reproduce, and shows as \"absent_file\" \nwarnings, i. e. files that are in the SVN checkout don't make it to the \nGit checkin.\n\nPerhaps this rings a bell with someone or there are similar reports that \nrelate to our issues...\n\n\n## ISSUE ## Once in a while, a git-svn clone or rebase fails with files \nmissing from commits, as mentioned above. We haven't been able to figure \nout under what circumstances this happens. If we try to reproduce this (i. \ne. kill the directory completely and re-run the git svn clone), another \nrevision (for instance 170) might show this, or the clone may succeed.\n This was observed with Git 1.6.0.4 under Ubuntu Linux Jaunty Jackalope \n(9.04) and on a different computer with Git 1.6.4.1.196.g31f0b (from the \nmaster branch) under a fully updated Cygwin 1.5.\n\n\nExample log of failed data (sorry, we are not allowed to let you access \nthe repository, so we must debug by proxy).\nI'm replacing substrings in the log below to maintain corporate \nconfidentiality levels:\n\n$ git svn clone --no-checkout https://svnserver.example.edu/project/\nInitialized empty Git repository in /tmp/compnet/.git/\nW: +empty_dir: branches\nW: +empty_dir: tags\nW: +empty_dir: trunk\nr1 = a1b0e99e3986da4d8d461944b623b6abb2460de4 (refs/remotes/git-svn)\n...\nr135 = 294077e62ee5f463b8ab97d961d9742ef89ae662 (refs/remotes/git-svn)\n A ... (~100 PDF files)\n ...\n A d1/u1/2009_06/f1.pdf\nW: +empty_dir: d1/u2/slides/2009_06\nW: absent_file: d1/u1/2009_06/d1/u1/2009_06/f2.pdf Insufficient \npermissions?\nW: absent_file: d1/u3/06_2009/f1/u3/2009_06/f3.pdf Insufficient \npermissions?\nW: absent_file: d1/u2/slides/2009_06/d1/u2/slides/2009_06/f4.pdf \nInsufficient permissions?\nr136 = 58523cf1fa867d33a74080dabdcbc85ae0ba99ec (refs/remotes/git-svn)\n...\nr139 = 2fb1647bc40d3815a5eefb32d43a375e03d2e871 (refs/remotes/git-svn)\nIncomplete data: Delta source ended unexpectedly at \n/usr/local/libexec/git-core/git-svn line 4605\n\n\nJust re-running \"git svn fetch\" reproduces the \"Incomplete data:\" error \nand aborts, but re-fetching an older version succeeds. No fiddling with \npermissions needed.\n\n\n## WORKAROUND ##\n\nIf this happens, I can force a checkout with, say, \"git checkout -t -b \nmaster remotes/git-svn\", revert to the last good revision with \"git svn \nreset -r135\" (this doesn't work without prior checkout) and then re-run \ngit svn fetch, which will usually succeed then.\n\n\nThe three absent_file lines are also recorded in \n.git/svn/refs/remotes/git-svn/unhandled.log for r136.\n\n\nQuestions:\n\n1. What causes these absent_file issues? How can we assist with debugging \nthis?\n\n2. What does \"Delta source ended unexpectedly\" mean? (the line number is \nbogus, it's just the finish_report call)\n\n3. Is this or a similar issue known? Is this an issue with the SVN server, \nthe SVN bindings, or the git svn adaptor?\n\n4. How can we avoid or fix this?\n\nWe're happy to test patches.\n\n\nThanks a lot in advance.\n\nCheers,\n\n-- \nMatthias Andree\n"},{"id":"121904","messageId":"20090827191902.GA3185@dcvr.yhbt.net","threadId":"20753","inReplyTo":"op.uzbs4bsq3myrm3@balu.cs.uni-paderborn.de","subject":"Re: git-svn intermittent issues with absent_file","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-08-27T19:19:02Z","receivedAt":"2009-08-27T19:19:02Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Matthias Andree wrote:\n> Greetings,\n>\n> we seem to have issues with checking out files from an SVN server via \n> https://. The problem is hard to reproduce, and shows as \"absent_file\" \n> warnings, i. e. files that are in the SVN checkout don't make it to the \n> Git checkin.\n>\n> Perhaps this rings a bell with someone or there are similar reports that \n> relate to our issues...\n\nHi Matthias,\n\nI don't recall any issues with the \"absent_file\" callback ever being\nreported to me. I don't think I've ever touched a repo that triggered\nit, either...\n\n> ## ISSUE ## Once in a while, a git-svn clone or rebase fails with files \n> missing from commits, as mentioned above. We haven't been able to figure \n> out under what circumstances this happens. If we try to reproduce this \n> (i. e. kill the directory completely and re-run the git svn clone), \n> another revision (for instance 170) might show this, or the clone may \n> succeed.\n> This was observed with Git 1.6.0.4 under Ubuntu Linux Jaunty Jackalope \n> (9.04) and on a different computer with Git 1.6.4.1.196.g31f0b (from the \n> master branch) under a fully updated Cygwin 1.5.\n\nI assume this happens with the latest git under Ubuntu Linux, too; but\ncan you give that a try just in case?\n\nThere have been a few odd bugs fixed since 1.6.0.4, but nothing rings a\nbell here. This doesn't seem to be a Cygwin/Win32-related issue, but\nI'd like to minimize the number of variables since I can't support\nCygwin/Win32 directly.\n\n> Example log of failed data (sorry, we are not allowed to let you access \n> the repository, so we must debug by proxy).\n> I'm replacing substrings in the log below to maintain corporate \n> confidentiality levels:\n>\n> $ git svn clone --no-checkout https://svnserver.example.edu/project/\n> Initialized empty Git repository in /tmp/compnet/.git/\n> W: +empty_dir: branches\n> W: +empty_dir: tags\n> W: +empty_dir: trunk\n> r1 = a1b0e99e3986da4d8d461944b623b6abb2460de4 (refs/remotes/git-svn)\n> ...\n> r135 = 294077e62ee5f463b8ab97d961d9742ef89ae662 (refs/remotes/git-svn)\n> A ... (~100 PDF files)\n> ...\n> A d1/u1/2009_06/f1.pdf\n> W: +empty_dir: d1/u2/slides/2009_06\n> W: absent_file: d1/u1/2009_06/d1/u1/2009_06/f2.pdf Insufficient \n> permissions?\n> W: absent_file: d1/u3/06_2009/f1/u3/2009_06/f3.pdf Insufficient \n> permissions?\n> W: absent_file: d1/u2/slides/2009_06/d1/u2/slides/2009_06/f4.pdf \n> Insufficient permissions?\n> r136 = 58523cf1fa867d33a74080dabdcbc85ae0ba99ec (refs/remotes/git-svn)\n> ...\n> r139 = 2fb1647bc40d3815a5eefb32d43a375e03d2e871 (refs/remotes/git-svn)\n> Incomplete data: Delta source ended unexpectedly at \n> /usr/local/libexec/git-core/git-svn line 4605\n>\n>\n> Just re-running \"git svn fetch\" reproduces the \"Incomplete data:\" error \n> and aborts, but re-fetching an older version succeeds. No fiddling with \n> permissions needed.\n>\n>\n> ## WORKAROUND ##\n>\n> If this happens, I can force a checkout with, say, \"git checkout -t -b \n> master remotes/git-svn\", revert to the last good revision with \"git svn \n> reset -r135\" (this doesn't work without prior checkout) and then re-run \n> git svn fetch, which will usually succeed then.\n>\n>\n> The three absent_file lines are also recorded in \n> .git/svn/refs/remotes/git-svn/unhandled.log for r136.\n>\n>\n> Questions:\n>\n> 1. What causes these absent_file issues? How can we assist with debugging \n> this?\n\nPermissions would be my first (and only) guess as I have no experience\nwith this condition...\n\nAre you using anything weird in ~/.subversion/config by any chance?\nAnd you're sure you're using the same user/account in all cases?\n\nAre the files you're hitting absent_file on unusually large and\nhitting some limit on the server side?\n\n> 2. What does \"Delta source ended unexpectedly\" mean? (the line number is \n> bogus, it's just the finish_report call)\n\nSometimes the server can just shut down/drop connections. Do you have\naccess to the server logs? How many simultaneous connections can\nit accept? Which version of the Subversion server is running?\nHow reliable is your network connection to the server?\n\nAnything weird with the network configuration? Mismatched MTU, odd\nfirewall rules, non-standard TCP stack configuration, VPN, proxies...\n\n> 3. Is this or a similar issue known? Is this an issue with the SVN \n> server, the SVN bindings, or the git svn adaptor?\n\nThe \"Incomplete data: Delta source ended unexpectedly\" happens\nfrom time to time on a few odd servers, but I've never found it\nreproducible and retrying \"git svn fetch\" always fixes it.\nThis is more than likely the fault of the network or server.\n\nThe \"absent_file\" issue I've never seen...\n\n> 4. How can we avoid or fix this?\n\nOther than your workaround, I don't know...\n\n> We're happy to test patches.\n\nCool. I don't have any ideas for patches right now, but does this\nproblem happen with other SVN clients? Even svk/svm could be\nworth a shot for testing...\n\n-- \nEric Won\n"},{"id":"121977","messageId":"op.uzc1njug3myrm3@merlin.emma.line.org","threadId":"20753","inReplyTo":"20090827191902.GA3185@dcvr.yhbt.net","subject":"Re: git-svn intermittent issues with absent_file","fromName":"Matthias Andree","fromEmail":"matthias.andree@uni-paderborn.de","sentAt":"2009-08-28T06:58:21Z","receivedAt":"2009-08-28T06:58:21Z","isPatch":false,"sender":{"key":"matthias.andree@uni-paderborn.de","avatar":null},"body":"Am 27.08.2009, 21:19 Uhr, schrieb Eric Wong :\n\n> Matthias Andree wrote:\n>> Greetings,\n>>\n>> we seem to have issues with checking out files from an SVN server via\n>> https://. The problem is hard to reproduce, and shows as \"absent_file\"\n>> warnings, i. e. files that are in the SVN checkout don't make it to the\n>> Git checkin.\n>>\n>> Perhaps this rings a bell with someone or there are similar reports that\n>> relate to our issues...\n>\n> Hi Matthias,\n>\n> I don't recall any issues with the \"absent_file\" callback ever being\n> reported to me. I don't think I've ever touched a repo that triggered\n> it, either...\n\nHi Eric,\n\nthank you for the prompt reply.\n\n>> ## ISSUE ## Once in a while, a git-svn clone or rebase fails with files\n>> missing from commits, as mentioned above. We haven't been able to figure\n>> out under what circumstances this happens. If we try to reproduce this\n>> (i. e. kill the directory completely and re-run the git svn clone),\n>> another revision (for instance 170) might show this, or the clone may\n>> succeed.\n>> This was observed with Git 1.6.0.4 under Ubuntu Linux Jaunty Jackalope\n>> (9.04) and on a different computer with Git 1.6.4.1.196.g31f0b (from the\n>> master branch) under a fully updated Cygwin 1.5.\n>\n> I assume this happens with the latest git under Ubuntu Linux, too; but\n> can you give that a try just in case?\n\nThe latest git was on Cygwin, but we'll see to trying it on Linux.\n\n> There have been a few odd bugs fixed since 1.6.0.4, but nothing rings a\n> bell here. This doesn't seem to be a Cygwin/Win32-related issue, but\n> I'd like to minimize the number of variables since I can't support\n> Cygwin/Win32 directly.\n\nThe other question is how many arcane APR or Subversion binding issues we\nhit here. Subversion 1.6 + bindings surely is rather picky on Cygwin 1.5,\nas discussed earlier on this list.\n\n>> Questions:\n>>\n>> 1. What causes these absent_file issues? How can we assist with \n>> debugging\n>> this?\n>\n> Permissions would be my first (and only) guess as I have no experience\n> with this condition...\n\nHm. In that case: who writes the file that goes missing, and why isn't the\nerror logged then? Is there a way to have the SVN bindings or the\ninterfacing code in Git-SVN log file creation/write errors with decoded\nerrno? That might help a big deal already.\n\n> Are you using anything weird in ~/.subversion/config by any chance?\n\nNot to my knowledge.\n\n> And you're sure you're using the same user/account in all cases?\n\nYes indeed. All under the same account.\n\n> Are the files you're hitting absent_file on unusually large and\n> hitting some limit on the server side?\n\nThe files aren't, the whole changeset might. We haven't yet observed this\non small (as in one .tex file) commits.\n\n>> 2. What does \"Delta source ended unexpectedly\" mean? (the line number is\n>> bogus, it's just the finish_report call)\n>\n> Sometimes the server can just shut down/drop connections. Do you have\n> access to the server logs? How many simultaneous connections can\n> it accept? Which version of the Subversion server is running?\n\nThe server doesn't permit user logins, so we need to ask the support\nstaff. Answer times are unpredictable.\n\n> How reliable is your network connection to the server?\n> Anything weird with the network configuration? Mismatched MTU, odd\n> firewall rules, non-standard TCP stack configuration, VPN, proxies...\n\nUncritical. Company network, \"direct connection\" (through switches).\n\n>> 3. Is this or a similar issue known? Is this an issue with the SVN\n>> server, the SVN bindings, or the git svn adaptor?\n>\n> The \"Incomplete data: Delta source ended unexpectedly\" happens\n> from time to time on a few odd servers, but I've never found it\n> reproducible and retrying \"git svn fetch\" always fixes it.\n> This is more than likely the fault of the network or server.\n\nSeems plausible, but...\n\n> The \"absent_file\" issue I've never seen...\n\n...in such a case, some part of the overall system (I'm not saying it's \ngit-svn though) must ignore an error and continue.\n\n>> We're happy to test patches.\n>\n> Cool. I don't have any ideas for patches right now, but does this\n> problem happen with other SVN clients? Even svk/svm could be\n> worth a shot for testing...\n\nI can try svk, but what's svm?\n\n-- \nMatthias Andree\n"},{"id":"121981","messageId":"op.uzc46sm43myrm3@balu.cs.uni-paderborn.de","threadId":"20753","inReplyTo":"op.uzc1njug3myrm3@merlin.emma.line.org","subject":"Re: git-svn intermittent issues with absent_file","fromName":"Matthias Andree","fromEmail":"matthias.andree@uni-paderborn.de","sentAt":"2009-08-28T08:14:42Z","receivedAt":"2009-08-28T08:14:42Z","isPatch":false,"sender":{"key":"matthias.andree@uni-paderborn.de","avatar":null},"body":"Am 28.08.2009, 08:58 Uhr, schrieb Matthias Andree \n:\n\n> The other question is how many arcane APR or Subversion binding issues we\n> hit here. Subversion 1.6 + bindings surely is rather picky on Cygwin 1.5,\n> as discussed earlier on this list.\n\nAs an update, Git-SVN 1.6.4.0 and 1.6.4.1 haven't seen such problems last \nnight when run on Ubuntu Linux.\n\nSo, the common idea set is apparently down to (a) intermittent server \nissues (I've asked for the relevant log excerpts) and (b) Cygwin issues, \nand we should keep in mind (c) git-svn, or svn bindings/libraries, losing \nthe causes error conditions from (a) or (b) and git-svn just reporting \nlater symptoms (absent files). More on the other experiments once I get \naround to them.\n\nThanks\nMatthias\n"},{"id":"122088","messageId":"efe2b6d70908291435w7d607151g370717ba2e034a2c@mail.gmail.com","threadId":"20753","inReplyTo":"op.uzc46sm43myrm3@balu.cs.uni-paderborn.de","subject":"Re: git-svn intermittent issues with absent_file","fromName":"Ealdwulf Wuffinga","fromEmail":"ealdwulf@googlemail.com","sentAt":"2009-08-29T21:35:24Z","receivedAt":"2009-08-29T21:35:24Z","isPatch":false,"sender":{"key":"ealdwulf@googlemail.com","avatar":null},"body":"On Fri, Aug 28, 2009 at 9:14 AM, Matthias\nAndree wrote:\n\n> So, the common idea set is apparently down to (a) intermittent server issues\n> (I've asked for the relevant log excerpts) and (b) Cygwin issues, and we\n> should keep in mind (c) git-svn, or svn bindings/libraries, losing the\n> causes error conditions from (a) or (b) and git-svn just reporting later\n> symptoms (absent files). More on the other experiments once I get around to\n> them.\n\nWon't help if it's a server issue, but if your suspicion towards\ngit-svn (or anything else you have the source for) increases, it may\nbe worth trying bbchop:\nhttp://github.com/Ealdwulf/bbchop\nThat is like git-bisect, except it can cope with intermittent bugs.\n\nEaldwulf\n"}]}