{"thread":{"id":"27112","subject":"Java Inflater problem decompressing packfile","startedAt":"2011-04-16T02:05:05Z","lastAt":"2011-04-17T04:36:07Z","messageCount":10,"participants":["madmarcos","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"165946","messageId":"1302919505984-6278154.post@n2.nabble.com","threadId":"27112","inReplyTo":null,"subject":"Java Inflater problem decompressing packfile","fromName":"madmarcos","fromEmail":"fru574@my.utsa.edu","sentAt":"2011-04-16T02:05:05Z","receivedAt":"2011-04-16T02:05:05Z","isPatch":false,"sender":{"key":"fru574@my.utsa.edu","avatar":null},"body":"This may be better suited for the Java forums but I will ask it here just in\ncase someone has run into it before.\n\nI have a packfile that I have saved as a file from the git-upload-pack\ncommand. I want to read through the packfile, decompressing each of the\nobjects. \n\nMy little inflater procedure works fine for a tiny HelloWorld project. So, I\ndecided to mix it up a little and use the jEdit source for a larger test. I\nam 99% certain the jEdit.git packfile itself is ok as I have passed it\nthrough directly to eGit's Import using an SSH proxy and eGit unpacked it\njust fine.\n\nSo, my inflater method decompresses the first 7 objects fine (a commit, a\ncouple of trees, and several blobs) and a cursory visual inspection of the\ndecompressed data seems fine. The eighth object becomes a problem, though.\nIt is a blob with the name build.xml that is 51,060 bytes decompressed\n(looking at the original pre-git-pushed jEdit source). The actual file size\nmatches the decompressed data content size in the packfile object header. \nThe inflater procedure outputs the decompressed data to System.out for\nvisual inspection. Approximately the first 1/3 looks like the original\nbuild.xml but after that, the output is garbled. The procedure continues\ndecompressing objects after the 8th, but garbled, object but it dies on the\n9th object with an \"unknown compression method\" error.\n\nSo I created a new test inflater to focus only on decompressing the 8th\nobject. Simply opening the packfile, copying out the compressed data (7793\nbytes), and inflating it yields the same 2/3 garbled xml. \n\nAs a further test, I then take the original build.xml file, compress it\nusing java's Deflater (yielding a 7793 byte array), and then inflate it\nusing my same procedure and it decompresses fine. All of the xml is\nreadable.\n\nNow, I have tried several variations of an inflater procedure, including an\npatchwork variation from jGit's WindowCursor.inflate method. But they all\nyield the same garbled result for the compressed build.xml data.\n\nAny suggestions. I can post some of my ugly code if asked.\n\nThanks!\n\n--\nView this message in context: http://git.661346.n2.nabble.com/Java-Inflater-problem-decompressing-packfile-tp6278154p6278154.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"165955","messageId":"20110416063729.GC28853@sigill.intra.peff.net","threadId":"27112","inReplyTo":"1302919505984-6278154.post@n2.nabble.com","subject":"Re: Java Inflater problem decompressing packfile","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-04-16T06:37:29Z","receivedAt":"2011-04-16T06:37:29Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 15, 2011 at 07:05:05PM -0700, madmarcos wrote:\n\n> So, my inflater method decompresses the first 7 objects fine (a commit, a\n> couple of trees, and several blobs) and a cursory visual inspection of the\n> decompressed data seems fine. The eighth object becomes a problem, though.\n> It is a blob with the name build.xml that is 51,060 bytes decompressed\n> (looking at the original pre-git-pushed jEdit source). The actual file size\n> matches the decompressed data content size in the packfile object header. \n> The inflater procedure outputs the decompressed data to System.out for\n> visual inspection. Approximately the first 1/3 looks like the original\n> build.xml but after that, the output is garbled. The procedure continues\n> decompressing objects after the 8th, but garbled, object but it dies on the\n> 9th object with an \"unknown compression method\" error.\n\nIs it possible that the blob is stored as a delta within the pack? In\nthat case the pack header will tell you what the eventual size of the\nblob will be, but the data will actually be a diff against another pack\nobject. Does your inflater handle delta-fied objects?\n\n-Peff\n"},{"id":"165961","messageId":"1302963832717-6279028.post@n2.nabble.com","threadId":"27112","inReplyTo":"20110416063729.GC28853@sigill.intra.peff.net","subject":"Re: Java Inflater problem decompressing packfile","fromName":"madmarcos","fromEmail":"fru574@my.utsa.edu","sentAt":"2011-04-16T14:23:52Z","receivedAt":"2011-04-16T14:23:52Z","isPatch":false,"sender":{"key":"fru574@my.utsa.edu","avatar":null},"body":"No, my inflater doesn't handle deltas, yet. \nBut there are a few reasons why I don't think that's the case.\n1. The project has only been pushed once to the git repository before my\ntests. No updates to the git repository project or anything like that. \n2. If it were a delta, would the first 1/3 of it be completely normal and\nreadable? There is no pattern that I can see to the remaining 2/3. It looks\nas if the characters in the 2/3 part were interleaved with the other\ncharacters about 10 times.\n3. The object type in the header is parsed as 3, or a blob. Aren't the delta\nobject types higher numbers than that?\n\n\n--\nView this message in context: http://git.661346.n2.nabble.com/Java-Inflater-problem-decompressing-packfile-tp6278154p6279028.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"165962","messageId":"1302964576373-6279050.post@n2.nabble.com","threadId":"27112","inReplyTo":"1302963832717-6279028.post@n2.nabble.com","subject":"Re: Java Inflater problem decompressing packfile","fromName":"madmarcos","fromEmail":"fru574@my.utsa.edu","sentAt":"2011-04-16T14:36:16Z","receivedAt":"2011-04-16T14:36:16Z","isPatch":false,"sender":{"key":"fru574@my.utsa.edu","avatar":null},"body":"1 more thing:\nwhen it is decompressing the garbled object, it throws an \"incorrect data\ncheck\" error.\n\n--\nView this message in context: http://git.661346.n2.nabble.com/Java-Inflater-problem-decompressing-packfile-tp6278154p6279050.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"165963","messageId":"1302965899845-6279085.post@n2.nabble.com","threadId":"27112","inReplyTo":"1302964576373-6279050.post@n2.nabble.com","subject":"Re: Java Inflater problem decompressing packfile","fromName":"madmarcos","fromEmail":"fru574@my.utsa.edu","sentAt":"2011-04-16T14:58:19Z","receivedAt":"2011-04-16T14:58:19Z","isPatch":false,"sender":{"key":"fru574@my.utsa.edu","avatar":null},"body":"here is some code that shows the problem. sorry if the formatting fails. not\nsure if I am supposed to use code tags or something.\n\n\n\t\ttry {\n\t\t\tbyte[] packFile = readFile(\"/Users/marcos/GitProxyCache/jedit.pack\");\n\t\t\t\n\t\t\t//THE BELOW OBJECT DECOMPRESSES FINE\n       \t\t//Object starts at index 8616\n       \t\t//Type = 3, Decompressed size = 2248 (uses 2 extra size bytes)\n\t\t\t//byte [] packDataWindow = new byte[8000];\n\t\t\t//System.arraycopy(packFile, 8619, packDataWindow, 0,\npackDataWindow.length); //works\n\t\t\t\n\t\t\t//THE BELOW OBJECT FAILS TO INFLATE\n\t\t\t//IT CAUSES an \"incorrect data check\" error\n       \t\t//Object starts at index 9470\n       \t\t//Type = 3, Decompressed size = 51060 (uses 2 extra size bytes)\n\t\t\tbyte [] packDataWindow = new byte[8000];\n\t\t\tSystem.arraycopy(packFile, 9473, packDataWindow, 0,\npackDataWindow.length); //does not work\n\n\t\t\tInflater decompresser = new Inflater();\n\t\t\tdecompresser.setInput(packDataWindow, 0, packDataWindow.length);\n\t\t\tbyte[] result = new byte[60000];\n\t\t\tint resultLength = 0;\n\t\t\tresultLength = decompresser.inflate(result);\n\t\t\tString outputString = new String(result, 0, resultLength, \"UTF-8\");\n\t\t\tSystem.out.println(outputString);\n\t\t\tdecompresser.end();\n\t\t} catch (Exception e) {\n\t\t\te.printStackTrace();\n\t\t}\n\n\n--\nView this message in context: http://git.661346.n2.nabble.com/Java-Inflater-problem-decompressing-packfile-tp6278154p6279085.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"165964","messageId":"1302969047343-6279183.post@n2.nabble.com","threadId":"27112","inReplyTo":"1302965899845-6279085.post@n2.nabble.com","subject":"Re: Java Inflater problem decompressing packfile","fromName":"madmarcos","fromEmail":"fru574@my.utsa.edu","sentAt":"2011-04-16T15:50:47Z","receivedAt":"2011-04-16T15:50:47Z","isPatch":false,"sender":{"key":"fru574@my.utsa.edu","avatar":null},"body":"if you want to play with the packfile, you can find it here:\n\nhttp://galadriel.cs.utsa.edu/utsadocs/jedit.pack\n\n\n\n--\nView this message in context: http://git.661346.n2.nabble.com/Java-Inflater-problem-decompressing-packfile-tp6278154p6279183.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"165994","messageId":"1303000824677-6280097.post@n2.nabble.com","threadId":"27112","inReplyTo":"1302969047343-6279183.post@n2.nabble.com","subject":"Re: Java Inflater problem decompressing packfile","fromName":"madmarcos","fromEmail":"fru574@my.utsa.edu","sentAt":"2011-04-17T00:40:24Z","receivedAt":"2011-04-17T00:40:24Z","isPatch":false,"sender":{"key":"fru574@my.utsa.edu","avatar":null},"body":"someone on the Java forums asked if I knew that the file was being read\ncompletely before inflating. Well... I just assumed (yes, I know not a good\nthing to do).\nSo here is my readFile code in case you want to see it:\n\npublic byte [] readFile(String fileName) {\n\t\tbyte [] input2 = null;\n\t    File tempPackInputFile2 = new File(fileName);\n\t\tInputStream tempPackInputStream2;\n\t\ttry {\n\t\t\ttempPackInputStream2 = new FileInputStream(tempPackInputFile2);\n\t\t\tlong tempPackLength2 = tempPackInputFile2.length();\n\t\t\tinput2 = new byte[(int) tempPackLength2];\n\t\t\t// Read in the bytes\n\t\t    int offset2 = 0;\n\t\t    int numRead2 = 0;\n\t\t    while (offset2 < input2.length\n\t\t           && (numRead2 = tempPackInputStream2.read(input2, offset2,\ninput2.length-offset2)) >= 0) {\n\t\t        offset2 += numRead2;\n\t\t    }\n\t\t    tempPackInputStream2.close();\n\t\t} catch (Exception e) {\n\t\t\te.printStackTrace();\n\t\t}\n\t    return input2;\n\t}\n\n--\nView this message in context: http://git.661346.n2.nabble.com/Java-Inflater-problem-decompressing-packfile-tp6278154p6280097.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"166001","messageId":"1303012970295-6280285.post@n2.nabble.com","threadId":"27112","inReplyTo":"1303000824677-6280097.post@n2.nabble.com","subject":"Re: Java Inflater problem decompressing packfile","fromName":"madmarcos","fromEmail":"fru574@my.utsa.edu","sentAt":"2011-04-17T04:02:50Z","receivedAt":"2011-04-17T04:02:50Z","isPatch":false,"sender":{"key":"fru574@my.utsa.edu","avatar":null},"body":"my entire testing class is below. just change the file path string in the\nfirst line in test4.\n\n\npackage server_test2;\n\nimport java.io.DataInputStream;\nimport java.io.File;\nimport java.io.FileInputStream;\nimport java.util.zip.Inflater;\n\npublic class InflaterTest2 {\n\n\tpublic InflaterTest2() {\n\t}\n\n\tpublic void test4() {\n\t\ttry {\n\t\t\tbyte[] packFile = readFile2(\"/Users/marcos/GitProxyCache/jedit.pack\");\n\n\t\t\tbyte [] packDataWindow = new byte[8000];\n\n\t\t\t//the below object is the .classpath blob. works fine\n\t\t\tSystem.arraycopy(packFile, 4828 + 2, packDataWindow, 0,\npackDataWindow.length); \n\n\t\t\t//the below object is the .project blob. works fine\n\t\t\t//System.arraycopy(packFile, 4978 + 2, packDataWindow, 0,\npackDataWindow.length); \n\t\t\t\n\t\t\t//the below object is the top-level tree. works fine\n\t\t\t//System.arraycopy(packFile, 5171 + 2, packDataWindow, 0,\npackDataWindow.length); \n\n\t\t\t//the below object is source code notes blob. works fine\n\t\t\t//System.arraycopy(packFile, 5760 + 3, packDataWindow, 0,\npackDataWindow.length); \n\n\t\t\t//the below object is a build file blob. works fine\n\t\t\t//System.arraycopy(packFile, 8619, packDataWindow, 0,\npackDataWindow.length); \n\t\t\t\n\t\t\t//THE BELOW OBJECT FAILS TO INFLATE\n\t\t\t//IT CAUSES an \"incorrect data check\" error\n       \t\t//Object starts at index 9470\n       \t\t//Type = 3, Decompressed size = 51060 (uses 2 extra size bytes)\n\t\t\t//System.arraycopy(packFile, 9470 + 3, packDataWindow, 0,\npackDataWindow.length); \n\n\t\t\tInflater decompresser = new Inflater();\n\t\t\tdecompresser.setInput(packDataWindow, 0, packDataWindow.length);\n\t\t\tbyte[] result = new byte[60000];\n\t\t\tint resultLength = 0;\n\t\t\tresultLength = decompresser.inflate(result);\n\t\t\t\n\t\t\tString outputString = new String(result, 0, resultLength, \"UTF-8\");\n\t\t\tSystem.out.println(outputString);\n\n\t\t\tSystem.out.println(\"------- End Decompressed Output -------\");\n\t\t\tint numCompressed = (int) decompresser.getBytesRead();\n\t\t\tSystem.out.println(\"# Bytes Compressed: \" + numCompressed);\n\t\t\tdecompresser.end();\n\n\t\t} catch (Exception e) {\n\t\t\te.printStackTrace();\n\t\t}\n\t}\n\n\tpublic byte [] readFile2(String fileName) {\n\t\tbyte [] input2 = null;\n\t    File tempPackInputFile2 = new File(fileName);\n\t\tDataInputStream tempPackInputStream2;\n\t\ttry {\n\t\t\ttempPackInputStream2 = new DataInputStream(new\nFileInputStream(tempPackInputFile2));\n\t\t\tlong tempPackLength2 = tempPackInputFile2.length();\n\t\t\tinput2 = new byte[(int) tempPackLength2];\n\t\t\ttempPackInputStream2.readFully(input2);\n\t\t    tempPackInputStream2.close();\n\t\t} catch (Exception e) {\n\t\t\t// TODO Auto-generated catch block\n\t\t\te.printStackTrace();\n\t\t}\n\t    return input2;\n\t}\n\n\tpublic static void main(String[] args) {\n\t\tInflaterTest2 app = new InflaterTest2();\n\t\tapp.test4();\n\t\tSystem.out.println(\"Done.\");\n\t}\n\n}\n\n--\nView this message in context: http://git.661346.n2.nabble.com/Java-Inflater-problem-decompressing-packfile-tp6278154p6280285.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"166002","messageId":"1303013175664-6280295.post@n2.nabble.com","threadId":"27112","inReplyTo":"1303012970295-6280285.post@n2.nabble.com","subject":"Re: Java Inflater problem decompressing packfile","fromName":"madmarcos","fromEmail":"fru574@my.utsa.edu","sentAt":"2011-04-17T04:06:15Z","receivedAt":"2011-04-17T04:06:15Z","isPatch":false,"sender":{"key":"fru574@my.utsa.edu","avatar":null},"body":"I hit send too quickly. \nbeneath the readFile call in the test4() method are several commented-out\nSystem.arraycopy lines. you can uncomment any one of them to see the blob or\ntree decompressed. all but the the last arraycopy line work fine. the last\none is the problem.\n\nthanks for any light you can shed on the problem. I am pulling my hair out.\n\n--\nView this message in context: http://git.661346.n2.nabble.com/Java-Inflater-problem-decompressing-packfile-tp6278154p6280295.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"166003","messageId":"20110417043607.GA10831@sigill.intra.peff.net","threadId":"27112","inReplyTo":"1302963832717-6279028.post@n2.nabble.com","subject":"Re: Java Inflater problem decompressing packfile","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-04-17T04:36:07Z","receivedAt":"2011-04-17T04:36:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Apr 16, 2011 at 07:23:52AM -0700, madmarcos wrote:\n\n> No, my inflater doesn't handle deltas, yet. \n> But there are a few reasons why I don't think that's the case.\n> 1. The project has only been pushed once to the git repository before my\n> tests. No updates to the git repository project or anything like that. \n\nYou can still have deltas between blobs in a single commit, but only if\nyou have similar blobs.\n\n> 2. If it were a delta, would the first 1/3 of it be completely normal and\n> readable? There is no pattern that I can see to the remaining 2/3. It looks\n> as if the characters in the 2/3 part were interleaved with the other\n> characters about 10 times.\n\nIt wouldn't be completely normal, but you might see chunks of the file\nalong with binary patch instructions (like \"put this chunk at offset\n...\").\n\n> 3. The object type in the header is parsed as 3, or a blob. Aren't the delta\n> object types higher numbers than that?\n\nYeah, if it is coming up as type 3, then it should definitely be a\nwhole, literal blob.\n\nAs far as your Java code, I don't see anything overtly wrong, but then,\nI know absolutely nothing about any of the classes you are using.\n\n-Peff\n"}]}