{"thread":{"id":"33890","subject":"Reading commit objects","startedAt":"2013-05-21T21:21:49Z","lastAt":"2013-06-04T10:18:36Z","messageCount":19,"participants":["Chico Sokol","Felipe Contreras","John Szakmeister","Junio C Hamano","Jonathan Nieder","Andreas Krey","Shawn Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"218098","messageId":"CABx5MBQ57-=MPamvV-peZUdD_KDLX+5cy9vD7CL7p_Vz9BkvTg@mail.gmail.com","threadId":"33890","inReplyTo":null,"subject":"Reading commit objects","fromName":"Chico Sokol","fromEmail":"chico.sokol@gmail.com","sentAt":"2013-05-21T21:21:49Z","receivedAt":"2013-05-21T21:21:49Z","isPatch":false,"sender":{"key":"chico.sokol@gmail.com","avatar":"https://gravatar.com/avatar/d15de6acd103ae702fa27f716c4569db8e12d8b70f8d5910db7864235c687fc2?d=mp&s=160"},"body":"Hello,\n\nI'm building a library to manipulate git repositories (interacting\ndirectly with the filesystem).\n\nCurrently, we're trying to parse commit objects. After decompressing\nthe contents of a commit object file we got the following output:\n\ncommit 191\nauthor Francisco Sokol 1369140112 -0300\ncommitter Francisco Sokol 1369140112 -0300\n\nfirst commit\n\nWe hoped to get the same output of a \"git cat-file -p \", but\nthat didn't happened. From a commit object, how can I find tree object\nhash of this commit?\n\nThanks,\n\n\n--\nChico Sokol\n"},{"id":"218099","messageId":"CAMP44s2YLK=EQ+YGh5s9HLGAtgvtNvgDdmXMPpw+PRbODD08RQ@mail.gmail.com","threadId":"33890","inReplyTo":"CABx5MBQ57-=MPamvV-peZUdD_KDLX+5cy9vD7CL7p_Vz9BkvTg@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-05-21T21:25:51Z","receivedAt":"2013-05-21T21:25:51Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, May 21, 2013 at 4:21 PM, Chico Sokol wrote:\n> Hello,\n>\n> I'm building a library to manipulate git repositories (interacting\n> directly with the filesystem).\n>\n> Currently, we're trying to parse commit objects. After decompressing\n> the contents of a commit object file we got the following output:\n>\n> commit 191\n> author Francisco Sokol 1369140112 -0300\n> committer Francisco Sokol 1369140112 -0300\n>\n> first commit\n>\n> We hoped to get the same output of a \"git cat-file -p \", but\n> that didn't happened. From a commit object, how can I find tree object\n> hash of this commit?\n\ngit rev-parse :\n\n-- \nFelipe Contreras\n"},{"id":"218100","messageId":"CAEBDL5XwrD8ZbRRSrM1iJGtcRgziH5bFVwRHzg9=_PYzaTfgAg@mail.gmail.com","threadId":"33890","inReplyTo":"CABx5MBQ57-=MPamvV-peZUdD_KDLX+5cy9vD7CL7p_Vz9BkvTg@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"John Szakmeister","fromEmail":"john@szakmeister.net","sentAt":"2013-05-21T21:37:45Z","receivedAt":"2013-05-21T21:37:45Z","isPatch":false,"sender":{"key":"john@szakmeister.net","avatar":"https://avatars.githubusercontent.com/u/448087?v=4"},"body":"On Tue, May 21, 2013 at 5:21 PM, Chico Sokol wrote:\n> Hello,\n>\n> I'm building a library to manipulate git repositories (interacting\n> directly with the filesystem).\n>\n> Currently, we're trying to parse commit objects. After decompressing\n> the contents of a commit object file we got the following output:\n>\n> commit 191\n> author Francisco Sokol 1369140112 -0300\n> committer Francisco Sokol 1369140112 -0300\n>\n> first commit\n\nDoes `git cat-file -p ` show a tree object? FWIW, I expected to\nsee a tree line there, so maybe this object was created without a\ntree? I also don't see a parent listed.\n\nI did this on one of my repos:\n\n>>> buf = open('.git/objects/cd/da219e4d7beceae55af73c44cb3c9e1ec56802', 'rb').read()\n>>> import zlib\n>>> zlib.decompress(buf)\n'commit 246\\x00tree 2abfe1a7bedb29672a223a5c5f266b7dc70a8d87\\nparent\n0636e7ff6b79470b0cd53ceacea88e7796f202ce\\nauthor John Szakmeister\n 1369168481 -0400\\ncommitter John Szakmeister\n 1369168481 -0400\\n\\nGot a file listing.\\n'\n\nSo at least creating the commits with Git, I see a tree. How was the\ncommit you're referencing created? Perhaps something is wrong with\nthat process?\n\n> We hoped to get the same output of a \"git cat-file -p \", but\n> that didn't happened. From a commit object, how can I find tree object\n> hash of this commit?\n\nI'd expect that too.\n\n-John\n"},{"id":"218101","messageId":"CABx5MBSnpZTthOHECqkbpdbFfkb4e_uSo-rh4owBc8B_oSKjJQ@mail.gmail.com","threadId":"33890","inReplyTo":"CAEBDL5XwrD8ZbRRSrM1iJGtcRgziH5bFVwRHzg9=_PYzaTfgAg@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Chico Sokol","fromEmail":"chico.sokol@gmail.com","sentAt":"2013-05-21T22:18:35Z","receivedAt":"2013-05-21T22:18:35Z","isPatch":false,"sender":{"key":"chico.sokol@gmail.com","avatar":"https://gravatar.com/avatar/d15de6acd103ae702fa27f716c4569db8e12d8b70f8d5910db7864235c687fc2?d=mp&s=160"},"body":"Ok, we discovered that the commit object actually contains the tree\nobject's sha1, by reading its contents with python zlib library.\n\nSo the bug must be with our java code (we're building a java lib).\n\nIs there any non-standard issue in git's zlib compression? We're\ndecompressing its contents with java default zlib api, so it should\nwork normally, here's our code, that's printing that wrong output:\n\nimport java.io.File;\nimport java.io.FileInputStream;\nimport java.util.zip.InflaterInputStream;\nimport org.apache.commons.io.IOUtils;\n...\nFile obj = new File(\".git/objects/25/0f67ef017fcb97b5371a302526872cfcadad21\");\nInflaterInputStream inflaterInputStream = new InflaterInputStream(new\nFileInputStream(obj));\nSystem.out.println(IOUtils.readLines(inflaterInputStream));\n\n\nI know that here it's not the right place to ask about java issues,\nbut we would appreciate any help any help.\n\n\n\n--\nChico Sokol\n\n\nOn Tue, May 21, 2013 at 6:37 PM, John Szakmeister wrote:\n> On Tue, May 21, 2013 at 5:21 PM, Chico Sokol wrote:\n>> Hello,\n>>\n>> I'm building a library to manipulate git repositories (interacting\n>> directly with the filesystem).\n>>\n>> Currently, we're trying to parse commit objects. After decompressing\n>> the contents of a commit object file we got the following output:\n>>\n>> commit 191\n>> author Francisco Sokol 1369140112 -0300\n>> committer Francisco Sokol 1369140112 -0300\n>>\n>> first commit\n>\n> Does `git cat-file -p ` show a tree object? FWIW, I expected to\n> see a tree line there, so maybe this object was created without a\n> tree? I also don't see a parent listed.\n>\n> I did this on one of my repos:\n>\n>>>> buf = open('.git/objects/cd/da219e4d7beceae55af73c44cb3c9e1ec56802', 'rb').read()\n>>>> import zlib\n>>>> zlib.decompress(buf)\n> 'commit 246\\x00tree 2abfe1a7bedb29672a223a5c5f266b7dc70a8d87\\nparent\n> 0636e7ff6b79470b0cd53ceacea88e7796f202ce\\nauthor John Szakmeister\n> 1369168481 -0400\\ncommitter John Szakmeister\n> 1369168481 -0400\\n\\nGot a file listing.\\n'\n>\n> So at least creating the commits with Git, I see a tree. How was the\n> commit you're referencing created? Perhaps something is wrong with\n> that process?\n>\n>> We hoped to get the same output of a \"git cat-file -p \", but\n>> that didn't happened. From a commit object, how can I find tree object\n>> hash of this commit?\n>\n> I'd expect that too.\n>\n> -John\n"},{"id":"218102","messageId":"7v4ndwm0u8.fsf@alter.siamese.dyndns.org","threadId":"33890","inReplyTo":"CABx5MBQ57-=MPamvV-peZUdD_KDLX+5cy9vD7CL7p_Vz9BkvTg@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-05-21T22:20:47Z","receivedAt":"2013-05-21T22:20:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Chico Sokol writes:\n\n> Hello,\n>\n> I'm building a library to manipulate git repositories (interacting\n> directly with the filesystem).\n>\n> Currently, we're trying to parse commit objects. After decompressing\n> the contents of a commit object file we got the following output:\n\nWho wrote this commit object you are trying to read? Us, or your\nlibrary (this question is to see if you are chasing the right\nproblem)?\n\n> commit 191\n> author Francisco Sokol 1369140112 -0300\n> committer Francisco Sokol 1369140112 -0300\n>\n> first commit\n>\n> We hoped to get the same output of a \"git cat-file -p \", but\n> that didn't happened. From a commit object, how can I find tree object\n> hash of this commit?\n\nIf you care about the byte-for-byte compatibility, never use\n\"cat-file -p\". That is meant for human consumption.\n\n\"git cat-file commit \" gives you the raw representation after\ninflating and stripping out the first \" SP LF\" line.\n"},{"id":"218103","messageId":"7vzjvokm7f.fsf@alter.siamese.dyndns.org","threadId":"33890","inReplyTo":"CABx5MBSnpZTthOHECqkbpdbFfkb4e_uSo-rh4owBc8B_oSKjJQ@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-05-21T22:22:12Z","receivedAt":"2013-05-21T22:22:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Chico Sokol writes:\n\n> Ok, we discovered that the commit object actually contains the tree\n> object's sha1, by reading its contents with python zlib library.\n>\n> So the bug must be with our java code (we're building a java lib).\n\nWhy aren't you using jgit?\n"},{"id":"218105","messageId":"CABx5MBQd8Q-NMdFb4p9hk91mpf4FgbTGc3E0oh1tHMfptZSyUQ@mail.gmail.com","threadId":"33890","inReplyTo":"7vzjvokm7f.fsf@alter.siamese.dyndns.org","subject":"Re: Reading commit objects","fromName":"Chico Sokol","fromEmail":"chico.sokol@gmail.com","sentAt":"2013-05-21T22:33:57Z","receivedAt":"2013-05-21T22:33:57Z","isPatch":false,"sender":{"key":"chico.sokol@gmail.com","avatar":"https://gravatar.com/avatar/d15de6acd103ae702fa27f716c4569db8e12d8b70f8d5910db7864235c687fc2?d=mp&s=160"},"body":"It was git who created that object.\n\nWe're trying to build a improved java library focused in our needs\n(jgit has a really confusing api focused in solving egit needs). But\nwe're about to get into their code to discover how to decompress git\nobjects.\n\n\n--\nChico Sokol\n\n\nOn Tue, May 21, 2013 at 7:22 PM, Junio C Hamano wrote:\n> Chico Sokol writes:\n>\n>> Ok, we discovered that the commit object actually contains the tree\n>> object's sha1, by reading its contents with python zlib library.\n>>\n>> So the bug must be with our java code (we're building a java lib).\n>\n> Why aren't you using jgit?\n"},{"id":"218112","messageId":"20130521233409.GV3657@google.com","threadId":"33890","inReplyTo":"CABx5MBQd8Q-NMdFb4p9hk91mpf4FgbTGc3E0oh1tHMfptZSyUQ@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-05-21T23:34:09Z","receivedAt":"2013-05-21T23:34:09Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Chico Sokol wrote:\n\n> We're trying to build a improved java library focused in our needs\n> (jgit has a really confusing api focused in solving egit needs).\n\nJGit is also open to contributions, including contributions that\nadd less confusing API calls. :) See\n\n http://wiki.eclipse.org/JGit/User_Guide\n http://wiki.eclipse.org/EGit/Contributor_Guide#JGit\n http://wiki.eclipse.org/EGit/Contributor_Guide#Using_Gerrit_at_https:.2F.2Fgit.eclipse.org.2Fr\n https://dev.eclipse.org/mailman/listinfo/jgit-dev\n\nThanks,\nJonathan\n"},{"id":"218145","messageId":"20130522045131.GA6257@inner.h.apk.li","threadId":"33890","inReplyTo":"CABx5MBSnpZTthOHECqkbpdbFfkb4e_uSo-rh4owBc8B_oSKjJQ@mail.gmail.com","subject":"java zlib woes (was: Reading commit objects)","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2013-05-22T04:51:31Z","receivedAt":"2013-05-22T04:51:31Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Tue, 21 May 2013 19:18:35 +0000, Chico Sokol wrote:\n> Ok, we discovered that the commit object actually contains the tree\n> object's sha1, by reading its contents with python zlib library.\n> \n> So the bug must be with our java code (we're building a java lib).\n\nThat's interesting. We ran in a similar problem: We had a fetch\nwith jget hanging within the zlib deflater code in what looked\nlike a busy loop. Unfortunately we don't yet have a publishable\nrepo on which this happens.\n\nAndreas\n"},{"id":"218146","messageId":"CAJo=hJsVMGPirxTb4tA1Z_Gt2So7jzEda2pUd=XM2nztcSYBzA@mail.gmail.com","threadId":"33890","inReplyTo":"CABx5MBQd8Q-NMdFb4p9hk91mpf4FgbTGc3E0oh1tHMfptZSyUQ@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2013-05-22T05:54:19Z","receivedAt":"2013-05-22T05:54:19Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Tue, May 21, 2013 at 3:33 PM, Chico Sokol wrote:\n> It was git who created that object.\n>\n> We're trying to build a improved java library focused in our needs\n> (jgit has a really confusing api focused in solving egit needs).\n\nJGit code... is confusing because its fast. We spent a lot of time\ntrying to make things fast on the JVM, and somewhat comparable with C\nGit even though its not in C. Some of the low-level APIs are fast\nbecause they bypass conventional Java wisdom and just tell the #@!*\nmachine what to do, with no pretty bits about it. Make it pretty, it\ngoes slower. Or uses more RAM. Java likes RAM.\n\nGood luck making an improved library. JGit of course is also\ninterested in contributions. The api package has been trying to make a\nsimpler calling convention for common use cases that match the command\nline interface user are familiar with, but its still incomplete and\nhides some optimizations that are possible with the lower-level calls.\n"},{"id":"218147","messageId":"CAJo=hJtkbCeJA4ao2CkPODrNX_QaKDo4uBS4qvBVTRQ=x-Os3A@mail.gmail.com","threadId":"33890","inReplyTo":"20130522045131.GA6257@inner.h.apk.li","subject":"Re: java zlib woes (was: Reading commit objects)","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2013-05-22T05:56:21Z","receivedAt":"2013-05-22T05:56:21Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Tue, May 21, 2013 at 9:51 PM, Andreas Krey wrote:\n> On Tue, 21 May 2013 19:18:35 +0000, Chico Sokol wrote:\n>> Ok, we discovered that the commit object actually contains the tree\n>> object's sha1, by reading its contents with python zlib library.\n>>\n>> So the bug must be with our java code (we're building a java lib).\n>\n> That's interesting. We ran in a similar problem: We had a fetch\n> with jget hanging within the zlib deflater code in what looked\n> like a busy loop. Unfortunately we don't yet have a publishable\n> repo on which this happens.\n\nThis was with JGit? A stack trace and JGit version (so we can\ncorrelate line numbers) would be a much more useful bug report than\nnothing at all.\n"},{"id":"218148","messageId":"CAJo=hJtqACW+CR5FkmDfwyK1Wg3Kcppy6DbW7P=On_qJyvsYvQ@mail.gmail.com","threadId":"33890","inReplyTo":"CABx5MBSnpZTthOHECqkbpdbFfkb4e_uSo-rh4owBc8B_oSKjJQ@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2013-05-22T05:59:33Z","receivedAt":"2013-05-22T05:59:33Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Tue, May 21, 2013 at 3:18 PM, Chico Sokol wrote:\n> Ok, we discovered that the commit object actually contains the tree\n> object's sha1, by reading its contents with python zlib library.\n>\n> So the bug must be with our java code (we're building a java lib).\n>\n> Is there any non-standard issue in git's zlib compression? We're\n> decompressing its contents with java default zlib api, so it should\n> work normally, here's our code, that's printing that wrong output:\n>\n> import java.io.File;\n> import java.io.FileInputStream;\n> import java.util.zip.InflaterInputStream;\n> import org.apache.commons.io.IOUtils;\n> ...\n> File obj = new File(\".git/objects/25/0f67ef017fcb97b5371a302526872cfcadad21\");\n> InflaterInputStream inflaterInputStream = new InflaterInputStream(new\n> FileInputStream(obj));\n> System.out.println(IOUtils.readLines(inflaterInputStream));\n...\n>>> Currently, we're trying to parse commit objects. After decompressing\n>>> the contents of a commit object file we got the following output:\n>>>\n>>> commit 191\n>>> author Francisco Sokol 1369140112 -0300\n>>> committer Francisco Sokol 1369140112 -0300\n>>>\n>>> first commit\n\nYour code is broken. IOUtils is probably corrupting what you get back.\nAfter inflating the stream you should see the object type (\"commit\"),\nspace, its length in bytes as a base 10 string, and then a NUL ('\\0').\nFollowing that is the tree line, and parent(s) if any. I wonder if\nIOUtils discarded the remainder of the line after the NUL and did not\nconsider the tree line.\n\nAnd you wonder why JGit code is confusing. We can't rely on \"standard\nJava APIs\" to do the right thing, because commonly used libraries have\nmade assumptions that disagree with the way Git works.\n"},{"id":"218178","messageId":"CABx5MBS9YgNmZD_tumMJ-MJVjHbRFCKbCjs9AZ347-OCwqO7qQ@mail.gmail.com","threadId":"33890","inReplyTo":"CAJo=hJtqACW+CR5FkmDfwyK1Wg3Kcppy6DbW7P=On_qJyvsYvQ@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Chico Sokol","fromEmail":"chico.sokol@gmail.com","sentAt":"2013-05-22T14:20:44Z","receivedAt":"2013-05-22T14:20:44Z","isPatch":false,"sender":{"key":"chico.sokol@gmail.com","avatar":"https://gravatar.com/avatar/d15de6acd103ae702fa27f716c4569db8e12d8b70f8d5910db7864235c687fc2?d=mp&s=160"},"body":"I'm not criticizing JGit, guys. It simply doesn't fit into our needs.\nWe're not interested in mapping git commands in java and don't have\nthe same RAM limitations.\n\nI know JGit team is doing a great job and we do not intend to build a\nlibrary with such completeness.\n\nAre you guys contributors of JGit? Can you guys point me out to the\ncode that unpacks git objects? The closest I could get was that class:\nhttps://github.com/eclipse/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/UnpackedObject.java\n\nIt seems to be a standard and a non standard format of the packed\nobject, as I read the comments of this method:\nhttps://github.com/eclipse/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/UnpackedObject.java#L272\n\nI suspect that the default inflater class of java api expect the\nobject to be in the standard format.\n\nWhat the following comment mean? What's the \"Experimental pack-based\"\nformat? Is there any docs on the specs of that?\n\nWe must determine if the buffer contains the standard\nzlib-deflated stream or the experimental format based\non the in-pack object format. Compare the header byte\nfor each format:\nRFC1950 zlib w/ deflate : 0www1000 : 0 <= www <= 7\nExperimental pack-based : Stttssss : ttt = 1,2,3,4\n\n\n--\nChico Sokol\n\n\nOn Wed, May 22, 2013 at 2:59 AM, Shawn Pearce wrote:\n> On Tue, May 21, 2013 at 3:18 PM, Chico Sokol wrote:\n>> Ok, we discovered that the commit object actually contains the tree\n>> object's sha1, by reading its contents with python zlib library.\n>>\n>> So the bug must be with our java code (we're building a java lib).\n>>\n>> Is there any non-standard issue in git's zlib compression? We're\n>> decompressing its contents with java default zlib api, so it should\n>> work normally, here's our code, that's printing that wrong output:\n>>\n>> import java.io.File;\n>> import java.io.FileInputStream;\n>> import java.util.zip.InflaterInputStream;\n>> import org.apache.commons.io.IOUtils;\n>> ...\n>> File obj = new File(\".git/objects/25/0f67ef017fcb97b5371a302526872cfcadad21\");\n>> InflaterInputStream inflaterInputStream = new InflaterInputStream(new\n>> FileInputStream(obj));\n>> System.out.println(IOUtils.readLines(inflaterInputStream));\n> ...\n>>>> Currently, we're trying to parse commit objects. After decompressing\n>>>> the contents of a commit object file we got the following output:\n>>>>\n>>>> commit 191\n>>>> author Francisco Sokol 1369140112 -0300\n>>>> committer Francisco Sokol 1369140112 -0300\n>>>>\n>>>> first commit\n>\n> Your code is broken. IOUtils is probably corrupting what you get back.\n> After inflating the stream you should see the object type (\"commit\"),\n> space, its length in bytes as a base 10 string, and then a NUL ('\\0').\n> Following that is the tree line, and parent(s) if any. I wonder if\n> IOUtils discarded the remainder of the line after the NUL and did not\n> consider the tree line.\n>\n> And you wonder why JGit code is confusing. We can't rely on \"standard\n> Java APIs\" to do the right thing, because commonly used libraries have\n> made assumptions that disagree with the way Git works.\n"},{"id":"218179","messageId":"CABx5MBSmCN=avRDCJ+RU8FoRDaGG=6uRfTdVR9m3A=SqpuKAjQ@mail.gmail.com","threadId":"33890","inReplyTo":"CAJo=hJtqACW+CR5FkmDfwyK1Wg3Kcppy6DbW7P=On_qJyvsYvQ@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Chico Sokol","fromEmail":"chico.sokol@gmail.com","sentAt":"2013-05-22T14:25:31Z","receivedAt":"2013-05-22T14:25:31Z","isPatch":false,"sender":{"key":"chico.sokol@gmail.com","avatar":"https://gravatar.com/avatar/d15de6acd103ae702fa27f716c4569db8e12d8b70f8d5910db7864235c687fc2?d=mp&s=160"},"body":"> Your code is broken. IOUtils is probably corrupting what you get back.\n> After inflating the stream you should see the object type (\"commit\"),\n> space, its length in bytes as a base 10 string, and then a NUL ('\\0').\n> Following that is the tree line, and parent(s) if any. I wonder if\n> IOUtils discarded the remainder of the line after the NUL and did not\n> consider the tree line.\n\n\nMaybe you're right, Shawn. I've also tried the following code:\n\nFile dotGit = new File(\"objects/25/0f67ef017fcb97b5371a302526872cfcadad21\");\nInflaterInputStream inflaterInputStream = new InflaterInputStream(new\nFileInputStream(dotGit));\nByteArrayOutputStream os = new ByteArrayOutputStream();\nIOUtils.copyLarge(inflaterInputStream, os);\nSystem.out.println(new String(os.toByteArray()));\n\nBut we got the same result, I'll try to read the bytes by myself\n(without apache IOUtils). Is the contents of a unpacked object utf-8\nencoded?\n"},{"id":"218183","messageId":"CABx5MBR=bvFEXhAiXgOfhkgVqhrYdjsWcWyHPc7+=bL-SW4vQA@mail.gmail.com","threadId":"33890","inReplyTo":"CABx5MBSmCN=avRDCJ+RU8FoRDaGG=6uRfTdVR9m3A=SqpuKAjQ@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Chico Sokol","fromEmail":"chico.sokol@gmail.com","sentAt":"2013-05-22T14:47:31Z","receivedAt":"2013-05-22T14:47:31Z","isPatch":false,"sender":{"key":"chico.sokol@gmail.com","avatar":"https://gravatar.com/avatar/d15de6acd103ae702fa27f716c4569db8e12d8b70f8d5910db7864235c687fc2?d=mp&s=160"},"body":"Solved! It was exaclty the problem pointed by Shawn.\n\nHere is the working code:\n\nFile dotGit = new File(\"objects/25/0f67ef017fcb97b5371a302526872cfcadad21\");\nInflaterInputStream inflaterInputStream = new InflaterInputStream(new\nFileInputStream(dotGit));\nInteger read = inflaterInputStream.read();\nwhile(read != 0) { //reading the bytes from 'commit \\0'\n read = inflaterInputStream.read();\n System.out.println((char)read.byteValue());\n}\nByteArrayOutputStream os = new ByteArrayOutputStream();\nIOUtils.copyLarge(inflaterInputStream, os);\nSystem.out.println(new String(os.toByteArray()));\n\nThank you all!\n\n\n\n--\nChico Sokol\n\n\nOn Wed, May 22, 2013 at 11:25 AM, Chico Sokol wrote:\n>> Your code is broken. IOUtils is probably corrupting what you get back.\n>> After inflating the stream you should see the object type (\"commit\"),\n>> space, its length in bytes as a base 10 string, and then a NUL ('\\0').\n>> Following that is the tree line, and parent(s) if any. I wonder if\n>> IOUtils discarded the remainder of the line after the NUL and did not\n>> consider the tree line.\n>\n>\n> Maybe you're right, Shawn. I've also tried the following code:\n>\n> File dotGit = new File(\"objects/25/0f67ef017fcb97b5371a302526872cfcadad21\");\n> InflaterInputStream inflaterInputStream = new InflaterInputStream(new\n> FileInputStream(dotGit));\n> ByteArrayOutputStream os = new ByteArrayOutputStream();\n> IOUtils.copyLarge(inflaterInputStream, os);\n> System.out.println(new String(os.toByteArray()));\n>\n> But we got the same result, I'll try to read the bytes by myself\n> (without apache IOUtils). Is the contents of a unpacked object utf-8\n> encoded?\n"},{"id":"218204","messageId":"CAJo=hJt7W4O3L6M6TmEvEfCKvjqwdXHAbSjCwazuzNsPzeKvhA@mail.gmail.com","threadId":"33890","inReplyTo":"CABx5MBSmCN=avRDCJ+RU8FoRDaGG=6uRfTdVR9m3A=SqpuKAjQ@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2013-05-22T19:59:07Z","receivedAt":"2013-05-22T19:59:07Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, May 22, 2013 at 7:25 AM, Chico Sokol wrote:\n>> Your code is broken. IOUtils is probably corrupting what you get back.\n>> After inflating the stream you should see the object type (\"commit\"),\n>> space, its length in bytes as a base 10 string, and then a NUL ('\\0').\n>> Following that is the tree line, and parent(s) if any. I wonder if\n>> IOUtils discarded the remainder of the line after the NUL and did not\n>> consider the tree line.\n>\n...\n> Is the contents of a unpacked object utf-8\n> encoded?\n\nIts more complicated than that. Commit objects are usually in utf-8,\nunless a repository configuration setting told you otherwise, or an\nencoding header appears in the commit. And sometimes that data lies\nanyway. ISO-8859-1 is one of the safer forms of reading a commit, but\nthat also isn't always accurate.\n"},{"id":"218205","messageId":"CAJo=hJuO23YGMJV69WvdKweW7EYCYVE-f93w5a=+N2_xh_e1+A@mail.gmail.com","threadId":"33890","inReplyTo":"CABx5MBS9YgNmZD_tumMJ-MJVjHbRFCKbCjs9AZ347-OCwqO7qQ@mail.gmail.com","subject":"Re: Reading commit objects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2013-05-22T20:02:06Z","receivedAt":"2013-05-22T20:02:06Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, May 22, 2013 at 7:20 AM, Chico Sokol wrote:\n> I'm not criticizing JGit, guys. It simply doesn't fit into our needs.\n> We're not interested in mapping git commands in java and don't have\n> the same RAM limitations.\n\nI guess you aren't trying to process the WebKit or Linux kernel\nrepositories. Or you can afford more RAM than I can[1]. :-)\n\n[1] $DAY_JOB has lots of RAM. Lots.\n\n> Are you guys contributors of JGit?\n\nNot really. I had nothing to do with JGit. :-)\n\n> Can you guys point me out to the\n> code that unpacks git objects? The closest I could get was that class:\n> https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/UnpackedObject.java\n\nThis class handles the loose object format in $GIT_DIR/objects, but\ndoes not handle objects contained in pack files. That is elsewhere,\nand well, more complex. Look at PackFile.java.\n\n> It seems to be a standard and a non standard format of the packed\n> object, as I read the comments of this method:\n> https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/UnpackedObject.java#L272\n\nThere are two formats, the official format that is used, and an\nexperimental format that was discarded but is still supported for\nlegacy reasons.\n\n> I suspect that the default inflater class of java api expect the\n> object to be in the standard format.\n>\n> What the following comment mean? What's the \"Experimental pack-based\"\n> format? Is there any docs on the specs of that?\n\nRead the code. This is the dead format that is no longer written, but\nis still supported.\n"},{"id":"218588","messageId":"20130527041146.GJ9448@inner.h.apk.li","threadId":"33890","inReplyTo":"CAJo=hJtkbCeJA4ao2CkPODrNX_QaKDo4uBS4qvBVTRQ=x-Os3A@mail.gmail.com","subject":"Re: java zlib woes (was: Reading commit objects)","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2013-05-27T04:11:46Z","receivedAt":"2013-05-27T04:11:46Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Tue, 21 May 2013 22:56:21 +0000, Shawn Pearce wrote:\n...\n> This was with JGit? A stack trace and JGit version (so we can\n> correlate line numbers) would be a much more useful bug report than\n> nothing at all.\n\nI now have a full test case (involving a generated repo just shy of 1GB)\nthat will reproduce that hang. Will look up the existing jgit bug to\nreport there.\n\nAndreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds \nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"219343","messageId":"20130604101836.GH22784@inner.h.apk.li","threadId":"33890","inReplyTo":"20130527041146.GJ9448@inner.h.apk.li","subject":"fetch delta resolution vs. checkout (was: java zlib woes)","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2013-06-04T10:18:36Z","receivedAt":"2013-06-04T10:18:36Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Mon, 27 May 2013 06:11:46 +0000, Andreas Krey wrote:\n...\n> \n> I now have a full test case (involving a generated repo just shy of 1GB)\n> that will reproduce that hang. Will look up the existing jgit bug to\n> report there.\n\nOn https://bugs.eclipse.org/bugs/show_bug.cgi?id=394078\n\nA question: The delta decoding. If I understand correctly,\ngit and jgit do verify the packfile content after fetching/cloning,\nand need to resolve any deltified files in the pack.\n\nAnd when checking out a commit it needs this to again for the\nfiles that are being checked out?\n\nBecause we now have the phenomenon that the packfile is fetched\nok, but a checkout then hangs (100%) CPU on one of the large files,\nand on one that should, according to core.bigfilethreshold, not\neven be deltified.\n\n(Setting core.bigfilethreshold to 20m in the source repo (C git)\ngets jgit to no longer hang in the fetch/delta resolution phase.\nAnd it doesn't look like jgit would repack the pack file, and\nuses it as it was received plus 20 bytes at the end.)\n\nAndreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds \nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"}]}