{"thread":{"id":"28988","subject":"Stack overflow at write_one()","startedAt":"2011-11-19T20:27:29Z","lastAt":"2011-11-20T02:00:31Z","messageCount":6,"participants":["Cesar Eduardo Barros","Junio C Hamano","Shawn Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"179748","messageId":"4EC81131.2010704@cesarb.net","threadId":"28988","inReplyTo":null,"subject":"Stack overflow at write_one()","fromName":"Cesar Eduardo Barros","fromEmail":"cesarb@cesarb.net","sentAt":"2011-11-19T20:27:29Z","receivedAt":"2011-11-19T20:27:29Z","isPatch":false,"sender":{"key":"cesarb@cesarb.net","avatar":"https://avatars.githubusercontent.com/u/31324?v=4"},"body":"I have found a stack overflow at builtin/pack-objects.c:write_one(), \nwhere it calls itself endlessly. This is caused by the object_entry e \nand e->delta->delta being the same. But I have no idea how that happened.\n\nFirst, the full story:\n\nI used Google's repo tool to mirror AOSP to my machine. This mirrors \nseveral kernel trees (six last time I counted), without sharing objects \none with another. To save space, I decided to point their \nobjects/info/alternates to my mirror of the Linus kernel tree (which \nshould be safe, since Linus makes it always fast-forward), and run \"git \ngc\" on them to create a smaller pack. This worked for all trees except \none, where it core dumped (abrt report at \nhttps://bugzilla.redhat.com/show_bug.cgi?id=755132).\n\nI compiled the latest git (v1.7.8-rc3-17-gf56ef11) to see if it still \nhappened, and here is what I could get from gdb. I attached to the \npack-objects process before it crashed (full command line \"git \npack-objects --keep-true-parents --honor-pack-keep --non-empty --all \n--reflog --unpack-unreachable --local --delta-base-offset \n/home/cesarb/src/bug755132/omap.git/objects/pack/.tmp-5171-pack\"), \ncontinued, and let it crash:\n\n(gdb) cont\nContinuing.\n[New Thread 0x7f3f2bad3700 (LWP 5205)]\n[New Thread 0x7f3f2b2d2700 (LWP 5206)]\n[New Thread 0x7f3f2aad1700 (LWP 5207)]\n[New Thread 0x7f3f2a2d0700 (LWP 5208)]\n[Thread 0x7f3f2b2d2700 (LWP 5206) exited]\n[Thread 0x7f3f2bad3700 (LWP 5205) exited]\n[Thread 0x7f3f2aad1700 (LWP 5207) exited]\n[Thread 0x7f3f2a2d0700 (LWP 5208) exited]\n\nProgram received signal SIGSEGV, Segmentation fault.\n0x00000000004472b9 in write_one (f=0x6a97db0, e=0x7f3f30233490,\n     offset=0x7fff79b53908) at builtin/pack-objects.c:415\n415\t{\n\nUnlike on Fedora's git binary, where it happened on a call instruction, \nthis time it happened on a push instruction:\n\n(gdb) disassemble\nDump of assembler code for function write_one:\n    0x00000000004472b0 <+0>:\tpush   %r15\n    0x00000000004472b2 <+2>:\tpush   %r14\n    0x00000000004472b4 <+4>:\tpush   %r13\n    0x00000000004472b6 <+6>:\tmov    %rdx,%r13\n=> 0x00000000004472b9 <+9>:\tpush   %r12\n    0x00000000004472bb <+11>:\tmov    %rdi,%r12\n\nThe last few frames on the stack show the endless recursion:\n\n(gdb) where\n#0  0x00000000004472b9 in write_one (f=0x6a97db0, e=0x7f3f30233490,\n     offset=0x7fff79b53908) at builtin/pack-objects.c:415\n#1  0x00000000004472ed in write_one (f=0x6a97db0, e=0x7f3f30277390,\n     offset=0x7fff79b53908) at builtin/pack-objects.c:423\n#2  0x00000000004472ed in write_one (f=0x6a97db0, e=0x7f3f30233490,\n     offset=0x7fff79b53908) at builtin/pack-objects.c:423\n#3  0x00000000004472ed in write_one (f=0x6a97db0, e=0x7f3f30277390,\n     offset=0x7fff79b53908) at builtin/pack-objects.c:423\n#4  0x00000000004472ed in write_one (f=0x6a97db0, e=0x7f3f30233490,\n     offset=0x7fff79b53908) at builtin/pack-objects.c:423\n\nAnd here is the loop in the data structures:\n\n(gdb) p e\n$1 = (struct object_entry *) 0x7f3f30233490\n(gdb) p e->delta\n$2 = (struct object_entry *) 0x7f3f30277390\n(gdb) p e->delta->delta\n$3 = (struct object_entry *) 0x7f3f30233490\n\nUnfortunately, I do not know enough of git's internals to debug further. \nIn case it helps, here is the contents of a few of these structures:\n\n(gdb) p *e\n$4 = {idx = {\n     sha1 = \"\\257>J\\241)\\266\\023\\064\\a\\342J\\320\\375ӆ\\262M\\245\", \n<incomplete sequence \\356>, crc32 = 0, offset = 0}, size = 20, in_pack = \n0x259b610,\n   in_pack_offset = 231061238, delta = 0x7f3f30277390,\n   delta_child = 0x7f3f30277390, delta_sibling = 0x7f3f30413b10,\n   delta_data = 0x0, delta_size = 20, z_delta_size = 0, hash = 2099915708,\n   type = OBJ_OFS_DELTA, in_pack_type = OBJ_OFS_DELTA,\n   in_pack_header_size = 5 '\\005', preferred_base = 0 '\\000',\n   no_try_delta = 0 '\\000', tagged = 0 '\\000', filled = 1 '\\001'}\n(gdb) p *(e->delta)\n$5 = {idx = {\n     sha1 = \n\"\\372\\307\\035\\372\\017\\350\\307\\f\\310R\\t\\236\\006\\034\\063N*T\\216\\253\",\n     crc32 = 0, offset = 0}, size = 14, in_pack = 0x259b610,\n   in_pack_offset = 39990, delta = 0x7f3f30233490,\n   delta_child = 0x7f3f30233490, delta_sibling = 0x0, delta_data = 0x0,\n   delta_size = 14, z_delta_size = 0, hash = 2099915708, type = \nOBJ_REF_DELTA,\n   in_pack_type = OBJ_REF_DELTA, in_pack_header_size = 21 '\\025',\n   preferred_base = 0 '\\000', no_try_delta = 0 '\\000', tagged = 0 '\\000',\n   filled = 1 '\\001'}\n(gdb) p *(e->in_pack)\n$6 = {next = 0x25a53c0, windows = 0x259bc40, pack_size = 449155894,\n   index_data = 0x7f3f4f0a9000, index_size = 58351420, num_objects = \n2083941,\n   num_bad_objects = 0, bad_object_sha1 = 0x0, index_version = 2,\n   mtime = 1321387261, pack_fd = -1, pack_local = 1, pack_keep = 0,\n   do_not_close = 0, sha1 = \n\"\\371Q4\\177.ȳv\\364\\246\\332Z\\234\\025?\\352ݠP\\210\",\n   pack_name = 0x259b671 \n\"/home/cesarb/src/bug755132/omap.git/objects/pack/pack-f951347f2ec8b376f4a6da5a9c153feadda05088.pack\"}\n\nI tried using \"git fsck\" to see if it could find anything strange, but \nit seems to get stuck (using 100% CPU) after these lines:\n\n[...]\nChecking commit fb630b9fc902e24209166b1659a8b375bf38099c\nChecking tree fc32c012c750084eb1d82782cee7c80a45a78289\nChecking blob fc7bbba585cee2c2b0d5282c42fb986bfb032a0a\nChecking commit fdcb23634c9b6649bb02c681033d4973491b0e35\nChecking tree fe773cf73ff553249be2f24ddf770f5dc43a41f1\nChecking blob fe67b5c79f0ff33d92ebe7469a89c5a5d044fc0a\nChecking blob fe73276e026bf263f494a917c84c6a3fcaeaaeda\nChecking tree fe30eda9d92d074816f9c3a47fd3ffb9b89ca835\nChecking tree fe9c75396e6d433b289d0e40c7e47921b91cad3a\nChecking blob ff3ed6086ce1c6b6b4b5111c034d14a208c0d045\nChecking blob ff66638ff54d5ad7067e4f246d392059eef1a7bf\nChecking tree ff126d2bc67017199049ddba761979f3bda57eb9\n\nUnfortunately, the reproducer I have (a copy of both trees with \nobjects/info/alternates modified) is 1.8G in size, and I do not know how \nto create a smaller reproducer. If you know of a command which would get \nmore relevant information from them, just ask; I plan on keeping them \naround for a while.\n\n-- \nCesar Eduardo Barros\ncesarb@cesarb.net\ncesar.barros@gmail.com\n"},{"id":"179750","messageId":"7vty5zizwn.fsf@alter.siamese.dyndns.org","threadId":"28988","inReplyTo":"4EC81131.2010704@cesarb.net","subject":"Re: Stack overflow at write_one()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-11-19T21:08:08Z","receivedAt":"2011-11-19T21:08:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Already found the real cause (jGit bug) and workaround posted, I think.\n\nSee $gmane/185573\n"},{"id":"179751","messageId":"4EC823A0.3010603@cesarb.net","threadId":"28988","inReplyTo":"7vty5zizwn.fsf@alter.siamese.dyndns.org","subject":"Re: Stack overflow at write_one()","fromName":"Cesar Eduardo Barros","fromEmail":"cesarb@cesarb.net","sentAt":"2011-11-19T21:46:08Z","receivedAt":"2011-11-19T21:46:08Z","isPatch":false,"sender":{"key":"cesarb@cesarb.net","avatar":"https://avatars.githubusercontent.com/u/31324?v=4"},"body":"Em 19-11-2011 19:08, Junio C Hamano escreveu:\n> Already found the real cause (jGit bug) and workaround posted, I think.\n\nI presume the cause then is what was fixed by \nhttp://egit.eclipse.org/w/?p=jgit.git;a=commit;h=2fbf296fda205446eac11a13abd4fcdb182f28d9 \n?\n\n> See $gmane/185573\n\nThat did it, thanks! The patch had an offset, a fuzz, and a reject, but \nit was easy to fix by hand.\n\n$ ../git/git gc\nCounting objects: 30254, done.\nDelta compression using up to 4 threads.\nCompressing objects: 100% (6614/6614), done.\nwarning: recursive delta detected for object \nfac71dfa0fe8c70cc852099e061c334e2a548eab\nwarning: recursive delta detected for object \n1b730f5b2e0bdb2a2206af8ed30170509e75a2f5\nwarning: recursive delta detected for object \n2f25a87e67fa3a226e367b9e080f11aa90c9f953\nwarning: recursive delta detected for object \nd5e5eefac91788da9a94efe9a15e0b928a77489e\nWriting objects: 100% (30254/30254), done.\nTotal 30254 (delta 24008), reused 28803 (delta 23266)\n\nAnd after that the repack does not break anymore:\n\n$ ../git/git gc\nCounting objects: 30254, done.\nDelta compression using up to 4 threads.\nCompressing objects: 100% (5876/5876), done.\nWriting objects: 100% (30254/30254), done.\nTotal 30254 (delta 24008), reused 30254 (delta 24008)\n\n-- \nCesar Eduardo Barros\ncesarb@cesarb.net\ncesar.barros@gmail.com\n"},{"id":"179752","messageId":"CAJo=hJv2aGEFcMjTPxJsyLerqUn3w3hc3hWnc1ScaDrSGihzyQ@mail.gmail.com","threadId":"28988","inReplyTo":"4EC823A0.3010603@cesarb.net","subject":"Re: Stack overflow at write_one()","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-11-19T23:30:35Z","receivedAt":"2011-11-19T23:30:35Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sat, Nov 19, 2011 at 13:46, Cesar Eduardo Barros <cesarb@cesarb.net> wrote:\n> Em 19-11-2011 19:08, Junio C Hamano escreveu:\n>>\n>> Already found the real cause (jGit bug) and workaround posted, I think.\n>\n> I presume the cause then is what was fixed by\n> http://egit.eclipse.org/w/?p=jgit.git;a=commit;h=2fbf296fda205446eac11a13abd4fcdb182f28d9\n> ?\n\nYes. The AOSP servers were all updated with the above JGit patch, so\nthe servers are no longer sending duplicate objects. But yes, for a\nperiod of time there were duplicates in the kernel repositories,\nparticularly kernal/omap.\n"},{"id":"179753","messageId":"4EC8437C.6000808@cesarb.net","threadId":"28988","inReplyTo":"CAJo=hJv2aGEFcMjTPxJsyLerqUn3w3hc3hWnc1ScaDrSGihzyQ@mail.gmail.com","subject":"Re: Stack overflow at write_one()","fromName":"Cesar Eduardo Barros","fromEmail":"cesarb@cesarb.net","sentAt":"2011-11-20T00:02:04Z","receivedAt":"2011-11-20T00:02:04Z","isPatch":false,"sender":{"key":"cesarb@cesarb.net","avatar":"https://avatars.githubusercontent.com/u/31324?v=4"},"body":"Em 19-11-2011 21:30, Shawn Pearce escreveu:\n> On Sat, Nov 19, 2011 at 13:46, Cesar Eduardo Barros<cesarb@cesarb.net>  wrote:\n>> Em 19-11-2011 19:08, Junio C Hamano escreveu:\n>>>\n>>> Already found the real cause (jGit bug) and workaround posted, I think.\n>>\n>> I presume the cause then is what was fixed by\n>> http://egit.eclipse.org/w/?p=jgit.git;a=commit;h=2fbf296fda205446eac11a13abd4fcdb182f28d9\n>> ?\n>\n> Yes. The AOSP servers were all updated with the above JGit patch, so\n> the servers are no longer sending duplicate objects. But yes, for a\n> period of time there were duplicates in the kernel repositories,\n> particularly kernal/omap.\n\nSo, would an alternative workaround in my situation be to delete \nkernel/omap.git and let repo sync recreate it? It seems repo does not \nhave extra metadata anywhere else, so just removing the directory should \nbe enough for it to clone again from scratch, hopefully getting a \ncorrected pack from the server.\n\n-- \nCesar Eduardo Barros\ncesarb@cesarb.net\ncesar.barros@gmail.com\n"},{"id":"179754","messageId":"CAJo=hJtBKmf_yuJDXoKPA7OXrXhaZtm+P6rQx1376dngAmQNiA@mail.gmail.com","threadId":"28988","inReplyTo":"4EC8437C.6000808@cesarb.net","subject":"Re: Stack overflow at write_one()","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-11-20T02:00:31Z","receivedAt":"2011-11-20T02:00:31Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sat, Nov 19, 2011 at 16:02, Cesar Eduardo Barros <cesarb@cesarb.net> wrote:\n> Em 19-11-2011 21:30, Shawn Pearce escreveu:\n>>\n>> On Sat, Nov 19, 2011 at 13:46, Cesar Eduardo Barros<cesarb@cesarb.net>\n>>  wrote:\n>>>\n>>> Em 19-11-2011 19:08, Junio C Hamano escreveu:\n>>>>\n>>>> Already found the real cause (jGit bug) and workaround posted, I think.\n>>>\n>>> I presume the cause then is what was fixed by\n>>>\n>>> http://egit.eclipse.org/w/?p=jgit.git;a=commit;h=2fbf296fda205446eac11a13abd4fcdb182f28d9\n>>> ?\n>>\n>> Yes. The AOSP servers were all updated with the above JGit patch, so\n>> the servers are no longer sending duplicate objects. But yes, for a\n>> period of time there were duplicates in the kernel repositories,\n>> particularly kernal/omap.\n>\n> So, would an alternative workaround in my situation be to delete\n> kernel/omap.git and let repo sync recreate it? It seems repo does not have\n> extra metadata anywhere else, so just removing the directory should be\n> enough for it to clone again from scratch, hopefully getting a corrected\n> pack from the server.\n\nYes. repo does not have extra state, so just removing the directory\nand running `repo sync` again to clone the repository would correct\nthe local repository.\n"}]}