{"thread":{"id":"27274","subject":"git failure on HP-UX","startedAt":"2011-05-05T18:04:38Z","lastAt":"2011-05-06T21:34:16Z","messageCount":6,"participants":["Kibler, Bill","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"167139","messageId":"88C5107F36BD7E4BBFDFDB140E3D81077FA78CF12C@GVW1362EXC.americas.hpqcorp.net","threadId":"27274","inReplyTo":null,"subject":"git failure on HP-UX","fromName":"Kibler, Bill","fromEmail":"bill.kibler@hp.com","sentAt":"2011-05-05T18:04:38Z","receivedAt":"2011-05-05T18:04:38Z","isPatch":false,"sender":{"key":"bill.kibler@hp.com","avatar":null},"body":"An outside HP service provides open source packages compiled to run on HP-UX systems. Our group within HP is currently moving to git with the majority of the git usage on linux systems. However, some older work still needs to be done on HP-UX systems and will require a working git set of tools. The current set of git tools works properly except for accessing gitolite, where ssh access and support is used. When trying to \"clone\" a gitolite hosted repo over ssh, the HP-UX git session fails with a core dump and \"SIGBUS\" error.\n\nI went through updating the HP-UX system with all the latest patches and was unable to alter the response in any way. After trying many options it became clear that the problem was inside git. I used wdb/gdb and tusc, where I tracked the failure down to a call to \"recv_sideband\" in sideband.c. It cores when the call is made. The calling process is \"sideband_demux\" in builtin/fetch-pack.c. \"tusc\" output showed a \"pipe\" of size 8192 being used prior to the core. I believe that git \"forks\" off a \"ssh user@hostanme upload-pack 'repo_name'\" that is \"piped\" back to the git stream process.\n\nIn looking at the code, \"sideband.h\" defines \"LARGE_PACKET_MAX 65520\" and is related to the passed flag \"side-band-64k\" as discussed in git document pack-protocol.txt. The current default usage seems to be 64K transfers, yet if we check the \"include/limits.h\" of HP-UX we see a \"PIPE_BUF\" set to 8192. Along with the tusc indication of 8K pipe size, I suspect that HP-UX is coring due to git trying to use a 64K pipe when 8K is max.\n\nI solved the probem for now, by changing the file sideband.h to use \"LARGE_PACKET_MAX 8208\". If you use 8192 or less, you get a failure of too small size in \"packet_read_line\". So I added 16 bytes to 8192 to get the 8208 value. I noticed the previous value and some comments indicating an 8 to 10 byte overhead was needed and thus a few bytes more is needed in this define.\n\nI suspect the correct process would be making the HP-UX git use/send \"side-band\" and not \"side-band-64k\" in the get packet protocol, but I was unable to find out how to do that. The pack-protocol.txt discussion completely ignores this topic and how to handle clients with smaller abilities than servers. It appeared to me that the server set the transfer size and the client is just suppose to accept it. Under HPUX as a client, that is not an option, as it has a 8K max limit.\n"},{"id":"167158","messageId":"7vfwot6k7a.fsf@alter.siamese.dyndns.org","threadId":"27274","inReplyTo":"88C5107F36BD7E4BBFDFDB140E3D81077FA78CF12C@GVW1362EXC.americas.hpqcorp.net","subject":"Re: git failure on HP-UX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-05T19:06:33Z","receivedAt":"2011-05-05T19:06:33Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Kibler, Bill\" <bill.kibler@hp.com> writes:\n\n> In looking at the code, \"sideband.h\" defines \"LARGE_PACKET_MAX 65520\"\n> and is related to the passed flag \"side-band-64k\" as discussed in git\n> document pack-protocol.txt. The current default usage seems to be 64K\n> transfers, yet if we check the \"include/limits.h\" of HP-UX we see a\n> \"PIPE_BUF\" set to 8192. Along with the tusc indication of 8K pipe size,\n> I suspect that HP-UX is coring due to git trying to use a 64K pipe when\n> 8K is max.\n>\n> I solved the probem for now, by changing the file sideband.h to use\n> \"LARGE_PACKET_MAX 8208\".\n\nThis does not make any sense.  We may make write(2) and read(2) system\ncalls with 64k (or maybe bit more) chunk, but that does not mean the\nimplementation of these system calls must take that as a whole.  Your\nwrite(2) is allowed to write only whatever fits your pipe buffer, and tell\nthe caller \"I wrote only 8192 bytes\", and the code is supposed to loop,\nadvancing the write pointer by 8k and calling write(2) again, until you\nwrite everything to whoever is reading the other end of the pipe.  The\nsame thing for the read(2).\n\nIf you can find a place where we make write(2)/read(2) and blindly assumes\nthat a non-negative return means everything was written/read successfully,\nthen you have found a bug.\n\nIf the symptom _were_ a deadlock where the writer of one pipe expected to\nbe able to send 64k to the other end of the pipe and then hear back from\nthe other side with a separate read, I would understand that could happen\n(actually we know a local pipe transfer without ssh has that kind of\npotential deadlock but I think the size we assume that can fit in the pipe\nbuffer is far smaller than 8k).  But I do not understand where a SIGBUS\ncan come from.\n"},{"id":"167189","messageId":"88C5107F36BD7E4BBFDFDB140E3D81077FA78CF26B@GVW1362EXC.americas.hpqcorp.net","threadId":"27274","inReplyTo":"7vfwot6k7a.fsf@alter.siamese.dyndns.org","subject":"RE: git failure on HP-UX","fromName":"Kibler, Bill","fromEmail":"bill.kibler@hp.com","sentAt":"2011-05-05T20:51:52Z","receivedAt":"2011-05-05T20:51:52Z","isPatch":false,"sender":{"key":"bill.kibler@hp.com","avatar":null},"body":"Let me say first off that my change seems to work, but clearly I feel it\nwas more a lucky guess on my part than hunting down the actual cause since\nI feel it is a combination of git actions and HP-UX libraries. I can't really\ndebug all of HP-UX libc, so I have to make an educated guess as to what might\nbe happening. For me, I felt that some mechanism inside of git should be\npossible to set on the client side to limit transfer buffer sizes without\na recompile.\n\nNow having said that and had a chance to consider my explanation as stated,\nit might be more accurate to say that what I think is happening is closer to\nthis - as I understand the SIGBUS and other messages around the action,\nI think the library call is setting up the pipe buffer as 8k of memory,\nwhile git is assuming(?) a 64K space and returns a pointer to the libc\nfunction that is well beyond - 48K beyond - the size of the buffer.\nAs I take what is happing, one of these processes is returning a pointer\nthat points outside approved space and causing the system to fault.\nThe debug steps all showed the values from git as being reasonable or\nwhat I thought them to be, yet HP-UX faulted when entering the called\nfunction. \n\nSince I was unable to date to clearly understand all the coding associated with\nthe fetch-pack process, just running out of time on the project, I was hoping\nto get more data from the git email group, that might highlight something I missed\ndebugging the problem. I clearly was unable to find enough text to explain both\nsides of the side-band handshake and how sideband values are used - maybe if\nI had more time to understand the code fully, but I don't.\n\nBill. \n\n-----Original Message-----\nFrom: Junio C Hamano [mailto:gitster@pobox.com] \nSent: Thursday, May 05, 2011 12:07 PM\nTo: Kibler, Bill\nCc: git@vger.kernel.org; Richard Lloyd\nSubject: Re: git failure on HP-UX\n\n\"Kibler, Bill\" <bill.kibler@hp.com> writes:\n\n> In looking at the code, \"sideband.h\" defines \"LARGE_PACKET_MAX 65520\"\n> and is related to the passed flag \"side-band-64k\" as discussed in git\n> document pack-protocol.txt. The current default usage seems to be 64K\n> transfers, yet if we check the \"include/limits.h\" of HP-UX we see a\n> \"PIPE_BUF\" set to 8192. Along with the tusc indication of 8K pipe size,\n> I suspect that HP-UX is coring due to git trying to use a 64K pipe when\n> 8K is max.\n>\n> I solved the probem for now, by changing the file sideband.h to use\n> \"LARGE_PACKET_MAX 8208\".\n\nThis does not make any sense.  We may make write(2) and read(2) system\ncalls with 64k (or maybe bit more) chunk, but that does not mean the\nimplementation of these system calls must take that as a whole.  Your\nwrite(2) is allowed to write only whatever fits your pipe buffer, and tell\nthe caller \"I wrote only 8192 bytes\", and the code is supposed to loop,\nadvancing the write pointer by 8k and calling write(2) again, until you\nwrite everything to whoever is reading the other end of the pipe.  The\nsame thing for the read(2).\n\nIf you can find a place where we make write(2)/read(2) and blindly assumes\nthat a non-negative return means everything was written/read successfully,\nthen you have found a bug.\n\nIf the symptom _were_ a deadlock where the writer of one pipe expected to\nbe able to send 64k to the other end of the pipe and then hear back from\nthe other side with a separate read, I would understand that could happen\n(actually we know a local pipe transfer without ssh has that kind of\npotential deadlock but I think the size we assume that can fit in the pipe\nbuffer is far smaller than 8k).  But I do not understand where a SIGBUS\ncan come from.\n"},{"id":"167274","messageId":"88C5107F36BD7E4BBFDFDB140E3D81077FA78CF67E@GVW1362EXC.americas.hpqcorp.net","threadId":"27274","inReplyTo":"7vfwot6k7a.fsf@alter.siamese.dyndns.org","subject":"RE: git failure on HP-UX - more data","fromName":"Kibler, Bill","fromEmail":"bill.kibler@hp.com","sentAt":"2011-05-06T19:02:50Z","receivedAt":"2011-05-06T19:02:50Z","isPatch":false,"sender":{"key":"bill.kibler@hp.com","avatar":null},"body":"I spent some time thinking about this problem and realized that guessing the\npipe size was the issue probably was wrong, when it is likely something more \ngeneric in nature. I did some more testing, and remembered a similar problem\nwhen trying to do a git clone of the arm kernel for my $99 netbook article\n(see www.kiblerelectronics./corner/ccii.shtml), where it failed much\nthe same as on HP-UX. My thinking is that this is a variable that is controlled\nby the OS/Libc settings and not something that can be selected as a\ngood enough value. My real problem and reason for putting in a bug report,\nis wanting to know how this value was intended to be used.\n\nI just ran several tests on hp-UX using various values for the\n\"LARGE_PACKET_MAX\", all trying to see what the actual failure value is.\nI thought it might be related to ssize_t, which in HP-UX is 32K, but\nvalues around 32K all worked fine. 65000 failed, while 48000 worked.\nI went through the normal trial and error process and found the value\nto vary over a number of tests - in short - not fixed. That suggested\nit is partly controlled by some amount of free space(?).\n\nI will test out my arm netbook and see if changing this value helps git\non my system, but for now I feel this value was selected somewhat\narbitrarily when it should be selected as the smallest value that can\nwork for your OS of choice. I thought of using 32K on HP-UX, but decided\nthat if I don't really know what the mechanism is controlling this value\nthen the smallest working value is the safest to use (8208).\n\nCan anyone in this group explain what is going on for me? Why was 64K\nselected in the first place?\nThanks.\nBill. \n\n-----Original Message-----\nFrom: Junio C Hamano [mailto:gitster@pobox.com] \nSent: Thursday, May 05, 2011 12:07 PM\nTo: Kibler, Bill\nCc: git@vger.kernel.org; Richard Lloyd\nSubject: Re: git failure on HP-UX\n\n\"Kibler, Bill\" <bill.kibler@hp.com> writes:\n\n> In looking at the code, \"sideband.h\" defines \"LARGE_PACKET_MAX 65520\"\n> and is related to the passed flag \"side-band-64k\" as discussed in git\n> document pack-protocol.txt. The current default usage seems to be 64K\n> transfers, yet if we check the \"include/limits.h\" of HP-UX we see a\n> \"PIPE_BUF\" set to 8192. Along with the tusc indication of 8K pipe size,\n> I suspect that HP-UX is coring due to git trying to use a 64K pipe when\n> 8K is max.\n>\n> I solved the probem for now, by changing the file sideband.h to use\n> \"LARGE_PACKET_MAX 8208\".\n\nThis does not make any sense.  We may make write(2) and read(2) system\ncalls with 64k (or maybe bit more) chunk, but that does not mean the\nimplementation of these system calls must take that as a whole.  Your\nwrite(2) is allowed to write only whatever fits your pipe buffer, and tell\nthe caller \"I wrote only 8192 bytes\", and the code is supposed to loop,\nadvancing the write pointer by 8k and calling write(2) again, until you\nwrite everything to whoever is reading the other end of the pipe.  The\nsame thing for the read(2).\n\nIf you can find a place where we make write(2)/read(2) and blindly assumes\nthat a non-negative return means everything was written/read successfully,\nthen you have found a bug.\n\nIf the symptom _were_ a deadlock where the writer of one pipe expected to\nbe able to send 64k to the other end of the pipe and then hear back from\nthe other side with a separate read, I would understand that could happen\n(actually we know a local pipe transfer without ssh has that kind of\npotential deadlock but I think the size we assume that can fit in the pipe\nbuffer is far smaller than 8k).  But I do not understand where a SIGBUS\ncan come from.\n"},{"id":"167278","messageId":"7vliyjzj0n.fsf@alter.siamese.dyndns.org","threadId":"27274","inReplyTo":"88C5107F36BD7E4BBFDFDB140E3D81077FA78CF67E@GVW1362EXC.americas.hpqcorp.net","subject":"Re: git failure on HP-UX - more data","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-06T20:11:36Z","receivedAt":"2011-05-06T20:11:36Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Kibler, Bill\" <bill.kibler@hp.com> writes:\n\n> I just ran several tests on hp-UX using various values for the\n> \"LARGE_PACKET_MAX\", ...\n\nThe original sideband protocol had a fixed receiver side buffer that was\nonly 1000 bytes long.  When we updated the protocol so that we can carry\nmore payload with a single logical pkt-line format, which has the maximum\npacket length of a bit less than 64k (it has a fixed 4 hexadecimal digits\nfield at the front that indicates its size, so the maximum payload size is\n64k minus 4 or something like that), we added a protocol extension that is\nnegotiated between the server and the client for both sides to make sure\nthat they have the updated implementation in which the receiver is\nprepared to accept a 64k packet, not just a small 1000-byte static buffer.\n\nBut all of that is at the logical protocol level.\n\nEven if the transfer goes over the Ethernet, this size is in no way\nlimited by its MTU of 1500 bytes, because the kernel will take care of\nbuffering and reassembling for us.\n\nIt is the same deal for the pkt-line protocol, where we issue a write(2)\nand expect that the system may write less than what we asked it to write,\nand return us how many bytes it has actually written. As long as write(2)\ncorrectly returns the number of bytes it wrote, and our code that calls\nwrite(2) correctly expects a short-write and loops until writing\neverything out, there is no need to worry about LARGE_PACKET_MAX.\n\nAt least, that is the theory.\n\nI think already said this in my previous message to you, but it is\npossible that we have a bug in our code that fails to expect write(2) to\nresult in short-write and loop until we write everything out.  My gut\nfeeling is that it is slightly more plausible that we have such a bug than\nthat your libc has a buggy implementation of write(2) that returns a bogus\nvalue (say 64k) when in fact it wrote only what would fit in your pipe\nbuffer (you said 8k, I think) when asked to write 64k.\n\nAnd the right thing to do is to find and fix such a bug. I am afraid you\nare wasting your time by futzing LARGE_PACKET_MAX. Even if you find a good\nsmall value that happens to work on _your_ machine, it would not be a real\nfix for the problem.\n"},{"id":"167287","messageId":"88C5107F36BD7E4BBFDFDB140E3D81077FA78CF78C@GVW1362EXC.americas.hpqcorp.net","threadId":"27274","inReplyTo":"7vliyjzj0n.fsf@alter.siamese.dyndns.org","subject":"RE: git failure on HP-UX - more data","fromName":"Kibler, Bill","fromEmail":"bill.kibler@hp.com","sentAt":"2011-05-06T21:34:16Z","receivedAt":"2011-05-06T21:34:16Z","isPatch":false,"sender":{"key":"bill.kibler@hp.com","avatar":null},"body":"Thanks for the response, it was very informative. I understand now\nthat the \"side-band\" values really are more of \"what version\" of\nthe protocol can you run, than an actual \"use this size\" handshake.\n\nI think it might help to say again, that on HP-UX I ran several tests\nof cloning using other processes, and they always worked correctly.\nI even did some tusc/traces which showed multiple pipe transfers of\n8K much as you would expected for your analysis of what is suppose\nto happen. It is only when doing this under the ssh pipe process\nthat the problem occurs.\n\nI did look, but not closely at the code, to see what might be handled\ndifferently between the ssh and say a http transfer as it related to\npack or sideband routines. I feel if we can just determine what really \nis different using ssh we would be pretty close to the problem.\n\nI am making it clear in my notes, that this fix is hopefully only a\ntemporary fix for what is still really an unknown problem.\n\n\n-----Original Message-----\nFrom: Junio C Hamano [mailto:gitster@pobox.com] \nSent: Friday, May 06, 2011 1:12 PM\nTo: Kibler, Bill\nCc: git@vger.kernel.org; Richard Lloyd; kibler@psyber.com\nSubject: Re: git failure on HP-UX - more data\n\n\"Kibler, Bill\" <bill.kibler@hp.com> writes:\n\n> I just ran several tests on hp-UX using various values for the\n> \"LARGE_PACKET_MAX\", ...\n\nThe original sideband protocol had a fixed receiver side buffer that was\nonly 1000 bytes long.  When we updated the protocol so that we can carry\nmore payload with a single logical pkt-line format, which has the maximum\npacket length of a bit less than 64k (it has a fixed 4 hexadecimal digits\nfield at the front that indicates its size, so the maximum payload size is\n64k minus 4 or something like that), we added a protocol extension that is\nnegotiated between the server and the client for both sides to make sure\nthat they have the updated implementation in which the receiver is\nprepared to accept a 64k packet, not just a small 1000-byte static buffer.\n\nBut all of that is at the logical protocol level.\n\nEven if the transfer goes over the Ethernet, this size is in no way\nlimited by its MTU of 1500 bytes, because the kernel will take care of\nbuffering and reassembling for us.\n\nIt is the same deal for the pkt-line protocol, where we issue a write(2)\nand expect that the system may write less than what we asked it to write,\nand return us how many bytes it has actually written. As long as write(2)\ncorrectly returns the number of bytes it wrote, and our code that calls\nwrite(2) correctly expects a short-write and loops until writing\neverything out, there is no need to worry about LARGE_PACKET_MAX.\n\nAt least, that is the theory.\n\nI think already said this in my previous message to you, but it is\npossible that we have a bug in our code that fails to expect write(2) to\nresult in short-write and loop until we write everything out.  My gut\nfeeling is that it is slightly more plausible that we have such a bug than\nthat your libc has a buggy implementation of write(2) that returns a bogus\nvalue (say 64k) when in fact it wrote only what would fit in your pipe\nbuffer (you said 8k, I think) when asked to write 64k.\n\nAnd the right thing to do is to find and fix such a bug. I am afraid you\nare wasting your time by futzing LARGE_PACKET_MAX. Even if you find a good\nsmall value that happens to work on _your_ machine, it would not be a real\nfix for the problem.\n"}]}