{"thread":{"id":"10780","subject":"git packs","startedAt":"2007-11-10T04:47:40Z","lastAt":"2007-11-12T14:15:56Z","messageCount":19,"participants":["bob","Nicolas Pitre","Luke Lu","Linus Torvalds","David Brown","Derek Fawcus","Martin Langhoff"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"59166","messageId":"F6DD8DCD-416B-4DDF-B384-7213C9ED5565@mac.com","threadId":"10780","inReplyTo":null,"subject":"git packs","fromName":"bob","fromEmail":"kranki@mac.com","sentAt":"2007-11-10T04:47:40Z","receivedAt":"2007-11-10T04:47:40Z","isPatch":false,"sender":{"key":"kranki@mac.com","avatar":null},"body":"When a repository is packed such as for a clone or fetch, is there  \njust one pack file created that is used for the transfer?\n"},{"id":"59167","messageId":"alpine.LFD.0.9999.0711100011150.21255@xanadu.home","threadId":"10780","inReplyTo":"F6DD8DCD-416B-4DDF-B384-7213C9ED5565@mac.com","subject":"Re: git packs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-11-10T05:13:36Z","receivedAt":"2007-11-10T05:13:36Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 9 Nov 2007, bob wrote:\n\n> When a repository is packed such as for a clone or fetch, is there just one\n> pack file created that is used for the transfer?\n\nYes.\n\nAnd modern Git is able to handle packs larger than 4GB too, assuming it \nis compiled using a toolchain with large file support.\n\n\nNicolas\n"},{"id":"59169","messageId":"FC175E4F-D9BE-42CC-B0BB-561B2EDCD941@mac.com","threadId":"10780","inReplyTo":"alpine.LFD.0.9999.0711100011150.21255@xanadu.home","subject":"Re: git packs","fromName":"bob","fromEmail":"kranki@mac.com","sentAt":"2007-11-10T06:00:27Z","receivedAt":"2007-11-10T06:00:27Z","isPatch":false,"sender":{"key":"kranki@mac.com","avatar":null},"body":"When you say toolchain, are you referring to the compiler\nand associated libraries or are you referring to OS programs\nsuch as ls, md5, cat, etc or both?\n\nThe reason that I ask is that I have been playing different\nscenarios using git 1.5.3.5 under MacOSX 10.4.10 mostly\nall day and every time that\n\nA) \ta file approaches or exceeds 2gig on an 'add', it\n\tresults in:\n\t\n\tfatal: Out of memory? mmap failed: Cannot allocate memory\n\n\n\nB) \tthe repository size less the .git subdirectory approaches\n\t4gig on a 'fetch' it results in:\n\n\tResolving 3356 deltas...\n\tfatal: serious inflate inconsistency: -3 (unknown compression method)\n\tfatal: index-pack died with error code 128\n\tfatal: Fetch failure: ../rmwHtmlOld\n\n\tUnder B, building the initial repository works fine.\n\n(I added a patch the Linus Torvalds gave out when a previous inflate  \nproblem\nwas being researched.)  Also, I have been looking in the source\nin particular in builtin-add.c builtin-pack-objects.c and associated  \nheaders\nand see int and unsigned long being used a lot, but not any unsigned  \nlong\nlongs.  I have been testing on my laptop which has a 32-bit Intel  \nCore Duo.\nAlso, I have run the same tests on a dual quad-core Intel processor\nwhich is 64 bit, (but not sure that Apple uses the 64 bits in  \n10.4.10).  I\nget the same results as above.\n\nThe zlib is at the latest revision of 1.2.3 and gcc is at 4.0.1\nwhich from what I can tell supports large files, because 'off_t' is 8  \nbytes\nwhich is the size used for a 'stat' file size.\n\nI am just wondering if these size limitations exist for MacOSX\nor maybe I am doing something wrong (which is probably\nthe case).\n\n\nOn Nov 10, 2007, at 12:13 AM, Nicolas Pitre wrote:\n\n> On Fri, 9 Nov 2007, bob wrote:\n>\n>> When a repository is packed such as for a clone or fetch, is there  \n>> just one\n>> pack file created that is used for the transfer?\n>\n> Yes.\n>\n> And modern Git is able to handle packs larger than 4GB too,  \n> assuming it\n> is compiled using a toolchain with large file support.\n>\n>\n> Nicolas\n> -\n"},{"id":"59174","messageId":"DF65F7E4-448A-4726-8B42-642776155A8F@vicaya.com","threadId":"10780","inReplyTo":"FC175E4F-D9BE-42CC-B0BB-561B2EDCD941@mac.com","subject":"Re: git packs","fromName":"Luke Lu","fromEmail":"git@vicaya.com","sentAt":"2007-11-10T06:36:03Z","receivedAt":"2007-11-10T06:36:03Z","isPatch":false,"sender":{"key":"git@vicaya.com","avatar":null},"body":"On Nov 9, 2007, at 10:00 PM, bob wrote:\n> When you say toolchain, are you referring to the compiler\n> and associated libraries or are you referring to OS programs\n> such as ls, md5, cat, etc or both?\n>\n> The reason that I ask is that I have been playing different\n> scenarios using git 1.5.3.5 under MacOSX 10.4.10 mostly\n> all day and every time that\n>\n> A) \ta file approaches or exceeds 2gig on an 'add', it\n> \tresults in:\n> \t\n> \tfatal: Out of memory? mmap failed: Cannot allocate memory\n>\n>\n>\n> B) \tthe repository size less the .git subdirectory approaches\n> \t4gig on a 'fetch' it results in:\n>\n> \tResolving 3356 deltas...\n> \tfatal: serious inflate inconsistency: -3 (unknown compression method)\n> \tfatal: index-pack died with error code 128\n> \tfatal: Fetch failure: ../rmwHtmlOld\n>\n> \tUnder B, building the initial repository works fine.\n>\n> (I added a patch the Linus Torvalds gave out when a previous  \n> inflate problem\n> was being researched.)  Also, I have been looking in the source\n> in particular in builtin-add.c builtin-pack-objects.c and  \n> associated headers\n> and see int and unsigned long being used a lot, but not any  \n> unsigned long\n> longs.  I have been testing on my laptop which has a 32-bit Intel  \n> Core Duo.\n> Also, I have run the same tests on a dual quad-core Intel processor\n> which is 64 bit, (but not sure that Apple uses the 64 bits in  \n> 10.4.10).  I\n> get the same results as above.\n>\n> The zlib is at the latest revision of 1.2.3 and gcc is at 4.0.1\n> which from what I can tell supports large files, because 'off_t' is  \n> 8 bytes\n> which is the size used for a 'stat' file size.\n\nmmap(2), which git uses by default, is subject to vm limits  \n(typically <2GB), regardless of large file support. file `which git`  \nwill probably tell you that it's a Mach-O executable i386 instead of  \nx86_64. In order to get 64 bit binaries on Mactel boxes, you'll need  \nthe -m64 flag for gcc. I suspect that compiling with NO_MMAP option  \nwork as well.\n\n__Luke\n"},{"id":"59175","messageId":"alpine.LFD.0.999.0711092211250.15101@woody.linux-foundation.org","threadId":"10780","inReplyTo":"FC175E4F-D9BE-42CC-B0BB-561B2EDCD941@mac.com","subject":"Re: git packs","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-11-10T06:38:05Z","receivedAt":"2007-11-10T06:38:05Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 10 Nov 2007, bob wrote:\n> \n> The reason that I ask is that I have been playing different\n> scenarios using git 1.5.3.5 under MacOSX 10.4.10 mostly\n> all day and every time that\n> \n> A) \ta file approaches or exceeds 2gig on an 'add', it\n> \tresults in:\n> \t\tfatal: Out of memory? mmap failed: Cannot allocate memory\n\nGit wants to handle single files as one single entity, so single big files \nreally do end up being very painful. The costs of compressing them and \ngenerating deltas would probably get prohibitively high *anyway*, but it \ndoes mean that if you have gigabyte files, you do want a 64-bit VM.\n\nI thought OS X could do 64 bits these days. Maybe not.\n\nAnyway, that explains the \"cannot allocate memory\". Git simply wants to \nmmap the whole file. You don't have enough VM space for it.\n\n(And if you seriously want to work with multi-gigabyte files, git probbaly \nisn't going to perform wonderfully well, even if it *should* work fine if \nyou just have a full 64-bit environment that allows the mmap).\n\n> B) \tthe repository size less the .git subdirectory approaches\n> \t4gig on a 'fetch' it results in:\n> \n> \tResolving 3356 deltas...\n> \tfatal: serious inflate inconsistency: -3 (unknown compression method)\n\nThat sounds really broken. I'm not seeing what would cause that, apart \nfrom some really bad data corruption and/or broken zlib implementation. \nBut if the pack-file really is 2GB+ in size, I could imagine some sign \nissues cropping up.\n\ngit will generally use \"unsigned long\" (which is probably just 32-bit on \nyour setup), but since git in those circumstances would be limited by the \nsize of the VM _anyway_, that's not really much of a limitation (although \nprobably broken on the crazy Windows \"LLP64\" model). But maybe we have \nsome place where we use a signed thing, or zlib does, and I could see that \ncausing breakage.\n\nBut that code-sequence really should never even come *close* to the 31-bit \nlimit, as long as the individual objects themselves aren't bigger than the \navailable VM space (and git currently assumes \"unsigned long\" is \nsufficiently big to cover the VM space, which is not technically correct, \nbut should be fine on OS X too).\n\nThat said, we should use \"off_t\" in that function. I suspect we have a \nnumber of people (read: me) who have grown too used to living in a 64-bit \nworld..\n\n> I have been testing on my laptop which has a 32-bit Intel Core Duo.\n\nOk, so you're 32-bit limited even if there is were to be some 64-bit \nsupport for OS X.\n\n> Also, I have run the same tests on a dual quad-core Intel processor\n> which is 64 bit, (but not sure that Apple uses the 64 bits in 10.4.10).  I\n> get the same results as above.\n\nI'm pretty sure OS X defaults to a 32-bit environment, but has at least \n*some* 64-bit support. It would definitely need to be enabled explicitly \n(since they made the *insane* decision to move over to Intel laptop chips \nsix months before they got 64-bit support! Somebody at Apple is a total \nidiot, and should get fired).\n\nSo it would be interesting to hear if a 64-bit build would make a \ndifference.\n\n> The zlib is at the latest revision of 1.2.3 and gcc is at 4.0.1\n> which from what I can tell supports large files, because 'off_t' is 8 bytes\n> which is the size used for a 'stat' file size.\n\nSee above: single files are size-limited, but with a large off_t like \nyours, you should be fine. Except we may have screwed up.\n\n> I am just wondering if these size limitations exist for MacOSX\n> or maybe I am doing something wrong (which is probably\n> the case).\n\nWe *have* had issues with broken implementations of \"pread()\" on some \nsystems.  \n\nYou could try setting NO_PREAD in the Makefile and compiling with the \ncompatibility function.. That's the only thing that comes to mind as being \nworth trying in that area.\n\nAnd if you have some script to generate the repository (ie you aren't \nusing \"live data\", but are testing the limits of the system), if you can \nmake that available, so that people with non-OSX environments can test, \nthat would be interesting..\n\nI certainly have some 32-bit environments too (old linux boxes), but I'm \ntoo lazy to write a test-case, so I was hoping you'd be using some simple \nscripts that I could just test and see if I can see the behaviour you \ndescribe myself.\n\nThat said, I have worked with a 3GB pack-file (one of the KDE trial \nrepos). That worked fine. But git does tend to want a *lot* of memory for \nreally big repositories, so I suspect that if you actually work with 2GB+ \npack-files, you'll be wanting a 64-bit environment just because you'll be \nwanting more than 2GB of physical RAM in order to be able to access it \nefficiently.\n\n\t\t\tLinus\n"},{"id":"59176","messageId":"alpine.LFD.0.999.0711092244080.15101@woody.linux-foundation.org","threadId":"10780","inReplyTo":"alpine.LFD.0.999.0711092211250.15101@woody.linux-foundation.org","subject":"Re: git packs","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-11-10T06:53:24Z","receivedAt":"2007-11-10T06:53:24Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 9 Nov 2007, Linus Torvalds wrote:\n> \n> That said, I have worked with a 3GB pack-file (one of the KDE trial \n> repos). That worked fine. But git does tend to want a *lot* of memory for \n> really big repositories, so I suspect that if you actually work with 2GB+ \n> pack-files, you'll be wanting a 64-bit environment just because you'll be \n> wanting more than 2GB of physical RAM in order to be able to access it \n> efficiently.\n\nJust double-checked. Yes, sirree. You definitely want 4GB+ if you are \ncloning a 3GB git pack-file. The \"git-pack-objects\" phase not only is \ngoing to walk all over the pack-file, it's going to add its own memory \nfootprint on top of that just keeping track of all the objects.\n\nSo I doubt 2GB+ pack-files are all that practical on 32-bit hosts. At \nleast not with the kind of performance behaviour *I* would accept.\n\n(Of course, since git packs things pretty damn well, it would need to be a \nreally really big project to be a 2GB+ pack-file, or just contain a lot of \ngenerally large non-deltable binary data file - one scenario where git \ndefinitely doesn't work wonderfully well, although I doubt many other \nSCM's do either..)\n\n\t\t\tLinus\n"},{"id":"59177","messageId":"alpine.LFD.0.999.0711092254130.15101@woody.linux-foundation.org","threadId":"10780","inReplyTo":"DF65F7E4-448A-4726-8B42-642776155A8F@vicaya.com","subject":"Re: git packs","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-11-10T06:58:16Z","receivedAt":"2007-11-10T06:58:16Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 9 Nov 2007, Luke Lu wrote:\n> \n> mmap(2), which git uses by default, is subject to vm limits (typically <2GB),\n> regardless of large file support. file `which git` will probably tell you that\n> it's a Mach-O executable i386 instead of x86_64. In order to get 64 bit\n> binaries on Mactel boxes, you'll need the -m64 flag for gcc. I suspect that\n> compiling with NO_MMAP option work as well.\n\nEven with NO_MMAP, git will still want to read in source files in their \nentirety (just with regular reads). So you'll still be VM size-limited: \nthe mmap() will just be replaced with a malloc+read in order to avoid \nsome broken windows mmap() behaviour.\n\nBut hearing whether -m64 makes a difference would be interesting. I'm \nhoping OS X is LP64, not some insane half-way thing like Vista is.\n\n\t\tLinus\n"},{"id":"59178","messageId":"B20E1D71-BCDB-4189-952F-3B809A342870@mac.com","threadId":"10780","inReplyTo":"alpine.LFD.0.999.0711092211250.15101@woody.linux-foundation.org","subject":"Re: git packs","fromName":"bob","fromEmail":"kranki@mac.com","sentAt":"2007-11-10T07:19:13Z","receivedAt":"2007-11-10T07:19:13Z","isPatch":false,"sender":{"key":"kranki@mac.com","avatar":null},"body":"I will try a few things and see if I can get a script put together\nthat generates the inflate problem.  The data that I am\nusing is a backup of my original repository.  So, I can\nplay all that I want.  But it would be a lot easier if I\ncould just generate some files using dd or something.\n\nI'll also try the 64-bit compile on my Mac Pro and see\nif it works.\n\nMy only reason for keeping this directory under git is\nthat I find git so easy to work with across multiple\nmachines.  I own 10+ computers and I use git to\nprovide easy transfers/updates from machine to\nmachine normally without any issues and a great\ndeal of reliability.\n\nThis directory is a website that I use\ninternally to keep track of things important to me.\nFor instance, the one large file is a movie of the\ninside of my house before they sheet-rocked it\nso that later I would have an easier time finding\nthings in the walls.  There is some html and php\nthat I wrote in it which I did want versioned.\n\nMaybe I should just drop back to using two\ndirectories, one of the large files which are\nstatic anyway and a git repo for the html and php.\nI was just trying to keep everything in the\nrepo for simplicity.\n\nNo matter which direction that I decide, I will still\ntry to provide the script.  Thank you both (Luke Lu)\nfor the feedback.\n\n\nOn Nov 10, 2007, at 1:38 AM, Linus Torvalds wrote:\n\n>\n>\n> On Sat, 10 Nov 2007, bob wrote:\n>>\n>> The reason that I ask is that I have been playing different\n>> scenarios using git 1.5.3.5 under MacOSX 10.4.10 mostly\n>> all day and every time that\n>>\n>> A) \ta file approaches or exceeds 2gig on an 'add', it\n>> \tresults in:\n>> \t\tfatal: Out of memory? mmap failed: Cannot allocate memory\n>\n> Git wants to handle single files as one single entity, so single  \n> big files\n> really do end up being very painful. The costs of compressing them and\n> generating deltas would probably get prohibitively high *anyway*,  \n> but it\n> does mean that if you have gigabyte files, you do want a 64-bit VM.\n>\n> I thought OS X could do 64 bits these days. Maybe not.\n>\n> Anyway, that explains the \"cannot allocate memory\". Git simply  \n> wants to\n> mmap the whole file. You don't have enough VM space for it.\n>\n> (And if you seriously want to work with multi-gigabyte files, git  \n> probbaly\n> isn't going to perform wonderfully well, even if it *should* work  \n> fine if\n> you just have a full 64-bit environment that allows the mmap).\n>\n>> B) \tthe repository size less the .git subdirectory approaches\n>> \t4gig on a 'fetch' it results in:\n>>\n>> \tResolving 3356 deltas...\n>> \tfatal: serious inflate inconsistency: -3 (unknown compression  \n>> method)\n>\n> That sounds really broken. I'm not seeing what would cause that, apart\n> from some really bad data corruption and/or broken zlib  \n> implementation.\n> But if the pack-file really is 2GB+ in size, I could imagine some sign\n> issues cropping up.\n>\n> git will generally use \"unsigned long\" (which is probably just 32- \n> bit on\n> your setup), but since git in those circumstances would be limited  \n> by the\n> size of the VM _anyway_, that's not really much of a limitation  \n> (although\n> probably broken on the crazy Windows \"LLP64\" model). But maybe we have\n> some place where we use a signed thing, or zlib does, and I could  \n> see that\n> causing breakage.\n>\n> But that code-sequence really should never even come *close* to the  \n> 31-bit\n> limit, as long as the individual objects themselves aren't bigger  \n> than the\n> available VM space (and git currently assumes \"unsigned long\" is\n> sufficiently big to cover the VM space, which is not technically  \n> correct,\n> but should be fine on OS X too).\n>\n> That said, we should use \"off_t\" in that function. I suspect we have a\n> number of people (read: me) who have grown too used to living in a  \n> 64-bit\n> world..\n>\n>> I have been testing on my laptop which has a 32-bit Intel Core Duo.\n>\n> Ok, so you're 32-bit limited even if there is were to be some 64-bit\n> support for OS X.\n>\n>> Also, I have run the same tests on a dual quad-core Intel processor\n>> which is 64 bit, (but not sure that Apple uses the 64 bits in  \n>> 10.4.10).  I\n>> get the same results as above.\n>\n> I'm pretty sure OS X defaults to a 32-bit environment, but has at  \n> least\n> *some* 64-bit support. It would definitely need to be enabled  \n> explicitly\n> (since they made the *insane* decision to move over to Intel laptop  \n> chips\n> six months before they got 64-bit support! Somebody at Apple is a  \n> total\n> idiot, and should get fired).\n>\n> So it would be interesting to hear if a 64-bit build would make a\n> difference.\n>\n>> The zlib is at the latest revision of 1.2.3 and gcc is at 4.0.1\n>> which from what I can tell supports large files, because 'off_t'  \n>> is 8 bytes\n>> which is the size used for a 'stat' file size.\n>\n> See above: single files are size-limited, but with a large off_t like\n> yours, you should be fine. Except we may have screwed up.\n>\n>> I am just wondering if these size limitations exist for MacOSX\n>> or maybe I am doing something wrong (which is probably\n>> the case).\n>\n> We *have* had issues with broken implementations of \"pread()\" on some\n> systems.\n>\n> You could try setting NO_PREAD in the Makefile and compiling with the\n> compatibility function.. That's the only thing that comes to mind  \n> as being\n> worth trying in that area.\n>\n> And if you have some script to generate the repository (ie you aren't\n> using \"live data\", but are testing the limits of the system), if  \n> you can\n> make that available, so that people with non-OSX environments can  \n> test,\n> that would be interesting..\n>\n> I certainly have some 32-bit environments too (old linux boxes),  \n> but I'm\n> too lazy to write a test-case, so I was hoping you'd be using some  \n> simple\n> scripts that I could just test and see if I can see the behaviour you\n> describe myself.\n>\n> That said, I have worked with a 3GB pack-file (one of the KDE trial\n> repos). That worked fine. But git does tend to want a *lot* of  \n> memory for\n> really big repositories, so I suspect that if you actually work  \n> with 2GB+\n> pack-files, you'll be wanting a 64-bit environment just because  \n> you'll be\n> wanting more than 2GB of physical RAM in order to be able to access it\n> efficiently.\n>\n> \t\t\tLinus\n"},{"id":"59179","messageId":"20071110075948.GA29288@old.davidb.org","threadId":"10780","inReplyTo":"alpine.LFD.0.999.0711092254130.15101@woody.linux-foundation.org","subject":"Re: git packs","fromName":"David Brown","fromEmail":"git@davidb.org","sentAt":"2007-11-10T07:59:48Z","receivedAt":"2007-11-10T07:59:48Z","isPatch":false,"sender":{"key":"git@davidb.org","avatar":"https://gravatar.com/avatar/94c86a2938470a74c2eac5e2b69afc0871f79a660295c02219597aba8cb101c1?d=mp&s=160"},"body":"On Fri, Nov 09, 2007 at 10:58:16PM -0800, Linus Torvalds wrote:\n\n>But hearing whether -m64 makes a difference would be interesting. I'm \n>hoping OS X is LP64, not some insane half-way thing like Vista is.\n\nSome casual tests with printf and sizeof makes it look like it is.  At\nleast sizeof (void *) and sizeof (long) are both 8.\n\nDavid\n"},{"id":"59224","messageId":"00593593-E943-4DA0-AA9B-FDBB866E7EFB@mac.com","threadId":"10780","inReplyTo":"F6DD8DCD-416B-4DDF-B384-7213C9ED5565@mac.com","subject":"Re: git packs","fromName":"bob","fromEmail":"kranki@mac.com","sentAt":"2007-11-10T17:40:16Z","receivedAt":"2007-11-10T17:40:16Z","isPatch":false,"sender":{"key":"kranki@mac.com","avatar":null},"body":"I compiled git under MacOSX 10.4.10 withr:\n\nA) -m64 -arch ppc64 on a dual G5\nB) -m64 -arch x86_64 on the dual quad-core\n\nIn both case, the link phase failed because there\nwas no 64-bit version of libz, libssl, libiconv and\nlibcrypto to link with.\n\nI then installed MacOSX 10.5 (Leopard) which\nwas just released last month on the dual\nquad-core machine with -m64 -arch x86_64.\ngit compiled and linked successfully.  However,\nit failed in the \"git add .\" which was the\nsecond command after \"git init\".  The message\nwas fairly cryptic, \"Bus error\".\n\nI am guessing that the \"Bus error\" is an Apple\nproblem and it did produce a crashreport.  So,\nI am going to submit it to Apple since it is easily\nreproducible.\n\nAnyway, those are the results.\n\nI am now going to split out the jpg(s), pdf(s)\nand (.mov) files from the repository  and just\nmanage them external to the git repository\nwhich will fix my git problem.\n\nI will report again when I get git running on\nLeopard.\n"},{"id":"59225","messageId":"20071110174559.GA2200@old.davidb.org","threadId":"10780","inReplyTo":"00593593-E943-4DA0-AA9B-FDBB866E7EFB@mac.com","subject":"Re: git packs","fromName":"David Brown","fromEmail":"git@davidb.org","sentAt":"2007-11-10T17:45:59Z","receivedAt":"2007-11-10T17:45:59Z","isPatch":false,"sender":{"key":"git@davidb.org","avatar":"https://gravatar.com/avatar/94c86a2938470a74c2eac5e2b69afc0871f79a660295c02219597aba8cb101c1?d=mp&s=160"},"body":"On Sat, Nov 10, 2007 at 12:40:16PM -0500, bob wrote:\n\n> I am guessing that the \"Bus error\" is an Apple\n> problem and it did produce a crashreport.  So,\n> I am going to submit it to Apple since it is easily\n> reproducible.\n\nThe crash report is probably not all that useful to Apple, since it occurs\nin a program you have built.  Perhaps you should send the stack trace from\nthe crash report here and people might have some ideas about what might be\nthe problem.\n\nDavid\n"},{"id":"59227","messageId":"134659C4-BA10-4B9E-9C64-2754A90D93F8@mac.com","threadId":"10780","inReplyTo":"20071110174559.GA2200@old.davidb.org","subject":"Re: git packs","fromName":"bob","fromEmail":"kranki@mac.com","sentAt":"2007-11-10T18:01:46Z","receivedAt":"2007-11-10T18:01:46Z","isPatch":false,"sender":{"key":"kranki@mac.com","avatar":null},"body":"It is fairly disappointing as far as indicating the problem.  Here is  \nthe entire report since it was so short.\n\n\n============ Begin =================\nProcess:         git [82703]\nPath:            git\nIdentifier:      git\nVersion:         ??? (???)\nCode Type:       X86-64 (Native)\nParent Process:  bash [261]\n\nDate/Time:       2007-11-10 11:23:33.976 -0500\nOS Version:      Mac OS X 10.5 (9A581)\nReport Version:  6\n\nException Type:  EXC_BAD_ACCESS (SIGBUS)\nException Codes: KERN_PROTECTION_FAILURE at 0x00007fff5fc00000\nCrashed Thread:  Unknown\n\nError Formulating Crash Report:\n*** -[NSCFDictionary setObject:forKey:]: attempt to insert nil value  \n(key: VMUSignaturePath)\n0x9501626b\n0x9586009b\n0x9501604b\n0x9501608a\n0x900d65e8\n0x00078308\n0x00087eb8\n0x0008800e\n0x000850a2\n0x00002d90\n0x000093be\n0x0000b57c\n0x0000b0c9\n0x919f8793\n0x0000a7d0\n0x919cc075\n0x919cbf32\n\nBacktrace not available\n\nUnknown thread crashed with X86 Thread State (64-bit):\n   rax: 0x000000000000003b  rbx: 0x00000001191c9338  rcx:  \n0x0000000000000392  rdx: 0x00000000aff4bc3b\n   rdi: 0x00007fff5fc00000  rsi: 0x0000000040000003  rbp:  \n0x00007fff5fbff180  rsp: 0x00007fff5fbff160\n    r8: 0x0000000000000000   r9: 0x058487f0858487f0  r10:  \n0x000000002cb27436  r11: 0x000000000ee2afe3\n   r12: 0x00000000181c84c0  r13: 0x00007fff5fbff1a0  r14:  \n0x000000010000000c  r15: 0x0000000101000000\n   rip: 0x00007fff80543ca5  rfl: 0x0000000000010202  cr2:  \n0x00007fff5fc00000\n\nBinary images description not available\n============ end =================\n\nMaybe there was another option to specifiy other than \"-m64 -arch  \nx86_64\".\n\n\nOn Nov 10, 2007, at 12:45 PM, David Brown wrote:\n\n> On Sat, Nov 10, 2007 at 12:40:16PM -0500, bob wrote:\n>\n>> I am guessing that the \"Bus error\" is an Apple\n>> problem and it did produce a crashreport.  So,\n>> I am going to submit it to Apple since it is easily\n>> reproducible.\n>\n> The crash report is probably not all that useful to Apple, since it  \n> occurs\n> in a program you have built.  Perhaps you should send the stack  \n> trace from\n> the crash report here and people might have some ideas about what  \n> might be\n> the problem.\n>\n> David\n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"59261","messageId":"alpine.LFD.0.9999.0711102331270.21255@xanadu.home","threadId":"10780","inReplyTo":"B20E1D71-BCDB-4189-952F-3B809A342870@mac.com","subject":"Re: git packs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-11-11T04:35:04Z","receivedAt":"2007-11-11T04:35:04Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 10 Nov 2007, bob wrote:\n\n> I will try a few things and see if I can get a script put together\n> that generates the inflate problem.  The data that I am\n> using is a backup of my original repository.  So, I can\n> play all that I want.  But it would be a lot easier if I\n> could just generate some files using dd or something.\n\nPlease see the patch I just posted to the list.  That should fix your \nproblem.  I even included a small script to create a repository \nconfirming the problem and allowing to test the fix.\n\n\nNicolas\n"},{"id":"59310","messageId":"20071111110942.A4013@edi-view2.cisco.com","threadId":"10780","inReplyTo":"134659C4-BA10-4B9E-9C64-2754A90D93F8@mac.com","subject":"Re: git packs","fromName":"Derek Fawcus","fromEmail":"dfawcus@cisco.com","sentAt":"2007-11-11T11:09:42Z","receivedAt":"2007-11-11T11:09:42Z","isPatch":false,"sender":{"key":"dfawcus@cisco.com","avatar":null},"body":"On Sat, Nov 10, 2007 at 01:01:46PM -0500, bob wrote:\n> It is fairly disappointing as far as indicating the problem.  Here is  \n> the entire report since it was so short.\n\n> Error Formulating Crash Report:\n> *** -[NSCFDictionary setObject:forKey:]: attempt to insert nil value  \n> (key: VMUSignaturePath)\n\nhuh?  The above looks like an ObjC invocation.  Last I checked,  git was C.\nSo why is that in the frame?\n\nDF\n"},{"id":"59316","messageId":"48B406B0-FA9D-4173-B2DF-A01B193E5338@mac.com","threadId":"10780","inReplyTo":"20071111110942.A4013@edi-view2.cisco.com","subject":"Re: git packs","fromName":"bob","fromEmail":"kranki@mac.com","sentAt":"2007-11-11T12:54:01Z","receivedAt":"2007-11-11T12:54:01Z","isPatch":false,"sender":{"key":"kranki@mac.com","avatar":null},"body":"Well, there are actually two problems here.  The first is that\ncrashreporter crashed trying to create the crash report of\nthe git failure.  The second was the error in git itself.\nHope that helps.\n\nI am submitting an Apple bugreport on the first and\nNicolas Pitre is probably correct in his fix.  When I\nwas looking at index-pack.c everything was using 32\nbit unsigned and signed number where off_t was\n64-bit which is what a stat() would return.  My problem\nis that I was not familiar enough with git internals to\nfigure out the solution.  My review was the first time\nthat I ever looked at git source.\n\nThanks, Nicolas\n\nOn Nov 11, 2007, at 6:09 AM, Derek Fawcus wrote:\n\n> On Sat, Nov 10, 2007 at 01:01:46PM -0500, bob wrote:\n>> It is fairly disappointing as far as indicating the problem.  Here is\n>> the entire report since it was so short.\n>\n>> Error Formulating Crash Report:\n>> *** -[NSCFDictionary setObject:forKey:]: attempt to insert nil value\n>> (key: VMUSignaturePath)\n>\n> huh?  The above looks like an ObjC invocation.  Last I checked,   \n> git was C.\n> So why is that in the frame?\n>\n> DF\n"},{"id":"59419","messageId":"B298202C-3D54-498D-A348-0338914FBA46@mac.com","threadId":"10780","inReplyTo":"alpine.LFD.0.9999.0711102331270.21255@xanadu.home","subject":"Re: git packs","fromName":"bob","fromEmail":"kranki@mac.com","sentAt":"2007-11-12T02:53:36Z","receivedAt":"2007-11-12T02:53:36Z","isPatch":false,"sender":{"key":"kranki@mac.com","avatar":null},"body":"I applied the patch and these commands:\n\ncd rmwHtmlOld\nrm -fr .git\ngit init\ngit config core.compression 0\ngit add .\n\nI then got the same error as before, \"Bus error\".  Rats!\n\nThen I modified your script since I do not have seq or\nyour test-genrandom.  I substituted:\n\ndd count=XX  if=/dev/random of=file_$i\n\nwhere XX is adjusted to meet dd's requirements.  Also,\nI found after searching for a while, that the following\nworks just like your seq command:\n\nxyzzy=\"1 2 3 4\"\nfor i in $xyzzy\ndo\n...\ndone\n\nYour script then ran flawlessly.\n\nI looked through index-pack.c some more, but it is\nvery hard to figure it out without doing a lot of research\nsince there doesn't seem to be anything that describes\nthe layout of a pack.  The link towards the end of the user's\nmanual doesn't work for me.\n\nThe difference between your test and my data is that\ninstead of having a few large files, I have 11,500 files\nof varying sizes.  On average though, the file size is\nabout 370k.\n\nHTH\n\nOn Nov 10, 2007, at 11:35 PM, Nicolas Pitre wrote:\n\n> On Sat, 10 Nov 2007, bob wrote:\n>\n>> I will try a few things and see if I can get a script put together\n>> that generates the inflate problem.  The data that I am\n>> using is a backup of my original repository.  So, I can\n>> play all that I want.  But it would be a lot easier if I\n>> could just generate some files using dd or something.\n>\n> Please see the patch I just posted to the list.  That should fix your\n> problem.  I even included a small script to create a repository\n> confirming the problem and allowing to test the fix.\n>\n>\n> Nicolas\n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"59421","messageId":"alpine.LFD.0.9999.0711112307070.21255@xanadu.home","threadId":"10780","inReplyTo":"B298202C-3D54-498D-A348-0338914FBA46@mac.com","subject":"Re: git packs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-11-12T04:21:58Z","receivedAt":"2007-11-12T04:21:58Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 11 Nov 2007, bob wrote:\n\n> I applied the patch and these commands:\n> \n> cd rmwHtmlOld\n> rm -fr .git\n> git init\n> git config core.compression 0\n> git add .\n\nNote that I did \"git config core.compression 0\" simply to disable \nzlib compression altogether when creating the test repo just so it gets \ncreated faster.  even then, auto-generating and cloning a 8GB test \nrepository isn't particularly quick.\n\n> I then got the same error as before, \"Bus error\".  Rats!\n\nDo you get that with a 32-bit or 64-bit build of Git?\n\n> Then I modified your script since I do not have seq or\n> your test-genrandom.\n\ntest-genrandom is built with Git.  It is just not installed anywhere.\n\n> I substituted:\n> \n> dd count=XX  if=/dev/random of=file_$i\n> \n> where XX is adjusted to meet dd's requirements.  Also,\n\nAgain I used test-genrandom instead of /dev/random or /dev/urandom \nsimply because the former is much faster.\n\n> I found after searching for a while, that the following\n> works just like your seq command:\n> \n> xyzzy=\"1 2 3 4\"\n> for i in $xyzzy\n> do\n> ...\n> done\n> \n> Your script then ran flawlessly.\n\nHowever 'seq -w 1 2 63' should be replaced  with \"01 03 05 07 09 11 13 \n15\" and so on up to 63, and 'seq -w 2 2 64' is \"02 04 06 08 10 12 16\" \nand so on.\n\n> I looked through index-pack.c some more, but it is\n> very hard to figure it out without doing a lot of research\n> since there doesn't seem to be anything that describes\n> the layout of a pack.  The link towards the end of the user's\n> manual doesn't work for me.\n\nLook at Documentation/technical/pack-format.txt in the Git source tree.\n\n> The difference between your test and my data is that\n> instead of having a few large files, I have 11,500 files\n> of varying sizes.  On average though, the file size is\n> about 370k.\n\nAre you saying that the test repo with big files works for you but not \nyour own data set?\n\nWould you please recap what your problem is?\n\nWith my one line patch you should not get the \"serious inflate \ninconsistency\" error anymore.  The bus error must be another issue.\n\n\nNicolas\n"},{"id":"59424","messageId":"46a038f90711112046y300dcd8amd37445895863557@mail.gmail.com","threadId":"10780","inReplyTo":"00593593-E943-4DA0-AA9B-FDBB866E7EFB@mac.com","subject":"Re: git packs","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2007-11-12T04:46:28Z","receivedAt":"2007-11-12T04:46:28Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Nov 11, 2007 6:40 AM, bob <kranki@mac.com> wrote:\n> I am now going to split out the jpg(s), pdf(s)\n> and (.mov) files from the repository  and just\n> manage them external to the git repository\n> which will fix my git problem.\n\nFor version-less synchronization of large files I'd suggest unison (as\na smarter rsync, it's excellent) or the newfangled tra.\n\nhth,\n\n\nmartin\n"},{"id":"59488","messageId":"904628F9-EE34-4C4B-A114-82159AFF0260@mac.com","threadId":"10780","inReplyTo":"alpine.LFD.0.9999.0711112307070.21255@xanadu.home","subject":"Re: git packs","fromName":"bob","fromEmail":"kranki@mac.com","sentAt":"2007-11-12T14:15:56Z","receivedAt":"2007-11-12T14:15:56Z","isPatch":false,"sender":{"key":"kranki@mac.com","avatar":null},"body":"\nOn Nov 11, 2007, at 11:21 PM, Nicolas Pitre wrote:\n\n> On Sun, 11 Nov 2007, bob wrote:\n>\n>> I applied the patch and these commands:\n>>\n>> cd rmwHtmlOld\n>> rm -fr .git\n>> git init\n>> git config core.compression 0\n>> git add .\n>\n> Note that I did \"git config core.compression 0\" simply to disable\n> zlib compression altogether when creating the test repo just so it  \n> gets\n> created faster.  even then, auto-generating and cloning a 8GB test\n> repository isn't particularly quick.\n\tI wasn't sure why you used that, but figured that I should put it\n\tin, because you did.  And you are very correct that these tests\n\tcan take a very long time.\n>\n>> I then got the same error as before, \"Bus error\".  Rats!\n>\n> Do you get that with a 32-bit or 64-bit build of Git?\n\tOn that machine, I changed the config.mak to generate nothing\n\tbut a 64-bit build.  And activity monitor shows it as 64-bit\n\twhen I am running it. Also, off_t is 64 bit on regular\n\tMacOSX and as a 64-bit program supported by the\n\tlatest MacOSX.\n>\n>> Then I modified your script since I do not have seq or\n>> your test-genrandom.\n>\n> test-genrandom is built with Git.  It is just not installed anywhere.\n>\n>> I substituted:\n>>\n>> dd count=XX  if=/dev/random of=file_$i\n>>\n>> where XX is adjusted to meet dd's requirements.  Also,\n>\n> Again I used test-genrandom instead of /dev/random or /dev/urandom\n> simply because the former is much faster.\n\tI didn't know that it existed, since I am not that familiar with the\n\tinternals and source directory for git.\n>\n>> I found after searching for a while, that the following\n>> works just like your seq command:\n>>\n>> xyzzy=\"1 2 3 4\"\n>> for i in $xyzzy\n>> do\n>> ...\n>> done\n>>\n>> Your script then ran flawlessly.\n>\n> However 'seq -w 1 2 63' should be replaced  with \"01 03 05 07 09 11 13\n> 15\" and so on up to 63, and 'seq -w 2 2 64' is \"02 04 06 08 10 12 16\"\n> and so on.\n\tThat I missed but the files added up to quite a bit over 8gigs,\n\tbecause my count calculation was off.  So, I thought that it\n\twas adequate for the testing.  Based on above, it was\n\tquite a few files less as well.\n>\n>> I looked through index-pack.c some more, but it is\n>> very hard to figure it out without doing a lot of research\n>> since there doesn't seem to be anything that describes\n>> the layout of a pack.  The link towards the end of the user's\n>> manual doesn't work for me.\n>\n> Look at Documentation/technical/pack-format.txt in the Git source  \n> tree.\nThanks.\n>\n>> The difference between your test and my data is that\n>> instead of having a few large files, I have 11,500 files\n>> of varying sizes.  On average though, the file size is\n>> about 370k.\n>\n> Are you saying that the test repo with big files works for you but not\n> your own data set?\n\tThat is correct, but you are probably correct that this is a separate\n\tproblem, because it is only occurring on the 64-bit side.  The 32-bit\n\tside is running without error.\n>\n> Would you please recap what your problem is?\n\tWith more testing last night, I believe that the problem on the 32-bit\n\tside has been fixed.  But the 'Bus error' on 64-bit side persists.  I\n\tagree that I was combining it and probably shouldn't have.\n> With my one line patch you should not get the \"serious inflate\n> inconsistency\" error anymore.  The bus error must be another issue.\nI no longer do on the 32-bit side.\n>\n>\n> Nicolas\n\n\n\nI am going to wait on the 64-bit problem until MacOSX 10.5 (Leopard)\nsettles down a bit.  It has a few bugs in it.  I will submit the  \ncrashreporter\nissue to Apple. Currently, I don't have the time to commit a lot to  \ngit right\nnow.\n\nAs Apple comes out with new subreleases, I will retry my data.\nIf it doesn't get resolved after a subrelease or two, then I dig  \ndeeper and\nhopefully have more time to devote to it at that point.\n\nThank you, I do appreciate your help and your patience.\n"}]}