{"thread":{"id":"7208","subject":"Re: Errors cloning large repo","startedAt":"2007-03-12T17:39:44Z","lastAt":"2007-03-12T18:40:38Z","messageCount":2,"participants":["Anton Tropashko","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"36926","messageId":"315943.12751.qm@web52606.mail.yahoo.com","threadId":"7208","inReplyTo":null,"subject":"Re: Errors cloning large repo","fromName":"Anton Tropashko","fromEmail":"atropashko@yahoo.com","sentAt":"2007-03-12T17:39:44Z","receivedAt":"2007-03-12T17:39:44Z","isPatch":false,"sender":{"key":"atropashko@yahoo.com","avatar":null},"body":"> For example, if \"du -sh\" says 8.5GB, it doesn't necessarily mean that \n> there really is 8.5GB of data there.\n\n> Its very likely this did fit in just under 4 GiB of packed data,\n> but as you said, without O_LARGEFILE we can't work with it.\n\n.git is 3.5GB according to du -H :)\n\n> As Linus said earlier in this thread; Nico and I are working on\n> pushing out the packfile limits, just not fast enough for some users\n> needs apparently (sorry about that!).  Troy's patch was rejected\n\nNo problem.\nYou're providing things to work around faster than I can process them :-)\n\n> So the \"git repack\" actually worked for you? It really shouldn't have \n> worked.\n\nIt did not complain. I did not check the exit status but there were no so\nmuch as a single warning message:\nindex file has overflown the kernel will panic shortly. please stand by...\n\n> Is the server side perhaps 64-bit? If so, the limit ends up being 4GB \n> instead of 2GB, and your 8.5GB project may actually fit.\n\nboth server and client are 32 bit.\n\n> If so, we can trivially fix it with the current index file even for a \n> 32-bit machine. The reason we limit pack-files to 2GB on 32-bit machines \n\nUnfortunately the server machine is managed by IT. I can't install whatever\nI want. The client is not and it's against the IT policy to have rogue linux boxes\non the net ;)\n\n> So, wouldn't the correct fix be to automatically split a pack\n> file in two pieces when it would become larger than 2 GB?\n\nJust curious why won't you use something like \nPostgreSQL for data storage at this point, but, then\nI know nothing about git internals :)\n\nAnyhow, I have a patch to apply now and a bash script to hone my\nbashing skills on. If you have anything else for me to test just shoot me\nan e-mail.\n\nI'm glad I can keep you all busy.\n\n\n\n\n\n\n \n____________________________________________________________________________________\nExpecting? Get great news right away with email Auto-Check. \nTry the Yahoo! Mail Beta.\nhttp://advision.webevents.yahoo.com/mailbeta/newmail_tools.html \n"},{"id":"36936","messageId":"Pine.LNX.4.64.0703121057530.9690@woody.linux-foundation.org","threadId":"7208","inReplyTo":"315943.12751.qm@web52606.mail.yahoo.com","subject":"Re: Errors cloning large repo","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-12T18:40:38Z","receivedAt":"2007-03-12T18:40:38Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 12 Mar 2007, Anton Tropashko wrote:\n> \n> > Its very likely this did fit in just under 4 GiB of packed data,\n> > but as you said, without O_LARGEFILE we can't work with it.\n> \n> .git is 3.5GB according to du -H :)\n\nOk, that's good.  That means that we really can use git without any major \nissues, and that it's literally apparently only receive-pack that has \nproblems.\n\nI didn't even realize that we have\n\n\t#define _FILE_OFFSET_BITS 64\n\nin the header file, but not only is that a glibc-specific thing, it also \nwon't really even cover all issues.\n\nFor example, if a file is opened from the shell (ie we're talking shell \nre-direction etc), that means that since the program that used \n_FILE_OFFSET_BITS wasn't the one opening, it was opened without \nO_LARGEFILE, and as such a write() will hit the LFS 31-bit limit.\n\nThat said, I'm not quite seeing why the _FILE_OFFSET_BITS trick doesn't \nhelp. We don't have any shell redirection in that path.\n\nI just did an \"strace -f\" on a git clone on x86, and all the git open's \nseemed to use O_LARGEFILE, but that's with a very recent git.\n\nI think you said that you had git-1.4.1 on the client, and I think that \nthe _FILE_OFFSET_BITS=64 hack went in after that, and if your client just \nupgrades to the current 1.5.x release, it will all \"just work\" for you.\n\n> Just curious why won't you use something like \n> PostgreSQL for data storage at this point, but, then\n> I know nothing about git internals :)\n\nI can pretty much guarantee that if we used a \"real\" database, we'd have\n\n - really really horrendously bad performance\n - total inability to actually recover from errors.\n\nOther SCM projects have used databases, and it *always* boils down that. \nMost either die off, or decide to just do their own homegrown database (eg \nswitching to FSFS for SVN).\n\nEven database people seem to have figured it out lately: relational \ndatabases are starting to lose ground to specialized ones. These days you \ncan google for something like\n\n\trelational specialized database performance\n\nand you'll see real papers that are actually finally being taken seriously \nabout how specialized databases often have performance-advantages of \norders of magnitude. There's a paper (the above will find it, but if you \nadd \"one size fits all\" you'll probably find it even better) that talks \nabout benchmarking specialized databases against RDBMS, and they are \n*literally* talking about three and four *orders*of*magnitude* speedups \n(ie not factors of 2 or three, but factors of _seven_hundred_).\n\nIn other words, the whole relational database hype is so seventies and \neighties. People have since figured out that yeah, they are convenient to \nprogram in if you want to do Visual Basic kind of things, but they really \nare *not* a replacement for good data structures.\n\nSo git has ended up writing its own data structures, but git is a lot \nbetter for it.\n\n\t\tLinus\n"}]}