{"thread":{"id":"7165","subject":"Errors cloning large repo","startedAt":"2007-03-09T19:20:35Z","lastAt":"2007-03-09T21:37:18Z","messageCount":2,"participants":["Anton Tropashko","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"36686","messageId":"284107.69764.qm@web52601.mail.yahoo.com","threadId":"7165","inReplyTo":null,"subject":"Errors cloning large repo","fromName":"Anton Tropashko","fromEmail":"atropashko@yahoo.com","sentAt":"2007-03-09T19:20:35Z","receivedAt":"2007-03-09T19:20:35Z","isPatch":false,"sender":{"key":"atropashko@yahoo.com","avatar":null},"body":"I managed to stuff 8.5 GB worth of files into a git repo (two two git commits since\nit was running out of memory when I gave it -a option)\n\nbut when I'm cloning to another linux box I get:\n\nGenerating pack...\nDone counting 152200 objects.\nDeltifying 152200 objects.\n0* 80% (122137/152200) donee\n 100% (152200/152200) done\n/usr/bin/git-clone: line 321:  2072 File size limit exceededgit-fetch-pack --all -k $quiet \"$repo\"\n\n\nWould be nice to be able to work around this somehow if the bug can not be fixed.\n1.5.0 on the server\n1.4.1 on the client\n\n\n\n\n\n \n____________________________________________________________________________________\nFood fight? Enjoy some healthy debate \nin the Yahoo! Answers Food & Drink Q&A.\nhttp://answers.yahoo.com/dir/?link=list&sid=396545367\n"},{"id":"36691","messageId":"Pine.LNX.4.64.0703091312530.10832@woody.linux-foundation.org","threadId":"7165","inReplyTo":"284107.69764.qm@web52601.mail.yahoo.com","subject":"Re: Errors cloning large repo","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-09T21:37:18Z","receivedAt":"2007-03-09T21:37:18Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 9 Mar 2007, Anton Tropashko wrote:\n>\n> I managed to stuff 8.5 GB worth of files into a git repo (two two git commits since\n> it was running out of memory when I gave it -a option)\n\nHeh. Your usage schenario may not be one where git is useful. If a single \ncommit generates that much data, git will likely perform horribly badly. \nBut it's an interesting test-case, and I don't think anybody has really \n*tried* this before, so don't give up yet.\n\nFirst off, you shouldn't really need two commits. It's true that \"git \ncommit -a\" will probably have memory usage issues (because a single \"git \nadd\" will keep it all in memory while it generates the objects), but it \nshould be possible to just use \"git add\" to add even 8.5GB worth of data \nin a few chunks, and then a single \"git commit\" should commit it.\n\nSo you might be able to do just do\n\n\tgit add dir1\n\tgit add dir2\n\tgit add dir3\n\t..\n\tgit commit\n\nor something.\n\nBut one caveat: git may not be the right tool for the job. May I inquire \nwhat the heck you're doing? We may be able to fix git even for your kinds \nof usage, but it's also possible that \n (a) git may not suit your needs\n (b) you might be better off using git differently\n\nEspecially when it comes to that \"(b)\" case, please realize that git is \nsomewhat different from something like CVS at a very fundamental level.\n\nCVS in many ways can more easily track *humongous* projects, for one very \nsimple reason: CVS really deep down just tracks individual files.\n\nSo people who have used CVS may get used to the notion of putting \neverything in one big repository, because in the end, it's just a ton of \nsmall files to CVS. CVS never really looks at the big picture - even doing \nsomething like merging or doing a full checkout is really just iterating \nover all the individual files.\n\nSo if you put a million files in a CVS repository, it's just going to \nbasically loop over those million files, but they are still just \nindividual files. There's never any operation that works on *all* of the \nfiles at once.\n\nGit really is *fundamentally* different here. Git takes completely the \nopposite approach, and git never tracks individual files at all at any \nlevel, really. Git almost doesn't care about file boundaries (I say \n\"almost\", because obviously git knows about them, and they are visible in \nmyriads of ways, but at the same time it's not entirely untrue to say that \ngit really doesn't care).\n\nSo git scales in a very different way from CVS. Many things are tons \nfaster (because git does many operations a full directory structure at a \ntime, and that makes merges that only touch a few subdirectories *much* \nfaster), but on the other hand, it means that git will consider everything \nto be *related* in a way that CVS never does.\n\nSo, for example, if your 8.5GB thing is something like your whole home \ndirectory, putting it as one git archive now ties everything together and \nthat can cause issues that really aren't very nice. Tying everything \ntogether is very important in a software project (the \"total state\" is \nwhat matters), but in your home directory, many things are simply totally \nindependent, and tying them together can be the wrong thing to do.\n\nSo I'm not saying that git won't work for you, I'm just warning that the \nwhole model of operation may or may not actually match what you want to \ndo. Do you really want to track that 8.5GB as *one* entity?\n\n> but when I'm cloning to another linux box I get:\n> \n> Generating pack...\n> Done counting 152200 objects.\n> Deltifying 152200 objects.\n\n.. this is the part makes me think git *should* be able to work for you. \nHaving lots of smallish files is much better for git than a few DVD \nimages, for example. And if those 152200 objects are just from two \ncommits, you obviously have lots of files ;)\n\nHowever, if it packs really badly (and without any history, that's quite \nlikely), maybe the resulting pack-file is bigger than 4GB, and then you'd \nhave trouble (in fact, I think you'd hit trouble at the 2GB pack-file \nmark).\n\nDoes \"git repack -a -d\" work for you?\n\n> /usr/bin/git-clone: line 321:  2072 File size limit exceededgit-fetch-pack --all -k $quiet \"$repo\"\n\n\"File size limit exceeded\" sounds like SIGXFSZ, which is either:\n\n - you have file limits enabled, and the resulting pack-file was just too \n   big for the limits.\n\n - the file size is bigger than MAX_NON_LFS (2GB-1), and we don't use \n   O_LARGEFILE.\n\nI suspect the second case. Shawn and Nico have worked on 64-bit packfile \nindexing, so they may have a patch / git tree for you to try out.\n\n\t\t\tLinus\n"}]}