{"thread":{"id":"7171","subject":"Re: Errors cloning large repo","startedAt":"2007-03-09T23:48:46Z","lastAt":"2007-03-10T02:12:46Z","messageCount":4,"participants":["Anton Tropashko","Linus Torvalds","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"36702","messageId":"20070309234846.7641.qmail@web52613.mail.yahoo.com","threadId":"7171","inReplyTo":null,"subject":"Re: Errors cloning large repo","fromName":"Anton Tropashko","fromEmail":"atropashko@yahoo.com","sentAt":"2007-03-09T23:48:46Z","receivedAt":"2007-03-09T23:48:46Z","isPatch":false,"sender":{"key":"atropashko@yahoo.com","avatar":null},"body":"answers inline prefixed with >>>> while I'm trying to figure out how to deal\nwith new \"improvement\" to yahoo mail beta.\n\nOn Fri, 9 Mar 2007, Anton Tropashko wrote:\n>\n> I managed to stuff 8.5 GB worth of files into a git repo (two two git commits since\n> it was running out of memory when I gave it -a option)\n\nSo you might be able to do just do\n\n git add dir1\n git add dir2\n git add dir3\n ..\n git commit\n\nor something.\n\n>>>>>>>>>>>> For some reason git add . swallowed the whole thing\n>>>>>>>>>>>> but git commit did not and I had to split it up. I trimmed the tree a bit\n>>>>>>>>>>>> since then by removing c & c++ files ;-)\n\nBut one caveat: git may not be the right tool for the job. May I inquire \nwhat the heck you're doing? We may be able to fix git even for your kinds \n\n>>>>>>>>>>>> I dumped a rather large SDK into it. Headers, libraries\n>>>>>>>>>>>> event crs.o from the toolchains that are part of SDK. The idea is to keep\n>>>>>>>>>>>> SDK versioned and being able to pull an arbitrary version once tagged.\n\nSo I'm not saying that git won't work for you, I'm just warning that the \nwhole model of operation may or may not actually match what you want to \ndo. Do you really want to track that 8.5GB as *one* entity?\n\n>>>>>>>>>>>> Yes. It would be nice if I won't have to prune pdfs, txts, and who\n>>>>>>>>>>>> knows what else people put in there just to reduce the size.\n\n> but when I'm cloning to another linux box I get:\n> \n> Generating pack...\n> Done counting 152200 objects.\n> Deltifying 152200 objects.\n\n.. this is the part makes me think git *should* be able to work for you. \nHaving lots of smallish files is much better for git than a few DVD \nimages, for example. And if those 152200 objects are just from two \ncommits, you obviously have lots of files ;)\n\nHowever, if it packs really badly (and without any history, that's quite \nlikely), maybe the resulting pack-file is bigger than 4GB, and then you'd \nhave trouble (in fact, I think you'd hit trouble at the 2GB pack-file \nmark).\n\nDoes \"git repack -a -d\" work for you?\n\n>>>>>>>>>>>> I'll tell you as soon as I get another failure. As you\n>>>>>>>>>>>> might guess it takes a while :-]\n\n> /usr/bin/git-clone: line 321: 2072 File size limit exceededgit-fetch-pack --all -k $quiet \"$repo\"\n\n\"File size limit exceeded\" sounds like SIGXFSZ, which is either:\n\n - you have file limits enabled, and the resulting pack-file was just too \n big for the limits.\n\n - the file size is bigger than MAX_NON_LFS (2GB-1), and we don't use \n O_LARGEFILE.\n\nI suspect the second case. Shawn and Nico have worked on 64-bit packfile \nindexing, so they may have a patch / git tree for you to try out.\n\n>>>>>>>>>>>> Ok. I think you're correct:\nfrom ulimit -a:\n...\nfile size (blocks, -f) unlimited\n...\n\nGood to know developers are ahead of the users.\n\nIs there way to get rid of pending (uncommitted) changes?\ngit revert does not work the same way as svn revert as I just discovered\nand git status still reports a ton of pending deletions\n(I changed my mind and need my object files back). I suppose I can move .git out\nof the way blow all the files move it back and git pull or whatever\ndoes a local checkout, but there must be a better way.\n \n\n\n\n\n\n \n____________________________________________________________________________________\nNo need to miss a message. Get email on-the-go \nwith Yahoo! Mail for Mobile. Get started.\nhttp://mobile.yahoo.com/mail \n"},{"id":"36706","messageId":"Pine.LNX.4.64.0703091641460.10832@woody.linux-foundation.org","threadId":"7171","inReplyTo":"20070309234846.7641.qmail@web52613.mail.yahoo.com","subject":"Re: Errors cloning large repo","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-10T00:54:44Z","receivedAt":"2007-03-10T00:54:44Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nOn Fri, 9 Mar 2007, Anton Tropashko wrote:\n>\n> > So you might be able to do just do\n> > \n> > git add dir1\n> > git add dir2\n> > git add dir3\n> > ..\n> > git commit\n> > \n> > or something.\n>\n> For some reason git add . swallowed the whole thing\n> but git commit did not and I had to split it up. I trimmed the tree a bit\n> since then by removing c & c++ files ;-)\n\nOk, that's a bit surprising, since \"git commit\" actually should do less \nthan \"git add .\", but it's entirely possible that just the status message \ngeneration ends up doing strange things for a repository with that many \nfiles in it.\n\nI should try it out with some made-up auto-generated directory setup, but \nI'm not sure I have the energy to do it ;)\n\n> > But one caveat: git may not be the right tool for the job. May I inquire \n> > what the heck you're doing? We may be able to fix git even for your kinds \n>\n> I dumped a rather large SDK into it. Headers, libraries\n> event crs.o from the toolchains that are part of SDK. The idea is to keep\n> SDK versioned and being able to pull an arbitrary version once tagged.\n\nOk. Assuming most of this doesn't change very often (ie the crs.o files \naren't actually *generated*, but come from some external thing), git \nshould do well enough once it's past the original hump.\n\nSo your usage scheanrio doesn't sound insane, and it's something we should \nbe able to support well enough. \n\n> > So I'm not saying that git won't work for you, I'm just warning that the \n> > whole model of operation may or may not actually match what you want to \n> > do. Do you really want to track that 8.5GB as *one* entity?\n>\n> Yes. It would be nice if I won't have to prune pdfs, txts, and who\n> knows what else people put in there just to reduce the size.\n\nSure. 8.5GB is absolutely huge, and clearly you're hitting some problems \nhere, but if we're talking things like having a whole development \nenvironment with big manuals etc, it might be a perfectly valid usage \nschenario.\n\nThat said, it might also be a good idea (regardless of anything else) to \nsplit things up, if only because it's quite possible that not everybody is \ninterested in having *everything*. Forcing people to work with a 8.5GB \nrepository when they might not care about it all could be a bad idea.\n\n> > - the file size is bigger than MAX_NON_LFS (2GB-1), and we don't use \n> > O_LARGEFILE.\n>\n> Ok. I think you're correct:\n> from ulimit -a:\n> ...\n> file size (blocks, -f) unlimited\n\nOk, then it's the 2GB limit that the OS puts on you unless you tell it to \nuse O_LARGEFILE.\n\nWhich is just as well, since the normal git pack-files won't index past \nthat size *anyway* (ok, so it should index all the way up to 4GB, but it's \nclose enough..)\n\n> Good to know developers are ahead of the users.\n\nWell, not \"ahead enough\" apparently ;)\n\nI was seriously hoping that we could keep off the 64-bit issues for a bit \nlonger, since the biggest real archive (firefox) we've seen so far was \nbarely over half a gigabyte.\n\n> Is there way to get rid of pending (uncommitted) changes?\n\n\"git reset --hard\" will do it for you. As will \"git checkout -f\", for that \nmatter.\n\n\"git revert\" will just undo an old commit (as you apparently already found \nout)\n\n\t\tLinus\n"},{"id":"36716","messageId":"Pine.LNX.4.64.0703091800110.10832@woody.linux-foundation.org","threadId":"7171","inReplyTo":"Pine.LNX.4.64.0703091641460.10832@woody.linux-foundation.org","subject":"Re: Errors cloning large repo","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-10T02:03:39Z","receivedAt":"2007-03-10T02:03:39Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 9 Mar 2007, Linus Torvalds wrote:\n> >\n> > For some reason git add . swallowed the whole thing\n> > but git commit did not and I had to split it up. I trimmed the tree a bit\n> > since then by removing c & c++ files ;-)\n> \n> Ok, that's a bit surprising, since \"git commit\" actually should do less \n> than \"git add .\", but it's entirely possible that just the status message \n> generation ends up doing strange things for a repository with that many \n> files in it.\n\nAhhh. Found it.\n\nIt's indeed \"git commit\" that takes tons of memory, but for all the wrong \nreasons. It does a \"git diff-tree\" to generate the diffstat, and *that* is \nextremely expensive:\n\n\tgit-diff-tree --shortstat --summary --root --no-commit-id HEAD --\n\nI suspect we shouldn't bother with the diffstat for the initial commit. \nJust removing \"--root\" migth be sufficient.\n\n\t\tLinus\n"},{"id":"36718","messageId":"7vabylzw0x.fsf@assigned-by-dhcp.cox.net","threadId":"7171","inReplyTo":"Pine.LNX.4.64.0703091800110.10832@woody.linux-foundation.org","subject":"Re: Errors cloning large repo","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-03-10T02:12:46Z","receivedAt":"2007-03-10T02:12:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> It's indeed \"git commit\" that takes tons of memory, but for all the wrong \n> reasons. It does a \"git diff-tree\" to generate the diffstat, and *that* is \n> extremely expensive:\n>\n> \tgit-diff-tree --shortstat --summary --root --no-commit-id HEAD --\n>\n> I suspect we shouldn't bother with the diffstat for the initial commit. \n> Just removing \"--root\" migth be sufficient.\n\nYes and no. It was added as a response to sugestions from\npeople in \"baby step tutorial\" camp.\n\nAn option to disable the last diff-tree step, just like git-pull\nhas --no-summary option, would be perfectly fine, though.\n"}]}