threads / discuss / 7171

Re: Errors cloning large repo

Subject: Re: Errors cloning large repo

## tl;dr

4 messages between Mar 9, 2007 and Mar 10, 2007.

replies: 3people: 3as markdown or json

Anton Tropashko· Mar 9, 2007, 23:48 UTC · lore

answers inline prefixed with >>>> while I'm trying to figure out how to deal with new "improvement" to yahoo mail beta.

On Fri, 9 Mar 2007, Anton Tropashko wrote:
>
> I managed to stuff 8.5 GB worth of files into a git repo (two two git commits since
> it was running out of memory when I gave it -a option)
So you might be able to do just do
    git add dir1
    git add dir2
    git add dir3
    ..
    git commit
or something.
>>>>>>>>>>>>  For some reason git add . swallowed the whole thing
>>>>>>>>>>>>  but git commit did not and I had to split it up. I trimmed the tree a bit
>>>>>>>>>>>>  since then by removing c & c++ files ;-)

But one caveat: git may not be the right tool for the job. May I inquire what the heck you're doing? We may be able to fix git even for your kinds 

>>>>>>>>>>>>  I dumped a rather large SDK into it. Headers, libraries
>>>>>>>>>>>> event crs.o from the toolchains that are part of SDK. The idea is to keep
>>>>>>>>>>>>  SDK versioned and being able to pull an arbitrary version once tagged.

So I'm not saying that git won't work for you, I'm just warning that the whole model of operation may or may not actually match what you want to do. Do you really want to track that 8.5GB as *one* entity?

>>>>>>>>>>>> Yes. It would be nice if I won't have to prune pdfs, txts, and who
>>>>>>>>>>>> knows what else people put in there just to reduce the size.
Show 5 quoted lines
> but when I'm cloning to another linux box I get:
> 
> Generating pack...
> Done counting 152200 objects.
> Deltifying 152200 objects.

.. this is the part makes me think git *should* be able to work for you. Having lots of smallish files is much better for git than a few DVD images, for example. And if those 152200 objects are just from two commits, you obviously have lots of files ;)

However, if it packs really badly (and without any history, that's quite likely), maybe the resulting pack-file is bigger than 4GB, and then you'd have trouble (in fact, I think you'd hit trouble at the 2GB pack-file mark).

Does "git repack -a -d" work for you?
>>>>>>>>>>>> I'll tell you as soon as I get another failure. As you
>>>>>>>>>>>> might guess it takes a while :-]
> /usr/bin/git-clone: line 321:  2072 File size limit exceededgit-fetch-pack --all -k $quiet "$repo"
"File size limit exceeded" sounds like SIGXFSZ, which is either:
 - you have file limits enabled, and the resulting pack-file was just too 
   big for the limits.
 - the file size is bigger than MAX_NON_LFS (2GB-1), and we don't use 
   O_LARGEFILE.

I suspect the second case. Shawn and Nico have worked on 64-bit packfile indexing, so they may have a patch / git tree for you to try out.

>>>>>>>>>>>> Ok. I think you're correct:

from ulimit -a: ... file size (blocks, -f) unlimited ...

Good to know developers are ahead of the users.
Is there way to get rid of pending (uncommitted) changes?
git revert does not work the same way as svn revert as I just discovered
and git status still reports a ton of pending deletions
(I changed my mind and need my object files back). I suppose I can move .git out
of the way blow all the files move it back and git pull or whatever
does a local checkout, but there must be a better way.
 
 
____________________________________________________________________________________
No need to miss a message. Get email on-the-go 
with Yahoo! Mail for Mobile. Get started.
http://mobile.yahoo.com/mail
Linus Torvalds· Mar 10, 2007, 00:54 UTC · re: Anton Tropashko · lore
On Fri, 9 Mar 2007, Anton Tropashko wrote:
Show 14 quoted lines
>
> > So you might be able to do just do
> > 
> >     git add dir1
> >     git add dir2
> >     git add dir3
> >     ..
> >     git commit
> > 
> > or something.
>
> For some reason git add . swallowed the whole thing
> but git commit did not and I had to split it up. I trimmed the tree a bit
> since then by removing c & c++ files ;-)

Ok, that's a bit surprising, since "git commit" actually should do less than "git add .", but it's entirely possible that just the status message generation ends up doing strange things for a repository with that many files in it.

I should try it out with some made-up auto-generated directory setup, but I'm not sure I have the energy to do it ;)

Show 6 quoted lines
> > But one caveat: git may not be the right tool for the job. May I inquire 
> > what the heck you're doing? We may be able to fix git even for your kinds 
>
> I dumped a rather large SDK into it. Headers, libraries
> event crs.o from the toolchains that are part of SDK. The idea is to keep
> SDK versioned and being able to pull an arbitrary version once tagged.

Ok. Assuming most of this doesn't change very often (ie the crs.o files aren't actually *generated*, but come from some external thing), git should do well enough once it's past the original hump.

So your usage scheanrio doesn't sound insane, and it's something we should be able to support well enough.

Show 6 quoted lines
> > So I'm not saying that git won't work for you, I'm just warning that the 
> > whole model of operation may or may not actually match what you want to 
> > do. Do you really want to track that 8.5GB as *one* entity?
>
> Yes. It would be nice if I won't have to prune pdfs, txts, and who
> knows what else people put in there just to reduce the size.

Sure. 8.5GB is absolutely huge, and clearly you're hitting some problems here, but if we're talking things like having a whole development environment with big manuals etc, it might be a perfectly valid usage schenario.

That said, it might also be a good idea (regardless of anything else) to split things up, if only because it's quite possible that not everybody is interested in having *everything*. Forcing people to work with a 8.5GB repository when they might not care about it all could be a bad idea.

Show 7 quoted lines
> >  - the file size is bigger than MAX_NON_LFS (2GB-1), and we don't use 
> >    O_LARGEFILE.
>
> Ok. I think you're correct:
> from ulimit -a:
> ...
> file size             (blocks, -f) unlimited

Ok, then it's the 2GB limit that the OS puts on you unless you tell it to use O_LARGEFILE.

Which is just as well, since the normal git pack-files won't index past that size *anyway* (ok, so it should index all the way up to 4GB, but it's close enough..)

> Good to know developers are ahead of the users.
Well, not "ahead enough" apparently ;)

I was seriously hoping that we could keep off the 64-bit issues for a bit longer, since the biggest real archive (firefox) we've seen so far was barely over half a gigabyte.

> Is there way to get rid of pending (uncommitted) changes?

"git reset --hard" will do it for you. As will "git checkout -f", for that matter.

"git revert" will just undo an old commit (as you apparently already found out)

		Linus
Linus Torvalds· Mar 10, 2007, 02:03 UTC · re: Linus Torvalds · lore
On Fri, 9 Mar 2007, Linus Torvalds wrote:
Show 9 quoted lines
> >
> > For some reason git add . swallowed the whole thing
> > but git commit did not and I had to split it up. I trimmed the tree a bit
> > since then by removing c & c++ files ;-)
> 
> Ok, that's a bit surprising, since "git commit" actually should do less 
> than "git add .", but it's entirely possible that just the status message 
> generation ends up doing strange things for a repository with that many 
> files in it.
Ahhh. Found it.

It's indeed "git commit" that takes tons of memory, but for all the wrong reasons. It does a "git diff-tree" to generate the diffstat, and *that* is extremely expensive:

	git-diff-tree --shortstat --summary --root --no-commit-id HEAD --

I suspect we shouldn't bother with the diffstat for the initial commit. Just removing "--root" migth be sufficient.

		Linus
Junio C Hamano· Mar 10, 2007, 02:12 UTC · re: Linus Torvalds · lore
Linus Torvalds <torvalds@linux-foundation.org> writes:
Show 8 quoted lines
> It's indeed "git commit" that takes tons of memory, but for all the wrong 
> reasons. It does a "git diff-tree" to generate the diffstat, and *that* is 
> extremely expensive:
>
> 	git-diff-tree --shortstat --summary --root --no-commit-id HEAD --
>
> I suspect we shouldn't bother with the diffstat for the initial commit. 
> Just removing "--root" migth be sufficient.

Yes and no. It was added as a response to sugestions from people in "baby step tutorial" camp.

An option to disable the last diff-tree step, just like git-pull has --no-summary option, would be perfectly fine, though.

← back to recent threads