Re: Errors cloning large repo

Anton Tropashko· Mar 12, 2007, 17:39 UTC · lore

> For example, if "du -sh" says 8.5GB, it doesn't necessarily mean that 
> there really is 8.5GB of data there.

> Its very likely this did fit in just under 4 GiB of packed data,
> but as you said, without O_LARGEFILE we can't work with it.

.git is 3.5GB according to du -H :)

> As Linus said earlier in this thread; Nico and I are working on
> pushing out the packfile limits, just not fast enough for some users
> needs apparently (sorry about that!).  Troy's patch was rejected

No problem. You're providing things to work around faster than I can process them :-)

> So the "git repack" actually worked for you? It really shouldn't have 
> worked.

It did not complain. I did not check the exit status but there were no so much as a single warning message: index file has overflown the kernel will panic shortly. please stand by...

> Is the server side perhaps 64-bit? If so, the limit ends up being 4GB 
> instead of 2GB, and your 8.5GB project may actually fit.

both server and client are 32 bit.

> If so, we can trivially fix it with the current index file even for a 
> 32-bit machine. The reason we limit pack-files to 2GB on 32-bit machines

Unfortunately the server machine is managed by IT. I can't install whatever I want. The client is not and it's against the IT policy to have rogue linux boxes on the net ;)

> So, wouldn't the correct fix be to automatically split a pack
> file in two pieces when it would become larger than 2 GB?

Just curious why won't you use something like PostgreSQL for data storage at this point, but, then I know nothing about git internals :)

Anyhow, I have a patch to apply now and a bash script to hone my bashing skills on. If you have anything else for me to test just shoot me an e-mail.

I'm glad I can keep you all busy.

 
____________________________________________________________________________________
Expecting? Get great news right away with email Auto-Check. 
Try the Yahoo! Mail Beta.
http://advision.webevents.yahoo.com/mailbeta/newmail_tools.html

Linus Torvalds· Mar 12, 2007, 18:40 UTC · re: Anton Tropashko · lore

On Mon, 12 Mar 2007, Anton Tropashko wrote:

Show 5 quoted lines

> 
> > Its very likely this did fit in just under 4 GiB of packed data,
> > but as you said, without O_LARGEFILE we can't work with it.
> 
> .git is 3.5GB according to du -H :)

Ok, that's good. That means that we really can use git without any major issues, and that it's literally apparently only receive-pack that has problems.

I didn't even realize that we have

	#define _FILE_OFFSET_BITS 64

in the header file, but not only is that a glibc-specific thing, it also won't really even cover all issues.

For example, if a file is opened from the shell (ie we're talking shell re-direction etc), that means that since the program that used _FILE_OFFSET_BITS wasn't the one opening, it was opened without O_LARGEFILE, and as such a write() will hit the LFS 31-bit limit.

That said, I'm not quite seeing why the _FILE_OFFSET_BITS trick doesn't help. We don't have any shell redirection in that path.

I just did an "strace -f" on a git clone on x86, and all the git open's seemed to use O_LARGEFILE, but that's with a very recent git.

I think you said that you had git-1.4.1 on the client, and I think that the _FILE_OFFSET_BITS=64 hack went in after that, and if your client just upgrades to the current 1.5.x release, it will all "just work" for you.

> Just curious why won't you use something like 
> PostgreSQL for data storage at this point, but, then
> I know nothing about git internals :)

I can pretty much guarantee that if we used a "real" database, we'd have

 - really really horrendously bad performance
 - total inability to actually recover from errors.

Other SCM projects have used databases, and it *always* boils down that. Most either die off, or decide to just do their own homegrown database (eg switching to FSFS for SVN).

Even database people seem to have figured it out lately: relational databases are starting to lose ground to specialized ones. These days you can google for something like

	relational specialized database performance

and you'll see real papers that are actually finally being taken seriously about how specialized databases often have performance-advantages of orders of magnitude. There's a paper (the above will find it, but if you add "one size fits all" you'll probably find it even better) that talks about benchmarking specialized databases against RDBMS, and they are *literally* talking about three and four *orders*of*magnitude* speedups (ie not factors of 2 or three, but factors of _seven_hundred_).

In other words, the whole relational database hype is so seventies and eighties. People have since figured out that yeah, they are convenient to program in if you want to do Visual Basic kind of things, but they really are *not* a replacement for good data structures.

So git has ended up writing its own data structures, but git is a lot better for it.

		Linus