Re: Errors cloning large repo

My problem is git-clone though since for commit it's no big deal to git commit [a-c]* , or use xargs as a workaround

For git clone I got this

Deltifying 144511 objects.
 100% (144511/144511) done
1625.375MB  (1713 kB/s)       
1729.057MB  (499 kB/s)       
/usr/bin/git-clone: line 321: 24360 File size limit exceededgit-fetch-pack --all -k $quiet "$repo"

again after git repack and don't see how to work around that aside from artifically splitting the tree at the top or resorting to a tarball on an ftp site. That 64 bit indexing code you previously mentioned would force me to upgrade git on both ends? Anywhere I can pull it out from?

 
____________________________________________________________________________________
Food fight? Enjoy some healthy debate 
in the Yahoo! Answers Food & Drink Q&A.
http://answers.yahoo.com/dir/?link=list&sid=396545367

Anton Tropashko <atropashko@yahoo.com> wrote:

I'm shocked you were able to repack an 8.5 GiB repository. The default git-repack script that we ship assumes you want to combine everything into one giant packfile; this is what is also happening during git-clone. Clearly your system is rejecting this packfile; and even if the OS allowed us to make that file the index offsets would all be wrong as they are only 32 bits wide. The repository becomes corrupt when those overflow.

Troy Telford (with the help of Eric Biederman) recently posted a patch that attempts to push the index to 64 bits:

  http://thread.gmane.org/gmane.comp.version-control.git/40680/focus=40999

You can try Troy's patch. Nico and my's 64 bit index work is *not* ready for anyone to use. It doesn't exist as a compileable chunk of code. ;-)

Just to warn you, I have (re)done some of Troy's changes and Junio has applied them to the current 'master' branch. So Troy's patch would need to be applied to something that is futher back, like around 2007-02-28 (when Troy sent the patch). But my changes alone are not enough to get "64 bit packfiles" working.

As Linus said earlier in this thread; Nico and I are working on pushing out the packfile limits, just not fast enough for some users needs apparently (sorry about that!). Troy's patch was rejected mainly because it is a file format change that is not backwards compatible (once you use the 64 bit index, anything accessing that repository *must* also support that).

Nico and I are working on other file format changes that are more extensive than just expanding the index out to 64 bits, and likewise are also not backwards compatible. To help users manage the upgrades, we want to do a single file format change in 2007, not two. So we are trying to be very sure that what we give Junio for final application really is the best we can do this year.

Otherwise we would have worked with Troy to help test his patch and get that into shape for application to main the git.git repository.

One thing that you could do is segment the repository into multiple packfiles yourself, and then clone using rsync or http, rather than using the native Git protocol.

For segmenting the repository, you would do something like:

	git rev-list --objects HEAD >S
	# segment S up into several files, e.g. T1, T2, T3
	foreach s in T*
	do
		name=$(git pack-objects tmp <$s)
		touch .git/objects/pack/pack-$name.keep
		mv tmp-$name.pack .git/objects/pack/pack-$name.pack
		mv tmp-$name.idx .git/objects/pack/pack-$name.idx
	done
	git prune-packed

The trick here is to segment S up into enough T1, T2, ... files such that when packed they each are less than 2 GiB. You can then clone this repository by copying the .git directory using more standard filesystem tools, which is what a clone with rsync or http is (more or less) doing.

Yes, the above process is horribly tedious and has a some trial and error involved in terms of selecting the packfile segmenting. We don't have anything that can automate this right now.

On Fri, 9 Mar 2007, Shawn O. Pearce wrote:

Side note - it would be nice to hear just how big the repository *really* is.

For example, if "du -sh" says 8.5GB, it doesn't necessarily mean that there really is 8.5GB of data there.

With a normal 4kB blocksize filesystem, and ~150.000 filesystem objects, you'd have an average of 300MB of just padding (roughly 2kB per file). Depending on the file statistics, it could be even more.

And if it's compressible, it's entirely possible that even without much delta compression, it could fit in a pack-file smaller than 4GB. At which point a 32-bit index file should work fine, just not with a 32-bit off_t.

So this really could be a situation where just small tweaks makes it work out for now. We'll need the full 64-bit index eventually for sure, but..

		Linus

Linus Torvalds <torvalds@linux-foundation.org> wrote:

Oh, good point.  Thanks for reminding me of reality.

I'm just so used to not looking at repository size unless the repository has been fully repacked first. So I somehow just read this thread has Anton having 8.5 GiB worth of *packed* data (where filesystem wastage in the tail block is minimal) and not 8.5 GiB of loose objects.

Its very likely this did fit in just under 4 GiB of packed data,
but as you said, without O_LARGEFILE we can't work with it.
 

Yes.  ;-)

hoi :)

On Sat, Mar 10, 2007 at 01:01:44AM -0500, Shawn O. Pearce wrote:

but newer git version can cope with it:

-r--r--r-- 1 martin martin 3847536413 18. Feb 10:36 pack-ffe867679d673ea5fbfa598b28aca1e58528b8cd.pack

On Sat, 10 Mar 2007, Martin Waitz wrote:

Are you sure you're not just running a 64-bit process?

64-bit processes don't need O_LARGEFILE to process files larger than 2GB, since for them, off_t is already 64-bit.

Grepping for O_LARGEFILE shows nothing.

Oh, except we have that 

	#define _FILE_OFFSET_BITS 64

which is just a horrible hack. That's nasty. We should just use O_LARGEFILE rather than depend on some internal glibc thing that works nowhere else.

		Linus

hoi :)

On Sat, Mar 10, 2007 at 02:46:35PM -0800, Linus Torvalds wrote:

pretty sure, yes :-)

but O_LARGEFILE is a Linux-only thing, right?

Well, if I remember correctly the *BSD systems always use 64bit now, its sad that glibc does not do the same out of the box for Linux. _FILE_OFFSET_BITS is the documented way to get 64bit file sizes on glibc, so I think it is the right thing for us (even when that define is really ugly).

Shawn O. Pearce wrote:

By the way, it would be nice to have talked about fetch / clone support for sending (and creating) _multiple_ pack files. Beside the situation where we must use more than one packfile because of size limits, it would also help clone as it could send existing packs and pack only loose objects (trading perhaps some bandwidth with CPU load on the server; think kernel.org).

Jakub Narebski <jnareb@gmail.com> wrote:

I've thought about adding that type of protocol extension on more than one occasion, but have now convinced myself that it is completely unnecessary. Well at least until a project has more than 2^32-1 objects anyway.

The reason is we can send any size packfile over the network; there is no index sent so there is no limit on how much data we transfer. We could easily just dump all existing packfiles as-is (just clip the header/footers and generate our own for the entire stream) and then send the loose objects on the end.

The client could easily segment that into multiple packfiles locally using two rules:

  - if the last object was not a OBJ_COMMIT and this object is
  an OBJ_COMMIT, start a new packfile with this object.

  - if adding this object to the current packfile exceeds my local
  filesize threshold, start a new packfile.

The first rule works because we sort objects by type, and commits appear at the front of a packfile. So if you see a non-commit followed by a commit, that's the packfile boundary that the server had.

The second rule is just common sense. But I'm not sure the first rule is even worthwhile; the server's packfile boundaries have no real interest for the client.

But Linus has already pointed all of this out (more or less) in a different fork of this thread. ;-)

Shawn O. Pearce wrote:

But what would happen if server supporting concatenated packfiles sends such stream to the old client? So I think some kind of protocol extension, or at least new request / new feature is needed for that.

Wouldn't it be better to pack loose objects into separate pack (and perhaps save it, if some threshold is crossed, and we have writing rights to repo), by the way?

Without first rule, wouldn't client end with strange packfile? Or would it have to rewrite a pack?

Jakub Narebski <jnareb@gmail.com> wrote:

No, a protocol extension is not required. The packfile format is: 12 byte header, objects, 20 byte SHA-1 footer. When sending concatenated packfiles to a client the server just needs to:

  - figure out how many objects total will be sent;
  - send its own (new) header with that count;
  - initialize a SHA-1 context and update it with the header;
  - for each packfile to be sent:
    - strip the first 12 bytes of the packfile;
    - send the remaining bytes, except the last 20;
    - update the SHA-1 context with the packfile data;
  - send its own footer with the SHA-1 context.

Very simple.  Even the oldest Git clients (pre multi-ack extension)
would understand that.  That's what's great about the way the
packfile protocol and disk format is organized.  ;-)
 

Perhaps.  Interesting food for thought, something nobody has tried
to experiment with.  Currently servers pack to update the fetching
client.  That means they may be sending a mixture of already-packed
(older) objects and loose (newer) objects.  But with the new kept
pack thing in receive-pack its more likely that things are already
packed on the server, and not loose.  (I suspect most public open
source users are pushing >100 objects when they do push to their
server.)
 

...

Nope. We don't care about the order of the objects in a packfile. Never have. Never will. Even in pack v4 where we have special object types that should only appear once in a packfile, they can appear at any position within the packfile. MUCH simpler code.

On Mon, 12 March 2007, Shawn O. Pearce wrote:

It would be a very nice thing to have, if it is backwards compatibile. It would ease load to server on clone, even if packs are divided into large tight archive pack and perhaps a few more current packs to make dumb transport do not neeed to download everything on [incremental] fetch.

On fetch... perhaps there should be some configuration variable which would change balance between load and bandwidth used...

And automatic splitting large pack on client side would help if for example we have huge repository (non-compressable binaries) and client has smaller filesystem limit on maximum file size than server.

On Fri, 9 Mar 2007, Anton Tropashko wrote:

Sure, but there were two problems.

The "git commit" problem is trivial, and in no way fundamental. The thing that uses tons of memory is literally just eyecandy, to show you *what* you're committing.

In fact, by the time it starts using tons of memory, the commit has literally already happened. It's just doing statistics afterwards that bloats it up.

The "git clone" problem is different, in that it's due to the 2GB pack-file limit. It's not "fundmentally hard" either, but it's at least not just a small tiny silly detail.

In fact, you can just do

	git add .
	git commit -q

and the "-q" flag (or "--quiet") will mean that the diffstat is never done, and the commit should be almost instantaneous (all the real work is done by the "git add .")

So "git commit" issue really is just a small beauty wart.

So the "git repack" actually worked for you? It really shouldn't have worked.

Is the server side perhaps 64-bit? If so, the limit ends up being 4GB instead of 2GB, and your 8.5GB project may actually fit.

If so, we can trivially fix it with the current index file even for a 32-bit machine. The reason we limit pack-files to 2GB on 32-bit machines is purely that we don't use O_LARGEFILE. If we enable O_LARGEFILE, that moves the limit up from 31 bits to 32 bits, and it might be enough for you. No new data structures for the index necessary at all.

		Linus

On Sat, 10 Mar 2007, Geert Bosch wrote:

I have to say, I think that's a good idea. Rather than supporting a 64-bit index, not generating big pack-files in general is probably a great idea.

For the streaming formats, we'd obviously generate arbitrarily large pack-files, but as you say, they never have an index at all, and the receiver always re-writes them *anyway* (ie we now always run "git-index-pack --fix-thin" on them), so we could just modify that "--fix-thin" logic to also split the pack when it reaches some arbitrary limit).

Some similar logic in git-pack-objects would mean that we'd never generate bigger packs in the first place..

It's not that 64-bit index file support is "wrong", but it does seem like it's not really necessary.

		Linus