threads / discuss / 3816

Re: Cygwin can't handle huge packfiles?

Subject: Re: Cygwin can't handle huge packfiles?

## tl;dr

3 messages between Apr 6, 2006 and Apr 7, 2006.

replies: 2people: 2as markdown or json

linux@horizon.com· Apr 6, 2006, 20:57 UTC · lore
Show 5 quoted lines
> Right now we LRU the pack files and evict older ones when we
> mmap too many, but the unit of eviction is the whole file, so it
> would not help the case like yours at all.  It might be possible
> to mmap only part of a packfile, but it would involve fairly
> major surgery to sha1_file.c.

The simplest solution seems to be to limit pack file size to a reasonable fraction of a 32-bit address space. Say, 0.5 G.

That should be a fairly straightforward hack to git-pack-objects. It already emits two files; just make it emit more.

You can tweak the heurisitics to try to find a good break point: start thinking about splitting the pack when you get to one size, but don't force a break until you hit a harder limit as long as the deltas are working well.

This can all be adjustable with a command line and/or config file option to allow for the eventual demise of 32-bit systems.

Junio C Hamano· Apr 6, 2006, 23:53 UTC · re: linux@horizon.com · lore
linux@horizon.com writes:
Show 8 quoted lines
>> Right now we LRU the pack files and evict older ones when we
>> mmap too many, but the unit of eviction is the whole file, so it
>> would not help the case like yours at all.  It might be possible
>> to mmap only part of a packfile, but it would involve fairly
>> major surgery to sha1_file.c.
>
> The simplest solution seems to be to limit pack file size to a reasonable
> fraction of a 32-bit address space.  Say, 0.5 G.

I do not think that would help the original poster's situation where only 5 revs result in a 1.5G pack. I would _almost_ say "do not pack such a repository", but there is the initial cloning over git-aware transports which always results in a repository with a single pack.

linux@horizon.com· Apr 7, 2006, 03:05 UTC · re: Junio C Hamano · lore
Show 5 quoted lines
> I do not think that would help the original poster's situation
> where only 5 revs result in a 1.5G pack.  I would _almost_ say
> "do not pack such a repository", but there is the initial
> cloning over git-aware transports which always results in a
> repository with a single pack.

Huh? Why not? That repository has a lot of files. For compression, you want all versions of a file in one pack, and with few versions that makes it easier to split up, not harder.

As for network transport of packs, I haven't studied the details, but if you allow "thin packs" that have deltas relative to objects not in the pack, then breaking up the pack anywhere should be legal.

Or, if necessary, you can stuff an arbitrarily large file through git-unpack-objects, which reads a stream from stdin without attempting to mmap it.

(Speaking of unpack-objects.c, what's that "static unsigned long eof" variable in there? It never seems to be set to a non-zero value.)

← back to recent threads