# Re: Cygwin can't handle huge packfiles?

3 messages from 2006-04-06 to 2006-04-07. Participants: linux@horizon.com, Junio C Hamano.
Thread: https://gitlist.dev/t/3816

## linux@horizon.com, 2006-04-06 20:57

Subject: Re: Cygwin can't handle huge packfiles?
Message-ID: <20060406205724.12216.qmail@science.horizon.com>
URL: https://gitlist.dev/e/20060406205724.12216.qmail%40science.horizon.com

```
> Right now we LRU the pack files and evict older ones when we
> mmap too many, but the unit of eviction is the whole file, so it
> would not help the case like yours at all.  It might be possible
> to mmap only part of a packfile, but it would involve fairly
> major surgery to sha1_file.c.

The simplest solution seems to be to limit pack file size to a reasonable
fraction of a 32-bit address space.  Say, 0.5 G.

That should be a fairly straightforward hack to git-pack-objects.
It already emits two files; just make it emit more.

You can tweak the heurisitics to try to find a good break point: start
thinking about splitting the pack when you get to one size, but don't
force a break until you hit a harder limit as long as the deltas are
working well.

This can all be adjustable with a command line and/or config file option
to allow for the eventual demise of 32-bit systems.

```

## Junio C Hamano, 2006-04-06 23:53

Subject: Re: Cygwin can't handle huge packfiles?
Message-ID: <7vk6a2uupy.fsf@assigned-by-dhcp.cox.net>
URL: https://gitlist.dev/e/7vk6a2uupy.fsf%40assigned-by-dhcp.cox.net
In-Reply-To: <20060406205724.12216.qmail@science.horizon.com>

```
linux@horizon.com writes:

>> Right now we LRU the pack files and evict older ones when we
>> mmap too many, but the unit of eviction is the whole file, so it
>> would not help the case like yours at all.  It might be possible
>> to mmap only part of a packfile, but it would involve fairly
>> major surgery to sha1_file.c.
>
> The simplest solution seems to be to limit pack file size to a reasonable
> fraction of a 32-bit address space.  Say, 0.5 G.

I do not think that would help the original poster's situation
where only 5 revs result in a 1.5G pack.  I would _almost_ say
"do not pack such a repository", but there is the initial
cloning over git-aware transports which always results in a
repository with a single pack.

```

## linux@horizon.com, 2006-04-07 03:05

Subject: Re: Cygwin can't handle huge packfiles?
Message-ID: <20060407030548.22399.qmail@science.horizon.com>
URL: https://gitlist.dev/e/20060407030548.22399.qmail%40science.horizon.com
In-Reply-To: <7vk6a2uupy.fsf@assigned-by-dhcp.cox.net>

```
> I do not think that would help the original poster's situation
> where only 5 revs result in a 1.5G pack.  I would _almost_ say
> "do not pack such a repository", but there is the initial
> cloning over git-aware transports which always results in a
> repository with a single pack.

Huh?  Why not?  That repository has a lot of files.  For compression,
you want all versions of a file in one pack, and with few versions that
makes it easier to split up, not harder.

As for network transport of packs, I haven't studied the details,
but if you allow "thin packs" that have deltas relative to
objects not in the pack, then breaking up the pack anywhere
should be legal.

Or, if necessary, you can stuff an arbitrarily large file through
git-unpack-objects, which reads a stream from stdin without
attempting to mmap it.


(Speaking of unpack-objects.c, what's that "static unsigned long eof"
variable in there?  It never seems to be set to a non-zero value.)

```