Re: Git as data archive

From: Andreas Kalz <andreas-kalz@gmx.de>
Date: Dec 8, 2019, 18:44 UTC
Message-ID: <76f7d2f5-45fe-a43f-2121-954e1a497b4f@gmx.de>
In-Reply-To: <1c81112d-5a1b-a791-b43f-d6a67d5b2e04@virtuell-zuhause.de>

Hi,

thanks to you both.

@Thomas: are you Thomas Braun who studied at FH Regensburg?

Well, currently the .git repository is 715GB and the maximum file size is 9.5GB, but I did not get error messages due to that even if the performance is quite low. The biggest pack* file is 24GB. There are some files which are modified, but most are not modified.

My question came up as I did not find a documentation about limits of git, only a lot of entries about github and forum users who are discussing about old bugs of git. I read about git-lfs and also that it is not working very stable, due to that I did not use it yet.

How can the delta compression settings and/or the big filethreshold limits be modified? Thanks in advance.

All the best, Andreas

Am 07.12.2019 um 19:04 schrieb Thomas Braun:

Show 53 quoted lines

> On 07.12.2019 17:54, Philip Oakley wrote:
>> Hi Andreas,
>>
>> On 06/12/2019 18:54, Andreas Kalz wrote:
>>> Hello,
>>> I am using git as archive and versioning also for photos. Apart from
>>> performance issues, I wanted to ask if there are hard limits and
>>> configurable limits (how to configure?) for maximum single file size and
>>> maximum .git archive size (Windows 64 Bit system)?
>>> Thanks in advance for your answer.
>>> All the best,
>>> Andreas
>> On Git the file size is currently limited to size of `long`, rather than
>> `size_t`. Hence on Git-for Windows the size limit is 32bit ~4GiB
>>
>> Any change will be a big change as it ripples through many places in the
>> code base and, for some, will feel 'wrong'. I did some work [1-4] on top
>> of those of many others that was almost there, but...
> Adding to what Philip said. On Windows the size of exported archives
> (git archive) is currently also limited to 4GB. The reason being also
> the long vs size_t issue (which is not present on linux though).
>
> So if you can switch to Linux or even MacOSX these issues are gone.
>
> The number of files in .git, only the number packfiles would be of
> interest here I guess, do not have the long vs size_t issue. So
> packfiles can be larger than 4GB on 64bit Windows (with 64bit git of
> course).
>
> But depending on how large the biggest files are, it might be worth
> tweaking some of the settings, so that the created packfiles are
> readable on all platforms. I once created a repo on linux which could
> not be checked on windows, and that is a bit annoying.
>
> So the questions are how large is each file? And what repository size do
> you expect? Are we talking about 20MB files and 10GB repository? Or a
> factor 100 more? And are you just adding files or are you modifying the
> added files? Depending on the file sizes it might then also be
> beneficial to tweak the delta compression settings and/or the big file
> threshold limits.
>
> Thomas
>
>> The alternative is git-lfs, which I don't personally use (see [4]).
>>
>> Philip
>>
>> [1] https://github.com/git-for-windows/git/pull/2179
>> [2] https://github.com/gitgitgadget/git/pull/115
>> [3] https://github.com/git-for-windows/git/issues/1063
>> [4] https://github.com/git-lfs/git-lfs/issues/2434
>>
>>