.gitignore for large files?

Has there been any discussion in the past on a method for ignoring large files via the .gitignore process?

It does appear to be a moderately common problem for folk to accidentally commit a large file which bloats their repository and they want rid of it, which causes history re-writes and such palaver.

Perhaps a simple '>' and '<' option (the latter to cover null or minimal files?) with a --warn postfix may be possible. Just an initial thought.

Where would the 'right place' be for me to look at the git code if it was beneficial.

Philip Oakley (UK: Scotland)

On Sun, Jul 24, 2011 at 3:00 AM, Philip Oakley <philipoakley@iee.org> wrote:

Once they are in, they cannot be ignored. Perhaps commit hooks at server side is a better place?

Or you can make use of .gitattributes, more flexible syntax.

In dir.c, add_exlude() does the parsing, excluded_from_list() handles the logic.

Duy,

The .git attributes is a good call. I'm thinking that it would be a global flag (i.e. is a max file limit in place at all on this repo), and then a set of limits, initially one for 'text' and one for 'binary' (and possibly a third for anything with an ext diff), and probably use a similar attribute format as used for setting diff attributes and its filters to set a per file type limit [if required]

The relevant function names gives me a place to start....

Philip
----- Original Message ----- 
From: "Nguyen Thai Ngoc Duy" <pclouds@gmail.com>
To: "Philip Oakley" <philipoakley@iee.org>
Cc: "Git List" <git@vger.kernel.org>
Sent: Sunday, July 24, 2011 3:59 AM
Subject: Re: .gitignore for large files?

On Mon, Jul 25, 2011 at 1:53 PM, Philip Oakley <philipoakley@iee.org> wrote:

While .gitattributes looks like a better place, it does not have "exclude" attribute equivalence to .gitignore. If I remember correctly, the way .gitignore and .gitattributes are implemented makes it very hard to turn .gitignore into part of .gitattributes implementation (gitattr checks .gitattributes of current dir first, then upward to parents, while .gitgnore follows the opposite direction).

But the other way might be simpler: introduce a new attribute, then make use of the attribute in exclude code like autogenerated .gitignore. I don't know. I have not thought through this idea.

But of course you can stay away from those by checking new attributes in builtin/add.c and refusing to execute if someone wants to add a file larger than a limit. You can avoid git-add internal business by checking entries in to-be-written index, at label "finish" in cmd_add(). This is the simplest way (and getting very close to using hooks).

Anyway, good luck. Come back with some patches ;-)

Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:

While I do not think it is necessarily a good idea to invent yet another way to exclude and add it to the attributes mechanism (unless we will be dropping the support for gitignore, which is not the case), I do not know why you think the direction of the scan matters.

A more important difference is that the attribute mechanism covers the actual paths, not intermediate directories, unlike gitignore does.

Subject: How to ignore large files?

The choice of any actual implementation would depend on both feasability and usefulness. Duy was pointing out (to me?) the different approaches used for respecting the gitattributes and (multiple) gitignore files. My first thought had been to use the .gitignore file but I then realised the potential problems of adding more metacharacters ( > and < ).

I've also just seen that 1.7.6 introduced the core.bigFileThreshold for memory management reasons, while this suggestion is to help folk avoid the mistake of unintentional committing of very large files without being warned.

Does the concept (of warning/ignoring when files are 'large') have any merit?

Philip