{"thread":{"id":"27899","subject":".gitignore for large files?","startedAt":"2011-07-23T20:00:48Z","lastAt":"2011-07-25T21:36:42Z","messageCount":6,"participants":["Philip Oakley","Nguyen Thai Ngoc Duy","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"171899","messageId":"B27BF8C0758741A68E1631A7308E17C6@PhilipOakley","threadId":"27899","inReplyTo":null,"subject":".gitignore for large files?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2011-07-23T20:00:48Z","receivedAt":"2011-07-23T20:00:48Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Has there been any discussion in the past on a method for ignoring large \nfiles via the .gitignore process?\n\nIt does appear to be a moderately common problem for folk to accidentally \ncommit a large file which bloats their repository and they want rid of it, \nwhich causes history re-writes and such palaver.\n\nPerhaps a simple '>' and '<' option (the latter to cover null or minimal \nfiles?) with a --warn postfix may be possible. Just an initial thought.\n\nWhere would the 'right place' be for me to look at the git code if it was \nbeneficial.\n\nPhilip Oakley\n(UK: Scotland) \n"},{"id":"171908","messageId":"CACsJy8C=ks1_TdQdNi4Wq8OCSAON9rhJ88zAp1kDZy3fBg2pmQ@mail.gmail.com","threadId":"27899","inReplyTo":"B27BF8C0758741A68E1631A7308E17C6@PhilipOakley","subject":"Re: .gitignore for large files?","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-07-24T02:59:45Z","receivedAt":"2011-07-24T02:59:45Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jul 24, 2011 at 3:00 AM, Philip Oakley wrote:\n> Has there been any discussion in the past on a method for ignoring large\n> files via the .gitignore process?\n>\n> It does appear to be a moderately common problem for folk to accidentally\n> commit a large file which bloats their repository and they want rid of it,\n> which causes history re-writes and such palaver.\n\nOnce they are in, they cannot be ignored. Perhaps commit hooks at\nserver side is a better place?\n\n> Perhaps a simple '>' and '<' option (the latter to cover null or minimal\n> files?) with a --warn postfix may be possible. Just an initial thought.\n\nOr you can make use of .gitattributes, more flexible syntax.\n\n> Where would the 'right place' be for me to look at the git code if it was\n> beneficial.\n\nIn dir.c, add_exlude() does the parsing, excluded_from_list() handles the logic.\n-- \nDuy\n"},{"id":"171976","messageId":"179943CA8CCC4BEE811E31B2BCC634D1@PhilipOakley","threadId":"27899","inReplyTo":"CACsJy8C=ks1_TdQdNi4Wq8OCSAON9rhJ88zAp1kDZy3fBg2pmQ@mail.gmail.com","subject":"Re: .gitignore for large files?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2011-07-25T06:53:44Z","receivedAt":"2011-07-25T06:53:44Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Duy,\n\nThe .git attributes is a good call. I'm thinking that it would be a global \nflag (i.e. is a max file limit in place at all on this repo), and then a set \nof limits, initially one for 'text' and one for 'binary' (and possibly a \nthird for anything with an ext diff), and probably use a similar attribute \nformat as used for setting diff attributes and its filters to set a per file \ntype limit [if required]\n\nThe relevant function names gives me a place to start....\n\nPhilip\n----- Original Message ----- \nFrom: \"Nguyen Thai Ngoc Duy\" \nTo: \"Philip Oakley\" \nCc: \"Git List\" \nSent: Sunday, July 24, 2011 3:59 AM\nSubject: Re: .gitignore for large files?\n\n\n> On Sun, Jul 24, 2011 at 3:00 AM, Philip Oakley \n> wrote:\n>> Has there been any discussion in the past on a method for ignoring large\n>> files via the .gitignore process?\n>>\n>> It does appear to be a moderately common problem for folk to accidentally\n>> commit a large file which bloats their repository and they want rid of \n>> it,\n>> which causes history re-writes and such palaver.\n>\n> Once they are in, they cannot be ignored. Perhaps commit hooks at\n> server side is a better place?\n>\n>> Perhaps a simple '>' and '<' option (the latter to cover null or minimal\n>> files?) with a --warn postfix may be possible. Just an initial thought.\n>\n> Or you can make use of .gitattributes, more flexible syntax.\n>\n>> Where would the 'right place' be for me to look at the git code if it was\n>> beneficial.\n>\n> In dir.c, add_exlude() does the parsing, excluded_from_list() handles the \n> logic.\n> -- \n> Duy\n>\n> ______________________________________________________________________\n> This email has been scanned by the MessageLabs Email Security System.\n> For more information please visit http://www.messagelabs.com/email\n> ______________________________________________________________________\n>\n>\n> -----\n> No virus found in this message.\n> Checked by AVG - www.avg.com\n> Version: 10.0.1390 / Virus Database: 1518/3783 - Release Date: 07/23/11\n> \n"},{"id":"171988","messageId":"CACsJy8B8kYU7bkD8SiK354z4u=sY3hHbe4JVwNT_1pxod1cqUw@mail.gmail.com","threadId":"27899","inReplyTo":"179943CA8CCC4BEE811E31B2BCC634D1@PhilipOakley","subject":"Re: .gitignore for large files?","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-07-25T13:17:47Z","receivedAt":"2011-07-25T13:17:47Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jul 25, 2011 at 1:53 PM, Philip Oakley wrote:\n> Duy,\n>\n> The .git attributes is a good call. I'm thinking that it would be a global\n> flag (i.e. is a max file limit in place at all on this repo), and then a set\n> of limits, initially one for 'text' and one for 'binary' (and possibly a\n> third for anything with an ext diff), and probably use a similar attribute\n> format as used for setting diff attributes and its filters to set a per file\n> type limit [if required]\n\nWhile .gitattributes looks like a better place, it does not have\n\"exclude\" attribute equivalence to .gitignore. If I remember\ncorrectly, the way .gitignore and .gitattributes are implemented makes\nit very hard to turn .gitignore into part of .gitattributes\nimplementation (gitattr checks .gitattributes of current dir first,\nthen upward to parents, while .gitgnore follows the opposite\ndirection).\n\nBut the other way might be simpler: introduce a new attribute, then\nmake use of the attribute in exclude code like autogenerated\n.gitignore. I don't know. I have not thought through this idea.\n\nBut of course you can stay away from those by checking new attributes\nin builtin/add.c and refusing to execute if someone wants to add a\nfile larger than a limit. You can avoid git-add internal business by\nchecking entries in to-be-written index, at label \"finish\" in\ncmd_add(). This is the simplest way (and getting very close to using\nhooks).\n\nAnyway, good luck. Come back with some patches ;-)\n-- \nDuy\n"},{"id":"172001","messageId":"7vd3gyrxtl.fsf@alter.siamese.dyndns.org","threadId":"27899","inReplyTo":"CACsJy8B8kYU7bkD8SiK354z4u=sY3hHbe4JVwNT_1pxod1cqUw@mail.gmail.com","subject":"Re: .gitignore for large files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-07-25T18:59:18Z","receivedAt":"2011-07-25T18:59:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy writes:\n\n> While .gitattributes looks like a better place, it does not have\n> \"exclude\" attribute equivalence to .gitignore. If I remember\n> correctly, the way .gitignore and .gitattributes are implemented makes\n> it very hard to turn .gitignore into part of .gitattributes\n> implementation (gitattr checks .gitattributes of current dir first,\n> then upward to parents, while .gitgnore follows the opposite\n> direction).\n\nWhile I do not think it is necessarily a good idea to invent yet another\nway to exclude and add it to the attributes mechanism (unless we will be\ndropping the support for gitignore, which is not the case), I do not know\nwhy you think the direction of the scan matters.\n\nA more important difference is that the attribute mechanism covers the\nactual paths, not intermediate directories, unlike gitignore does.\n"},{"id":"172015","messageId":"0F3C5B6B6420446A9D64B9394AA2BC4D@PhilipOakley","threadId":"27899","inReplyTo":"7vd3gyrxtl.fsf@alter.siamese.dyndns.org","subject":"Re: .gitignore for large files?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2011-07-25T21:36:42Z","receivedAt":"2011-07-25T21:36:42Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Subject: How to ignore large files?\n> Nguyen Thai Ngoc Duy writes:\n>\n>> While .gitattributes looks like a better place, it does not have\n>> \"exclude\" attribute equivalence to .gitignore. If I remember\n>> correctly, the way .gitignore and .gitattributes are implemented makes\n>> it very hard to turn .gitignore into part of .gitattributes\n>> implementation (gitattr checks .gitattributes of current dir first,\n>> then upward to parents, while .gitgnore follows the opposite\n>> direction).\n>\n> While I do not think it is necessarily a good idea to invent yet another\n> way to exclude and add it to the attributes mechanism (unless we will be\n> dropping the support for gitignore, which is not the case), I do not know\n> why you think the direction of the scan matters.\n>\n> A more important difference is that the attribute mechanism covers the\n> actual paths, not intermediate directories, unlike gitignore does.\n\nThe choice of any actual implementation would depend on both feasability and \nusefulness. Duy was pointing out (to me?) the different approaches used for \nrespecting the gitattributes and (multiple) gitignore files. My first \nthought had been to use the .gitignore file but I then realised the \npotential problems of adding more metacharacters ( > and < ).\n\nI've also just seen that 1.7.6 introduced the core.bigFileThreshold for \nmemory management reasons, while this suggestion is to help folk avoid the \nmistake of unintentional committing of very large files without being \nwarned.\n\nDoes the concept (of warning/ignoring when files are 'large') have any \nmerit?\n\nPhilip \n"}]}