{"thread":{"id":"46448","subject":"Should I store large text files on Git LFS?","startedAt":"2017-07-24T02:02:00Z","lastAt":"2017-07-25T21:38:49Z","messageCount":14,"participants":["Farshid Zavareh","Andrew Ardill","David Lang","Jeff King","Junio C Hamano","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"324930","messageId":"AC260BC2-FED8-4811-9F5D-220EF1DAAA53@gmail.com","threadId":"46448","inReplyTo":null,"subject":"Should I store large text files on Git LFS?","fromName":"Farshid Zavareh","fromEmail":"fhzavareh@gmail.com","sentAt":"2017-07-24T02:01:50Z","receivedAt":"2017-07-24T02:02:00Z","isPatch":false,"sender":{"key":"fhzavareh@gmail.com","avatar":null},"body":"Hey all. \n\nI'v been handed over a project that uses Git LFS for storing large CSV files.\n\nMy understanding is that the main benefit of using Git LFS is to keep the repository small for binary files, where Git can't keep track of the changes and ends up storing whole files for each revision. For a text file, that problem does not exist to begin with and Git can store only the changes. At the same time, this is going to make checkouts unnecessarily slow, not to mention the financial cost of storing the whole file for each revision.\n\nIs there something I'm missing here?\n\nThanks"},{"id":"324932","messageId":"CAH5451mrL=GE6WrX6juoyGPV6trcQhXXthKhjT2=qCDCiffeeA@mail.gmail.com","threadId":"46448","inReplyTo":"AC260BC2-FED8-4811-9F5D-220EF1DAAA53@gmail.com","subject":"Re: Should I store large text files on Git LFS?","fromName":"Andrew Ardill","fromEmail":"andrew.ardill@gmail.com","sentAt":"2017-07-24T02:29:23Z","receivedAt":"2017-07-24T02:30:08Z","isPatch":false,"sender":{"key":"andrew.ardill@gmail.com","avatar":"https://gravatar.com/avatar/da14cb7c091dd44dc6c63a4d3361b149acaf25226dc78eb4131a17b93d9b0993?d=mp&s=160"},"body":"Hi Farshid,\n\nOn 24 July 2017 at 12:01, Farshid Zavareh <fhzavareh@gmail.com> wrote:\n> I'v been handed over a project that uses Git LFS for storing large CSV files.\n>\n> My understanding is that the main benefit of using Git LFS is to keep the repository small for binary files, where Git can't keep track of the changes and ends up storing whole files for each revision. For a text file, that problem does not exist to begin with and Git can store only the changes. At the same time, this is going to make checkouts unnecessarily slow, not to mention the financial cost of storing the whole file for each revision.\n>\n> Is there something I'm missing here?\n\nGit LFS gives benefits when working on *large* files, not just large\n*binary* files.\n\nI can imagine a few reasons for using LFS for some CSV files\n(especially the kinds of files I deal with sometimes!).\n\nThe main one is that many users don't need or want to download the\nlarge files, or all versions of the large file. Moreover, you probably\ndon't care about changes between those files, or there would be so\nmany that using the git machinery for comparing them would be\ncumbersome and ineffective.\n\nFor me, if I was storing any CSV file over a couple of hundred\nmegabyte I would consider using something like LFS. An example would\nbe a large Dunn & Bradstreet data file, which I do an analysis on\nevery quarter. I want to include the file in the repository, so that\nthe analysis can be replicated later on, but I don't want to add 4GB\nof data to the repo every single time the dataset gets updated (also\nevery quarter). Storing that in LFS would be a good solution then.\n\nRegards,\n\nAndrew Ardill\n"},{"id":"324933","messageId":"FF08CB42-35AC-4B97-BB02-2473BEDE66A1@gmail.com","threadId":"46448","inReplyTo":"CAH5451mrL=GE6WrX6juoyGPV6trcQhXXthKhjT2=qCDCiffeeA@mail.gmail.com","subject":"Re: Should I store large text files on Git LFS?","fromName":"Farshid Zavareh","fromEmail":"fhzavareh@gmail.com","sentAt":"2017-07-24T03:46:26Z","receivedAt":"2017-07-24T03:46:36Z","isPatch":false,"sender":{"key":"fhzavareh@gmail.com","avatar":null},"body":"Hi Andrew.\n\nThanks for your reply.\n\nI'll probably test this myself, but would modifying and committing a 4GB text file actually add 4GB to the repository's size? I anticipate that it won't, since Git keeps track of the changes only, instead of storing a copy of the whole file (whereas this is not the case with binary files, hence the need for LFS).\n\nKind regards,\nFarshid\n\n> On 24 Jul 2017, at 12:29 pm, Andrew Ardill <andrew.ardill@gmail.com> wrote:\n> \n> Hi Farshid,\n> \n> On 24 July 2017 at 12:01, Farshid Zavareh <fhzavareh@gmail.com> wrote:\n>> I'v been handed over a project that uses Git LFS for storing large CSV files.\n>> \n>> My understanding is that the main benefit of using Git LFS is to keep the repository small for binary files, where Git can't keep track of the changes and ends up storing whole files for each revision. For a text file, that problem does not exist to begin with and Git can store only the changes. At the same time, this is going to make checkouts unnecessarily slow, not to mention the financial cost of storing the whole file for each revision.\n>> \n>> Is there something I'm missing here?\n> \n> Git LFS gives benefits when working on *large* files, not just large\n> *binary* files.\n> \n> I can imagine a few reasons for using LFS for some CSV files\n> (especially the kinds of files I deal with sometimes!).\n> \n> The main one is that many users don't need or want to download the\n> large files, or all versions of the large file. Moreover, you probably\n> don't care about changes between those files, or there would be so\n> many that using the git machinery for comparing them would be\n> cumbersome and ineffective.\n> \n> For me, if I was storing any CSV file over a couple of hundred\n> megabyte I would consider using something like LFS. An example would\n> be a large Dunn & Bradstreet data file, which I do an analysis on\n> every quarter. I want to include the file in the repository, so that\n> the analysis can be replicated later on, but I don't want to add 4GB\n> of data to the repo every single time the dataset gets updated (also\n> every quarter). Storing that in LFS would be a good solution then.\n> \n> Regards,\n> \n> Andrew Ardill\n\n"},{"id":"324935","messageId":"45DFB089-EDA2-4F51-9948-18B84C933FB1@gmail.com","threadId":"46448","inReplyTo":"nycvar.QRO.7.76.6.1707232109480.20418@qynat-yncgbc","subject":"Re: Should I store large text files on Git LFS?","fromName":"Farshid Zavareh","fromEmail":"fhzavareh@gmail.com","sentAt":"2017-07-24T04:18:54Z","receivedAt":"2017-07-24T04:19:04Z","isPatch":false,"sender":{"key":"fhzavareh@gmail.com","avatar":null},"body":"I see your point. So I guess it really comes down to how the file is anticipated to change. If only one or two line are going to change every now and then, then LFS is not really necessary. But, as you mentioned, text files that change drastically will affect the repository in the same way that binaries do.\n\n\n> On 24 Jul 2017, at 2:13 pm, David Lang <david@lang.hm> wrote:\n> \n> On Mon, 24 Jul 2017, Farshid Zavareh wrote:\n> \n>> I'll probably test this myself, but would modifying and committing a 4GB text file actually add 4GB to the repository's size? I anticipate that it won't, since Git keeps track of the changes only, instead of storing a copy of the whole file (whereas this is not the case with binary files, hence the need for LFS).\n> \n> well, it wouldn't be 4G because text compresses well, but if the file changes drastically from version to version (say a quarterly report), the diff won't help.\n> \n> David Lang\n\n"},{"id":"324936","messageId":"nycvar.QRO.7.76.6.1707232119210.20418@qynat-yncgbc","threadId":"46448","inReplyTo":"CANENsPpdQzBqStGjq4jUsAB0-7U8_SQq+=kjmJe6pJtiXxnYFg@mail.gmail.com","subject":"Re: Should I store large text files on Git LFS?","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2017-07-24T04:19:57Z","receivedAt":"2017-07-24T04:20:03Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Mon, 24 Jul 2017, Farshid Zavareh wrote:\n\n> I see your point. So I guess it really comes down to how the file is\n> anticipated to change. If only one or two line are going to change every\n> now and then, then LFS is not really necessary. But, as you mentioned, text\n> files that change drastically will affect the repository in the same way\n> that binaries do.\n\nNot quite the same way that binaries do, because text files compress well. but \nclose.\n\nDavid Lang\n"},{"id":"324937","messageId":"nycvar.QRO.7.76.6.1707232109480.20418@qynat-yncgbc","threadId":"46448","inReplyTo":"FF08CB42-35AC-4B97-BB02-2473BEDE66A1@gmail.com","subject":"Re: Should I store large text files on Git LFS?","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2017-07-24T04:13:02Z","receivedAt":"2017-07-24T04:23:23Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Mon, 24 Jul 2017, Farshid Zavareh wrote:\n\n> I'll probably test this myself, but would modifying and committing a 4GB text \n> file actually add 4GB to the repository's size? I anticipate that it won't, \n> since Git keeps track of the changes only, instead of storing a copy of the \n> whole file (whereas this is not the case with binary files, hence the need for \n> LFS).\n\nwell, it wouldn't be 4G because text compresses well, but if the file changes \ndrastically from version to version (say a quarterly report), the diff won't \nhelp.\n\nDavid Lang\n"},{"id":"324938","messageId":"CAH5451m0P4eZMXj8ojgbd+q-8scoJpwn9UcZLvqYKM=+8hhWPg@mail.gmail.com","threadId":"46448","inReplyTo":"CANENsPr271w=a4YNOYdrp9UM4L_eA1VZMRP_UrH+NZ+2PWM_qg@mail.gmail.com","subject":"Re: Should I store large text files on Git LFS?","fromName":"Andrew Ardill","fromEmail":"andrew.ardill@gmail.com","sentAt":"2017-07-24T04:58:38Z","receivedAt":"2017-07-24T04:59:24Z","isPatch":false,"sender":{"key":"andrew.ardill@gmail.com","avatar":"https://gravatar.com/avatar/da14cb7c091dd44dc6c63a4d3361b149acaf25226dc78eb4131a17b93d9b0993?d=mp&s=160"},"body":"Hi Farshid,\n\nOn 24 July 2017 at 13:45, Farshid Zavareh <fhzavareh@gmail.com> wrote:\n> I'll probably test this myself, but would modifying and committing a 4GB\n> text file actually add 4GB to the repository's size? I anticipate that it\n> won't, since Git keeps track of the changes only, instead of storing a copy\n> of the whole file (whereas this is not the case with binary files, hence the\n> need for LFS).\n\nI decided to do a little test myself. I add three versions of the same\ndata set (sometimes slightly different cuts of the parent data set,\nwhich I don't have) each between 2 and 4GB in size.\nEach time I added a new version it added ~500MB to the repository, and\noperations on the repository took 35-45 seconds to complete.\nRunning `git gc` compressed the objects fairly well, saving ~400MB of\nspace. I would imagine that even more space would be saved\n(proportionally) if there were a lot more similar files in the repo.\nThe time to checkout different commits didn't change much, I presume\nthat most of the time is spent copying the large file into the working\ndirectory, but I didn't test that. I did test adding some other small\nfiles, and sometimes it was slow (when cold I think?) and other times\nfast.\n\nOverall, I think as long as the files change rarely, and the\nrepository remains responsive, having these large files in the\nrepository is ok. They're still big, and if most people will never use\nthem it will be annoying for people to clone and checkout updated\nversions of the files. If you have a lot of the files, or they update\noften, or most people don't need all the files, using something like\nLFS will help a lot.\n\n$ git version  # running on my windows machine at work\ngit version 2.6.3.windows.1\n\n$ git init git-csv-test && cd git-csv-test\n$ du -h --max-depth=2  # including here to compare after large data\nfiles are added\n35K     ./.git/hooks\n1.0K    ./.git/info\n0       ./.git/objects\n0       ./.git/refs\n43K     ./.git\n43K     .\n\n$ git add data.txt  # first version of the data file, 3.2 GB\n$ git commit\n$ du -h --max-depth=2  # the data gets compressed down to ~580M of\nobjects in the git store\n35K     ./.git/hooks\n1.0K    ./.git/info\n2.0K    ./.git/logs\n580M    ./.git/objects\n1.0K    ./.git/refs\n581M    ./.git\n3.7G    .\n\n\n$ git add data.txt  # second version of the data file, 3.6 GB\n$ git commit\n$ du -h --max-depth=1  # an extra ~520M of objects added\n1.2G    ./.git\n4.7G    .\n\n\n$ time git add data.txt  # 42.344s - second version of the data file, 2.2 GB\n$ git commit  # takes about 30 seconds to load editor\n$ du -h --max-depth=1\n1.7G    ./.git\n3.9G    .\n\n$ time git checkout HEAD^  # 36.509s\n$ time git checkout HEAD^  # 44.658s\n$ time git checkout master  # 38.267s\n\n$ git gc\n$ du -h --max-depth=1\n1.3G    ./.git\n3.4G    .\n\n$ time git checkout HEAD^  # 34.743s\n$ time git checkout HEAD^  # 41.226s\n\nRegards,\n\nAndrew Ardill\n"},{"id":"324965","messageId":"20170724181118.ntqjqfihhblbvwmi@sigill.intra.peff.net","threadId":"46448","inReplyTo":"CAH5451m0P4eZMXj8ojgbd+q-8scoJpwn9UcZLvqYKM=+8hhWPg@mail.gmail.com","subject":"Re: Should I store large text files on Git LFS?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-24T18:11:18Z","receivedAt":"2017-07-24T18:11:37Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 24, 2017 at 02:58:38PM +1000, Andrew Ardill wrote:\n\n> On 24 July 2017 at 13:45, Farshid Zavareh <fhzavareh@gmail.com> wrote:\n> > I'll probably test this myself, but would modifying and committing a 4GB\n> > text file actually add 4GB to the repository's size? I anticipate that it\n> > won't, since Git keeps track of the changes only, instead of storing a copy\n> > of the whole file (whereas this is not the case with binary files, hence the\n> > need for LFS).\n> \n> I decided to do a little test myself. I add three versions of the same\n> data set (sometimes slightly different cuts of the parent data set,\n> which I don't have) each between 2 and 4GB in size.\n> Each time I added a new version it added ~500MB to the repository, and\n> operations on the repository took 35-45 seconds to complete.\n> Running `git gc` compressed the objects fairly well, saving ~400MB of\n> space. I would imagine that even more space would be saved\n> (proportionally) if there were a lot more similar files in the repo.\n\nDid you tweak core.bigfilethreshold? Git won't actually try to find\ndeltas on files larger than that (500MB by default). So you might be\nseeing just the effects of zlib compression, and not deltas.\n\nYou can always check the delta status after a gc by running:\n\n  git rev-list --objects --all |\n  git cat-file --batch-check='%(objectsize:disk) %(objectsize) %(deltabase) %(rest)'\n\nThat should give you a sense of how much you're saving due to zlib (by\ncomparing the first two numbers for a copy that isn't a delta; i.e.,\nwith an all-zeros delta base) and how much due to deltas (how much\nsmaller the first number is for an entry that _is_ a delta).\n\n-Peff\n"},{"id":"324979","messageId":"xmqqvamhd3m0.fsf@gitster.mtv.corp.google.com","threadId":"46448","inReplyTo":"20170724181118.ntqjqfihhblbvwmi@sigill.intra.peff.net","subject":"Re: Should I store large text files on Git LFS?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-24T19:41:43Z","receivedAt":"2017-07-24T19:41:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Mon, Jul 24, 2017 at 02:58:38PM +1000, Andrew Ardill wrote:\n>\n>> On 24 July 2017 at 13:45, Farshid Zavareh <fhzavareh@gmail.com> wrote:\n>> > I'll probably test this myself, but would modifying and committing a 4GB\n>> > text file actually add 4GB to the repository's size? I anticipate that it\n>> > won't, since Git keeps track of the changes only, instead of storing a copy\n>> > of the whole file (whereas this is not the case with binary files, hence the\n>> > need for LFS).\n>> \n>> I decided to do a little test myself. I add three versions of the same\n>> data set (sometimes slightly different cuts of the parent data set,\n>> which I don't have) each between 2 and 4GB in size.\n>> Each time I added a new version it added ~500MB to the repository, and\n>> operations on the repository took 35-45 seconds to complete.\n>> Running `git gc` compressed the objects fairly well, saving ~400MB of\n>> space. I would imagine that even more space would be saved\n>> (proportionally) if there were a lot more similar files in the repo.\n>\n> Did you tweak core.bigfilethreshold? Git won't actually try to find\n> deltas on files larger than that (500MB by default). So you might be\n> seeing just the effects of zlib compression, and not deltas.\n>\n> You can always check the delta status after a gc by running:\n>\n>   git rev-list --objects --all |\n>   git cat-file --batch-check='%(objectsize:disk) %(objectsize) %(deltabase) %(rest)'\n>\n> That should give you a sense of how much you're saving due to zlib (by\n> comparing the first two numbers for a copy that isn't a delta; i.e.,\n> with an all-zeros delta base) and how much due to deltas (how much\n> smaller the first number is for an entry that _is_ a delta).\n\nIn addition to that, people need to take into account that \"binary\nvs text\" is a secondary criteria when considering how effective our\ndeltifying algorithm works on their data.\n\nWe use the same xdelta algorithm that is oblivious to line breaks,\nso given two pairs of input files (T1, T2) and (B1, B2), where T1\nand B1 are comparative sizes and T2 and B2 are comparative sizes,\nand the change made to T1 to produce T2 (e.g. copy byte range X-Y of\nT1 to byte ranage starting from offset O of T2, insert this literal\nbyte string of length L, etc.) and the change made to B1 to produce\nB2 are of comparative sizes (i.e. X-Ys and Os are similar), when T's\nare text and B's are binary, you should get similarly sized delta to\nrepresent T2 as a delta to T1 and B2 as a delta to B1.  \n\nThe reason why typical \"binary\" file does not delta well is not\ninherent to their \"binary\"-ness but lies elsewhere.  It is because\ntools that produce \"binary\" files tend not to care too much about\npreserving original and only effect changes to limited part.  That\nis what makes their data not delta well across versions.\n\nExceptions are editing exif data without changing the actual image\nbits in jpeg files or editing id3 data without changing the actual\nsound bits in mp3 files.  Binary files across these kind of\noperations delta very well with Git, as \"edit\" is not done by\ncompletely rewriting everything but is confined in a small area of\nthe file.\n\n\n"},{"id":"325028","messageId":"CAH5451nbY+Xo0Fpe2OdsxwJeRV1ddZmYX7v-bPYgRsbS2kNJSg@mail.gmail.com","threadId":"46448","inReplyTo":"20170724181118.ntqjqfihhblbvwmi@sigill.intra.peff.net","subject":"Re: Should I store large text files on Git LFS?","fromName":"Andrew Ardill","fromEmail":"andrew.ardill@gmail.com","sentAt":"2017-07-25T08:06:49Z","receivedAt":"2017-07-25T08:07:38Z","isPatch":false,"sender":{"key":"andrew.ardill@gmail.com","avatar":"https://gravatar.com/avatar/da14cb7c091dd44dc6c63a4d3361b149acaf25226dc78eb4131a17b93d9b0993?d=mp&s=160"},"body":"On 25 July 2017 at 04:11, Jeff King <peff@peff.net> wrote:\n> On Mon, Jul 24, 2017 at 02:58:38PM +1000, Andrew Ardill wrote:\n>\n>> On 24 July 2017 at 13:45, Farshid Zavareh <fhzavareh@gmail.com> wrote:\n>> > I'll probably test this myself, but would modifying and committing a 4GB\n>> > text file actually add 4GB to the repository's size? I anticipate that it\n>> > won't, since Git keeps track of the changes only, instead of storing a copy\n>> > of the whole file (whereas this is not the case with binary files, hence the\n>> > need for LFS).\n>>\n>> I decided to do a little test myself. I add three versions of the same\n>> data set (sometimes slightly different cuts of the parent data set,\n>> which I don't have) each between 2 and 4GB in size.\n>> Each time I added a new version it added ~500MB to the repository, and\n>> operations on the repository took 35-45 seconds to complete.\n>> Running `git gc` compressed the objects fairly well, saving ~400MB of\n>> space. I would imagine that even more space would be saved\n>> (proportionally) if there were a lot more similar files in the repo.\n>\n> Did you tweak core.bigfilethreshold? Git won't actually try to find\n> deltas on files larger than that (500MB by default). So you might be\n> seeing just the effects of zlib compression, and not deltas.\n\nI tweaked nothing!\n\nThe space saving I assumed was pretty much just zlib compression, and\nI wasn't sure how much delta we actually could get, and how long that\nmight take to run.\n\n> You can always check the delta status after a gc by running:\n>\n>   git rev-list --objects --all |\n>   git cat-file --batch-check='%(objectsize:disk) %(objectsize) %(deltabase) %(rest)'\n>\n> That should give you a sense of how much you're saving due to zlib (by\n> comparing the first two numbers for a copy that isn't a delta; i.e.,\n> with an all-zeros delta base) and how much due to deltas (how much\n> smaller the first number is for an entry that _is_ a delta).\n\nLet's have a look:\n\n$ git rev-list --objects --all |\n  git cat-file --batch-check='%(objectsize:disk) %(objectsize)\n%(deltabase) %(rest)'\n174 262 0000000000000000000000000000000000000000\n171 260 0000000000000000000000000000000000000000\n139 212 0000000000000000000000000000000000000000\n47 36 0000000000000000000000000000000000000000\n377503831 2310238304 0000000000000000000000000000000000000000 data.txt\n47 36 0000000000000000000000000000000000000000\n500182546 3740427683 0000000000000000000000000000000000000000 data.txt\n47 36 0000000000000000000000000000000000000000\n447340264 3357717475 0000000000000000000000000000000000000000 data.txt\n\nYep, all zlib.\n\nWhat do you think is a reasonable config for storing text files this\nlarge, to get good delta compression, or is it more of a trial and\nerror to find out what works best?\n\nRegards,\n\nAndrew Ardill\n"},{"id":"325058","messageId":"20170725191347.e2p7goxho2rcemz4@sigill.intra.peff.net","threadId":"46448","inReplyTo":"CAH5451nbY+Xo0Fpe2OdsxwJeRV1ddZmYX7v-bPYgRsbS2kNJSg@mail.gmail.com","subject":"Re: Should I store large text files on Git LFS?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-25T19:13:47Z","receivedAt":"2017-07-25T19:13:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 25, 2017 at 06:06:49PM +1000, Andrew Ardill wrote:\n\n> Let's have a look:\n> \n> $ git rev-list --objects --all |\n>   git cat-file --batch-check='%(objectsize:disk) %(objectsize)\n> %(deltabase) %(rest)'\n> 174 262 0000000000000000000000000000000000000000\n> 171 260 0000000000000000000000000000000000000000\n> 139 212 0000000000000000000000000000000000000000\n> 47 36 0000000000000000000000000000000000000000\n> 377503831 2310238304 0000000000000000000000000000000000000000 data.txt\n> 47 36 0000000000000000000000000000000000000000\n> 500182546 3740427683 0000000000000000000000000000000000000000 data.txt\n> 47 36 0000000000000000000000000000000000000000\n> 447340264 3357717475 0000000000000000000000000000000000000000 data.txt\n> \n> Yep, all zlib.\n\nOK, that makes sense.\n\n> What do you think is a reasonable config for storing text files this\n> large, to get good delta compression, or is it more of a trial and\n> error to find out what works best?\n\nI think it would really depend on what's in your repo. If you just have\ngigantic text files and no big binaries, and you have enough RAM to do\ndiffs on the text files, it's not unreasonable to just send\ncore.bigfilethreshold to something really big and not worry about it.\n\nIn general, a diff is going to want memory at least 2x the size of the\nfile (for the old and new images). And we tend to keep in memory all of\nthe images for a single tree-diff at one time (so if you touched two\ngigantic files in one commit, then \"git log -p\" is probably going to\npeak at having all four before/after images in memory at once).\n\nIf you just want deltas but not diffs, you can probably do:\n\n  echo '*.gigantic -diff' >.gitattributes\n  git config core.bigfilethreshold 10G\n\nI think that will turn off streaming of the blobs in some code paths,\ntoo. But hopefully a _single_ copy of each file would be OK to hold in\nRAM. If it's not, you might also be able to get away with packing once\nwith:\n\n  git -c core.bigfilethreshold=10G repack -adf\n\nand then further repacks will carry those deltas forward. I think we\nonly apply the limit when actively searching for new deltas, not when\nreusing existing ones.\n\nAs you can see, core.bigfilethreshold is a pretty blunt instrument. It\nmight be nice if .gitattributes understood other types of patterns\nbesides filenames, so you could do something like:\n\n  echo '[size > 500MB] delta -diff' >.gitattributes\n\nor something like that. I don't think it's come up enough for anybody to\ncare too much about it or work on it.\n\n-Peff\n"},{"id":"325071","messageId":"xmqq7eywutlt.fsf@gitster.mtv.corp.google.com","threadId":"46448","inReplyTo":"20170725191347.e2p7goxho2rcemz4@sigill.intra.peff.net","subject":"Re: Should I store large text files on Git LFS?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-25T20:52:46Z","receivedAt":"2017-07-25T20:52:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> As you can see, core.bigfilethreshold is a pretty blunt instrument. It\n> might be nice if .gitattributes understood other types of patterns\n> besides filenames, so you could do something like:\n>\n>   echo '[size > 500MB] delta -diff' >.gitattributes\n>\n> or something like that. I don't think it's come up enough for anybody to\n> care too much about it or work on it.\n\nBut attributes is about paths, at which a blob may or may not exist,\nso it is a bad fit to add conditionals that are based on sizes and\ntypes.\n"},{"id":"325074","messageId":"20170725211300.vwlpioy5jes55273@sigill.intra.peff.net","threadId":"46448","inReplyTo":"xmqq7eywutlt.fsf@gitster.mtv.corp.google.com","subject":"Re: Should I store large text files on Git LFS?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-25T21:13:01Z","receivedAt":"2017-07-25T21:16:28Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 25, 2017 at 01:52:46PM -0700, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> > As you can see, core.bigfilethreshold is a pretty blunt instrument. It\n> > might be nice if .gitattributes understood other types of patterns\n> > besides filenames, so you could do something like:\n> >\n> >   echo '[size > 500MB] delta -diff' >.gitattributes\n> >\n> > or something like that. I don't think it's come up enough for anybody to\n> > care too much about it or work on it.\n> \n> But attributes is about paths, at which a blob may or may not exist,\n> so it is a bad fit to add conditionals that are based on sizes and\n> types.\n\nDo attributes _have_ to be about paths? In practice we often use them to\ndescribe objects, and paths are just the only mechanism we give to refer\nto objects.  But it is not actually a correct or rigorous mechanism in\nsome cases.  For example, imagine I have a .gitattributes with:\n\n  foo -delta\n  bar delta\n\nand then imagine I have a tree with both \"foo\" and \"bar\" pointing to the\nsame blob. When I run pack-objects, it wants to know whether to delta\nthe object. What should it do?\n\nThe delta decision is really a property of the object. But the only\nmechanism we give for selecting an object is by path, which we know is\nnot a one-to-one mapping with objects. So the results you get will\ndepend on which name we happened to see the object under first while\ntraversing.\n\nI think the case you are getting at is something like clean filters,\nwhere we might not have an object at all. In that case I would argue\nthat a property of an object could never be satisfied (so neither\n\"size > 500\" nor \"size <= 500\" could match). Whether object properties\nare meaningful is in the eye of the code that is looking up the value.\nOr more generally, the set of properties to be matched is in the eye of\nthe caller. So looking up a clean filter might want to define the size\nproperty based no the working tree size.\n\n-Peff\n"},{"id":"325079","messageId":"CAGZ79kaFR0d4Z2kcUawHw8PmHpa9gaj4sBMktZjYoeAc27ywyQ@mail.gmail.com","threadId":"46448","inReplyTo":"20170725211300.vwlpioy5jes55273@sigill.intra.peff.net","subject":"Re: Should I store large text files on Git LFS?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-25T21:38:43Z","receivedAt":"2017-07-25T21:38:49Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jul 25, 2017 at 2:13 PM, Jeff King <peff@peff.net> wrote:\n> On Tue, Jul 25, 2017 at 01:52:46PM -0700, Junio C Hamano wrote:\n>\n>> Jeff King <peff@peff.net> writes:\n>>\n>> > As you can see, core.bigfilethreshold is a pretty blunt instrument. It\n>> > might be nice if .gitattributes understood other types of patterns\n>> > besides filenames, so you could do something like:\n>> >\n>> >   echo '[size > 500MB] delta -diff' >.gitattributes\n>> >\n>> > or something like that. I don't think it's come up enough for anybody to\n>> > care too much about it or work on it.\n>>\n>> But attributes is about paths, at which a blob may or may not exist,\n>> so it is a bad fit to add conditionals that are based on sizes and\n>> types.\n>\n> Do attributes _have_ to be about paths? In practice we often use them to\n> describe objects, and paths are just the only mechanism we give to refer\n> to objects.  But it is not actually a correct or rigorous mechanism in\n> some cases.  For example, imagine I have a .gitattributes with:\n>\n>   foo -delta\n>   bar delta\n>\n> and then imagine I have a tree with both \"foo\" and \"bar\" pointing to the\n> same blob. When I run pack-objects, it wants to know whether to delta\n> the object. What should it do?\n>\n> The delta decision is really a property of the object. But the only\n> mechanism we give for selecting an object is by path, which we know is\n> not a one-to-one mapping with objects. So the results you get will\n> depend on which name we happened to see the object under first while\n> traversing.\n>\n> I think the case you are getting at is something like clean filters,\n> where we might not have an object at all. In that case I would argue\n> that a property of an object could never be satisfied (so neither\n> \"size > 500\" nor \"size <= 500\" could match). Whether object properties\n> are meaningful is in the eye of the code that is looking up the value.\n> Or more generally, the set of properties to be matched is in the eye of\n> the caller. So looking up a clean filter might want to define the size\n> property based no the working tree size.\n>\n> -Peff\n\nI recall a similar discussion on the different \"big repo\" approaches.\nLooking at the interface of LFS, there are things such as:\n\n  git lfs fetch --recent\n  git lfs fetch --all\n  git lfs fetch [--exclude] <pathspec>\n\nso LFS provides both the way to address objects via time or by path,\nmaybe even combined \"I want everything from <pathspec 1> but only\n'recent' things from <pathspec 2>\".\n\nattributes can already be queried from pathspecs, and I think when\ndesigning from scratch we might put it the other way round:\n\n    delta:\n        bar\n        everything <500m\n    -delta\n        foo\n        binaries\n\nSo in the far future, attributes may learn about more than just\npathspecs that we currently use to assign labels, but could\n* include size\n* properties derived from the 'file' utility\n* be specific about certain objects (historic paths)\n"}]}