{"thread":{"id":"44492","subject":"Bug with disabling compression and 'binary' files.","startedAt":"2016-11-16T00:21:20Z","lastAt":"2016-11-16T05:23:07Z","messageCount":4,"participants":["Douglas Cox","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"306025","messageId":"CA+i4re65SsxcaLcpGyMDnJygQFmAq4X_x_uxrkqB0yqQkEYPUQ@mail.gmail.com","threadId":"44492","inReplyTo":null,"subject":"Bug with disabling compression and 'binary' files.","fromName":"Douglas Cox","fromEmail":"ziflin@gmail.com","sentAt":"2016-11-16T00:21:14Z","receivedAt":"2016-11-16T00:21:20Z","isPatch":false,"sender":{"key":"ziflin@gmail.com","avatar":null},"body":"I was doing some experiments today with large-ish (100-200MB) binary\nfiles and was trying to determine the best configuration for Git. Here\nare the steps and timings I saw:\n\ngit init Test\ncp .../largemovie.mp4 .\ntime git add largemovie.mp4\n\nThis took 6.5s for a 200MB file.\n\nThis file compressed a little bit, but not enough to matter, so I\nwanted to see how much faster the add would be with compression\ndisabled. So I ran:\n\ngit config core.compression = 0\n\nI then completely reran the test above starting with a clean\nrepository. This time the add took only 2.08s. I repeated these two\ntests about 10 times using the same file each time to verify it wasn't\nrelated to disk caching, etc.\n\nAt this point I decided that this was likely a good setting for this\nrepository, so I also created a .gitattributes file and added a few\nentries often seen in suggestions for dealing with binary files:\n\n*.mp4 binary -delta\n\nThe goal here was to disable any delta compression done during a\ngc/pack and use the other settings 'binary' applies. Unfortunately\nwhen I ran the test again (still using compression = 0), the test was\nback to taking 6.5s and the file inside the .git/objects/ folder was\ncompressed again.\n\nI narrowed this down to the '-text' attribute that is set when\nspecifying 'binary'. For some reason this flag is cancelling out the\ncore.compression = 0 setting and I think this is a bug?\n\nUnfortunately core.compression = 0 is also global. Ideally it would be\ngreat if there was a separate 'compression' attribute that could be\nspecified in .gitattributes per wildcard similar to how -delta can be\nused. This way we would still be able to get compression for\ntext/source files, while still getting the speed of skipping\ncompression for binary files that do not compress well.\n\nHas there been any discussion on having an attribute similar to this?\n\nThanks,\n-Doug\n"},{"id":"306031","messageId":"xmqqk2c4ryj8.fsf@gitster.mtv.corp.google.com","threadId":"44492","inReplyTo":"CA+i4re65SsxcaLcpGyMDnJygQFmAq4X_x_uxrkqB0yqQkEYPUQ@mail.gmail.com","subject":"Re: Bug with disabling compression and 'binary' files.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-16T01:42:03Z","receivedAt":"2016-11-16T01:42:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Douglas Cox writes:\n\n> I narrowed this down to the '-text' attribute that is set when\n> specifying 'binary'. For some reason this flag is cancelling out the\n> core.compression = 0 setting and I think this is a bug?\n>\n> Unfortunately core.compression = 0 is also global. Ideally it would be\n> great if there was a separate 'compression' attribute that could be\n> specified in .gitattributes per wildcard similar to how -delta can be\n> used. This way we would still be able to get compression for\n> text/source files, while still getting the speed of skipping\n> compression for binary files that do not compress well.\n>\n> Has there been any discussion on having an attribute similar to this?\n\nNope. \n\nI do not offhand think of a way for '-text' attribute (or any\nattribute for what matter) to interfere with compression level, but\nwhile reading the various parts of the system that futz with the\ncompression level configuration, I noticed one thing. When we do an\ninitial \"bulk-checkin\" optimization that sends all objects to a\nsingle packfile upon \"git add\", the packfile creation uses its own\ncompression level that is not affected by any configuration or\ncommand line option. This may or may not be related to the symptom\nyou are observing (if it is, then you would see a packfile created\nin objects/pack/, not in loose objects in object/??/ directories).\n\n\n\n"},{"id":"306033","messageId":"CA+i4re4FZhshheftQnsogaY7601wgf_GfsFUxy+doLcOYPWv7Q@mail.gmail.com","threadId":"44492","inReplyTo":"xmqqk2c4ryj8.fsf@gitster.mtv.corp.google.com","subject":"Re: Bug with disabling compression and 'binary' files.","fromName":"Douglas Cox","fromEmail":"ziflin@gmail.com","sentAt":"2016-11-16T03:12:04Z","receivedAt":"2016-11-16T03:12:11Z","isPatch":false,"sender":{"key":"ziflin@gmail.com","avatar":null},"body":"> This may or may not be related to the symptom\n> you are observing (if it is, then you would see a packfile created\n> in objects/pack/, not in loose objects in object/??/ directories).\n\nNo, the file is loose (it's in .git/objects/eb in this case). This is\nseen immediately after the add, though I believe it's the same way\nwhen doing a commit on a changed file.\n\n\nOn Tue, Nov 15, 2016 at 8:42 PM, Junio C Hamano wrote:\n> Douglas Cox writes:\n>\n>> I narrowed this down to the '-text' attribute that is set when\n>> specifying 'binary'. For some reason this flag is cancelling out the\n>> core.compression = 0 setting and I think this is a bug?\n>>\n>> Unfortunately core.compression = 0 is also global. Ideally it would be\n>> great if there was a separate 'compression' attribute that could be\n>> specified in .gitattributes per wildcard similar to how -delta can be\n>> used. This way we would still be able to get compression for\n>> text/source files, while still getting the speed of skipping\n>> compression for binary files that do not compress well.\n>>\n>> Has there been any discussion on having an attribute similar to this?\n>\n> Nope.\n>\n> I do not offhand think of a way for '-text' attribute (or any\n> attribute for what matter) to interfere with compression level, but\n> while reading the various parts of the system that futz with the\n> compression level configuration, I noticed one thing. When we do an\n> initial \"bulk-checkin\" optimization that sends all objects to a\n> single packfile upon \"git add\", the packfile creation uses its own\n> compression level that is not affected by any configuration or\n> command line option. This may or may not be related to the symptom\n> you are observing (if it is, then you would see a packfile created\n> in objects/pack/, not in loose objects in object/??/ directories).\n>\n>\n>\n"},{"id":"306035","messageId":"xmqq37isroaz.fsf@gitster.mtv.corp.google.com","threadId":"44492","inReplyTo":"CA+i4re4FZhshheftQnsogaY7601wgf_GfsFUxy+doLcOYPWv7Q@mail.gmail.com","subject":"Re: Bug with disabling compression and 'binary' files.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-16T05:23:00Z","receivedAt":"2016-11-16T05:23:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Douglas Cox writes:\n\n>> This may or may not be related to the symptom\n>> you are observing (if it is, then you would see a packfile created\n>> in objects/pack/, not in loose objects in object/??/ directories).\n>\n> No, the file is loose (it's in .git/objects/eb in this case). This is\n> seen immediately after the add, though I believe it's the same way\n> when doing a commit on a changed file.\n\nThen I do not have a guess as to where the symptom you are seeing is\ncoming from.\n"}]}