{"thread":{"id":"23786","subject":"serious performance issues with images, audio files, and other \"non-code\" data","startedAt":"2010-05-12T18:53:53Z","lastAt":"2010-05-25T17:47:39Z","messageCount":28,"participants":["John","Jakub Narebski","Jeff King","Dirk Süsserott","Sverre Rabbelier","Nicolas Pitre","Junio C Hamano","Michael J Gruber"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"141541","messageId":"4BEAF941.6040609@puckerupgames.com","threadId":"23786","inReplyTo":null,"subject":"serious performance issues with images, audio files, and other \"non-code\" data","fromName":"John","fromEmail":"john@puckerupgames.com","sentAt":"2010-05-12T18:53:53Z","receivedAt":"2010-05-12T18:53:53Z","isPatch":false,"sender":{"key":"john@puckerupgames.com","avatar":null},"body":"Hi,\n\nWe're seeing serious performance issues with repos that store media files, even relatively small \nfiles. For example, a web site with less than 100 MB of images can take minutes to commit, push, or \npull when images have changed.\n\nOur first guess was that git is repeatedly attempting to compress/decompress data that had already \nbeen compressed. We tried these configuration settings (shooting in the dark) to no avail:\n\n    core.compression 0   ## Docs say this disables compression. Didn't seem to work.\n    pack.depth 1     ## Unclear what this does.\n    pack.window 0    ## No idea what this does.\n    gc.auto 0        ## We hope this disables automatic packing.\n\nOur guess that re-compression is to blame may not even be valid since we can manually re-compress \nthese files in seconds, not minutes.\n\nIs there a trick to getting git to simply \"copy files as is\"?  In other words, don't attempt to \ncompress them, don't attempt to \"diff\" them, just store/copy/transfer the files as-is?\n\nThanks,\n  -John\n"},{"id":"141543","messageId":"m339xxdj71.fsf@localhost.localdomain","threadId":"23786","inReplyTo":"4BEAF941.6040609@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-05-12T19:15:41Z","receivedAt":"2010-05-12T19:15:41Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"John <john@puckerupgames.com> writes:\n\n> We're seeing serious performance issues with repos that store media\n> files, even relatively small files. For example, a web site with less\n> than 100 MB of images can take minutes to commit, push, or pull when\n> images have changed.\n> \n> Our first guess was that git is repeatedly attempting to\n> compress/decompress data that had already been compressed. We tried\n> these configuration settings (shooting in the dark) to no avail:\n> \n>     core.compression 0   ## Docs say this disables compression. Didn't seem to work.\n>     pack.depth 1     ## Unclear what this does.\n>     pack.window 0    ## No idea what this does.\n>     gc.auto 0        ## We hope this disables automatic packing.\n> \n> Our guess that re-compression is to blame may not even be valid since\n> we can manually re-compress these files in seconds, not minutes.\n> \n> Is there a trick to getting git to simply \"copy files as is\"?  In\n> other words, don't attempt to compress them, don't attempt to \"diff\"\n> them, just store/copy/transfer the files as-is?\n\nSearch for `delta` attribute, which should be unset for files that you\ndon't want for git to attempt (binary) delta against, in gitattributes\nmanpage.\n\nP.S. There is also git-bigfiles project that migth be of interest to\nyou.\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"141634","messageId":"20100514051049.GF6075@coredump.intra.peff.net","threadId":"23786","inReplyTo":"4BEAF941.6040609@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-14T05:10:49Z","receivedAt":"2010-05-14T05:10:49Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 12, 2010 at 02:53:53PM -0400, John wrote:\n\n> We're seeing serious performance issues with repos that store media\n> files, even relatively small files. For example, a web site with less\n> than 100 MB of images can take minutes to commit, push, or pull when\n> images have changed.\n\nThat sounds way too slow from my experiences. I have a repository with 3\ngigabytes of photos and videos. Committing 20M of new images takes a\nsecond or two. The biggest slowdown is doing the sha1 over the new data\n(which actually happens during \"git add\").\n\nWhat version of git are you using? Have you tried \"commit -q\" to\nsuppress the diff at the end of commit?\n\nCan you show us exactly what commands you're using, along with timings\nso we can see where the slowness is?\n\nFor pushing and pulling, you're probably seeing delta compression, which\ncan be slow for large files (though again, minutes seems kind of slow to\nme). It _can_ be worth doing for images, if you do things like change\nonly exif tags but not the image data itself. But if the images\nthemselves are changing, you probably want to try setting the \"-delta\"\nattribute. Like:\n\n  echo '*.jpg -delta' >.gitattributes\n\nAlso, consider repacking your repository, which will generate a packfile\nthat will be re-used during push and pull.\n\n> Our first guess was that git is repeatedly attempting to\n> compress/decompress data that had already been compressed. We tried\n\nGit does spend a fair bit of time in zlib for some workloads, but it\nshould not create problems on the order of minutes.\n\n>    core.compression 0   ## Docs say this disables compression. Didn't seem to work.\n\nThat should disable zlib compression of loose objects and objects within\npackfiles. It can save a little time for objects which won't compress,\nbut you will lose the size benefits for any text files.\n\nBut it won't turn off delta compression, which is what the\n\"compressing...\" phase during push and pull is doing. And which is much\nmore likely the cause of slowness.\n\n>    pack.depth 1     ## Unclear what this does.\n\nIt says you can't make a chain of deltas deeper than 1. It's probably\nnot what you want.\n\n>    pack.window 0    ## No idea what this does.\n\nIt sets the number of other objects git will consider when doing delta\ncompression. Setting it low should improve your push/pull times. But you\nwill lose the substantial benefit of delta-compression of your non-image\nfiles (and git's meta objects). So the \"-delta\" option above for\nspecific files is a much better solution.\n\n>    gc.auto 0        ## We hope this disables automatic packing.\n\nIt disables automatic repacking when you have a lot of objects. You\n_have_ to pack when pushing and pulling, since packfiles are the\non-the-wire format. What will help is:\n\n  1. Having repositories already packed, since git can re-use the packed\n     data.\n\n  2. Using -delta so that things which delta poorly are just copied into\n     the packfile as-is.\n\n> Is there a trick to getting git to simply \"copy files as is\"?  In\n> other words, don't attempt to compress them, don't attempt to \"diff\"\n> them, just store/copy/transfer the files as-is?\n\nHopefully you can pick out the answer to that question from the above\nstatements. :)\n\n-Peff\n"},{"id":"141669","messageId":"4BED47EA.9090905@puckerupgames.com","threadId":"23786","inReplyTo":"20100514051049.GF6075@coredump.intra.peff.net","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"John","fromEmail":"john@puckerupgames.com","sentAt":"2010-05-14T12:54:02Z","receivedAt":"2010-05-14T12:54:02Z","isPatch":false,"sender":{"key":"john@puckerupgames.com","avatar":null},"body":"Thanks so much. It's version 1.5.6.5. I compiled it 3 months ago. For example, \nin one repo, there are 1200 source files, each on average 109K in size, for a \ntotal size of 127M. The largest source file is 82M. Most of the non-text source \nfiles are already compressed.\n\nI packed the bare repo, then ran `gc --aggressive`. Then I did a `git pull`, \nwhich took 35 minutes. The git processes in `top` seemed to peak at around 300M \nof memory. Since then, I added 'binary -delta' to the .gitattributes for various \nfiles, based on suggestions from this mailing list, but by that time did not \nwish to repeat the 35 minute pull to test it out. Let's hope that made a difference.\n\nYou can simulate it all by generating a batch of 1-100 MB files from \n/dev/urandom (since they won't compress), commit them, then do it again many \ntimes to simulate edits. Every few iterates, push it somewhere.\n\n\nI noticed some other folks on this list apparently having the same issues, but \nthey don't know it yet (\"git hangs while compressing objects\", etc.). That's \nprobably the first symptom they'll see. It *appears* to hang, but it's really \nspinning away on the `pack` gizmo.\n\nI'm open to alternative suggestions -- some kind of dual-mode, where text files \nare \"fully\" version'd, diff'd, delta'd, index'd, stash'd, pack'd, compress'd, \nobject'd and whatever else git needs to do, while non-text files are archived in \na \"lesser\" manner.  On the other hand, I get the sense that the LAST thing git \nneeds is another \"mode\"!\n\n\n\n\nOn 05/14/2010 01:10 AM, Jeff King wrote:\n> On Wed, May 12, 2010 at 02:53:53PM -0400, John wrote:\n>\n>> We're seeing serious performance issues with repos that store media\n>> files, even relatively small files. For example, a web site with less\n>> than 100 MB of images can take minutes to commit, push, or pull when\n>> images have changed.\n>\n> That sounds way too slow from my experiences. I have a repository with 3\n> gigabytes of photos and videos. Committing 20M of new images takes a\n> second or two. The biggest slowdown is doing the sha1 over the new data\n> (which actually happens during \"git add\").\n>\n> What version of git are you using? Have you tried \"commit -q\" to\n> suppress the diff at the end of commit?\n>\n> Can you show us exactly what commands you're using, along with timings\n> so we can see where the slowness is?\n>\n> For pushing and pulling, you're probably seeing delta compression, which\n> can be slow for large files (though again, minutes seems kind of slow to\n> me). It _can_ be worth doing for images, if you do things like change\n> only exif tags but not the image data itself. But if the images\n> themselves are changing, you probably want to try setting the \"-delta\"\n> attribute. Like:\n>\n>    echo '*.jpg -delta'>.gitattributes\n>\n> Also, consider repacking your repository, which will generate a packfile\n> that will be re-used during push and pull.\n>\n>> Our first guess was that git is repeatedly attempting to\n>> compress/decompress data that had already been compressed. We tried\n>\n> Git does spend a fair bit of time in zlib for some workloads, but it\n> should not create problems on the order of minutes.\n>\n>>     core.compression 0   ## Docs say this disables compression. Didn't seem to work.\n>\n> That should disable zlib compression of loose objects and objects within\n> packfiles. It can save a little time for objects which won't compress,\n> but you will lose the size benefits for any text files.\n>\n> But it won't turn off delta compression, which is what the\n> \"compressing...\" phase during push and pull is doing. And which is much\n> more likely the cause of slowness.\n>\n>>     pack.depth 1     ## Unclear what this does.\n>\n> It says you can't make a chain of deltas deeper than 1. It's probably\n> not what you want.\n>\n>>     pack.window 0    ## No idea what this does.\n>\n> It sets the number of other objects git will consider when doing delta\n> compression. Setting it low should improve your push/pull times. But you\n> will lose the substantial benefit of delta-compression of your non-image\n> files (and git's meta objects). So the \"-delta\" option above for\n> specific files is a much better solution.\n>\n>>     gc.auto 0        ## We hope this disables automatic packing.\n>\n> It disables automatic repacking when you have a lot of objects. You\n> _have_ to pack when pushing and pulling, since packfiles are the\n> on-the-wire format. What will help is:\n>\n>    1. Having repositories already packed, since git can re-use the packed\n>       data.\n>\n>    2. Using -delta so that things which delta poorly are just copied into\n>       the packfile as-is.\n>\n>> Is there a trick to getting git to simply \"copy files as is\"?  In\n>> other words, don't attempt to compress them, don't attempt to \"diff\"\n>> them, just store/copy/transfer the files as-is?\n>\n> Hopefully you can pick out the answer to that question from the above\n> statements. :)\n>\n> -Peff\n"},{"id":"141685","messageId":"4BED87E3.60000@dirk.my1.cc","threadId":"23786","inReplyTo":"4BED47EA.9090905@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Dirk Süsserott","fromEmail":"newsletter@dirk.my1.cc","sentAt":"2010-05-14T17:26:59Z","receivedAt":"2010-05-14T17:26:59Z","isPatch":false,"sender":{"key":"newsletter@dirk.my1.cc","avatar":null},"body":"Am 14.05.2010 14:54 schrieb John:\n> Thanks so much. It's version 1.5.6.5. I compiled it 3 months ago. For \n> example, in one repo, there are 1200 source files, each on average 109K \n> in size, for a total size of 127M. The largest source file is 82M. Most \n> of the non-text source files are already compressed.\n> \n> I packed the bare repo, then ran `gc --aggressive`. Then I did a `git \n> pull`, which took 35 minutes. The git processes in `top` seemed to peak \n> at around 300M of memory. Since then, I added 'binary -delta' to the \n> .gitattributes for various files, based on suggestions from this mailing \n> list, but by that time did not wish to repeat the 35 minute pull to test \n> it out. Let's hope that made a difference.\n> \n> You can simulate it all by generating a batch of 1-100 MB files from \n> /dev/urandom (since they won't compress), commit them, then do it again \n> many times to simulate edits. Every few iterates, push it somewhere.\n> \n> \n> I noticed some other folks on this list apparently having the same \n> issues, but they don't know it yet (\"git hangs while compressing \n> objects\", etc.). That's probably the first symptom they'll see. It \n> *appears* to hang, but it's really spinning away on the `pack` gizmo.\n> \n> I'm open to alternative suggestions -- some kind of dual-mode, where \n> text files are \"fully\" version'd, diff'd, delta'd, index'd, stash'd, \n> pack'd, compress'd, object'd and whatever else git needs to do, while \n> non-text files are archived in a \"lesser\" manner.  On the other hand, I \n> get the sense that the LAST thing git needs is another \"mode\"!\n> \n> \n> \n> \n> On 05/14/2010 01:10 AM, Jeff King wrote:\n>> On Wed, May 12, 2010 at 02:53:53PM -0400, John wrote:\n>>\n>>> We're seeing serious performance issues with repos that store media\n>>> files, even relatively small files. For example, a web site with less\n>>> than 100 MB of images can take minutes to commit, push, or pull when\n>>> images have changed.\n>>\n>> That sounds way too slow from my experiences. I have a repository with 3\n>> gigabytes of photos and videos. Committing 20M of new images takes a\n>> second or two. The biggest slowdown is doing the sha1 over the new data\n>> (which actually happens during \"git add\").\n>>\n>> What version of git are you using? Have you tried \"commit -q\" to\n>> suppress the diff at the end of commit?\n>>\n>> Can you show us exactly what commands you're using, along with timings\n>> so we can see where the slowness is?\n>>\n>> For pushing and pulling, you're probably seeing delta compression, which\n>> can be slow for large files (though again, minutes seems kind of slow to\n>> me). It _can_ be worth doing for images, if you do things like change\n>> only exif tags but not the image data itself. But if the images\n>> themselves are changing, you probably want to try setting the \"-delta\"\n>> attribute. Like:\n>>\n>>    echo '*.jpg -delta'>.gitattributes\n>>\n>> Also, consider repacking your repository, which will generate a packfile\n>> that will be re-used during push and pull.\n>>\n>>> Our first guess was that git is repeatedly attempting to\n>>> compress/decompress data that had already been compressed. We tried\n>>\n>> Git does spend a fair bit of time in zlib for some workloads, but it\n>> should not create problems on the order of minutes.\n>>\n>>>     core.compression 0   ## Docs say this disables compression. \n>>> Didn't seem to work.\n>>\n>> That should disable zlib compression of loose objects and objects within\n>> packfiles. It can save a little time for objects which won't compress,\n>> but you will lose the size benefits for any text files.\n>>\n>> But it won't turn off delta compression, which is what the\n>> \"compressing...\" phase during push and pull is doing. And which is much\n>> more likely the cause of slowness.\n>>\n>>>     pack.depth 1     ## Unclear what this does.\n>>\n>> It says you can't make a chain of deltas deeper than 1. It's probably\n>> not what you want.\n>>\n>>>     pack.window 0    ## No idea what this does.\n>>\n>> It sets the number of other objects git will consider when doing delta\n>> compression. Setting it low should improve your push/pull times. But you\n>> will lose the substantial benefit of delta-compression of your non-image\n>> files (and git's meta objects). So the \"-delta\" option above for\n>> specific files is a much better solution.\n>>\n>>>     gc.auto 0        ## We hope this disables automatic packing.\n>>\n>> It disables automatic repacking when you have a lot of objects. You\n>> _have_ to pack when pushing and pulling, since packfiles are the\n>> on-the-wire format. What will help is:\n>>\n>>    1. Having repositories already packed, since git can re-use the packed\n>>       data.\n>>\n>>    2. Using -delta so that things which delta poorly are just copied into\n>>       the packfile as-is.\n>>\n>>> Is there a trick to getting git to simply \"copy files as is\"?  In\n>>> other words, don't attempt to compress them, don't attempt to \"diff\"\n>>> them, just store/copy/transfer the files as-is?\n>>\n>> Hopefully you can pick out the answer to that question from the above\n>> statements. :)\n>>\n>> -Peff\n> \n\nHi John,\n\nPeff explained it very well. Some time ago, I had a similar problem:\nhttp://www.mentby.com/Group/git/how-to-prevent-git-from-compressing-certain-files.html\nand he helped me as well. Probably you may want to have a look at that \nthread.\n\n\tDirk\n\n\n> -- \n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n"},{"id":"141845","messageId":"20100517231642.GB12092@coredump.intra.peff.net","threadId":"23786","inReplyTo":"4BED47EA.9090905@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-17T23:16:43Z","receivedAt":"2010-05-17T23:16:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 14, 2010 at 08:54:02AM -0400, John wrote:\n\n> Thanks so much. It's version 1.5.6.5. I compiled it 3 months ago. For\n\nBy git standards, that version is ancient. You may want to try with a\nmore recent version of git (at the very least, multithreaded delta\ncompression has been enabled by default since then).\n\n> I packed the bare repo, then ran `gc --aggressive`.\n\nNote that \"gc --aggressive\" will repack from scratch, throwing away the\nprevious pack.\n\n> Then I did a `git pull`, which took 35 minutes.\n\nThat sounds like a long time. What was taking so long? Was delta\ncompression pegging the CPU? Was it limited during the \"Writing objects\"\nphase, which is going to be limited by either disk I/O or network speed?\n\nHow big is your packed repo? Given the pattern you describe below, I am\nbeginning to wonder if it is simply the case that even though a single\ncheckout of your repo isn't that large, the complete history of your\nproject may simply be gigantic (e.g., because you are repeatedly writing\nnew apparently-random versions of each file, so your repository size\nwill grow quite quickly).\n\nRemember that a git clone transfers the full history (and a pull will\ntransfer all of the intermediate history). If you have rewritten those\nfiles many times, you may be transferring many times your working\ndirectory size in history.\n\n> You can simulate it all by generating a batch of 1-100 MB files from\n> /dev/urandom (since they won't compress), commit them, then do it\n> again many times to simulate edits. Every few iterates, push it\n> somewhere.\n\nI tried this script to make a 100M working directory with a 400M .git\ndirectory:\n\n-- >8 --\n#!/bin/sh\n\nrm -rf big-repo\nmkdir big-repo && cd big-repo && git init\n\nmark() {\n  echo \"`date` $*\"\n}\n\nrandomize() {\n  mark randomize start\n  for i in `seq 1 100`; do\n    openssl rand $((1024*1024)) >$i.rand\n  done\n  mark randomize end\n}\n\ncommit() {\n  mark add start\n  git add .\n  mark add end\n  mark commit start\n  git commit -m \"$1\"\n  mark commit end\n}\n\nrandomize; commit base\nrandomize; commit one\nrandomize; commit two\nrandomize; commit three\n-- 8< --\n\nHere are a few timings I noted:\n\n  - it takes about 5 seconds to generate and write the random data\n\n  - git add runs in about 13 seconds. It pegs the CPU hashing all of the\n    data.\n\n  - the first commit is nearly instantaneous, as the summary diff takes\n    no work; subsequent commits spend about 9 seconds to create the\n    summary diff.  Changing commit to \"commit -q\" drops that to back to\n    near-instantaneous.\n\n  - with no attributes set, \"time git gc --aggressive\" reports:\n\n      real    1m31.983s\n      user    2m29.621s\n      sys     0m3.732s\n\n    Note the real/user discrepancy. It's a dual-core machine, and recent\n    git will multi-thread the delta phase, which is what dominates the\n    time. This should correspond roughly to the delta-compression phase\n    of your pull time, as that was just making a pack on the fly (but\n    now that we are packed, pulls will be limited only by the time to\n    transfer the objects themselves).\n\n  - Turning off delta compression for the .rand files makes repacking\n    much faster:\n\n      $ echo '*.rand -delta' >.gitattributes\n      $ time git gc --aggressive\n      ...\n      real    0m25.354s\n      user    0m22.057s\n      sys     0m1.316s\n\n    The delta compression phase is very quick, and we spend most of our\n    time writing out the packfile to disk.\n\nSo I stand by my earlier statements:\n\n  1. Use \"git commit -q\" to avoid wasting time on the commit diff\n     summary (we should perhaps have a commit.quiet config option for\n     repos like this where you would almost always want to suppress it).\n\n  2. Make sure your upstream repo is packed so pullers do not have to\n     generate a new packfile all the time.\n\n  3. Use -delta where appropriate to avoid useless delta compression.\n\nIf things are still slow after that, you'll need to be more specific\nabout your exact workload and exactly what is slow (I am still not sure\nif delta compression or network bandwidth is the limiting factor for\nyour slow pulls).\n\n-Peff\n"},{"id":"141847","messageId":"AANLkTikAoZgvYXN-iGPGjMP-vdgrOuKp3_gJzGYaojec@mail.gmail.com","threadId":"23786","inReplyTo":"20100517231642.GB12092@coredump.intra.peff.net","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-05-17T23:33:35Z","receivedAt":"2010-05-17T23:33:35Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Tue, May 18, 2010 at 01:16, Jeff King <peff@peff.net> wrote:\n>  1. Use \"git commit -q\" to avoid wasting time on the commit diff\n>     summary (we should perhaps have a commit.quiet config option for\n>     repos like this where you would almost always want to suppress it).\n\nDo we respect the .gitattributef and not try to generate the diffstat\nfor files that are uncompressable?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"141874","messageId":"4BF2E168.2020706@puckerupgames.com","threadId":"23786","inReplyTo":"20100517231642.GB12092@coredump.intra.peff.net","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"John","fromEmail":"john@puckerupgames.com","sentAt":"2010-05-18T18:50:16Z","receivedAt":"2010-05-18T18:50:16Z","isPatch":false,"sender":{"key":"john@puckerupgames.com","avatar":null},"body":"On 05/17/2010 07:16 PM, Jeff King wrote:\n> On Fri, May 14, 2010 at 08:54:02AM -0400, John wrote:\n>\n>> Thanks so much. It's version 1.5.6.5. I compiled it 3 months ago. For\n>\n> By git standards, that version is ancient. You may want to try with a\n> more recent version of git (at the very least, multithreaded delta\n> compression has been enabled by default since then).\n\nI just compiled the latest git. It got worse!!\n\n$  git --version\ngit version 1.5.6.5\n$ time git gc --aggressive\nCounting objects: 2086, done.\nCompressing objects: 100% (2054/2054), done.\nWriting objects: 100% (2086/2086), done.\nTotal 2086 (delta 676), reused 0 (delta 0)\n\nreal    4m28.573s\nuser    3m38.650s\nsys     0m5.156s\n$  git --version\ngit version 1.7.1\n$ time git gc --aggressive\nCounting objects: 2086, done.\nCompressing objects: 100% (2054/2054), done.\nWriting objects: 100% (2086/2086), done.\nTotal 2086 (delta 676), reused 0 (delta 0)\n\nreal    6m16.406s\nuser    5m28.665s\nsys     0m6.196s\n$ du -hs .git\n203M\t.git\n\n\n>\n>> I packed the bare repo, then ran `gc --aggressive`.\n>\n> Note that \"gc --aggressive\" will repack from scratch, throwing away the\n> previous pack.\n>\n>> Then I did a `git pull`, which took 35 minutes.\n>\n> That sounds like a long time. What was taking so long? Was delta\n> compression pegging the CPU? Was it limited during the \"Writing objects\"\n> phase, which is going to be limited by either disk I/O or network speed?\n\nThe compressing objects phase. Yes, pegging the CPU and hogging memory.\n\n\n> How big is your packed repo? Given the pattern you describe below, I am\n> beginning to wonder if it is simply the case that even though a single\n> checkout of your repo isn't that large, the complete history of your\n> project may simply be gigantic (e.g., because you are repeatedly writing\n> new apparently-random versions of each file, so your repository size\n> will grow quite quickly).\n\nThe packed .git dir is 203 MB. Yes, we make frequent changes to these files, and push/pull \nfrequently as well. Just a normal development pattern, though. It's all manually done -- i.e., \nthere's no automated bot doing excessive git operations.\n\n>\n> Remember that a git clone transfers the full history (and a pull will\n> transfer all of the intermediate history). If you have rewritten those\n> files many times, you may be transferring many times your working\n> directory size in history.\n>\n>> You can simulate it all by generating a batch of 1-100 MB files from\n>> /dev/urandom (since they won't compress), commit them, then do it\n>> again many times to simulate edits. Every few iterates, push it\n>> somewhere.\n>\n> I tried this script to make a 100M working directory with a 400M .git\n> directory:\n>\n> -- >8 --\n> #!/bin/sh\n>\n> rm -rf big-repo\n> mkdir big-repo&&  cd big-repo&&  git init\n>\n> mark() {\n>    echo \"`date` $*\"\n> }\n>\n> randomize() {\n>    mark randomize start\n>    for i in `seq 1 100`; do\n>      openssl rand $((1024*1024))>$i.rand\n>    done\n>    mark randomize end\n> }\n>\n> commit() {\n>    mark add start\n>    git add .\n>    mark add end\n>    mark commit start\n>    git commit -m \"$1\"\n>    mark commit end\n> }\n>\n> randomize; commit base\n> randomize; commit one\n> randomize; commit two\n> randomize; commit three\n> -- 8<  --\n>\n> Here are a few timings I noted:\n>\n>    - it takes about 5 seconds to generate and write the random data\n>\n>    - git add runs in about 13 seconds. It pegs the CPU hashing all of the\n>      data.\n>\n>    - the first commit is nearly instantaneous, as the summary diff takes\n>      no work; subsequent commits spend about 9 seconds to create the\n>      summary diff.  Changing commit to \"commit -q\" drops that to back to\n>      near-instantaneous.\n>\n>    - with no attributes set, \"time git gc --aggressive\" reports:\n>\n>        real    1m31.983s\n>        user    2m29.621s\n>        sys     0m3.732s\n>\n>      Note the real/user discrepancy. It's a dual-core machine, and recent\n>      git will multi-thread the delta phase, which is what dominates the\n>      time. This should correspond roughly to the delta-compression phase\n>      of your pull time, as that was just making a pack on the fly (but\n>      now that we are packed, pulls will be limited only by the time to\n>      transfer the objects themselves).\n>\n>    - Turning off delta compression for the .rand files makes repacking\n>      much faster:\n>\n>        $ echo '*.rand -delta'>.gitattributes\n>        $ time git gc --aggressive\n>        ...\n>        real    0m25.354s\n>        user    0m22.057s\n>        sys     0m1.316s\n>\n>      The delta compression phase is very quick, and we spend most of our\n>      time writing out the packfile to disk.\n>\n> So I stand by my earlier statements:\n>\n>    1. Use \"git commit -q\" to avoid wasting time on the commit diff\n>       summary (we should perhaps have a commit.quiet config option for\n>       repos like this where you would almost always want to suppress it).\n\nThanks, I will try that,\n\n>\n>    2. Make sure your upstream repo is packed so pullers do not have to\n>       generate a new packfile all the time.\n\nGot that in cron now.\n\n\n>    3. Use -delta where appropriate to avoid useless delta compression.\n\nAlready in there (thanks to your previous advice).\n\n\n> If things are still slow after that, you'll need to be more specific\n> about your exact workload and exactly what is slow (I am still not sure\n> if delta compression or network bandwidth is the limiting factor for\n> your slow pulls).\n\nIt's definitely the pull/push in git. Not knowing my way around git internals at all, I don't know \n(nor do I really want to know, to be honest) which \"sub-processes\" of `git pull` or `git push` are \nthe culprit. Yes, network bandwidth is always a factor, but I guess my expectation is that git \nshouldn't transfer too much more info than the amount of recent changes. For example, if we change \n10 files for a total of 10MB, then my admittedly naive expectation is that git will send that 10MB \nof changes, plus some small constant amount of meta info... not the whole repo every time. No?\n"},{"id":"141875","messageId":"AANLkTintESjlaE_YSYBcZxF0lP7GhXnXdYmYzb4phjas@mail.gmail.com","threadId":"23786","inReplyTo":"4BF2E168.2020706@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-05-18T18:54:19Z","receivedAt":"2010-05-18T18:54:19Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Tue, May 18, 2010 at 20:50, John <john@puckerupgames.com> wrote:\n> I just compiled the latest git. It got worse!!\n\nI think that's got --aggressive got more aggressive :). We now do\n--window=200 and --depth=200 for --aggressive gc's.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"141877","messageId":"20100518190706.GA2383@coredump.intra.peff.net","threadId":"23786","inReplyTo":"AANLkTikAoZgvYXN-iGPGjMP-vdgrOuKp3_gJzGYaojec@mail.gmail.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-18T19:07:06Z","receivedAt":"2010-05-18T19:07:06Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 18, 2010 at 01:33:35AM +0200, Sverre Rabbelier wrote:\n\n> On Tue, May 18, 2010 at 01:16, Jeff King <peff@peff.net> wrote:\n> >  1. Use \"git commit -q\" to avoid wasting time on the commit diff\n> >     summary (we should perhaps have a commit.quiet config option for\n> >     repos like this where you would almost always want to suppress it).\n> \n> Do we respect the .gitattributef and not try to generate the diffstat\n> for files that are uncompressable?\n\nNo, not to my knowledge. Even the \"binary\" attribute just says \"this\nfile is binary, don't text diff it\". I think we will always still do\nrewrite-detection for operations like \"git status\" and the diff summary\nof \"git commit\".\n\n-Peff\n"},{"id":"141878","messageId":"AANLkTiltNPIR5gAWMOqZ2Y_azFUU93kH54ddHuCFFeCp@mail.gmail.com","threadId":"23786","inReplyTo":"20100518190706.GA2383@coredump.intra.peff.net","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-05-18T19:10:58Z","receivedAt":"2010-05-18T19:10:58Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Tue, May 18, 2010 at 21:07, Jeff King <peff@peff.net> wrote:\n> No, not to my knowledge. Even the \"binary\" attribute just says \"this\n> file is binary, don't text diff it\". I think we will always still do\n> rewrite-detection for operations like \"git status\" and the diff summary\n> of \"git commit\".\n\nWould that not be a very sensible optimization that would help John\n(and other users of big files) a lot?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"141879","messageId":"20100518191933.GB2383@coredump.intra.peff.net","threadId":"23786","inReplyTo":"4BF2E168.2020706@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-18T19:19:33Z","receivedAt":"2010-05-18T19:19:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 18, 2010 at 02:50:16PM -0400, John wrote:\n\n> I just compiled the latest git. It got worse!!\n\nI think Sverre is right that this is simply that --aggressive got more\nso in the last few versions. But do note that aggressive implies that we\nshould pack from scratch, not reusing previously found deltas (or\naccepting that we didn't find deltas previously).\n\nSo you might want \"git gc --aggressive\" the _first_ time you pack, or\npossibly even very occasionally. But if you are packing every day, you\nshould just use \"git gc\", which will run much more quickly (and would\nprobably have acceptable behavior even without the -delta attribute, as\nit would only have to look at _new_ objects).\n\nIt will have to write the whole 200M packfile out each time, though.\nFrom your timings that looks to take about 50 seconds or so (just\nlooking at the difference between wall clock time and CPU time, which is\npresumably spent in I/O).\n\nPacking nightly won't hurt, but is perhaps excessive. It sounds like you\nactually have a fairly normal workload.\n\n> >How big is your packed repo? Given the pattern you describe below, I am\n> [...]\n> The packed .git dir is 203 MB. Yes, we make frequent changes to these\n> files, and push/pull frequently as well. Just a normal development\n> pattern, though. It's all manually done -- i.e., there's no automated\n> bot doing excessive git operations.\n\nOK, that is not very big. Once packed, you really should not see\nperformance issues.\n\n> culprit. Yes, network bandwidth is always a factor, but I guess my\n> expectation is that git shouldn't transfer too much more info than\n> the amount of recent changes. For example, if we change 10 files for\n> a total of 10MB, then my admittedly naive expectation is that git\n> will send that 10MB of changes, plus some small constant amount of\n> meta info... not the whole repo every time. No?\n\nYour assumption is correct. Git should transmit at _worst_ 10MB in such\na scenario (i.e., often much less because of delta compression, but in\nyour case of apparently-random media files, probably about 10MB).\n\nI wasn't clear from your message: you indicated the changes you made,\nbut are you still having performance problems, or are you still waiting\nto get data?\n\n-Peff\n"},{"id":"141880","messageId":"20100518192758.GC2383@coredump.intra.peff.net","threadId":"23786","inReplyTo":"AANLkTiltNPIR5gAWMOqZ2Y_azFUU93kH54ddHuCFFeCp@mail.gmail.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-18T19:27:58Z","receivedAt":"2010-05-18T19:27:58Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 18, 2010 at 09:10:58PM +0200, Sverre Rabbelier wrote:\n\n> On Tue, May 18, 2010 at 21:07, Jeff King <peff@peff.net> wrote:\n> > No, not to my knowledge. Even the \"binary\" attribute just says \"this\n> > file is binary, don't text diff it\". I think we will always still do\n> > rewrite-detection for operations like \"git status\" and the diff summary\n> > of \"git commit\".\n> \n> Would that not be a very sensible optimization that would help John\n> (and other users of big files) a lot?\n\nIt might help some, but I worry about overloading the meaning of\n\"-delta\". Right now it has a very clear meaning: don't delta for\npackfiles. But that doesn't mean I might not want to see break detection\n(or inexact rename detection, for that matter) at some time.\n\nLarge binary files shouldn't be taxing on regular diffs.  If you have\nmarked a file as \"binary\" and we are not creating a binary diff (i.e.,\njust printing \"binary files differ\"), then we shouldn't even need to\npull the blob from storage (since we can tell from the sha1 that it is\ndifferent). I haven't checked to see if we do that simple optimization\n(if you haven't marked it with a binary attribute, then obviously we do\nhave to look at the blob to find out that it is binary).\n\nSo:\n\n  1. I think it would need a separate attribute that is about diffing\n     (possibly even just options to a custom diff filter).\n\n  2. I am not clear exactly what options would work best. Do you want to\n     disable diffing entirely? Disable just inexact rename detection and\n     break detection? If break detection is disabled, do you assume it\n     is _always_ a rewrite, or never?\n\nSo I am open to the idea, but I think we would need a more concrete\nproposal and some timings to show how it is a benefit.\n\n-Peff\n"},{"id":"141881","messageId":"alpine.LFD.2.00.1005181528550.12758@xanadu.home","threadId":"23786","inReplyTo":"20100518191933.GB2383@coredump.intra.peff.net","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-05-18T19:33:58Z","receivedAt":"2010-05-18T19:33:58Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 18 May 2010, Jeff King wrote:\n\n> So you might want \"git gc --aggressive\" the _first_ time you pack, or\n> possibly even very occasionally. But if you are packing every day, you\n> should just use \"git gc\", which will run much more quickly (and would\n> probably have acceptable behavior even without the -delta attribute, as\n> it would only have to look at _new_ objects).\n> \n> It will have to write the whole 200M packfile out each time, though.\n\nNo.  gc will only create a pack with new loose objects by default.  \nOnly if the number of packs grow too large will it combine them into one \npack.\n\n> Packing nightly won't hurt, but is perhaps excessive. It sounds like you\n> actually have a fairly normal workload.\n\nPacking nightly with a simple \"git gc\" i.e. without extra options should \nbe perfectly fine.\n\n\nNicolas\n"},{"id":"141882","messageId":"alpine.LFD.2.00.1005181535130.12758@xanadu.home","threadId":"23786","inReplyTo":"20100518192758.GC2383@coredump.intra.peff.net","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-05-18T19:37:40Z","receivedAt":"2010-05-18T19:37:40Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 18 May 2010, Jeff King wrote:\n\n> On Tue, May 18, 2010 at 09:10:58PM +0200, Sverre Rabbelier wrote:\n> \n> > On Tue, May 18, 2010 at 21:07, Jeff King <peff@peff.net> wrote:\n> > > No, not to my knowledge. Even the \"binary\" attribute just says \"this\n> > > file is binary, don't text diff it\". I think we will always still do\n> > > rewrite-detection for operations like \"git status\" and the diff summary\n> > > of \"git commit\".\n> > \n> > Would that not be a very sensible optimization that would help John\n> > (and other users of big files) a lot?\n> \n> It might help some, but I worry about overloading the meaning of\n> \"-delta\". Right now it has a very clear meaning: don't delta for\n> packfiles. But that doesn't mean I might not want to see break detection\n> (or inexact rename detection, for that matter) at some time.\n\nIndeed. Please keep the delta attribute for what it is named after: \ndeltas. And those are meant to be used in the context of object packing \nonly.\n\n\nNicolas\n"},{"id":"141883","messageId":"20100518194105.GA4723@coredump.intra.peff.net","threadId":"23786","inReplyTo":"alpine.LFD.2.00.1005181528550.12758@xanadu.home","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-18T19:41:05Z","receivedAt":"2010-05-18T19:41:05Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 18, 2010 at 03:33:58PM -0400, Nicolas Pitre wrote:\n\n> > It will have to write the whole 200M packfile out each time, though.\n> \n> No.  gc will only create a pack with new loose objects by default.  \n> Only if the number of packs grow too large will it combine them into one \n> pack.\n\nI think that is only \"gc --auto\". With regular gc:\n\n  $ git init\n  $ echo content >file && git add file && git commit -m one\n  $ git gc\n  Counting objects: 3, done.\n  Writing objects: 100% (3/3), done.\n  Total 3 (delta 0), reused 0 (delta 0)\n  $ du -a .git/objects/pack\n  4  .git/objects/pack/pack-5f6fe4b14529d73f51d7c8efa69306edd35f2302.idx\n  4  .git/objects/pack/pack-5f6fe4b14529d73f51d7c8efa69306edd35f2302.pack\n  12 .git/objects/pack\n\n  $ echo content >>file && git commit -a -m two\n  $ git gc\n  Counting objects: 6, done.\n  Delta compression using up to 2 threads.\n  Compressing objects: 100% (2/2), done.\n  Writing objects: 100% (6/6), done.\n  Total 6 (delta 0), reused 3 (delta 0)\n  $ du -a .git/objects/pack\n  4  .git/objects/pack/pack-ecf41a1c120eb911f50fdd2c159e94d5832974f7.idx\n  4  .git/objects/pack/pack-ecf41a1c120eb911f50fdd2c159e94d5832974f7.pack\n  12 .git/objects/pack\n\nSo six objects written in the second gc, and obviously a brand new\nsingle pack.\n\n-Peff\n"},{"id":"141885","messageId":"alpine.LFD.2.00.1005181557250.12758@xanadu.home","threadId":"23786","inReplyTo":"20100518194105.GA4723@coredump.intra.peff.net","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-05-18T19:59:51Z","receivedAt":"2010-05-18T19:59:51Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 18 May 2010, Jeff King wrote:\n\n> On Tue, May 18, 2010 at 03:33:58PM -0400, Nicolas Pitre wrote:\n> \n> > > It will have to write the whole 200M packfile out each time, though.\n> > \n> > No.  gc will only create a pack with new loose objects by default.  \n> > Only if the number of packs grow too large will it combine them into one \n> > pack.\n> \n> I think that is only \"gc --auto\".\n\nArgh. You're right.  And \"gc --auto\" is already ran by many commands \nalready.\n\nIt is \"git repack\" that doesn't combine packs by default.\n\n\nNicolas\n"},{"id":"142171","messageId":"4BF9C678.6010108@puckerupgames.com","threadId":"23786","inReplyTo":"alpine.LFD.2.00.1005181557250.12758@xanadu.home","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"John","fromEmail":"john@puckerupgames.com","sentAt":"2010-05-24T00:21:12Z","receivedAt":"2010-05-24T00:21:12Z","isPatch":false,"sender":{"key":"john@puckerupgames.com","avatar":null},"body":"Just to follow up, the two solutions which have had a noticeable effect are, \nfirst to run daily `gc`s, and, second, to configure a \".gitattributes\" file as such:\n\n*.jpg  binary -delta\n*.png  binary -delta\n*.psd  binary -delta\n*.gz  binary -delta\n*.bz2  binary -delta\n.. and so on.\n\nOn my first go-round with \".gitattributes\" (earlier in this thread), my patterns \nwere setup incorrectly, as in,\n\n*.{gz,bz2,tgz,psd,png,jpg} binary -delta\n\nSince git does not perform brace expansion, the above patterns never matched. \nAfter revising the .gitattributes file, a ~6 minute gc dropped down to just \nunder ~3 minutes.\n\nIs there any reason why someone would NOT want the above \".gitattributes\" \ndefined by default?\n\n\n\n\nOn 05/18/2010 03:59 PM, Nicolas Pitre wrote:\n> On Tue, 18 May 2010, Jeff King wrote:\n>\n>> On Tue, May 18, 2010 at 03:33:58PM -0400, Nicolas Pitre wrote:\n>>\n>>>> It will have to write the whole 200M packfile out each time, though.\n>>>\n>>> No.  gc will only create a pack with new loose objects by default.\n>>> Only if the number of packs grow too large will it combine them into one\n>>> pack.\n>>\n>> I think that is only \"gc --auto\".\n>\n> Argh. You're right.  And \"gc --auto\" is already ran by many commands\n> already.\n>\n> It is \"git repack\" that doesn't combine packs by default.\n>\n>\n> Nicolas\n"},{"id":"142172","messageId":"7vaarq14me.fsf@alter.siamese.dyndns.org","threadId":"23786","inReplyTo":"4BF9C678.6010108@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-05-24T01:16:09Z","receivedAt":"2010-05-24T01:16:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John <john@puckerupgames.com> writes:\n\n> Is there any reason why someone would NOT want the above\n> \".gitattributes\" defined by default?\n\nOther than that our originally intended target audience are people who use\ngit as a source code control system, not much.\n"},{"id":"142173","messageId":"20100524053955.GA4528@coredump.intra.peff.net","threadId":"23786","inReplyTo":"4BF9C678.6010108@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-24T05:39:55Z","receivedAt":"2010-05-24T05:39:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, May 23, 2010 at 08:21:12PM -0400, John wrote:\n\n> *.jpg  binary -delta\n> *.png  binary -delta\n> *.psd  binary -delta\n> *.gz  binary -delta\n> *.bz2  binary -delta\n> .. and so on.\n>\n> [...]\n>\n> Is there any reason why someone would NOT want the above\n> \".gitattributes\" defined by default?\n\nI delta jpgs in one of my repositories. It is useful if the exif\nmetadata changes but the image data does not. I assume you could do the\nsame with other formats which have compressed and uncompressed portions\n(I also do it with video containers).  I don't think it would ever make\nsense to try to delta gzip'd or bzip'd contents.\n\nI also don't use \"binary\", as I use a custom diff driver instead (binary\nimplies \"-diff\").\n\nAs for what should be the default, until now the default has always\nbeen that no gitattributes are defined by default. This is nice because\nit's simple to understand; git doesn't care about filenames unless you\ntell it to. The downside obviously is that it may not perform optimally\nfor some unusual workloads without extra configuration.\n\nWe could probably do defaults for some common extensions, but I'm not\nreally sure where such a thing should end up. For example, I consider\n*.psd a uselessly obscure extension, as Adobe doesn't write software for\nmy platform of choice. Not that I mind having it in git, but rather that\nwe are inevitably going to miss somebody's pet extension, and then we\nare right back where we started with them needing to configure, except\nnow they also have to figure out which extensions have default\nattributes.\n\n-Peff\n"},{"id":"142175","messageId":"4BFA2054.6060002@puckerupgames.com","threadId":"23786","inReplyTo":"20100524053955.GA4528@coredump.intra.peff.net","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"John","fromEmail":"john@puckerupgames.com","sentAt":"2010-05-24T06:44:36Z","receivedAt":"2010-05-24T06:44:36Z","isPatch":false,"sender":{"key":"john@puckerupgames.com","avatar":null},"body":"I agree, no defaults are better than arbitrary defaults.  So why is the default \n\"text\"?\n\n\nOn 05/24/2010 01:39 AM, Jeff King wrote:\n> On Sun, May 23, 2010 at 08:21:12PM -0400, John wrote:\n>\n>> *.jpg  binary -delta\n>> *.png  binary -delta\n>> *.psd  binary -delta\n>> *.gz  binary -delta\n>> *.bz2  binary -delta\n>> .. and so on.\n>>\n>> [...]\n>>\n>> Is there any reason why someone would NOT want the above\n>> \".gitattributes\" defined by default?\n>\n> I delta jpgs in one of my repositories. It is useful if the exif\n> metadata changes but the image data does not. I assume you could do the\n> same with other formats which have compressed and uncompressed portions\n> (I also do it with video containers).  I don't think it would ever make\n> sense to try to delta gzip'd or bzip'd contents.\n>\n> I also don't use \"binary\", as I use a custom diff driver instead (binary\n> implies \"-diff\").\n>\n> As for what should be the default, until now the default has always\n> been that no gitattributes are defined by default. This is nice because\n> it's simple to understand; git doesn't care about filenames unless you\n> tell it to. The downside obviously is that it may not perform optimally\n> for some unusual workloads without extra configuration.\n>\n> We could probably do defaults for some common extensions, but I'm not\n> really sure where such a thing should end up. For example, I consider\n> *.psd a uselessly obscure extension, as Adobe doesn't write software for\n> my platform of choice. Not that I mind having it in git, but rather that\n> we are inevitably going to miss somebody's pet extension, and then we\n> are right back where we started with them needing to configure, except\n> now they also have to figure out which extensions have default\n> attributes.\n>\n> -Peff\n"},{"id":"142176","messageId":"20100524064554.GA12262@coredump.intra.peff.net","threadId":"23786","inReplyTo":"4BFA2054.6060002@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-24T06:45:54Z","receivedAt":"2010-05-24T06:45:54Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 24, 2010 at 02:44:36AM -0400, John wrote:\n\n> I agree, no defaults are better than arbitrary defaults.  So why is\n> the default \"text\"?\n\nBecause git was designed as a source control system?\n\n-Peff\n"},{"id":"142178","messageId":"4BFA2451.6090507@puckerupgames.com","threadId":"23786","inReplyTo":"7vaarq14me.fsf@alter.siamese.dyndns.org","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"John","fromEmail":"john@puckerupgames.com","sentAt":"2010-05-24T07:01:37Z","receivedAt":"2010-05-24T07:01:37Z","isPatch":false,"sender":{"key":"john@puckerupgames.com","avatar":null},"body":"Ok, fair enough. It's your project, and you are defining \"source control\" as \nthat which git supports: non-binary, line-by-line text only, C, bash .. no \nimages, documents, etc.\n\nI only wish that definition of \"source\" had been more clear from the get-go.\n\nPerhaps a front and center blurb on the git home page or mission statement might \nclarify things for those of us who have different definitions of \"source\"?  That \nway, you wouldn't have to be bothered by folks trying to version all their \nproject assets with git. For example, you could specify that non-text is out of \nscope for git, (or however you wish to define \"source\").\n\n\nOn 05/23/2010 09:16 PM, Junio C Hamano wrote:\n> John<john@puckerupgames.com>  writes:\n>\n>> Is there any reason why someone would NOT want the above\n>> \".gitattributes\" defined by default?\n>\n> Other than that our originally intended target audience are people who use\n> git as a source code control system, not much.\n>\n"},{"id":"142237","messageId":"20100525063354.GC10898@coredump.intra.peff.net","threadId":"23786","inReplyTo":"4BFA2451.6090507@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-05-25T06:33:54Z","receivedAt":"2010-05-25T06:33:54Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 24, 2010 at 03:01:37AM -0400, John wrote:\n\n> Ok, fair enough. It's your project, and you are defining \"source\n> control\" as that which git supports: non-binary, line-by-line text\n> only, C, bash .. no images, documents, etc.\n>\n> I only wish that definition of \"source\" had been more clear from the get-go.\n\nI think both Junio's and my responses were not \"we can't do a better job\nwith non-text sources\" but rather \"this is how we ended up with the\ncurrent state\".\n\nI'll admit I have some reservations about trying to figure out a sane\nset of extensions for default .gitattributes, but that doesn't mean you\ncan't try if you want to. :)\n\n> Perhaps a front and center blurb on the git home page or mission\n> statement might clarify things for those of us who have different\n> definitions of \"source\"?  That way, you wouldn't have to be bothered\n> by folks trying to version all their project assets with git. For\n> example, you could specify that non-text is out of scope for git, (or\n> however you wish to define \"source\").\n\nI don't know that we want to explicitly discourage such use. Obviously\ncertain workflows don't work as well with randomly-changing binary blobs\n(e.g., reading format-patch output is next to useless, though it does\nstill work as a transport if your project relies on emailed patch\nsubmissions).\n\nIn general, I think we are happy to take patches making binary storage\nmore pleasant (e.g., textconv) as long as they don't somehow make the\n\"normal\" case of text worse. There are some things for which git is\nsimply not well suited (single files in the gigabytes, for example), and\nthose aren't likely to change because the some of the issues are\nfundamental to how git works (though there are often workarounds, like\nputting gigantic files in their own individual packs). But certainly\n100M of jpgs does not seem like an unusable workload to me (as I\nmentioned, I have a several-gigabyte photo repository that git does just\nfine with).\n\n-Peff\n"},{"id":"142248","messageId":"4BFB7C24.10609@drmicha.warpmail.net","threadId":"23786","inReplyTo":"7vaarq14me.fsf@alter.siamese.dyndns.org","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2010-05-25T07:28:36Z","receivedAt":"2010-05-25T07:28:36Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Junio C Hamano venit, vidit, dixit 24.05.2010 03:16:\n> John <john@puckerupgames.com> writes:\n> \n>> Is there any reason why someone would NOT want the above\n>> \".gitattributes\" defined by default?\n> \n> Other than that our originally intended target audience are people who use\n> git as a source code control system, not much.\n> \n\nand other than that many people use clean/smudge filters to make git\nhappily and efficiently deltify compressed file formats (such as gz,\nbz2, zip) and still keep compressed checkouts...\n\nand other than that which you (plural) and I are not thinking of right now.\n\nLet the defaults be as they are (fit for source control in the proper\nsense), it's easy enough to change them for other use cases.\n\nMichael\n"},{"id":"142284","messageId":"4BFBF701.6060901@puckerupgames.com","threadId":"23786","inReplyTo":"4BFB7C24.10609@drmicha.warpmail.net","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"John","fromEmail":"john@puckerupgames.com","sentAt":"2010-05-25T16:12:49Z","receivedAt":"2010-05-25T16:12:49Z","isPatch":false,"sender":{"key":"john@puckerupgames.com","avatar":null},"body":"On 05/25/2010 03:28 AM, Michael J Gruber wrote:\n> Junio C Hamano venit, vidit, dixit 24.05.2010 03:16:\n>> John<john@puckerupgames.com>  writes:\n>>\n>>> Is there any reason why someone would NOT want the above\n>>> \".gitattributes\" defined by default?\n>>\n>> Other than that our originally intended target audience are people who use\n>> git as a source code control system, not much.\n>>\n>\n> and other than that many people use clean/smudge filters to make git\n> happily and efficiently deltify compressed file formats (such as gz,\n> bz2, zip) and still keep compressed checkouts...\n>\n> and other than that which you (plural) and I are not thinking of right now.\n>\n> Let the defaults be as they are (fit for source control in the proper\n> sense), it's easy enough to change them for other use cases.\n\nThat's fine. We all have different ideas what revision control means. So long as it's clear what git \nconsiders \"source\" and what it considers out of scope, what the defaults are, and what the \nlimitations are, potential users can more fairly evaluate git to see if it fits their needs.\n\nFor example, code libraries and shell utilities may not require anything more complicated than \nline-by-line text-based patches in revision control.\n\nOn the other hand, projects such as web sites, mobile phone apps, desktop applications, (and games \n:) have lots of \"source\" that is not code.  Even XML, which is text-based, but not line-based (and \nneed not contain any newlines), may present a problem for git in this respect.\n\nPerhaps a section in the manual with a header such as \"Handling non-text files\", or \"Revision \ncontrol for media, XML, and other non line-oriented files\" would clear this all up. You could almost \ncull the body of it from this thread and other similar threads.\n"},{"id":"142288","messageId":"alpine.LFD.2.00.1005251317550.24526@xanadu.home","threadId":"23786","inReplyTo":"4BFBF701.6060901@puckerupgames.com","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-05-25T17:18:49Z","receivedAt":"2010-05-25T17:18:49Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 25 May 2010, John wrote:\n\n> Perhaps a section in the manual with a header such as \"Handling non-text\n> files\", or \"Revision control for media, XML, and other non line-oriented\n> files\" would clear this all up. You could almost cull the body of it from this\n> thread and other similar threads.\n\nThat is indeed a good idea.\n\nDo you volunteer?\n\n\nNicolas\n"},{"id":"142290","messageId":"4BFC0D3B.4010804@puckerupgames.com","threadId":"23786","inReplyTo":"alpine.LFD.2.00.1005251317550.24526@xanadu.home","subject":"Re: serious performance issues with images, audio files, and other \"non-code\" data","fromName":"John","fromEmail":"john@puckerupgames.com","sentAt":"2010-05-25T17:47:39Z","receivedAt":"2010-05-25T17:47:39Z","isPatch":false,"sender":{"key":"john@puckerupgames.com","avatar":null},"body":"On 05/25/2010 01:18 PM, Nicolas Pitre wrote:\n> On Tue, 25 May 2010, John wrote:\n>\n>> Perhaps a section in the manual with a header such as \"Handling non-text\n>> files\", or \"Revision control for media, XML, and other non line-oriented\n>> files\" would clear this all up. You could almost cull the body of it from this\n>> thread and other similar threads.\n>\n> That is indeed a good idea.\n>\n> Do you volunteer?\n\nYes, of course. If y'all are not up to it, I'd be happy to give it a shot.\n"}]}