{"thread":{"id":"19525","subject":"Problem with large files on different OSes","startedAt":"2009-05-27T10:52:12Z","lastAt":"2009-05-28T21:21:06Z","messageCount":31,"participants":["Christopher Jefferson","Andreas Ericsson","John Tapsell","Tomas Carnecky","Jakub Narebski","Linus Torvalds","Nicolas Pitre","Jeff King","Alan Manuel Gloria","Eric Raible","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"114792","messageId":"submission.1M9Gk0-0000N8-MQ@mail.cs.st-andrews.ac.uk","threadId":"19525","inReplyTo":null,"subject":"Problem with large files on different OSes","fromName":"Christopher Jefferson","fromEmail":"caj@cs.st-andrews.ac.uk","sentAt":"2009-05-27T10:52:12Z","receivedAt":"2009-05-27T10:52:12Z","isPatch":false,"sender":{"key":"caj@cs.st-andrews.ac.uk","avatar":null},"body":"I recently came across a very annoying problem, characterised by the \nfollowing example:\n\nOn a recent ubuntu install:\n\ndd if=/dev/zero of=file bs=1300k count=1k\ngit commit file -m \"Add huge file\"\n\n\nThe repository can be pulled and pushed successfully to other ubuntu \ninstalls, but on Mac OS X, 10.5.7 machine with 4GB ram git pull \nproduces:\n\nremote: Counting objects: 6, done.\nremote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) failed \n(error code=12)\nremote: *** error: can't allocate region\nremote: *** set a breakpoint in malloc_error_break to debug\nremote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) failed \n(error code=12)\nremote: *** error: can't allocate region\nremote: *** set a breakpoint in malloc_error_break to debug\nremote: fatal: Out of memory, malloc failed\nerror: git upload-pack: git-pack-objects died with error.\nfatal: git upload-pack: aborting due to possible repository corruption \non the remote side.\nremote: aborting due to possible repository corruption on the remote \nside.\nfatal: protocol error: bad pack header\n\n\nThe problem appears to be the different maximum mmap sizes available \non different OSes. Whic I don't really mind the maximum file size \nrestriction git imposes, this restriction varying from OS to OS is \nvery annoying, fixing this required rewriting history to remove the \ncommit, which caused problems as the commit had already been pulled, \nand built on, by a number of developers.\n\nIf the requirement that all files can be mmapped cannot be easily \nremoved, would be it perhaps be acceptable to impose a (soft?) \n1GB(ish) file size limit? I suggest 1GB as all the OSes I can get hold \nof easily (freeBSD, windows, Mac OS X, linux) support a mmap of size > \n1GB.\n"},{"id":"114794","messageId":"4A1D2614.5060303@op5.se","threadId":"19525","inReplyTo":"submission.1M9Gk0-0000N8-MQ@mail.cs.st-andrews.ac.uk","subject":"Re: Problem with large files on different OSes","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-05-27T11:37:56Z","receivedAt":"2009-05-27T11:37:56Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Christopher Jefferson wrote:\n> I recently came across a very annoying problem, characterised by the \n> following example:\n> \n> On a recent ubuntu install:\n> \n> dd if=/dev/zero of=file bs=1300k count=1k\n> git commit file -m \"Add huge file\"\n> \n> \n> The repository can be pulled and pushed successfully to other ubuntu \n> installs, but on Mac OS X, 10.5.7 machine with 4GB ram git pull produces:\n> \n> remote: Counting objects: 6, done.\n> remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) failed \n> (error code=12)\n> remote: *** error: can't allocate region\n> remote: *** set a breakpoint in malloc_error_break to debug\n> remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) failed \n> (error code=12)\n> remote: *** error: can't allocate region\n> remote: *** set a breakpoint in malloc_error_break to debug\n> remote: fatal: Out of memory, malloc failed\n> error: git upload-pack: git-pack-objects died with error.\n> fatal: git upload-pack: aborting due to possible repository corruption \n> on the remote side.\n> remote: aborting due to possible repository corruption on the remote side.\n> fatal: protocol error: bad pack header\n> \n> \n> The problem appears to be the different maximum mmap sizes available on \n> different OSes. Whic I don't really mind the maximum file size \n> restriction git imposes, this restriction varying from OS to OS is very \n> annoying, fixing this required rewriting history to remove the commit, \n> which caused problems as the commit had already been pulled, and built \n> on, by a number of developers.\n> \n> If the requirement that all files can be mmapped cannot be easily \n> removed, would be it perhaps be acceptable to impose a (soft?) 1GB(ish) \n> file size limit?\n\nMost definitely not. Why should we limit a cross-platform system for\nthe benefit of one particular developer's lacking hardware?\n\nSuch a convention should, if anything, be enforced by social policy,\nbut not by the tool itself.\n\nOtherwise, why not just restrict the tool that created the huge file\nso that it makes smaller files that fit into git on all platforms\ninstead? (No, that wasn't a real suggestion. It was just to make the\npoint that your suggestion for git to impose artificial limits is\nequally ludicrous)\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"114795","messageId":"submission.1M9Im6-0003Hs-40@mail.cs.st-andrews.ac.uk","threadId":"19525","inReplyTo":"4A1D2614.5060303@op5.se","subject":"Re: Problem with large files on different OSes","fromName":"Christopher Jefferson","fromEmail":"caj@cs.st-andrews.ac.uk","sentAt":"2009-05-27T13:02:29Z","receivedAt":"2009-05-27T13:02:29Z","isPatch":false,"sender":{"key":"caj@cs.st-andrews.ac.uk","avatar":null},"body":"\nOn 27 May 2009, at 12:37, Andreas Ericsson wrote:\n\n> Christopher Jefferson wrote:\n>> I recently came across a very annoying problem, characterised by \n>> the following example:\n>> On a recent ubuntu install:\n>> dd if=/dev/zero of=file bs=1300k count=1k\n>> git commit file -m \"Add huge file\"\n>> The repository can be pulled and pushed successfully to other \n>> ubuntu installs, but on Mac OS X, 10.5.7 machine with 4GB ram git \n>> pull produces:\n>> remote: Counting objects: 6, done.\n>> remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) \n>> failed (error code=12)\n>> remote: *** error: can't allocate region\n>> remote: *** set a breakpoint in malloc_error_break to debug\n>> remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) \n>> failed (error code=12)\n>> remote: *** error: can't allocate region\n>> remote: *** set a breakpoint in malloc_error_break to debug\n>> remote: fatal: Out of memory, malloc failed\n>> error: git upload-pack: git-pack-objects died with error.\n>> fatal: git upload-pack: aborting due to possible repository \n>> corruption on the remote side.\n>> remote: aborting due to possible repository corruption on the \n>> remote side.\n>> fatal: protocol error: bad pack header\n>> The problem appears to be the different maximum mmap sizes \n>> available on different OSes. Whic I don't really mind the maximum \n>> file size restriction git imposes, this restriction varying from OS \n>> to OS is very annoying, fixing this required rewriting history to \n>> remove the commit, which caused problems as the commit had already \n>> been pulled, and built on, by a number of developers.\n>> If the requirement that all files can be mmapped cannot be easily \n>> removed, would be it perhaps be acceptable to impose a (soft?) \n>> 1GB(ish) file size limit?\n>\n> Most definitely not. Why should we limit a cross-platform system for\n> the benefit of one particular developer's lacking hardware?\n\nOut of curiosity, why do you say lacking hardware? I am running \nubuntu, windows and Mac OS X on exactly the same machine, which is not \nrunning out of physical memory, never mind swap, when using git on any \nOS. The problem is purely a software (and OS) problem.\n\n\nChris\n"},{"id":"114796","messageId":"43d8ce650905270628o2eb4ed3cqf6b78e00933198af@mail.gmail.com","threadId":"19525","inReplyTo":"4A1D2614.5060303@op5.se","subject":"Re: Problem with large files on different OSes","fromName":"John Tapsell","fromEmail":"johnflux@gmail.com","sentAt":"2009-05-27T13:28:58Z","receivedAt":"2009-05-27T13:28:58Z","isPatch":false,"sender":{"key":"johnflux@gmail.com","avatar":"https://gravatar.com/avatar/25f70d4c0f96396b84a2e34bcd9bdc233462c7b4be29b5fdca8266fc53f30b0c?d=mp&s=160"},"body":"2009/5/27 Andreas Ericsson :\n> Christopher Jefferson wrote:\n>> If the requirement that all files can be mmapped cannot be easily removed,\n>> would be it perhaps be acceptable to impose a (soft?) 1GB(ish) file size\n>> limit?\n>\n> Most definitely not. Why should we limit a cross-platform system for\n> the benefit of one particular developer's lacking hardware?\n\nPerhaps a simple warning would suffice \"Warning: Files larger than\n2GB may cause problems when trying to checkout on Windows.\"\n\nJohn\n"},{"id":"114797","messageId":"submission.1M9JCt-00047m-4a@mail.cs.st-andrews.ac.uk","threadId":"19525","inReplyTo":"43d8ce650905270628o2eb4ed3cqf6b78e00933198af@mail.gmail.com","subject":"Re: Problem with large files on different OSes","fromName":"Christopher Jefferson","fromEmail":"caj@cs.st-andrews.ac.uk","sentAt":"2009-05-27T13:30:10Z","receivedAt":"2009-05-27T13:30:10Z","isPatch":false,"sender":{"key":"caj@cs.st-andrews.ac.uk","avatar":null},"body":"\nOn 27 May 2009, at 14:28, John Tapsell wrote:\n\n> 2009/5/27 Andreas Ericsson :\n>> Christopher Jefferson wrote:\n>>> If the requirement that all files can be mmapped cannot be easily \n>>> removed,\n>>> would be it perhaps be acceptable to impose a (soft?) 1GB(ish) \n>>> file size\n>>> limit?\n>>\n>> Most definitely not. Why should we limit a cross-platform system for\n>> the benefit of one particular developer's lacking hardware?\n>\n> Perhaps a simple warning would suffice \"Warning: Files larger than\n> 2GB may cause problems when trying to checkout on Windows.\"\n>\n\nSomething like that, except that limit seems to be only 1.3GB on Mac \nOS X\n\nChris\n"},{"id":"114798","messageId":"43d8ce650905270632t2a2914b4j340a5cfd756023a4@mail.gmail.com","threadId":"19525","inReplyTo":"submission.1M9JCt-00047m-4a@mail.cs.st-andrews.ac.uk","subject":"Re: Problem with large files on different OSes","fromName":"John Tapsell","fromEmail":"johnflux@gmail.com","sentAt":"2009-05-27T13:32:40Z","receivedAt":"2009-05-27T13:32:40Z","isPatch":false,"sender":{"key":"johnflux@gmail.com","avatar":"https://gravatar.com/avatar/25f70d4c0f96396b84a2e34bcd9bdc233462c7b4be29b5fdca8266fc53f30b0c?d=mp&s=160"},"body":"2009/5/27 Christopher Jefferson :\n> Something like that, except that limit seems to be only 1.3GB on Mac OS X\n\nDoes linux have a similar limitation, lower than the limit imposed by\nthe filesystem?\nCould this be solved by having a fallback solution for mmap?\n(switching to opening the file normally) Or would this fallback be\ntoo intrusive/large of a change?\n\nJohn\n"},{"id":"114801","messageId":"C933C1BA-0170-4469-8099-2C0E95C61D24@dbservice.com","threadId":"19525","inReplyTo":"submission.1M9Gk0-0000N8-MQ@mail.cs.st-andrews.ac.uk","subject":"Re: Problem with large files on different OSes","fromName":"Tomas Carnecky","fromEmail":"tom@dbservice.com","sentAt":"2009-05-27T14:01:57Z","receivedAt":"2009-05-27T14:01:57Z","isPatch":false,"sender":{"key":"tom@dbservice.com","avatar":"https://gravatar.com/avatar/900a300bdd1a8bbe086008ad78210bbee2ad2803b7d50a5cba04c1e9404bd6d2?d=mp&s=160"},"body":"\nOn May 27, 2009, at 12:52 PM, Christopher Jefferson wrote:\n\n> I recently came across a very annoying problem, characterised by the \n> following example:\n>\n> On a recent ubuntu install:\n>\n> dd if=/dev/zero of=file bs=1300k count=1k\n> git commit file -m \"Add huge file\"\n>\n>\n> The repository can be pulled and pushed successfully to other ubuntu \n> installs, but on Mac OS X, 10.5.7 machine with 4GB ram git pull \n> produces:\n>\n> remote: Counting objects: 6, done.\n> remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) \n> failed (error code=12)\n> remote: *** error: can't allocate region\n> remote: *** set a breakpoint in malloc_error_break to debug\n> remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) \n> failed (error code=12)\n> remote: *** error: can't allocate region\n> remote: *** set a breakpoint in malloc_error_break to debug\n> remote: fatal: Out of memory, malloc failed\n> error: git upload-pack: git-pack-objects died with error.\n> fatal: git upload-pack: aborting due to possible repository \n> corruption on the remote side.\n> remote: aborting due to possible repository corruption on the remote \n> side.\n> fatal: protocol error: bad pack header\n>\n>\n> The problem appears to be the different maximum mmap sizes available \n> on different OSes. Whic I don't really mind the maximum file size \n> restriction git imposes, this restriction varying from OS to OS is \n> very annoying, fixing this required rewriting history to remove the \n> commit, which caused problems as the commit had already been pulled, \n> and built on, by a number of developers.\n>\n> If the requirement that all files can be mmapped cannot be easily \n> removed, would be it perhaps be acceptable to impose a (soft?) \n> 1GB(ish) file size limit? I suggest 1GB as all the OSes I can get \n> hold of easily (freeBSD, windows, Mac OS X, linux) support a mmap of \n> size > 1GB.\n\nI think this is a limitation of a 32bit build of git. I just tried \nwith a 64bit build and it added the file just fine. The compiler on \nMacOSX (gcc) produces 32bit builds by default, even if the system \nsupports 64bit executables. But gcc on 64bit Linux (at least the \ninstallations I have at home) produces a 64bit executables by default. \nSolaris/OpenSolaris behaves like MacOSX, no idea about *BSD or \nWindows. Maybe this is why git works on Linux but not MacOSX even on \nthe same hardware.\nBtw, I built git with: make install prefix=... CC=\"gcc -m64\", no \nmodifications needed (MacOSX 10.5.7).\n\ntom\n"},{"id":"114803","messageId":"submission.1M9JpE-0005AW-92@mail.cs.st-andrews.ac.uk","threadId":"19525","inReplyTo":"C933C1BA-0170-4469-8099-2C0E95C61D24@dbservice.com","subject":"Re: Problem with large files on different OSes","fromName":"Christopher Jefferson","fromEmail":"caj@cs.st-andrews.ac.uk","sentAt":"2009-05-27T14:09:47Z","receivedAt":"2009-05-27T14:09:47Z","isPatch":false,"sender":{"key":"caj@cs.st-andrews.ac.uk","avatar":null},"body":"\nOn 27 May 2009, at 15:01, Tomas Carnecky wrote:\n\n>>\n>> The problem appears to be the different maximum mmap sizes \n>> available on different OSes. Whic I don't really mind the maximum \n>> file size restriction git imposes, this restriction varying from OS \n>> to OS is very annoying, fixing this required rewriting history to \n>> remove the commit, which caused problems as the commit had already \n>> been pulled, and built on, by a number of developers.\n>>\n>> If the requirement that all files can be mmapped cannot be easily \n>> removed, would be it perhaps be acceptable to impose a (soft?) \n>> 1GB(ish) file size limit? I suggest 1GB as all the OSes I can get \n>> hold of easily (freeBSD, windows, Mac OS X, linux) support a mmap \n>> of size > 1GB.\n>\n> I think this is a limitation of a 32bit build of git. I just tried \n> with a 64bit build and it added the file just fine. The compiler on \n> MacOSX (gcc) produces 32bit builds by default, even if the system \n> supports 64bit executables. But gcc on 64bit Linux (at least the \n> installations I have at home) produces a 64bit executables by \n> default. Solaris/OpenSolaris behaves like MacOSX, no idea about *BSD \n> or Windows. Maybe this is why git works on Linux but not MacOSX even \n> on the same hardware.\n> Btw, I built git with: make install prefix=... CC=\"gcc -m64\", no \n> modifications needed (MacOSX 10.5.7).\n\nThe git installs I am using are all 32bit, this machine doesn't have a \n64bit processor (it is one of the few macs released without one). It's \nnice to know long term this problem will go away, that all suggests \nintroducing some limit is not approriate, as while 32bit users have \nsome arbitary limit above which they cannot go, I am sure all 64-bit \nOSes will manage to easily mmap any file. Of course warning such users \nthey are producing packs that are not going to work on 32bit compiles \nof git isn't a stupid idea.\n\nChris\n"},{"id":"114804","messageId":"4A1D4CBE.1010905@op5.se","threadId":"19525","inReplyTo":"submission.1M9JpE-0005AW-92@mail.cs.st-andrews.ac.uk","subject":"Re: Problem with large files on different OSes","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-05-27T14:22:54Z","receivedAt":"2009-05-27T14:22:54Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Christopher Jefferson wrote:\n> \n> On 27 May 2009, at 15:01, Tomas Carnecky wrote:\n> \n>>>\n>>> The problem appears to be the different maximum mmap sizes available \n>>> on different OSes. Whic I don't really mind the maximum file size \n>>> restriction git imposes, this restriction varying from OS to OS is \n>>> very annoying, fixing this required rewriting history to remove the \n>>> commit, which caused problems as the commit had already been pulled, \n>>> and built on, by a number of developers.\n>>>\n>>> If the requirement that all files can be mmapped cannot be easily \n>>> removed, would be it perhaps be acceptable to impose a (soft?) \n>>> 1GB(ish) file size limit? I suggest 1GB as all the OSes I can get \n>>> hold of easily (freeBSD, windows, Mac OS X, linux) support a mmap of \n>>> size > 1GB.\n>>\n>> I think this is a limitation of a 32bit build of git. I just tried \n>> with a 64bit build and it added the file just fine. The compiler on \n>> MacOSX (gcc) produces 32bit builds by default, even if the system \n>> supports 64bit executables. But gcc on 64bit Linux (at least the \n>> installations I have at home) produces a 64bit executables by default. \n>> Solaris/OpenSolaris behaves like MacOSX, no idea about *BSD or \n>> Windows. Maybe this is why git works on Linux but not MacOSX even on \n>> the same hardware.\n>> Btw, I built git with: make install prefix=... CC=\"gcc -m64\", no \n>> modifications needed (MacOSX 10.5.7).\n> \n> The git installs I am using are all 32bit, this machine doesn't have a \n> 64bit processor (it is one of the few macs released without one). It's \n> nice to know long term this problem will go away, that all suggests \n> introducing some limit is not approriate, as while 32bit users have some \n> arbitary limit above which they cannot go, I am sure all 64-bit OSes \n> will manage to easily mmap any file. Of course warning such users they \n> are producing packs that are not going to work on 32bit compiles of git \n> isn't a stupid idea.\n> \n\nmmap()'ing large files (> 4GB) work just fine on Linux. You can't mmap()\nmore than 4GB at a time though (I think; I didn't try), but since we\ndon't do that anyway I doubt that was the problem.\n\nThe file you produced with your dd command should have ended up being\n1239MB, or 1.21GB, so the real hard limit for MacOSX seem to be 1GB\nif, indeed, there is one. On the other hand, the error message you\ngot (\"fatal: Out of memory, malloc failed\") seems to indicate the\nsystem actually had no memory left when you tried to garbage-collect\nyour repository. Are you using a dual-core system? If so, please try\nagain with\n\n pack.threads = 1\n\nset in the .git/config file of that particular repository. Each\nthread will allocate roughly the same amount of memory, so if\nboth of them had to handle that huge blob at the same time, they'd\nhave exploded memory usage up to 1.3GB + the compressed size of\nthem + DAG-bookkeeping etc etc.\n\nI'm guessing we'd have seen error reports from other OSX users\nif it was actually impossible to mmap() 1GB files in git on OSX.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"114805","messageId":"m3y6siboij.fsf@localhost.localdomain","threadId":"19525","inReplyTo":"submission.1M9Gk0-0000N8-MQ@mail.cs.st-andrews.ac.uk","subject":"Re: Problem with large files on different OSes","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-27T14:37:33Z","receivedAt":"2009-05-27T14:37:33Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Christopher Jefferson writes:\n\n> I recently came across a very annoying problem, characterised by the\n> following example:\n> \n> On a recent ubuntu install:\n> \n> dd if=/dev/zero of=file bs=1300k count=1k\n> git commit file -m \"Add huge file\"\n> \n> \n> The repository can be pulled and pushed successfully to other ubuntu\n> installs, but on Mac OS X, 10.5.7 machine with 4GB ram git pull\n> produces:\n\nDo seting `pack.packSizeLimit`, or adjusting values of\n`core.packedGitWindowSize` and/or `core.packedGitLimit`\n(see git-config(1)) help in your situation?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"114811","messageId":"alpine.LFD.2.01.0905270922250.3435@localhost.localdomain","threadId":"19525","inReplyTo":"m3y6siboij.fsf@localhost.localdomain","subject":"Re: Problem with large files on different OSes","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-27T16:30:18Z","receivedAt":"2009-05-27T16:30:18Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 27 May 2009, Jakub Narebski wrote:\n> \n> Do seting `pack.packSizeLimit`, or adjusting values of\n> `core.packedGitWindowSize` and/or `core.packedGitLimit`\n> (see git-config(1)) help in your situation?\n\nNo, that will help just the packfile mmap (and even there, it won't help \nwith things like index file size - we'll always mmap the whole index \nfile). It's definitely worth doing, though - but I think we already \ndefault to 32MB pack-file windows on 32-bit architectures.\n\nIndividual files we always handle in one go. It's what git was designed \nfor, after all - fairly small files. And so git is limited to files \nsmaller than the virtual address space.\n\nOn a 32-bit setup, that often limits you to roughly a gigabyte. You have \n4GB of virtual address space, of which one or two is used for the OS \nkernel. So say you have 2GB for user mode - you then have the executable \nmapping and libraries and stack, all spread out in that 2GB virtual \naddress space.\n\nIn fact, even if it's 3GB for user (I don't know what OS X does), getting \none contiguous area may well be limited to ~1GB depending on layout of \nshared library mappings etc VM fragmentation. Older Linux systems tended \nto map things in ways that made it hard to get more than 1GB of contiguous \ndata mapping if you compiled with dynamic libraries.\n\n64-bit mostly makes this a non-issue.\n\nIn fact, if you do \"diff\", you're going to be even _more_ limited, since \nfor simplicity, our version of xdiff really wants both sources in memory \nat a time. So you can't really diff >500MB files. Not that you generally \nwant to, of course.\n\nI'll see if I can make us handle the \"big file without diff\" case better \nby chunking.\n\n\t\tLinus\n"},{"id":"114813","messageId":"alpine.LFD.2.01.0905270942580.3435@localhost.localdomain","threadId":"19525","inReplyTo":"alpine.LFD.2.01.0905270922250.3435@localhost.localdomain","subject":"Re: Problem with large files on different OSes","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-27T16:59:21Z","receivedAt":"2009-05-27T16:59:21Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 27 May 2009, Linus Torvalds wrote:\n> \n> I'll see if I can make us handle the \"big file without diff\" case better \n> by chunking.\n\nHmm. No. Looking at it some more, we could add some nasty code to do \n_some_ things chunked (like adding a new file as a single object), but it \ndoesn't really help. For any kind of useful thing, we'd need to handle the \n\"read from pack\" case in multiple chunks too, and that gets really nasty \nreally quickly.\n\nThe whole \"each object as one allocation\" design is pretty core, and it \nlooks pointless to have a few special cases, when any actual relevant use \nwould need a whole lot more than the few simple ones.\n\nGit really doesn't like big individual objects.\n\nI've occasionally thought about handling big files as multiple big \nobjects: we'd split them into a \"pseudo-directory\" (it would have some new \nobject ID), and then treat them as a magical special kind of directory \nthat just happens to be represented as one large file on the filesystem.\n\nThat would mean that if you have a huge file, git internally would never \nthink of it as one big file, but as a collection of many smaller objects. \nBy just making the point where you break up files be a consistent rule \n(\"always break into 256MB pieces\"), it would be a well-behaved design (ie \nthings like behaviour convergence wrt the same big file being created \ndifferent ways).\n\nHOWEVER.\n\nWhile that would fit in the git design (ie it would be just a fairly \nstraightforward extension - another level of indirection, kind of the way \nwe added subprojects), it would still be a rewrite of some core stuff. The \nactual number of lines might not be too horrid, but quite frankly, I \nwouldn't want to do it personally. It would be a lot of work with lots of \ncareful special case handling - and no real upside for normal use.\n\nSo I'm kind of down on it. I would suggest just admitting that git isn't \nvery good at big individual files - especially not if you have a limited \naddress space.\n\nSo \"don't do it then\" or \"make sure you are 64-bit and have lots of \nmemory if you do it\" may well be the right solution.\n\n[ And it's really really sad how Apple migrated to x86-32. It was totally \n unforgivably stupid, and I said so at the time. When Apple did the \n PowerPC -> x86 transition, they should have just transitioned to x86-64, \n and never had a 32-bit space.\n\n But Apple does stupid things, that seem to be driven by marketing rather \n than thinking deeply about the technology, and now they basically _have_ \n to default to that 32-bit environment. ]\n\nOh well. \n\n\t\t\tLinus\n"},{"id":"114815","messageId":"submission.1M9Mpl-0001CW-M0@mail.cs.st-andrews.ac.uk","threadId":"19525","inReplyTo":"alpine.LFD.2.01.0905270942580.3435@localhost.localdomain","subject":"Re: Problem with large files on different OSes","fromName":"Christopher Jefferson","fromEmail":"caj@cs.st-andrews.ac.uk","sentAt":"2009-05-27T17:22:33Z","receivedAt":"2009-05-27T17:22:33Z","isPatch":false,"sender":{"key":"caj@cs.st-andrews.ac.uk","avatar":null},"body":"\nOn 27 May 2009, at 17:59, Linus Torvalds wrote:\n\n>\n>\n> On Wed, 27 May 2009, Linus Torvalds wrote:\n>>\n>> I'll see if I can make us handle the \"big file without diff\" case \n>> better\n>> by chunking.\n\n> So \"don't do it then\" or \"make sure you are 64-bit and have lots of\n> memory if you do it\" may well be the right solution.\n\nThank you for that description of the problem, I can see how hard it is.\n\nPerhaps it might be useful to think about how to codify \"don't do it \nthen\" in a reasonably simple, automatic way?\n\nI've been trying to write a pre-commit hook (I think that's the right \nplace?) which would refuse commits larger than some file size (512MB \nas a random number I decided), but am having trouble getting it to \nwork right, and generally. Would such a thing be easy, and would that \nbe the right place to put it?\n\nWhile I wouldn't suggest this become default, providing such a hook, \nand describing why you might want to use it, would seem to avoid the \naccidental part of the problem.\n\nOf course, people should really notice that they are submitting large \nfiles, but it's easy(ish) to commit some output file from a program, \nwithout realising the file ended up being the wrong side of 1GB.\n\nChris\n"},{"id":"114816","messageId":"200905271930.30496.jnareb@gmail.com","threadId":"19525","inReplyTo":"submission.1M9Mpl-0001CW-M0@mail.cs.st-andrews.ac.uk","subject":"Re: Problem with large files on different OSes","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-27T17:30:28Z","receivedAt":"2009-05-27T17:30:28Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 27 May 2009, Christopher Jefferson wrote:\n> On 27 May 2009, at 17:59, Linus Torvalds wrote:\n>> On Wed, 27 May 2009, Linus Torvalds wrote:\n>>>\n>>> I'll see if I can make us handle the \"big file without diff\" case \n>>> better by chunking.\n \n>> So \"don't do it then\" or \"make sure you are 64-bit and have lots of\n>> memory if you do it\" may well be the right solution.\n> \n> Thank you for that description of the problem, I can see how hard it is.\n> \n> Perhaps it might be useful to think about how to codify \"don't do it \n> then\" in a reasonably simple, automatic way?\n> \n> I've been trying to write a pre-commit hook (I think that's the right \n> place?) which would refuse commits larger than some file size (512MB \n> as a random number I decided), but am having trouble getting it to \n> work right, and generally. Would such a thing be easy, and would that \n> be the right place to put it?\n> \n> While I wouldn't suggest this become default, providing such a hook, \n> and describing why you might want to use it, would seem to avoid the \n> accidental part of the problem.\n\nHmmm... this is another issue (beside checking for portability of\nfilenames) that would be neatly solved if there was 'pre-add' hook,\nrather than trying to use 'pre-commit' hook for that. It should not,\nI think, be that hard to add it...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"114817","messageId":"alpine.LFD.2.00.0905271312220.3906@xanadu.home","threadId":"19525","inReplyTo":"alpine.LFD.2.01.0905270942580.3435@localhost.localdomain","subject":"Re: Problem with large files on different OSes","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-27T17:37:26Z","receivedAt":"2009-05-27T17:37:26Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 27 May 2009, Linus Torvalds wrote:\n\n> Hmm. No. Looking at it some more, we could add some nasty code to do \n> _some_ things chunked (like adding a new file as a single object), but it \n> doesn't really help. For any kind of useful thing, we'd need to handle the \n> \"read from pack\" case in multiple chunks too, and that gets really nasty \n> really quickly.\n> \n> The whole \"each object as one allocation\" design is pretty core, and it \n> looks pointless to have a few special cases, when any actual relevant use \n> would need a whole lot more than the few simple ones.\n> \n> Git really doesn't like big individual objects.\n> \n> I've occasionally thought about handling big files as multiple big \n> objects: we'd split them into a \"pseudo-directory\" (it would have some new \n> object ID), and then treat them as a magical special kind of directory \n> that just happens to be represented as one large file on the filesystem.\n> \n> That would mean that if you have a huge file, git internally would never \n> think of it as one big file, but as a collection of many smaller objects. \n> By just making the point where you break up files be a consistent rule \n> (\"always break into 256MB pieces\"), it would be a well-behaved design (ie \n> things like behaviour convergence wrt the same big file being created \n> different ways).\n> \n> HOWEVER.\n> \n> While that would fit in the git design (ie it would be just a fairly \n> straightforward extension - another level of indirection, kind of the way \n> we added subprojects), it would still be a rewrite of some core stuff. The \n> actual number of lines might not be too horrid, but quite frankly, I \n> wouldn't want to do it personally. It would be a lot of work with lots of \n> careful special case handling - and no real upside for normal use.\n\nMy idea for handling big files is simply to:\n\n 1) Define a new parameter to determine what is considered a big file.\n\n 2) Store any file larger than the treshold defined in (1) directly into \n a pack of their own at \"git add\" time.\n\n 3) Never attempt to diff nor delta large objects, again according to \n (1) above. It is typical for large files not to be deltifiable, and \n a diff for files in the thousands of megabytes cannot possibly be \n sane.\n\nThe idea is to avoid ever needing to load such object's content entirely \nin memory. So with the data already in a pack, the pack data reuse logic \n(which already does its copy in chunks) could be triggered during a \nrepack/fetch/push.\n\nThis is also quite trivial to implement with very few special cases, and \nthen git would handle huge repositories with lots of huge files just as \nwell as any other SCMs. The usual git repository compactness won't be \nthere of course, but I doubt people dealing with repositories in the \nhundreds of gigabytes really care.\n\n\nNicolas\n"},{"id":"114843","messageId":"20090527215314.GA10362@coredump.intra.peff.net","threadId":"19525","inReplyTo":"alpine.LFD.2.00.0905271312220.3906@xanadu.home","subject":"Re: Problem with large files on different OSes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-05-27T21:53:14Z","receivedAt":"2009-05-27T21:53:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 27, 2009 at 01:37:26PM -0400, Nicolas Pitre wrote:\n\n> My idea for handling big files is simply to:\n> \n> 1) Define a new parameter to determine what is considered a big file.\n> \n> 2) Store any file larger than the treshold defined in (1) directly into \n> a pack of their own at \"git add\" time.\n> \n> 3) Never attempt to diff nor delta large objects, again according to \n> (1) above. It is typical for large files not to be deltifiable, and \n> a diff for files in the thousands of megabytes cannot possibly be \n> sane.\n\nWhat about large files that have a short metadata section that may\nchange? Versions with only the metadata changed delta well, and with a\ncustom diff driver, can produce useful diffs. And I don't think that is\nan impractical or unlikely example; large files can often be tagged\nmedia.\n\nLinus' \"split into multiple objects\" approach means you could perhaps\nsplit intelligently into metadata and \"uninteresting data\" sections\nbased on the file type. That would make things like rename detection\nvery fast. Of course it has the downside that you are cementing whatever\nsplit you made into history for all time. And it means that two people\nadding the same content might end up with different trees. Both things\nthat git tries to avoid.\n\nI wonder if it would be useful to make such a split at _read_ time. That\nis, still refer to the sha-1 of the whole content in the tree objects,\nbut have a separate cache that says \"hash X splits to the concatenation\nof Y,Z\". Thus you can always refer to the \"pure\" object, both as a user,\nand in the code. So we could avoid retrofitting all of the code -- just\nsome parts like diff might want to handle an object in multiple\nsegments.\n\n-Peff\n"},{"id":"114845","messageId":"alpine.LFD.2.01.0905271457310.3435@localhost.localdomain","threadId":"19525","inReplyTo":"20090527215314.GA10362@coredump.intra.peff.net","subject":"Re: Problem with large files on different OSes","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-27T22:07:49Z","receivedAt":"2009-05-27T22:07:49Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 27 May 2009, Jeff King wrote:\n>\n> Linus' \"split into multiple objects\" approach means you could perhaps\n> split intelligently into metadata and \"uninteresting data\" sections\n> based on the file type.\n\nI suspect you wouldn't even need to. A regular delta algorithm would just \nwork fairly well to find the common parts.\n\nSure, if the offset of the data changes a lot, then you'd miss all the \ndeltas between two (large) objects that now have data that traverses \nobject boundaries, but especially if the split size is pretty large (ie \nseveral tens of MB, possibly something like 256M), that's still going to \nbe a pretty rare event.\n\nIOW, imagine that you have a big file that is 2GB in size, and you prepend \n100kB of data to it (that's why it's so big - you keep prepending data to \nit as some kind of odd ChangeLog file). What happens? It would still delta \nfairly well, even if the delta's would now be:\n\n - 100kB of new data\n - 256M - 100kB of old data as a small delta entry\n\nand the _next_ chunk woul be:\n\n - 100kB of \"new\" data (old data from the previous chunk)\n - 256M - 100kB of old data as a small delta entry\n\n.. and so on for each chunk. So if the whole file is 2GB, it would be \nroughly 8 256MB chunks, and it would delta perfectly well: except for the \noverlap, that would now be 8x 100kB \"slop\" deltas.\n\nSo even a totally unmodified delta algorithm would shrink down the two \ncopies of a ~2GB file to one copy + 900kB of extra delta.\n\nSure, a perfect xdelta thing that would have treated it as one huge file \nwould have had just 100kB of delta data, but 900kB would still be a *big* \nsaving over duplicating the whole 2GB.\n\n> That would make things like rename detection very fast. Of course it has \n> the downside that you are cementing whatever split you made into history \n> for all time. And it means that two people adding the same content might \n> end up with different trees. Both things that git tries to avoid.\n\nIt's the \"I can no longer see that the files are the same by comparing \nSHA1's\" that I personally dislike.\n\nSo my \"fixed chunk\" approach would be nice in that if you have this kind \nof \"chunkblob\" entry, in the tree (and index) it would literally be one \nentry, and look like that:\n\n 100644 chunkblob \n\nso you could compare two trees that have the same chunkblob entry, and \njust see that they are the same without ever looking at the (humongous) \ndata.\n\nThe type itself would then look like just an array of SHA1's, \nie it would literally be an object that only points to other blobs. Kind \nof a \"simplified tree object\", if you will.\n\nI think it would fit very well in the git model. But it's a nontrivial \namount of changes.\n\n\t\t\tLinus\n"},{"id":"114862","messageId":"f95910c20905271609u63d04965oa38b8af34d7704c1@mail.gmail.com","threadId":"19525","inReplyTo":"alpine.LFD.2.01.0905271457310.3435@localhost.localdomain","subject":"Re: Problem with large files on different OSes","fromName":"Alan Manuel Gloria","fromEmail":"almkglor@gmail.com","sentAt":"2009-05-27T23:09:14Z","receivedAt":"2009-05-27T23:09:14Z","isPatch":false,"sender":{"key":"almkglor@gmail.com","avatar":null},"body":"On Thu, May 28, 2009 at 6:07 AM, Linus Torvalds\n wrote:\n>\n>\n> On Wed, 27 May 2009, Jeff King wrote:\n>>\n>> Linus' \"split into multiple objects\" approach means you could perhaps\n>> split intelligently into metadata and \"uninteresting data\" sections\n>> based on the file type.\n>\n> I suspect you wouldn't even need to. A regular delta algorithm would just\n> work fairly well to find the common parts.\n>\n> Sure, if the offset of the data changes a lot, then you'd miss all the\n> deltas between two (large) objects that now have data that traverses\n> object boundaries, but especially if the split size is pretty large (ie\n> several tens of MB, possibly something like 256M), that's still going to\n> be a pretty rare event.\n>\n> IOW, imagine that you have a big file that is 2GB in size, and you prepend\n> 100kB of data to it (that's why it's so big - you keep prepending data to\n> it as some kind of odd ChangeLog file). What happens? It would still delta\n> fairly well, even if the delta's would now be:\n>\n>  - 100kB of new data\n>  - 256M - 100kB of old data as a small delta entry\n>\n> and the _next_ chunk woul be:\n>\n>  - 100kB of \"new\" data (old data from the previous chunk)\n>  - 256M - 100kB of old data as a small delta entry\n>\n> .. and so on for each chunk. So if the whole file is 2GB, it would be\n> roughly 8 256MB chunks, and it would delta perfectly well: except for the\n> overlap, that would now be 8x 100kB \"slop\" deltas.\n>\n> So even a totally unmodified delta algorithm would shrink down the two\n> copies of a ~2GB file to one copy + 900kB of extra delta.\n>\n> Sure, a perfect xdelta thing that would have treated it as one huge file\n> would have had just 100kB of delta data, but 900kB would still be a *big*\n> saving over duplicating the whole 2GB.\n>\n>> That would make things like rename detection very fast. Of course it has\n>> the downside that you are cementing whatever split you made into history\n>> for all time. And it means that two people adding the same content might\n>> end up with different trees. Both things that git tries to avoid.\n>\n> It's the \"I can no longer see that the files are the same by comparing\n> SHA1's\" that I personally dislike.\n>\n> So my \"fixed chunk\" approach would be nice in that if you have this kind\n> of \"chunkblob\" entry, in the tree (and index) it would literally be one\n> entry, and look like that:\n>\n>   100644 chunkblob \n>\n> so you could compare two trees that have the same chunkblob entry, and\n> just see that they are the same without ever looking at the (humongous)\n> data.\n>\n> The type itself would then look like just an array of SHA1's,\n> ie it would literally be an object that only points to other blobs. Kind\n> of a \"simplified tree object\", if you will.\n>\n> I think it would fit very well in the git model. But it's a nontrivial\n> amount of changes.\n>\n>                        Linus\n\nI'd like to pitch in that our mother company uses Subversion, and they\nconsistently push very large binaries onto their Subvesion\nrepositories (I know it's not a good idea. They do it nevertheless.\nThe very large binary is a description of a design in a proprietary\nformat by a proprietary tool; they don't want to keep running that\ntool because of licensing etc issues, so they archive it on\nSubversion).\n\nI'm trying to convince the mother company to switch to git, mostly\nbecause our company (the daughter company) doesn't have direct access\nto their Subversion repo (we're in another country), and I've become\nconvinced that distributed repos like git are the way to go. But the\nfact that large binaries require me to turn off gc.auto and otherwise\navoid packing large filles makes my case a harder sell; quite a bit of\nthe mother company's workflow has been integrated with Subversion.\n\nNote that in my case \"large binary\" is really a 164Mb file, but my\nwork system is a dual-core 512Mb computer, so I suppose my hardware is\nreally the limitation; still, some of the computers at the mother\ncompany are even lousier.\n\nIf you'd prefer someone else to hack it, can you at least give me some\npointers on which code files to start looking? I'd really like to\nhave proper large-file-packing support, where large file is anything\nmuch bigger than a megabyte or so.\n\nAdmittedly I'm not a filesystems guy and I can just barely grok git's\nblobs (they're the actual files, right? except they're named with\ntheir hash), but not packs (err, a bunch of files?) and trees (brown\nand green stuff you plant?). Still, I can try to learn it.\n\nSincerely,\nAmkG\n"},{"id":"114863","messageId":"alpine.LFD.2.00.0905271834280.3906@xanadu.home","threadId":"19525","inReplyTo":"20090527215314.GA10362@coredump.intra.peff.net","subject":"Re: Problem with large files on different OSes","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-27T23:29:02Z","receivedAt":"2009-05-27T23:29:02Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 27 May 2009, Jeff King wrote:\n\n> On Wed, May 27, 2009 at 01:37:26PM -0400, Nicolas Pitre wrote:\n> \n> > My idea for handling big files is simply to:\n> > \n> > 1) Define a new parameter to determine what is considered a big file.\n> > \n> > 2) Store any file larger than the treshold defined in (1) directly into \n> > a pack of their own at \"git add\" time.\n> > \n> > 3) Never attempt to diff nor delta large objects, again according to \n> > (1) above. It is typical for large files not to be deltifiable, and \n> > a diff for files in the thousands of megabytes cannot possibly be \n> > sane.\n> \n> What about large files that have a short metadata section that may\n> change? Versions with only the metadata changed delta well, and with a\n> custom diff driver, can produce useful diffs. And I don't think that is\n> an impractical or unlikely example; large files can often be tagged\n> media.\n\nSure... but what is the actual data pattern currently used out there?\nWhat does P4 or CVS or SVN do with multiple versions of almost identical \n2GB+ files?\n\nMy point is, if the tool people are already using with gigantic \nrepositories is not bothering with delta compression then we don't lose \nmuch in making git usable with those repositories by doing the same. \nAnd this can be achieved pretty easily with fairly minor changes. Plus, \nmy proposal doesn't introduce any incompatibility in the git repository \nformat while not denying possible future enhancements.\n\nFor example, that would be non trivial but still doable to make git work \non data streams instead of buffers. The current code for blob \nread/write/delta could be kept for performance, along with another \nversion in parallel doing the same but with file descriptors and \npread/pwrite for big files.\n\n> Linus' \"split into multiple objects\" approach means you could perhaps\n> split intelligently into metadata and \"uninteresting data\" sections\n> based on the file type. That would make things like rename detection\n> very fast. Of course it has the downside that you are cementing whatever\n> split you made into history for all time. And it means that two people\n> adding the same content might end up with different trees. Both things\n> that git tries to avoid.\n\nExact. And honnestly I don't think that would be worth trying to do \ninexact rename detection for huge files anyway. It is rarely the case \nthat moving/renaming a movie file needs to change its content in some \nway.\n\n> I wonder if it would be useful to make such a split at _read_ time. That\n> is, still refer to the sha-1 of the whole content in the tree objects,\n> but have a separate cache that says \"hash X splits to the concatenation\n> of Y,Z\". Thus you can always refer to the \"pure\" object, both as a user,\n> and in the code. So we could avoid retrofitting all of the code -- just\n> some parts like diff might want to handle an object in multiple\n> segments.\n\nUnless there are real world scenarios where diffing (as we know it) two \nhuge files is a common and useful operation, I don't think we should \neven try to consider that problem. What people are doing with huge \nfiles is storing them and retrieving them, so we probably should only \nlimit ourselves to making those operations work for now.\n\nAnd to that effect I don't think it would be wise to introduce \nartificial segmentations in the object structure that would make both \nthe code and the git model more complex. We could just as well limit \nthe complexity to the code for dealing with blobs without having to load \nthem all in memory at once instead and keep the git repository model \nsimple.\n\nSo if we want to do the real thing and deal with huge blobs, there is \nonly a small set of operations that need to be considered:\n\n - Creation of new blobs (or \"git add\") for huge files: can be done \n trivially in chunks. Open issue is whether or not the SHA1 of the \n file is computed with a first pass over the file, and if the object \n doesn't exist then perform a second pass to deflate it if desired, or \n do both the SHA1 summing and deflate in the same pass and discard the \n result if the object happens to already exist. Still trivial to \n implement.\n\n - Checkout of huge files: still trivial to perform if non delta. In \n the delta case, that _could_ be quite simple if the base objects \n were not deflated by recursively parsing deltas. But again that \n remains to be seen if 1) deflating or even 2) deltifying huge files\n is useful in practice with real world data.\n\n - repack/fetch/pull: In the pack data reuse case, the code is already \n fine as it streams small blocks from the source to the destination. \n Delta compression can be done by using coarse indexing of the source \n object and loading/discarding portions of the source data while the \n target object is processed in a streaming fashion.\n\nOther than that, I don't see how git could be useful for huge files. \nThe above operations (read/write/delta of huge blobs) would need to be \ndone with a separate set of functions, and a configurable size treshold \nwould select the regular or the chunked set. Nothing fundamentally \ndifficult in my mind.\n\n\nNicolas\n"},{"id":"114864","messageId":"alpine.LFD.2.01.0905271825520.3435@localhost.localdomain","threadId":"19525","inReplyTo":"f95910c20905271609u63d04965oa38b8af34d7704c1@mail.gmail.com","subject":"Re: Problem with large files on different OSes","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-28T01:56:49Z","receivedAt":"2009-05-28T01:56:49Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 28 May 2009, Alan Manuel Gloria wrote:\n> \n> If you'd prefer someone else to hack it, can you at least give me some\n> pointers on which code files to start looking? I'd really like to\n> have proper large-file-packing support, where large file is anything\n> much bigger than a megabyte or so.\n> \n> Admittedly I'm not a filesystems guy and I can just barely grok git's\n> blobs (they're the actual files, right? except they're named with\n> their hash), but not packs (err, a bunch of files?) and trees (brown\n> and green stuff you plant?). Still, I can try to learn it.\n\nThe packs is a big part of the complexity.\n\nIf you were to keep the big files as unpacked blobs, that would be \nfairly simple - but the pack-file format is needed for fetching and \npushing things, so it's not really an option.\n\nFor your particular case, the simplest approach is probably to just \nlimit the delta search. Something like just saying \"if the object is \nlarger than X, don't even bother to try to delta it, and just pack it \nwithout delta compression\". \n\nThe code would still load that whole object in one go, but it sounds like \nyou can handle _one_ object at a time. So for your case, I don't think you \nneed a fundamental git change - you'd be ok with just an inefficient pack \nformat for large files that are very expensive to pack otherwise.\n\nYou can already do that by using .gitattributes to not delta entries \nby name, but maybe it's worth doing explicitly by size too.\n\nI realize that the \"delta\" attribute is apparently almost totally \nundocumented. But if your big blobs have a particular name pattern, what \nyou should try is to do something like\n\n - in your '.gitattributes' file (or .git/info/attributes if you don't \n want to check it in), add a line like\n\n\t*.img !delta\n\n which now sets the 'delta' attribute to false for all objects that \n match the '*.img' pattern.\n\n - see if pack creation is now acceptable (ie do a \"git gc\" or try to push \n somewhere)\n\nSomething like the following may also work, as a more generic \"just don't \neven bother trying to delta huge files\".\n\nTotally untested. Maybe it works. Maybe it doesn't.\n\n\t\tLinus\n\n---\n Documentation/config.txt | 7 +++++++\n builtin-pack-objects.c | 9 +++++++++\n 2 files changed, 16 insertions(+), 0 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 2c03162..8c21027 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1238,6 +1238,13 @@ older version of git. If the `{asterisk}.pack` file is smaller than 2 GB, howeve\n you can use linkgit:git-index-pack[1] on the *.pack file to regenerate\n the `{asterisk}.idx` file.\n \n+pack.packDeltaLimit::\n+\tThe default maximum size of objects that we try to delta.\n++\n+Big files can be very expensive to delta, and if they are large binary\n+blobs, there is likely little upside to it anyway. So just pack them\n+as-is, and don't waste time on them.\n+\n pack.packSizeLimit::\n \tThe default maximum size of a pack. This setting only affects\n \tpacking to a file, i.e. the git:// protocol is unaffected. It\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 9742b45..9a0072b 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -85,6 +85,7 @@ static struct progress *progress_state;\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n static int pack_compression_seen;\n \n+static unsigned long pack_delta_limit = 64*1024*1024;\n static unsigned long delta_cache_size = 0;\n static unsigned long max_delta_cache_size = 0;\n static unsigned long cache_max_small_delta_size = 1000;\n@@ -1270,6 +1271,10 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (trg_entry->type != src_entry->type)\n \t\treturn -1;\n \n+\t/* If we limit delta generation, don't even bother for larger blobs */\n+\tif (pack_delta_limit && trg_entry->size >= pack_delta_limit)\n+\t\treturn -1;\n+\n \t/*\n \t * We do not bother to try a delta that we discarded\n \t * on an earlier try, but only when reusing delta data.\n@@ -1865,6 +1870,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tpack_size_limit_cfg = git_config_ulong(k, v);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.packdeltalimit\")) {\n+\t\tpack_delta_limit = git_config_ulong(k, v);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n"},{"id":"114875","messageId":"alpine.LFD.2.00.0905272312370.3906@xanadu.home","threadId":"19525","inReplyTo":"alpine.LFD.2.01.0905271825520.3435@localhost.localdomain","subject":"Re: Problem with large files on different OSes","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-28T03:26:06Z","receivedAt":"2009-05-28T03:26:06Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 27 May 2009, Linus Torvalds wrote:\n\n> Something like the following may also work, as a more generic \"just don't \n> even bother trying to delta huge files\".\n> \n> Totally untested. Maybe it works. Maybe it doesn't.\n> \n> \t\tLinus\n> \n> ---\n> Documentation/config.txt | 7 +++++++\n> builtin-pack-objects.c | 9 +++++++++\n> 2 files changed, 16 insertions(+), 0 deletions(-)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 2c03162..8c21027 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1238,6 +1238,13 @@ older version of git. If the `{asterisk}.pack` file is smaller than 2 GB, howeve\n> you can use linkgit:git-index-pack[1] on the *.pack file to regenerate\n> the `{asterisk}.idx` file.\n> \n> +pack.packDeltaLimit::\n> +\tThe default maximum size of objects that we try to delta.\n\nThe option name feels a bit wrong here, like if it meant the max number \nof deltas in a pack. Nothing better comes to my mind at the moment \nthough.\n\n> diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\n> index 9742b45..9a0072b 100644\n> --- a/builtin-pack-objects.c\n> +++ b/builtin-pack-objects.c\n> @@ -85,6 +85,7 @@ static struct progress *progress_state;\n> static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n> static int pack_compression_seen;\n> \n> +static unsigned long pack_delta_limit = 64*1024*1024;\n> static unsigned long delta_cache_size = 0;\n> static unsigned long max_delta_cache_size = 0;\n> static unsigned long cache_max_small_delta_size = 1000;\n> @@ -1270,6 +1271,10 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n> \tif (trg_entry->type != src_entry->type)\n> \t\treturn -1;\n> \n> +\t/* If we limit delta generation, don't even bother for larger blobs */\n> +\tif (pack_delta_limit && trg_entry->size >= pack_delta_limit)\n> +\t\treturn -1;\n\nI'd suggest filtering delta candidates out of delta_list up front in \nprepare_pack() instead.\n\n\nNicolas\n"},{"id":"114877","messageId":"loom.20090528T041831-21@post.gmane.org","threadId":"19525","inReplyTo":"alpine.LFD.2.00.0905272312370.3906@xanadu.home","subject":"Re: Problem with large files on different OSes","fromName":"Eric Raible","fromEmail":"raible@gmail.com","sentAt":"2009-05-28T04:21:11Z","receivedAt":"2009-05-28T04:21:11Z","isPatch":false,"sender":{"key":"raible@gmail.com","avatar":null},"body":"Nicolas Pitre cam.org> writes:\n\n> On Wed, 27 May 2009, Linus Torvalds wrote:\n> \n> > +pack.packDeltaLimit::\n> > +\tThe default maximum size of objects that we try to delta.\n> \n> The option name feels a bit wrong here, like if it meant the max number \n> of deltas in a pack. Nothing better comes to my mind at the moment \n> though.\n\npack.maxDeltaSize sounds weird when said aloud.\nHow about pack.deltaMaxSize?\n\n- Eric\n"},{"id":"114878","messageId":"20090528043019.GA30527@spearce.org","threadId":"19525","inReplyTo":"loom.20090528T041831-21@post.gmane.org","subject":"Re: Problem with large files on different OSes","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-28T04:30:19Z","receivedAt":"2009-05-28T04:30:19Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Eric Raible wrote:\n> Nicolas Pitre cam.org> writes:\n> > On Wed, 27 May 2009, Linus Torvalds wrote:\n> > \n> > > +pack.packDeltaLimit::\n> > > +\tThe default maximum size of objects that we try to delta.\n> > \n> > The option name feels a bit wrong here, like if it meant the max number \n> > of deltas in a pack. Nothing better comes to my mind at the moment \n> > though.\n> \n> pack.maxDeltaSize sounds weird when said aloud.\n> How about pack.deltaMaxSize?\n\nThat sounds like, how big should a delta be? E.g. set it to 200\nand any delta instruction stream over 200 bytes would be discarded,\ncausing the whole object to be stored instead. Which is obviously\nsomewhat silly, but that's the way I'd read that option...\n\n-- \nShawn.\n"},{"id":"114884","messageId":"279b37b20905272252i3d98b6bv698ff636d61ee058@mail.gmail.com","threadId":"19525","inReplyTo":"20090528043019.GA30527@spearce.org","subject":"Re: Problem with large files on different OSes","fromName":"Eric Raible","fromEmail":"raible@gmail.com","sentAt":"2009-05-28T05:52:37Z","receivedAt":"2009-05-28T05:52:37Z","isPatch":false,"sender":{"key":"raible@gmail.com","avatar":null},"body":"On Wed, May 27, 2009 at 9:30 PM, Shawn O. Pearce wrote:\n> Eric Raible wrote:\n>> Nicolas Pitre cam.org> writes:\n>> > On Wed, 27 May 2009, Linus Torvalds wrote:\n>> >\n>> > > +pack.packDeltaLimit::\n>> > > + The default maximum size of objects that we try to delta.\n>> >\n>> > The option name feels a bit wrong here, like if it meant the max number\n>> > of deltas in a pack.  Nothing better comes to my mind at the moment\n>> > though.\n>>\n>> pack.maxDeltaSize sounds weird when said aloud.\n>> How about pack.deltaMaxSize?\n>\n> That sounds like, how big should a delta be?  E.g. set it to 200\n> and any delta instruction stream over 200 bytes would be discarded,\n> causing the whole object to be stored instead.  Which is obviously\n> somewhat silly, but that's the way I'd read that option...\n>\n> --\n> Shawn.\n\nYou're right, that _is_ a strange color for the bike shed...\n"},{"id":"114905","messageId":"4A1E50D4.2010203@op5.se","threadId":"19525","inReplyTo":"279b37b20905272252i3d98b6bv698ff636d61ee058@mail.gmail.com","subject":"Re: Problem with large files on different OSes","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-05-28T08:52:36Z","receivedAt":"2009-05-28T08:52:36Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Eric Raible wrote:\n> On Wed, May 27, 2009 at 9:30 PM, Shawn O. Pearce wrote:\n>> Eric Raible wrote:\n>>> Nicolas Pitre cam.org> writes:\n>>>> On Wed, 27 May 2009, Linus Torvalds wrote:\n>>>>\n>>>>> +pack.packDeltaLimit::\n>>>>> + The default maximum size of objects that we try to delta.\n>>>> The option name feels a bit wrong here, like if it meant the max number\n>>>> of deltas in a pack. Nothing better comes to my mind at the moment\n>>>> though.\n>>> pack.maxDeltaSize sounds weird when said aloud.\n>>> How about pack.deltaMaxSize?\n>> That sounds like, how big should a delta be? E.g. set it to 200\n>> and any delta instruction stream over 200 bytes would be discarded,\n>> causing the whole object to be stored instead. Which is obviously\n>> somewhat silly, but that's the way I'd read that option...\n>>\n>> --\n>> Shawn.\n> \n> You're right, that _is_ a strange color for the bike shed...\n\nSince 'delta' names both the action and the result of the action, it's\ntricky to get it unambiguous without helping the grammar along a little.\n\n pack.maxFileSizeToDelta\n\nis probably the shortest we're going to get it while avoiding ambiguity.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"114935","messageId":"alpine.LFD.2.00.0905281339030.3906@xanadu.home","threadId":"19525","inReplyTo":"loom.20090528T041831-21@post.gmane.org","subject":"Re: Problem with large files on different OSes","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-28T17:41:37Z","receivedAt":"2009-05-28T17:41:37Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"[ please don't drop original sender address and CC unless asked to ]\n\nOn Thu, 28 May 2009, Eric Raible wrote:\n\n> Nicolas Pitre cam.org> writes:\n> \n> > On Wed, 27 May 2009, Linus Torvalds wrote:\n> > \n> > > +pack.packDeltaLimit::\n> > > +\tThe default maximum size of objects that we try to delta.\n> > \n> > The option name feels a bit wrong here, like if it meant the max number \n> > of deltas in a pack. Nothing better comes to my mind at the moment \n> > though.\n> \n> pack.maxDeltaSize sounds weird when said aloud.\n> How about pack.deltaMaxSize?\n\npack.MaxSizeForDelta\n\nWhatever...\n\n\nNicolas\n"},{"id":"114950","messageId":"20090528194348.GH13499@coredump.intra.peff.net","threadId":"19525","inReplyTo":"alpine.LFD.2.01.0905271457310.3435@localhost.localdomain","subject":"Re: Problem with large files on different OSes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-05-28T19:43:48Z","receivedAt":"2009-05-28T19:43:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 27, 2009 at 03:07:49PM -0700, Linus Torvalds wrote:\n\n> I suspect you wouldn't even need to. A regular delta algorithm would just \n> work fairly well to find the common parts.\n> \n> Sure, if the offset of the data changes a lot, then you'd miss all the \n> deltas between two (large) objects that now have data that traverses \n> object boundaries, but especially if the split size is pretty large (ie \n> several tens of MB, possibly something like 256M), that's still going to \n> be a pretty rare event.\n\nI confess that I'm not just interested in the _size_ of the deltas, but\nalso speeding up deltification and rename detection. And I'm interested\nin files where we can benefit from their semantics a bit. So yes, with\nsome overlap you would end up with pretty reasonable deltas for\narbitrary binary, as you describe.\n\nBut I was thinking something more like splitting a JPEG into a small\nfirst chunk that contains EXIF data, and a big secondary chunk that\ncontains the actual image data. The second half is marked as not\ncompressible (since it is already lossily compressed), and not\ninteresting for deltification. When we consider two images for\ndeltification, either:\n\n 1. they have the same \"uninteresting\" big part. In that case, you can\n trivially make a delta by just replacing the smaller first part (or\n even finding the optimal delta between the small parts). You never\n even need to look at the second half.\n\n 2. they don't have the same uninteresting part. You can reject them as\n delta candidates, because there is little chance the big parts will\n be related, even for a different version of the same image.\n\nAnd that extends to rename detection, as well. You can avoid looking at\nthe big part at all if you assume big parts with differing hashes are\ngoing to be drastically different.\n\n> > That would make things like rename detection very fast. Of course it has \n> > the downside that you are cementing whatever split you made into history \n> > for all time. And it means that two people adding the same content might \n> > end up with different trees. Both things that git tries to avoid.\n> \n> It's the \"I can no longer see that the files are the same by comparing \n> SHA1's\" that I personally dislike.\n\nRight. I don't think splitting in the git data structure itself is worth\nit for that reason. But deltification and rename detection keeping a\ncache of smart splits that says \"You can represent as this\nconcatenation of s\" means they can still get some advantage (over\nmultiple runs, certainly, but possibly even over a single run: a smart\nsplitter might not even have to look at the entire file contents).\n\n> So my \"fixed chunk\" approach would be nice in that if you have this kind \n> of \"chunkblob\" entry, in the tree (and index) it would literally be one \n> entry, and look like that:\n> \n> 100644 chunkblob \n\nBut if I am understanding you correctly, you _are_ proposing to munge\nthe git data structure here. Which means that pre-chunkblob trees will\npoint to the raw blob, and then post-chunkblob trees will point to the\nchunked representation. And that means not being able to use the sha-1\nto see that they eventually point to the same content.\n\n-Peff\n"},{"id":"114951","messageId":"alpine.LFD.2.01.0905281246330.3435@localhost.localdomain","threadId":"19525","inReplyTo":"20090528194348.GH13499@coredump.intra.peff.net","subject":"Re: Problem with large files on different OSes","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-28T19:49:12Z","receivedAt":"2009-05-28T19:49:12Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 28 May 2009, Jeff King wrote:\n> \n> > So my \"fixed chunk\" approach would be nice in that if you have this kind \n> > of \"chunkblob\" entry, in the tree (and index) it would literally be one \n> > entry, and look like that:\n> > \n> > 100644 chunkblob \n> \n> But if I am understanding you correctly, you _are_ proposing to munge\n> the git data structure here. Which means that pre-chunkblob trees will\n> point to the raw blob, and then post-chunkblob trees will point to the\n> chunked representation. And that means not being able to use the sha-1\n> to see that they eventually point to the same content.\n\nYes. If we were to do this, and people have large chunks, then once you \nstart using the chunkblob (for lack of a better word) model, you'll see \nthe same object with two different SHA1's. But it's a one-time (and \none-way - since once it's a chunkblob, older models can't touch it) thing, \nit can never cause any long-term confusion.\n\n(We'll end up with something similar if somebody ever breaks SHA-1 enough \nfor us to care - the logical way to handle it is likely to just accept the \nSHA512-160 object name \"aliases\")\n\n\t\t\tLinus\n"},{"id":"114953","messageId":"20090528200039.GI13499@coredump.intra.peff.net","threadId":"19525","inReplyTo":"alpine.LFD.2.00.0905271834280.3906@xanadu.home","subject":"Re: Problem with large files on different OSes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-05-28T20:00:39Z","receivedAt":"2009-05-28T20:00:39Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 27, 2009 at 07:29:02PM -0400, Nicolas Pitre wrote:\n\n> > What about large files that have a short metadata section that may\n> > change? Versions with only the metadata changed delta well, and with a\n> > custom diff driver, can produce useful diffs. And I don't think that is\n> > an impractical or unlikely example; large files can often be tagged\n> > media.\n> \n> Sure... but what is the actual data pattern currently used out there?\n\nI'm not sure what you mean by \"out there\", but I just exactly described\nthe data pattern of a repo I have (a few thousand 5 megapixel JPEGs and\nshort (a few dozens of megabytes) AVIs, frequent additions, infrequently\nchanging photo contents, and moderately changing metadata). I don't know\nhow that matches other peoples' needs.\n\nGame designers have been mentioned before in large media checkins, and I\nthink they focus less on metadata changes. Media is either there to\nstay, or it is replaced as a whole.\n\n> What does P4 or CVS or SVN do with multiple versions of almost identical \n> 2GB+ files?\n\nI only ever tried this with CVS, which just stored the entire binary\nversion as a whole. And of course running \"diff\" was useless, but then\nit was also useless on text files. ;) I suspect CVS would simply choke\non a 2G file.\n\nBut I don't want to do as well as those other tools. I want to be able\nto do all of the useful things git can do but with large files.\n\n> My point is, if the tool people are already using with gigantic \n> repositories is not bothering with delta compression then we don't lose \n> much in making git usable with those repositories by doing the same.\n> And this can be achieved pretty easily with fairly minor changes. Plus, \n> my proposal doesn't introduce any incompatibility in the git repository \n> format while not denying possible future enhancements.\n\nRight. I think in some ways we are perhaps talking about two different\nproblems. I am really interested in moderately large files (a few\nmegabytes up to a few dozens or even a hundred megabytes), but I want\ngit to be _fast_ at dealing with them, and doing useful operations on\nthem (like rename detection, diffing, etc).\n\nA smart splitter would probably want to mark part of the split as \"this\nsection is large and uninteresting for compression, deltas, diffing, and\nrenames\". And that half may be stored in the way that you are proposing\n(in a separate single-object pack, no compression, no delta, etc). So in\na sense I think what I am talking about would built on top of what you\nwant to do.\n\n> > very fast. Of course it has the downside that you are cementing whatever\n> > split you made into history for all time. And it means that two people\n> > adding the same content might end up with different trees. Both things\n> > that git tries to avoid.\n> \n> Exact. And honnestly I don't think that would be worth trying to do \n> inexact rename detection for huge files anyway. It is rarely the case \n> that moving/renaming a movie file needs to change its content in some \n> way.\n\nI should have been more clear in my other email: I think splitting that\nis represented in the actual git trees is not going to be worth the\nhassle.\n\nBut I do think we can get some of the benefits by maintaining a split\ncache for viewers. And again, maybe my use case is crazy, but in my repo\nI have renames and metadata content changes together.\n\n> Unless there are real world scenarios where diffing (as we know it) two \n> huge files is a common and useful operation, I don't think we should \n> even try to consider that problem. What people are doing with huge \n> files is storing them and retrieving them, so we probably should only \n> limit ourselves to making those operations work for now.\n\nAgain, this is motivated by a real use case that I have.\n\n> And to that effect I don't think it would be wise to introduce \n> artificial segmentations in the object structure that would make both \n> the code and the git model more complex. We could just as well limit \n> the complexity to the code for dealing with blobs without having to load \n> them all in memory at once instead and keep the git repository model \n> simple.\n\nI do agree with this; I don't want to make any changes to the repository\nmodel.\n\n> So if we want to do the real thing and deal with huge blobs, there is \n> only a small set of operations that need to be considered:\n\nI think everything you say here is sensible; I just want more operations\nfor my use case.\n\n-Peff\n"},{"id":"114961","messageId":"alpine.LFD.2.00.0905281608060.3906@xanadu.home","threadId":"19525","inReplyTo":"20090528200039.GI13499@coredump.intra.peff.net","subject":"Re: Problem with large files on different OSes","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-28T20:54:28Z","receivedAt":"2009-05-28T20:54:28Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 28 May 2009, Jeff King wrote:\n\n> On Wed, May 27, 2009 at 07:29:02PM -0400, Nicolas Pitre wrote:\n> \n> > > What about large files that have a short metadata section that may\n> > > change? Versions with only the metadata changed delta well, and with a\n> > > custom diff driver, can produce useful diffs. And I don't think that is\n> > > an impractical or unlikely example; large files can often be tagged\n> > > media.\n> > \n> > Sure... but what is the actual data pattern currently used out there?\n> \n> I'm not sure what you mean by \"out there\", but I just exactly described\n> the data pattern of a repo I have (a few thousand 5 megapixel JPEGs and\n> short (a few dozens of megabytes) AVIs, frequent additions, infrequently\n> changing photo contents, and moderately changing metadata). I don't know\n> how that matches other peoples' needs.\n\nHow do diffing à la 'git diff' JPEGs or AVIs make sense?\n\nAlso, you certainly have little to delta against as you add new photos \nmore often than modifying existing ones?\n\n> Game designers have been mentioned before in large media checkins, and I\n> think they focus less on metadata changes. Media is either there to\n> stay, or it is replaced as a whole.\n\nRight. And my proposal fits that scenario pretty well.\n\n> > What does P4 or CVS or SVN do with multiple versions of almost identical \n> > 2GB+ files?\n> \n> I only ever tried this with CVS, which just stored the entire binary\n> version as a whole. And of course running \"diff\" was useless, but then\n> it was also useless on text files. ;) I suspect CVS would simply choke\n> on a 2G file.\n> \n> But I don't want to do as well as those other tools. I want to be able\n> to do all of the useful things git can do but with large files.\n\nRight now git simply do much worse. So doing as well is still a worthy \ngoal.\n\n> Right. I think in some ways we are perhaps talking about two different\n> problems. I am really interested in moderately large files (a few\n> megabytes up to a few dozens or even a hundred megabytes), but I want\n> git to be _fast_ at dealing with them, and doing useful operations on\n> them (like rename detection, diffing, etc).\n\nI still can't see how diffing big files is useful. Certainly you'll \nneed a specialized external diff tool, in which case it is not git's \nproblem anymore except for writing content to temporary files.\n\nRename detection: either you deal with the big files each time, or you \n(re)create a cache with that information so no analysis is needed the \nsecond time around. This is something that even small files might \npossibly benefit from. But in any case, there is no other ways but to \nbite the bullet at least initially, and big files will be slower to \nprocess no matter what.\n\n> A smart splitter would probably want to mark part of the split as \"this\n> section is large and uninteresting for compression, deltas, diffing, and\n> renames\". And that half may be stored in the way that you are proposing\n> (in a separate single-object pack, no compression, no delta, etc). So in\n> a sense I think what I am talking about would built on top of what you\n> want to do.\n\nLooks to me like you wish for git to do what a specialized database \nwould be much more suited for the task. Isn't there tools to gather \npicture metadata info, just like itunes does with MP3s already?\n\n> But I do think we can get some of the benefits by maintaining a split\n> cache for viewers.\n\nSure.\n\nBut being able to deal with large (1GB and more) files remains a totally \ndifferent problem.\n\n\nNicolas\n"},{"id":"114965","messageId":"20090528212106.GA14874@coredump.intra.peff.net","threadId":"19525","inReplyTo":"alpine.LFD.2.00.0905281608060.3906@xanadu.home","subject":"Re: Problem with large files on different OSes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-05-28T21:21:06Z","receivedAt":"2009-05-28T21:21:06Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 28, 2009 at 04:54:28PM -0400, Nicolas Pitre wrote:\n\n> > I'm not sure what you mean by \"out there\", but I just exactly described\n> > the data pattern of a repo I have (a few thousand 5 megapixel JPEGs and\n> > short (a few dozens of megabytes) AVIs, frequent additions, infrequently\n> > changing photo contents, and moderately changing metadata). I don't know\n> > how that matches other peoples' needs.\n> \n> How do diffing à la 'git diff' JPEGs or AVIs make sense?\n\nIt is useful to see the changes in a text representation of the\nmetadata, with a single-line mention if the image or movie data has\nchanged. It's why I wrote the textconv feature.\n\n> Also, you certainly have little to delta against as you add new photos \n> more often than modifying existing ones?\n\nI do add new photos more than modifying existing ones. But I do modify\nthe old ones (tag corrections, new tags I didn't think of initially,\nupdates to the tagging schema, etc), too.\n\nThe sum of the sizes for all objects in the repo is 8.3G. The fully\npacked repo is 3.3G. So there clearly is some benefit from deltas, and I\ndon't want to just turn them off. Doing the actual repack is painfully\nslow.\n\n> I still can't see how diffing big files is useful. Certainly you'll \n> need a specialized external diff tool, in which case it is not git's \n> problem anymore except for writing content to temporary files.\n\nWriting content to temporarily files is actually quite slow when the\nfiles are hundreds of megabytes (even \"git show\" can be painful, let\nalone \"git log -p\"). But that is something that can be dealt with by\nimproving the interface to external diff and textconv to avoid writing\nout the whole file (and is something I have patches in the works for,\nbut they need finished and cleaned up).\n\n> Rename detection: either you deal with the big files each time, or you \n> (re)create a cache with that information so no analysis is needed the \n> second time around. This is something that even small files might \n> possibly benefit from. But in any case, there is no other ways but to \n> bite the bullet at least initially, and big files will be slower to \n> process no matter what.\n\nRight. What I am proposing is basically to create such a cache. But it\nis one that is general enough that it could be used for more than just\nthe rename detection (though arguably rename detection and deltification\ncould actually share more of the same techniques, in which case a cache\nfor one would help the other).\n\n> Looks to me like you wish for git to do what a specialized database \n> would be much more suited for the task. Isn't there tools to gather \n> picture metadata info, just like itunes does with MP3s already?\n\nYes, I already have tools for handling picture metadata info. How do I\nversion control that information? How do I keep it in sync across\nmultiple checkouts? How do I handle merging concurrent changes from\nmultiple sources? How do I keep that metadata connected to the pictures\nthat it describes? The things I want to do are conceptually\nno different what I do with other files; it's merely the size of the\nfiles that makes working with them in git less convenient (but it does\n_work_; I am using git for this _now_, and I have been for a few years).\n\n> But being able to deal with large (1GB and more) files remains a totally \n> different problem.\n\nRight, that is why I think I will end up building on top of what you do.\nI am trying to make a way for some operations to avoid looking at the\nentire file, even streaming, which should drastically speed up those\noperations. But it is unavoidable that some operations (e.g., \"git\nadd\") will have to look at the entire file. And that is what your\nproposal is about; streaming is basically the only way forward there.\n\n-Peff\n"}]}