# git and binary files 32 messages from 2008-01-16 to 2008-01-18. Participants: Petko Manolov, David Symonds, Johannes Schindelin, Jakub Narebski, Jeff King, Wincent Colaiuta, Nicolas Pitre, Rogan Dawes, Florian Weimer, Junio C Hamano, David Brown. Thread: https://gitlist.dev/t/11642 ## Petko Manolov, 2008-01-16 10:34 Subject: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161222150.3889%40bender.nucleusys.com ``` Hello there, I've searched the web for an answer, but i didn't find it and decided to take the risk of being yelled at. And post it here. Some of my projects require having binary files (firmware and other stuff) somewhere in the tree structure. Unfortunately these files are big - 50MB and more. After a couple of new versions arrive (and get committed) i end up with a repository way too big than necessary. The nature of these binary files is such that i don't care neither about their history nor older versions. Hence the question: is there an easy way to tell git not to bother about the history of these particular files and keep the most recent version only? cheers, Petko ``` ## David Symonds, 2008-01-16 10:54 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/ee77f5c20801160254u53e07773qb4125ffd90cf7619%40mail.gmail.com In-Reply-To: ``` On Jan 16, 2008 9:34 PM, Petko Manolov wrote: > > The nature of these binary files is such that i don't care neither about > their history nor older versions. Hence the question: is there an easy > way to tell git not to bother about the history of these particular files > and keep the most recent version only? If you don't care about versioning those files, why would you use a version control system? Just store them somewhere else, and use symlinks. Dave. ``` ## Johannes Schindelin, 2008-01-16 11:54 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.LSU.1.00.0801161113170.17650%40racer.site In-Reply-To: ``` Hi, On Wed, 16 Jan 2008, Petko Manolov wrote: > Some of my projects require having binary files (firmware and other > stuff) somewhere in the tree structure. Unfortunately these files are > big - 50MB and more. After a couple of new versions arrive (and get > committed) i end up with a repository way too big than necessary. > > The nature of these binary files is such that i don't care neither about > their history nor older versions. Hence the question: is there an easy > way to tell git not to bother about the history of these particular > files and keep the most recent version only? Your subject is a little bit misleading, no? It's not about the binariness (git handles binary files just fine, thankyouverymuch), but about the not-tracking them. The answer is no. You cannot ask git to have the newest version of something, but not the old ones. It contradicts the distributedness of git, too. Hth, Dscho ``` ## Petko Manolov, 2008-01-16 13:21 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161517260.5260%40bender.nucleusys.com In-Reply-To: ``` On Wed, 16 Jan 2008, David Symonds wrote: > If you don't care about versioning those files, why would you use a > version control system? Just store them somewhere else, and use > symlinks. That is certainly a way of doing it. However, it will be much simpler and fast to be able to "git clone" and then "git pull" every once in a while. The alternative involves "cp -a" or most likely "scp -r" the binaries along with the repository and you can never be sure that both are in sync. Petko ``` ## Petko Manolov, 2008-01-16 13:39 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161521500.5260%40bender.nucleusys.com In-Reply-To: ``` On Wed, 16 Jan 2008, Johannes Schindelin wrote: > Your subject is a little bit misleading, no? It's not about the > binariness (git handles binary files just fine, thankyouverymuch), but > about the not-tracking them. You're absobloodylutely correct. I was too preoccupied defining my problem in a better way, which left the subject kind of dumb. Well, quite dumb. :-) > The answer is no. You cannot ask git to have the newest version of > something, but not the old ones. It contradicts the distributedness of > git, too. I don't agree here. Assume that whatever you're working on require firmware for a device that won't change during the lifetime of the software project. The newest version of the said firmware is mostly bugfixes and you basically don't want to revert to the older ones. Consider the microcode for modern Pentiums, Core 2, etc. What i am trying to suggest is that there might be cases when you need something in the repository, but you don't want GIT to keep it's history nor it's predecessors. Leaving it out breaks the atomicity of such repository and makes the project management more complex. There's a few examples out there that shows how to solve this, but it seems inconvenient and involves branching, cloning, etc. Isn't it possible to add something like: "git nohistory firmware.bin" or "git nohistory -i-understand-this-might-be-dangerous firmware.bin" cheers, Petko ``` ## Johannes Schindelin, 2008-01-16 13:42 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.LSU.1.00.0801161341430.17650%40racer.site In-Reply-To: ``` Hi, On Wed, 16 Jan 2008, Petko Manolov wrote: > On Wed, 16 Jan 2008, David Symonds wrote: > > > If you don't care about versioning those files, why would you use a > > version control system? Just store them somewhere else, and use > > symlinks. > > That is certainly a way of doing it. However, it will be much simpler > and fast to be able to "git clone" and then "git pull" every once in a > while. The alternative involves "cp -a" or most likely "scp -r" the > binaries along with the repository and you can never be sure that both > are in sync. I think that you're missing the point of version control. It's not only about having an up-to-date source tree, but also about being able to go back to a certain revision. What you want is most likely covered by "rsync -au". Hth, Dscho ``` ## Jakub Narebski, 2008-01-16 13:53 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/m37ii9nagt.fsf%40roke.D-201 In-Reply-To: ``` Petko Manolov writes: > On Wed, 16 Jan 2008, Johannes Schindelin wrote: > > > The answer is no. You cannot ask git to have the newest version of > > something, but not the old ones. It contradicts the distributedness > > of git, too. > > I don't agree here. Assume that whatever you're working on require > firmware for a device that won't change during the lifetime of the > software project. The newest version of the said firmware is mostly > bugfixes and you basically don't want to revert to the older > ones. Consider the microcode for modern Pentiums, Core 2, etc. > > What i am trying to suggest is that there might be cases when you need > something in the repository, but you don't want GIT to keep it's > history nor it's predecessors. Leaving it out breaks the atomicity of > such repository and makes the project management more complex. > > There's a few examples out there that shows how to solve this, but it > seems inconvenient and involves branching, cloning, etc. Isn't it > possible to add something like: > > "git nohistory firmware.bin" > > or > "git nohistory -i-understand-this-might-be-dangerous firmware.bin" You can always tag a blob (like junio-gpg-pub tag in git.git repository), but it wouldn't be in a working directory. But it would get distributed on clone. BTW. if those large binary files doesn't differ much between version, they should get well compressed even if you would store them normally, all revisions. -- Jakub Narebski Poland ShadeHawk on #git ``` ## Jeff King, 2008-01-16 13:54 Subject: Re: git and binary files Message-ID: <20080116135420.GA21588@coredump.intra.peff.net> URL: https://gitlist.dev/e/20080116135420.GA21588%40coredump.intra.peff.net In-Reply-To: ``` On Wed, Jan 16, 2008 at 03:39:06PM +0200, Petko Manolov wrote: > What i am trying to suggest is that there might be cases when you need > something in the repository, but you don't want GIT to keep it's history > nor it's predecessors. Leaving it out breaks the atomicity of such > repository and makes the project management more complex. But not versioning some files while versioning others breaks the atomicity of project version, which is at the core of git's model. There is no such thing as "this file is at revision X, but that one is at revision Y." There is only "the project is at revision X." > There's a few examples out there that shows how to solve this, but it > seems inconvenient and involves branching, cloning, etc. Isn't it > possible to add something like: > > "git nohistory firmware.bin" > > or > "git nohistory -i-understand-this-might-be-dangerous firmware.bin" Not easily. It goes against the underlying data model at the core of git. How big are your firmware files? How often do they change, and how large are the changes? IOW, have you confirmed that repacking does not produce an acceptable delta, meaning you get versioning for very low space cost? -Peff ``` ## Petko Manolov, 2008-01-16 13:58 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161549140.5260%40bender.nucleusys.com In-Reply-To: ``` On Wed, 16 Jan 2008, Johannes Schindelin wrote: > I think that you're missing the point of version control. It's not only > about having an up-to-date source tree, but also about being able to go > back to a certain revision. No contradiction here. In my case old source code will work perfectly with new binaries/firmware. That's why i don't _need_ the history, only the latest stuff in order to save space. I do realize that what i am talking about is statistically microscopic scenario, but it does exist. If there's no such feature then i don't have much choice, but stick with my current way of doing things. > What you want is most likely covered by "rsync -au". Yeah, just like in the old days when "git pull" didn't do everything for you. thanks, Petko ``` ## Petko Manolov, 2008-01-16 14:04 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161600030.5260%40bender.nucleusys.com In-Reply-To: ``` On Wed, 16 Jan 2008, Jakub Narebski wrote: > You can always tag a blob (like junio-gpg-pub tag in git.git repository), > but it wouldn't be in a working directory. But it would get distributed > on clone. Hm, how does it work? > BTW. if those large binary files doesn't differ much between version, > they should get well compressed even if you would store them normally, > all revisions. Unfortunately this is not the case. These binary blobs are already compressed and/or encrypted and adding even a few bytes ends up storing new version in full size. cheers, Petko ``` ## Johannes Schindelin, 2008-01-16 14:07 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.LSU.1.00.0801161405330.17650%40racer.site In-Reply-To: ``` Hi, On Wed, 16 Jan 2008, Petko Manolov wrote: > On Wed, 16 Jan 2008, Johannes Schindelin wrote: > > > I think that you're missing the point of version control. It's not > > only about having an up-to-date source tree, but also about being able > > to go back to a certain revision. > > No contradiction here. In my case old source code will work perfectly > with new binaries/firmware. That's why i don't _need_ the history, only > the latest stuff in order to save space. No, you _do_ miss the point here. You might _think_ that they work perfectly, but with revision control you want to have _exactly_ the same setup. You want to be able to go back to a certain _revision_ (including the then-current firmware). And that's what you don't want. So git is not for you. Ciao, Dscho ``` ## Petko Manolov, 2008-01-16 14:14 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161606160.5260%40bender.nucleusys.com In-Reply-To: <20080116135420.GA21588@coredump.intra.peff.net> ``` On Wed, 16 Jan 2008, Jeff King wrote: > On Wed, Jan 16, 2008 at 03:39:06PM +0200, Petko Manolov wrote: > >> What i am trying to suggest is that there might be cases when you need >> something in the repository, but you don't want GIT to keep it's history >> nor it's predecessors. Leaving it out breaks the atomicity of such >> repository and makes the project management more complex. > > But not versioning some files while versioning others breaks the > atomicity of project version, which is at the core of git's model. There > is no such thing as "this file is at revision X, but that one is at > revision Y." There is only "the project is at revision X." Sigh. You are right. However, the said project is kind of exception. The binaries are there from the very beginning - they are indivisible part of the project and it won't work without them. This is why i am not worried if i revert to previous source code version, but actually check-out fresh binary - in my case it won't break things. >> There's a few examples out there that shows how to solve this, but it >> seems inconvenient and involves branching, cloning, etc. Isn't it >> possible to add something like: >> >> "git nohistory firmware.bin" >> >> or >> "git nohistory -i-understand-this-might-be-dangerous firmware.bin" > > Not easily. It goes against the underlying data model at the core of > git. Damn, i knew you'd say something like this. :-) > How big are your firmware files? How often do they change, and how large > are the changes? IOW, have you confirmed that repacking does not produce > an acceptable delta, meaning you get versioning for very low space cost? Changes don't happen too often, but the size of everything binary in the tree easily goes to about 100MB. Three commits later it ends up at about 300MB... cheers, Petko ``` ## Jeff King, 2008-01-16 14:18 Subject: Re: git and binary files Message-ID: <20080116141836.GA22639@coredump.intra.peff.net> URL: https://gitlist.dev/e/20080116141836.GA22639%40coredump.intra.peff.net In-Reply-To: ``` On Wed, Jan 16, 2008 at 04:14:30PM +0200, Petko Manolov wrote: >> How big are your firmware files? How often do they change, and how large >> are the changes? IOW, have you confirmed that repacking does not produce >> an acceptable delta, meaning you get versioning for very low space cost? > > Changes don't happen too often, but the size of everything binary in the > tree easily goes to about 100MB. Three commits later it ends up at about > 300MB... Right, as loose objects. Did you try running "git-gc" to repack? -Peff ``` ## Jakub Narebski, 2008-01-16 14:20 Subject: Re: git and binary files Message-ID: <200801161520.44668.jnareb@gmail.com> URL: https://gitlist.dev/e/200801161520.44668.jnareb%40gmail.com In-Reply-To: ``` Petko Manolov wrote: > On Wed, 16 Jan 2008, Jakub Narebski wrote: > > > You can always tag a blob (like junio-gpg-pub tag in git.git repository), > > but it wouldn't be in a working directory. But it would get distributed > > on clone. > > Hm, how does it work? You use git-hash-object to put file (-t blob) into the object database. It would return sha1 of added object. Use git-tag to create tag to blob (use returned sha1 for head). You can get file (to stdout) with "git cat-file blob tagname^{blob}". The file would be in object database, but not in working directory by default. > > BTW. if those large binary files doesn't differ much between version, > > they should get well compressed even if you would store them normally, > > all revisions. > > Unfortunately this is not the case. These binary blobs are already > compressed and/or encrypted and adding even a few bytes ends up storing > new version in full size. Can't you store them uncompressed? -- Jakub Narebski Poland ``` ## Petko Manolov, 2008-01-16 14:21 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161615300.5260%40bender.nucleusys.com In-Reply-To: ``` On Wed, 16 Jan 2008, Johannes Schindelin wrote: > No, you _do_ miss the point here. You might _think_ that they work > perfectly, but with revision control you want to have _exactly_ the same > setup. You want to be able to go back to a certain _revision_ > (including the then-current firmware). I _know_ older code will work with new binaries. I know because i've done it many times and the application is the sort that is not going to forgive any frivolity. Unfortunately this is very specific to what i'm doing and does not apply for 99.99% of what people usually need. > And that's what you don't want. So git is not for you. I use git for SCV from day one. It's great. I was just thinking aloud about something i've stumbled upon. ;-) Petko ``` ## Petko Manolov, 2008-01-16 14:25 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161622030.5260%40bender.nucleusys.com In-Reply-To: <20080116141836.GA22639@coredump.intra.peff.net> ``` On Wed, 16 Jan 2008, Jeff King wrote: > On Wed, Jan 16, 2008 at 04:14:30PM +0200, Petko Manolov wrote: > >>> How big are your firmware files? How often do they change, and how large >>> are the changes? IOW, have you confirmed that repacking does not produce >>> an acceptable delta, meaning you get versioning for very low space cost? >> >> Changes don't happen too often, but the size of everything binary in the >> tree easily goes to about 100MB. Three commits later it ends up at about >> 300MB... > > Right, as loose objects. Did you try running "git-gc" to repack? I did "git repack -f -a -d", but it didn't reduce the repository size. Those binaries are already compressed so any change adds up their size once again. cheers, Petko ``` ## Jeff King, 2008-01-16 14:32 Subject: Re: git and binary files Message-ID: <20080116143219.GA22744@coredump.intra.peff.net> URL: https://gitlist.dev/e/20080116143219.GA22744%40coredump.intra.peff.net In-Reply-To: ``` On Wed, Jan 16, 2008 at 04:25:28PM +0200, Petko Manolov wrote: >> Right, as loose objects. Did you try running "git-gc" to repack? > > I did "git repack -f -a -d", but it didn't reduce the repository size. > Those binaries are already compressed so any change adds up their size > once again. OK, that was the answer I was looking for; it looks like you are out of luck. BTW, the main space-saver in repacking is _not_ compression, but rather finding deltas between similar objects (e.g., two versions of the same file that, although large, differ only by a small amount). So even compressed files can still produce space savings during a repack, though perhaps not as well because of randomness introduced by the compression. As an experiment, it might be worth trying to store the uncompressed versions instead (git will delta _and_ compress them for you). -Peff ``` ## Wincent Colaiuta, 2008-01-16 14:34 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/D3716EB3-10B1-4D96-AB12-BD86CBB189CB%40wincent.com In-Reply-To: ``` El 16/1/2008, a las 14:58, Petko Manolov escribió: > On Wed, 16 Jan 2008, Johannes Schindelin wrote: > >> I think that you're missing the point of version control. It's not >> only about having an up-to-date source tree, but also about being >> able to go back to a certain revision. > > No contradiction here. In my case old source code will work > perfectly with new binaries/firmware. That's why i don't _need_ the > history, only the latest stuff in order to save space. You may be interested in the history, but the entire purpose of any version control system (not just Git) is to record exactly that: history. If the exact contents of these large binaries *really* don't matter, as you say they don't, than why don't you just commit one and never touch it again? Cheers, Wincent ``` ## Petko Manolov, 2008-01-16 14:39 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161634080.5260%40bender.nucleusys.com In-Reply-To: <20080116143219.GA22744@coredump.intra.peff.net> ``` On Wed, 16 Jan 2008, Jeff King wrote: > OK, that was the answer I was looking for; it looks like you are out > of luck. Story of my life. :-) > As an experiment, it might be worth trying to store the uncompressed > versions instead (git will delta _and_ compress them for you). I don't have them uncompressed. I can try to convert those files into ascii format and then save them in the repository. Since most changes are incremental git should be able to generate relatively small delta, which should compress well enough. Thanks for the hint. Petko ``` ## Petko Manolov, 2008-01-16 14:43 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161640010.5260%40bender.nucleusys.com In-Reply-To: <200801161520.44668.jnareb@gmail.com> ``` On Wed, 16 Jan 2008, Jakub Narebski wrote: > Petko Manolov wrote: >> On Wed, 16 Jan 2008, Jakub Narebski wrote: >> >>> You can always tag a blob (like junio-gpg-pub tag in git.git repository), >>> but it wouldn't be in a working directory. But it would get distributed >>> on clone. >> >> Hm, how does it work? > > You use git-hash-object to put file (-t blob) into the object database. > It would return sha1 of added object. Use git-tag to create tag to blob > (use returned sha1 for head). You can get file (to stdout) with > "git cat-file blob tagname^{blob}". Sounds like i'll have to play with the above. Thanks for the tip. > The file would be in object database, but not in working directory > by default. Not a big problem. >> Unfortunately this is not the case. These binary blobs are already >> compressed and/or encrypted and adding even a few bytes ends up storing >> new version in full size. > > Can't you store them uncompressed? Not really, but i can convert them into ascii format and store only the delta. This will admittedly increase the initial size of the repository, but hopefully not by much. Petko ``` ## Petko Manolov, 2008-01-16 14:45 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161643480.5260%40bender.nucleusys.com In-Reply-To: ``` On Wed, 16 Jan 2008, Wincent Colaiuta wrote: > If the exact contents of these large binaries *really* don't matter, as > you say they don't, than why don't you just commit one and never touch > it again? Unfortunately those binaries does change, although the process is slow and not very frequent. And this is why it pokes me in the eye - for changing a few bytes i end up with much larger repository. Petko ``` ## Nicolas Pitre, 2008-01-16 15:01 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.LFD.1.00.0801160958170.25841%40xanadu.home In-Reply-To: ``` On Wed, 16 Jan 2008, Petko Manolov wrote: > On Wed, 16 Jan 2008, Jakub Narebski wrote: > > > Petko Manolov wrote: > > > > > > Unfortunately this is not the case. These binary blobs are already > > > compressed and/or encrypted and adding even a few bytes ends up storing > > > new version in full size. > > > > Can't you store them uncompressed? > > Not really, but i can convert them into ascii format and store only the delta. If you don't have the original uncompressed unencrypted file, what will converting them to ascii actually give you? Nicolas ``` ## Rogan Dawes, 2008-01-16 15:05 Subject: Re: git and binary files Message-ID: <478E1D55.3050108@dawes.za.net> URL: https://gitlist.dev/e/478E1D55.3050108%40dawes.za.net In-Reply-To: ``` Petko Manolov wrote: > On Wed, 16 Jan 2008, Jeff King wrote: > >> OK, that was the answer I was looking for; it looks like you are out >> of luck. > > Story of my life. :-) > >> As an experiment, it might be worth trying to store the uncompressed >> versions instead (git will delta _and_ compress them for you). > > I don't have them uncompressed. > > I can try to convert those files into ascii format and then save them in > the repository. Since most changes are incremental git should be able > to generate relatively small delta, which should compress well enough. > > Thanks for the hint. > > > Petko That is unlikely to help, since git can find deltas in binary files just as easily as in text files. All you are doing is changing the encoding. Rogan ``` ## Petko Manolov, 2008-01-16 15:18 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161715570.5260%40bender.nucleusys.com In-Reply-To: ``` On Wed, 16 Jan 2008, Nicolas Pitre wrote: > On Wed, 16 Jan 2008, Petko Manolov wrote: > >> On Wed, 16 Jan 2008, Jakub Narebski wrote: >> >>> Petko Manolov wrote: >>>> >>>> Unfortunately this is not the case. These binary blobs are already >>>> compressed and/or encrypted and adding even a few bytes ends up storing >>>> new version in full size. >>> >>> Can't you store them uncompressed? >> >> Not really, but i can convert them into ascii format and store only the delta. > > If you don't have the original uncompressed unencrypted file, what will > converting them to ascii actually give you? I hope that in the case of incremental changes (0 to 5MB file is the same, last 64KB are actually new) the delta will be small and should be able to compress well. This won't work for random changes along the length of the whole file. Petko ``` ## Nicolas Pitre, 2008-01-16 15:58 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.LFD.1.00.0801161056500.25841%40xanadu.home In-Reply-To: ``` On Wed, 16 Jan 2008, Petko Manolov wrote: > On Wed, 16 Jan 2008, Nicolas Pitre wrote: > > > On Wed, 16 Jan 2008, Petko Manolov wrote: > > > > > On Wed, 16 Jan 2008, Jakub Narebski wrote: > > > > > > > Petko Manolov wrote: > > > > > > > > > > Unfortunately this is not the case. These binary blobs are already > > > > > compressed and/or encrypted and adding even a few bytes ends up > > > > > storing > > > > > new version in full size. > > > > > > > > Can't you store them uncompressed? > > > > > > Not really, but i can convert them into ascii format and store only the > > > delta. > > > > If you don't have the original uncompressed unencrypted file, what will > > converting them to ascii actually give you? > > I hope that in the case of incremental changes (0 to 5MB file is the same, > last 64KB are actually new) the delta will be small and should be able to > compress well. > > This won't work for random changes along the length of the whole file. But my question remains. If you cannot create good deltas out of your binary files, converting those binaries into ascii will do nothing to compression performance. Nicolas ``` ## Petko Manolov, 2008-01-16 16:06 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161802370.5260%40bender.nucleusys.com In-Reply-To: ``` On Wed, 16 Jan 2008, Nicolas Pitre wrote: > On Wed, 16 Jan 2008, Petko Manolov wrote: > >> On Wed, 16 Jan 2008, Nicolas Pitre wrote: >> >>> On Wed, 16 Jan 2008, Petko Manolov wrote: >>> >>>> On Wed, 16 Jan 2008, Jakub Narebski wrote: >>>> >>>>> Petko Manolov wrote: >>>>>> >>>>>> Unfortunately this is not the case. These binary blobs are already >>>>>> compressed and/or encrypted and adding even a few bytes ends up >>>>>> storing >>>>>> new version in full size. >>>>> >>>>> Can't you store them uncompressed? >>>> >>>> Not really, but i can convert them into ascii format and store only the >>>> delta. >>> >>> If you don't have the original uncompressed unencrypted file, what will >>> converting them to ascii actually give you? >> >> I hope that in the case of incremental changes (0 to 5MB file is the same, >> last 64KB are actually new) the delta will be small and should be able to >> compress well. >> >> This won't work for random changes along the length of the whole file. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > But my question remains. > > If you cannot create good deltas out of your binary files, converting > those binaries into ascii will do nothing to compression performance. I did agree with you above. Binary or ascii it doesn't matter. The experiments so far shows that git is doing a good job finding deltas in binary files. This may be a solution for certain part of my files... Petko ``` ## Petko Manolov, 2008-01-16 16:09 Subject: Re: git and binary files Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0801161806220.5260%40bender.nucleusys.com In-Reply-To: ``` On Wed, 16 Jan 2008, Nicolas Pitre wrote: > On Wed, 16 Jan 2008, Petko Manolov wrote: > >> On Wed, 16 Jan 2008, Jakub Narebski wrote: >> >>> Petko Manolov wrote: >>>> >>>> Unfortunately this is not the case. These binary blobs are already >>>> compressed and/or encrypted and adding even a few bytes ends up storing >>>> new version in full size. >>> >>> Can't you store them uncompressed? >> >> Not really, but i can convert them into ascii format and store only the delta. > > If you don't have the original uncompressed unencrypted file, what will > converting them to ascii actually give you? Not much. I was blindly hoping that bzip2 may compress such files better, though i haven't tested this hypothesis yet. Will let you know if the result is different from the obvious one. :-) Petko ``` ## Jakub Narebski, 2008-01-16 16:34 Subject: Re: git and binary files Message-ID: <200801161734.13481.jnareb@gmail.com> URL: https://gitlist.dev/e/200801161734.13481.jnareb%40gmail.com In-Reply-To: ``` On Wed, 16 January 2008, Petko Manolov wrote: > On Wed, 16 Jan 2008, Jakub Narebski wrote: >> Petko Manolov wrote: >>> On Wed, 16 Jan 2008, Jakub Narebski wrote: >>> >>>> You can always tag a blob (like junio-gpg-pub tag in git.git repository), >>>> but it wouldn't be in a working directory. But it would get distributed >>>> on clone. >>> >>> Hm, how does it work? >> >> You use git-hash-object to put file (-t blob) into the object database. >> It would return sha1 of added object. Use git-tag to create tag to blob >> (use returned sha1 for head). You can get file (to stdout) with >> "git cat-file blob tagname^{blob}". > > Sounds like i'll have to play with the above. Thanks for the tip. > >> The file would be in object database, but not in working directory >> by default. > > Not a big problem. Please read carefully section about retagging / changing a tag in git-tag(1) manpage; you should take care about propagating change if you ever change the binary blob. Nothe that I haven't used the technique described. -- Jakub Narebski Poland ``` ## Florian Weimer, 2008-01-16 16:41 Subject: Re: git and binary files Message-ID: <82r6gh9104.fsf@mid.bfk.de> URL: https://gitlist.dev/e/82r6gh9104.fsf%40mid.bfk.de In-Reply-To: ``` * Petko Manolov: >>> Unfortunately this is not the case. These binary blobs are already >>> compressed and/or encrypted and adding even a few bytes ends up storing >>> new version in full size. >> >> Can't you store them uncompressed? > > Not really, but i can convert them into ascii format and store only > the delta. This will admittedly increase the initial size of the > repository, but hopefully not by much. If the encryption is halfway decent, a new IV/nonce will be chosen for each new revision, and you can't tell that two ciphertexts share a common prefix without fully decrypting them. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 ``` ## Junio C Hamano, 2008-01-16 18:02 Subject: Re: git and binary files Message-ID: <7vwsq9r6mf.fsf@gitster.siamese.dyndns.org> URL: https://gitlist.dev/e/7vwsq9r6mf.fsf%40gitster.siamese.dyndns.org In-Reply-To: ``` Petko Manolov writes: > On Wed, 16 Jan 2008, Wincent Colaiuta wrote: > >> If the exact contents of these large binaries *really* don't matter, >> as you say they don't, than why don't you just commit one and never >> touch it again? > > Unfortunately those binaries does change, although the process is slow > and not very frequent. And this is why it pokes me in the eye - for > changing a few bytes i end up with much larger repository. For changing a few bytes you get with much larger repository? What happened to our packfiles? Is it possible for us to take a look at two versions of such a binary blob, one before the change and one after such a change that touches only a few bytes? ``` ## Junio C Hamano, 2008-01-16 18:09 Subject: Re: git and binary files Message-ID: <7vprw1r6ai.fsf@gitster.siamese.dyndns.org> URL: https://gitlist.dev/e/7vprw1r6ai.fsf%40gitster.siamese.dyndns.org In-Reply-To: <7vwsq9r6mf.fsf@gitster.siamese.dyndns.org> ``` Junio C Hamano writes: > Petko Manolov writes: > >> On Wed, 16 Jan 2008, Wincent Colaiuta wrote: >> >>> If the exact contents of these large binaries *really* don't matter, >>> as you say they don't, than why don't you just commit one and never >>> touch it again? >> >> Unfortunately those binaries does change, although the process is slow >> and not very frequent. And this is why it pokes me in the eye - for >> changing a few bytes i end up with much larger repository. > > For changing a few bytes you get with much larger repository? > What happened to our packfiles? > > Is it possible for us to take a look at two versions of such a > binary blob, one before the change and one after such a change > that touches only a few bytes? I replied before reading the rest of the thread. Please ignore me on this one. ``` ## David Brown, 2008-01-18 06:52 Subject: Re: git and binary files Message-ID: <20080118065201.GA4478@old.davidb.org> URL: https://gitlist.dev/e/20080118065201.GA4478%40old.davidb.org In-Reply-To: ``` On Wed, Jan 16, 2008 at 04:39:19PM +0200, Petko Manolov wrote: >> As an experiment, it might be worth trying to store the uncompressed >> versions instead (git will delta _and_ compress them for you). > > I don't have them uncompressed. > > I can try to convert those files into ascii format and then save them in > the repository. Since most changes are incremental git should be able to > generate relatively small delta, which should compress well enough. This won't help, only make things slightly worse. The problem is when you compress the files, as soon as there is a difference between the two files, the output of the compressor will be entirely different for the remainder of the file. Unless you changes always are at the end of the file, you will end up pretty much entire versions in the repo. I'm using some repos that get regular 50MB tarballs checked into them. I convinced them to not compress the tarballs, and git does a fairly decent job of doing delta compression between them (although it needs quite a bit of RAM to do so). Dave ```