git and binary files

>
> The nature of these binary files is such that i don't care neither about
> their history nor older versions.  Hence the question:  is there an easy
> way to tell git not to bother about the history of these particular files
> and keep the most recent version only?

> On Wed, 16 Jan 2008, David Symonds wrote:
> 
> > If you don't care about versioning those files, why would you use a 
> > version control system? Just store them somewhere else, and use 
> > symlinks.
> 
> That is certainly a way of doing it.  However, it will be much simpler 
> and fast to be able to "git clone" and then "git pull" every once in a 
> while. The alternative involves "cp -a" or most likely "scp -r" the 
> binaries along with the repository and you can never be sure that both 
> are in sync.

> On Wed, 16 Jan 2008, Johannes Schindelin wrote:
> 
> > I think that you're missing the point of version control.  It's not 
> > only about having an up-to-date source tree, but also about being able 
> > to go back to a certain revision.
> 
> No contradiction here.  In my case old source code will work perfectly 
> with new binaries/firmware.  That's why i don't _need_ the history, only 
> the latest stuff in order to save space.

> On Wed, 16 Jan 2008, Johannes Schindelin wrote:
>
>> I think that you're missing the point of version control.  It's not  
>> only about having an up-to-date source tree, but also about being  
>> able to go back to a certain revision.
>
> No contradiction here.  In my case old source code will work  
> perfectly with new binaries/firmware.  That's why i don't _need_ the  
> history, only the latest stuff in order to save space.

> On Wed, 16 Jan 2008, Wincent Colaiuta wrote:
>
>> If the exact contents of these large binaries *really* don't matter,
>> as you say they don't, than why don't you just commit one and never
>> touch it again?
>
> Unfortunately those binaries does change, although the process is slow
> and not very frequent.  And this is why it pokes me in the eye - for
> changing a few bytes i end up with much larger repository.

> Petko Manolov <petkan@nucleusys.com> writes:
>
>> On Wed, 16 Jan 2008, Wincent Colaiuta wrote:
>>
>>> If the exact contents of these large binaries *really* don't matter,
>>> as you say they don't, than why don't you just commit one and never
>>> touch it again?
>>
>> Unfortunately those binaries does change, although the process is slow
>> and not very frequent.  And this is why it pokes me in the eye - for
>> changing a few bytes i end up with much larger repository.
>
> For changing a few bytes you get with much larger repository?
> What happened to our packfiles?
>
> Is it possible for us to take a look at two versions of such a
> binary blob, one before the change and one after such a change
> that touches only a few bytes?

> Some of my projects require having binary files (firmware and other 
> stuff) somewhere in the tree structure.  Unfortunately these files are 
> big - 50MB and more.  After a couple of new versions arrive (and get 
> committed) i end up with a repository way too big than necessary.
> 
> The nature of these binary files is such that i don't care neither about 
> their history nor older versions.  Hence the question:  is there an easy 
> way to tell git not to bother about the history of these particular 
> files and keep the most recent version only?

> On Wed, 16 Jan 2008, Johannes Schindelin wrote:
> 
> > The answer is no.  You cannot ask git to have the newest version of
> > something, but not the old ones.  It contradicts the distributedness
> > of git, too.
> 
> I don't agree here.  Assume that whatever you're working on require
> firmware for a device that won't change during the lifetime of the
> software project.  The newest version of the said firmware is mostly
> bugfixes and you basically don't want to revert to the older
> ones. Consider the microcode for modern Pentiums, Core 2, etc.
> 
> What i am trying to suggest is that there might be cases when you need
> something in the repository, but you don't want GIT to keep it's
> history nor it's predecessors.  Leaving it out breaks the atomicity of
> such repository and makes the project management more complex.
> 
> There's a few examples out there that shows how to solve this, but it
> seems inconvenient and involves branching, cloning, etc.  Isn't it
> possible to add something like:
> 
>  	"git nohistory firmware.bin"
> 
> or
>  	"git nohistory -i-understand-this-might-be-dangerous firmware.bin"

> On Wed, 16 Jan 2008, Jakub Narebski wrote:
> 
> > You can always tag a blob (like junio-gpg-pub tag in git.git repository),
> > but it wouldn't be in a working directory. But it would get distributed
> > on clone.
> 
> Hm, how does it work?

> > BTW. if those large binary files doesn't differ much between version, 
> > they should get well compressed even if you would store them normally, 
> > all revisions.
> 
> Unfortunately this is not the case.  These binary blobs are already 
> compressed and/or encrypted and adding even a few bytes ends up storing 
> new version in full size.

> Petko Manolov wrote:
>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>
>>> You can always tag a blob (like junio-gpg-pub tag in git.git repository),
>>> but it wouldn't be in a working directory. But it would get distributed
>>> on clone.
>>
>> Hm, how does it work?
>
> You use git-hash-object to put file (-t blob) into the object database.
> It would return sha1 of added object. Use git-tag to create tag to blob
> (use returned sha1 for head). You can get file (to stdout) with
> "git cat-file blob tagname^{blob}".

>> Unfortunately this is not the case.  These binary blobs are already
>> compressed and/or encrypted and adding even a few bytes ends up storing
>> new version in full size.
>
> Can't you store them uncompressed?

> On Wed, 16 Jan 2008, Jakub Narebski wrote:
> 
> > Petko Manolov wrote:
> > > 
> > > Unfortunately this is not the case.  These binary blobs are already
> > > compressed and/or encrypted and adding even a few bytes ends up storing
> > > new version in full size.
> > 
> > Can't you store them uncompressed?
> 
> Not really, but i can convert them into ascii format and store only the delta.

> On Wed, 16 Jan 2008, Petko Manolov wrote:
>
>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>
>>> Petko Manolov wrote:
>>>>
>>>> Unfortunately this is not the case.  These binary blobs are already
>>>> compressed and/or encrypted and adding even a few bytes ends up storing
>>>> new version in full size.
>>>
>>> Can't you store them uncompressed?
>>
>> Not really, but i can convert them into ascii format and store only the delta.
>
> If you don't have the original uncompressed unencrypted file, what will
> converting them to ascii actually give you?

> On Wed, 16 Jan 2008, Nicolas Pitre wrote:
> 
> > On Wed, 16 Jan 2008, Petko Manolov wrote:
> > 
> > > On Wed, 16 Jan 2008, Jakub Narebski wrote:
> > > 
> > > > Petko Manolov wrote:
> > > > > 
> > > > > Unfortunately this is not the case.  These binary blobs are already
> > > > > compressed and/or encrypted and adding even a few bytes ends up
> > > > > storing
> > > > > new version in full size.
> > > > 
> > > > Can't you store them uncompressed?
> > > 
> > > Not really, but i can convert them into ascii format and store only the
> > > delta.
> > 
> > If you don't have the original uncompressed unencrypted file, what will
> > converting them to ascii actually give you?
> 
> I hope that in the case of incremental changes (0 to 5MB file is the same,
> last 64KB are actually new) the delta will be small and should be able to
> compress well.
> 
> This won't work for random changes along the length of the whole file.

> On Wed, 16 Jan 2008, Petko Manolov wrote:
>
>> On Wed, 16 Jan 2008, Nicolas Pitre wrote:
>>
>>> On Wed, 16 Jan 2008, Petko Manolov wrote:
>>>
>>>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>>>
>>>>> Petko Manolov wrote:
>>>>>>
>>>>>> Unfortunately this is not the case.  These binary blobs are already
>>>>>> compressed and/or encrypted and adding even a few bytes ends up
>>>>>> storing
>>>>>> new version in full size.
>>>>>
>>>>> Can't you store them uncompressed?
>>>>
>>>> Not really, but i can convert them into ascii format and store only the
>>>> delta.
>>>
>>> If you don't have the original uncompressed unencrypted file, what will
>>> converting them to ascii actually give you?
>>
>> I hope that in the case of incremental changes (0 to 5MB file is the same,
>> last 64KB are actually new) the delta will be small and should be able to
>> compress well.
>>
>> This won't work for random changes along the length of the whole file.

> On Wed, 16 Jan 2008, Petko Manolov wrote:
>
>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>
>>> Petko Manolov wrote:
>>>>
>>>> Unfortunately this is not the case.  These binary blobs are already
>>>> compressed and/or encrypted and adding even a few bytes ends up storing
>>>> new version in full size.
>>>
>>> Can't you store them uncompressed?
>>
>> Not really, but i can convert them into ascii format and store only the delta.
>
> If you don't have the original uncompressed unencrypted file, what will
> converting them to ascii actually give you?

> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>> Petko Manolov wrote:
>>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>>
>>>> You can always tag a blob (like junio-gpg-pub tag in git.git repository),
>>>> but it wouldn't be in a working directory. But it would get distributed
>>>> on clone.
>>>
>>> Hm, how does it work?
>>
>> You use git-hash-object to put file (-t blob) into the object database.
>> It would return sha1 of added object. Use git-tag to create tag to blob
>> (use returned sha1 for head). You can get file (to stdout) with
>> "git cat-file blob tagname^{blob}".
> 
> Sounds like i'll have to play with the above.  Thanks for the tip.
> 
>> The file would be in object database, but not in working directory
>> by default.
> 
> Not a big problem.

>>> Unfortunately this is not the case.  These binary blobs are already
>>> compressed and/or encrypted and adding even a few bytes ends up storing
>>> new version in full size.
>>
>> Can't you store them uncompressed?
>
> Not really, but i can convert them into ascii format and store only
> the delta.  This will admittedly increase the initial size of the
> repository, but hopefully not by much.

> There's a few examples out there that shows how to solve this, but it  
> seems inconvenient and involves branching, cloning, etc.  Isn't it  
> possible to add something like:
>
> 	"git nohistory firmware.bin"
>
> or
> 	"git nohistory -i-understand-this-might-be-dangerous firmware.bin"

> On Wed, Jan 16, 2008 at 03:39:06PM +0200, Petko Manolov wrote:
>
>> What i am trying to suggest is that there might be cases when you need
>> something in the repository, but you don't want GIT to keep it's history
>> nor it's predecessors.  Leaving it out breaks the atomicity of such
>> repository and makes the project management more complex.
>
> But not versioning some files while versioning others breaks the
> atomicity of project version, which is at the core of git's model. There
> is no such thing as "this file is at revision X, but that one is at
> revision Y." There is only "the project is at revision X."

>> There's a few examples out there that shows how to solve this, but it
>> seems inconvenient and involves branching, cloning, etc.  Isn't it
>> possible to add something like:
>>
>> 	"git nohistory firmware.bin"
>>
>> or
>> 	"git nohistory -i-understand-this-might-be-dangerous firmware.bin"
>
> Not easily. It goes against the underlying data model at the core of
> git.

>> How big are your firmware files? How often do they change, and how large 
>> are the changes? IOW, have you confirmed that repacking does not produce 
>> an acceptable delta, meaning you get versioning for very low space cost?
>
> Changes don't happen too often, but the size of everything binary in the  
> tree easily goes to about 100MB.  Three commits later it ends up at about  
> 300MB...

> On Wed, Jan 16, 2008 at 04:14:30PM +0200, Petko Manolov wrote:
>
>>> How big are your firmware files? How often do they change, and how large
>>> are the changes? IOW, have you confirmed that repacking does not produce
>>> an acceptable delta, meaning you get versioning for very low space cost?
>>
>> Changes don't happen too often, but the size of everything binary in the
>> tree easily goes to about 100MB.  Three commits later it ends up at about
>> 300MB...
>
> Right, as loose objects. Did you try running "git-gc" to repack?

>> Right, as loose objects. Did you try running "git-gc" to repack?
>
> I did "git repack -f -a -d", but it didn't reduce the repository size.  
> Those binaries are already compressed so any change adds up their size  
> once again.

> On Wed, 16 Jan 2008, Jeff King wrote:
> 
>> OK, that was the answer I was looking for; it looks like you are out
>> of luck.
> 
> Story of my life. :-)
> 
>> As an experiment, it might be worth trying to store the uncompressed
>> versions instead (git will delta _and_ compress them for you).
> 
> I don't have them uncompressed.
> 
> I can try to convert those files into ascii format and then save them in 
> the repository.  Since most changes are incremental git should be able 
> to generate relatively small delta, which should compress well enough.
> 
> Thanks for the hint.
> 
> 
>         Petko

>> As an experiment, it might be worth trying to store the uncompressed
>> versions instead (git will delta _and_ compress them for you).
>
> I don't have them uncompressed.
>
> I can try to convert those files into ascii format and then save them in 
> the repository.  Since most changes are incremental git should be able to 
> generate relatively small delta, which should compress well enough.