threads / discuss / 26789

About Summer of code idea -- better big-file support

Subject: About Summer of code idea -- better big-file support

## tl;dr

5 messages between Mar 19, 2011 and Mar 20, 2011.

replies: 4people: 2as markdown or json

徐迪· Mar 19, 2011, 13:15 UTC · lore

Hi, there

I am wondering start implementing the idea about better big-file support during summer of code. Here are my idea: often gigantic files is media file, such as rmvb, swf, pdf, dll, etc. And those file themselves are already being compressed, as we all know, git uses zlib to compress all the object in repository, and it is the common sense that if we use zlib to compress those file, the compress ratio will be unbelievable low, from my test if we compress 521MiB mkv file it will be 520MiB after compressed, how ridiculous it is! Also I test that if we use "git hash-object" to calculate SHA-1, and copy all content to repository, it will costs 1 minutes, once we use "git hash-object -w" it will costs 2 minutes. So we sacrificed a lot, but gained a little. So, how do you guys think about that we can test the file type, and then decide to use zlib or not? Cheer Di

Nguyen Thai Ngoc Duy· Mar 19, 2011, 13:48 UTC · re: 徐迪 · lore

Re: About Summer of code idea -- better big-file support

2011/3/19 徐迪 <xudifsd@gmail.com>:

> So, how do you guys think about that we can test the file type, and then
> decide to use zlib or not?

We already have core.bigFileThreshold, which can be used to determine storage type (traditional one with zlib, or some fancy ones for big files). If that's not enough, .gitattributes can be used to mark certain files as 'big'.

See below link for recent discussions regarding big file support:

http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956

On the store of big files, bup (at github) model might fit in as well if large files change a small portion regularly.

-- 
Duy

徐迪· Mar 19, 2011, 14:28 UTC · re: Nguyen Thai Ngoc Duy · lore

Re: About Summer of code idea -- better big-file support

2011/3/19, Nguyen Thai Ngoc Duy <pclouds@gmail.com>:

Show 18 quoted lines

> 2011/3/19 徐迪 <xudifsd@gmail.com>:
>> So, how do you guys think about that we can test the file type, and then
>> decide to use zlib or not?
>
> We already have core.bigFileThreshold, which can be used to determine
> storage type (traditional one with zlib, or some fancy ones for big
> files). If that's not enough, .gitattributes can be used to mark
> certain files as 'big'.
>
> See below link for recent discussions regarding big file support:
>
> http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956
>
> On the store of big files, bup (at github) model might fit in as well
> if large files change a small portion regularly.
> --
> Duy
>

But, why those features haven't merged into upstream developement? Sorry for this question, but I'm fresh here.

Nguyen Thai Ngoc Duy· Mar 19, 2011, 15:13 UTC · re: 徐迪 · lore

Re: About Summer of code idea -- better big-file support

2011/3/19 徐迪 <xudifsd@gmail.com>:

Show 10 quoted lines

> 2011/3/19, Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>> See below link for recent discussions regarding big file support:
>>
>> http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956
>>
>> On the store of big files, bup (at github) model might fit in as well
>> if large files change a small portion regularly.
>>
> But, why those features haven't merged into upstream developement?
> Sorry for this question, but I'm fresh here.

This is just my guess. For specific apps, you should pull their authors in.

All apps experiment different ways of dealing with big files and work in certain situations. None is generic enough to be merged upstream. Take bup as an example, it aims at big files _only_. A file is treated as a directory of many smaller pieces. In order to get it merged, we need to separate those big files from the rest because it's just too different in handling them.

-- 
Duy

徐迪· Mar 20, 2011, 12:33 UTC · re: Nguyen Thai Ngoc Duy · lore

Re: About Summer of code idea -- better big-file support

2011/3/19 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:

Show 23 quoted lines

> 2011/3/19 徐迪 <xudifsd@gmail.com>:
>> 2011/3/19, Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>>> See below link for recent discussions regarding big file support:
>>>
>>> http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956
>>>
>>> On the store of big files, bup (at github) model might fit in as well
>>> if large files change a small portion regularly.
>>>
>> But, why those features haven't merged into upstream developement?
>> Sorry for this question, but I'm fresh here.
>
> This is just my guess. For specific apps, you should pull their authors in.
>
> All apps experiment different ways of dealing with big files and work
> in certain situations. None is generic enough to be merged upstream.
> Take bup as an example, it aims at big files _only_. A file is treated
> as a directory of many smaller pieces. In order to get it merged, we
> need to separate those big files from the rest because it's just too
> different in handling them.
> --
> Duy
>

Thanks for that timely link, I think I should do some more research now :P