# About Summer of code idea -- better big-file support 5 messages from 2011-03-19 to 2011-03-20. Participants: 徐迪, Nguyen Thai Ngoc Duy. Thread: https://gitlist.dev/t/26789 ## 徐迪, 2011-03-19 13:15 Subject: About Summer of code idea -- better big-file support Message-ID: URL: https://gitlist.dev/e/AANLkTi%3DZ_4cGBSPQW2YVuj%3Ddd8mcO4YQ0G2owknt7vfw%40mail.gmail.com ``` Hi, there I am wondering start implementing the idea about better big-file support during summer of code. Here are my idea: often gigantic files is media file, such as rmvb, swf, pdf, dll, etc. And those file themselves are already being compressed, as we all know, git uses zlib to compress all the object in repository, and it is the common sense that if we use zlib to compress those file, the compress ratio will be unbelievable low, from my test if we compress 521MiB mkv file it will be 520MiB after compressed, how ridiculous it is! Also I test that if we use "git hash-object" to calculate SHA-1, and copy all content to repository, it will costs 1 minutes, once we use "git hash-object -w" it will costs 2 minutes. So we sacrificed a lot, but gained a little. So, how do you guys think about that we can test the file type, and then decide to use zlib or not? Cheer Di ``` ## Nguyen Thai Ngoc Duy, 2011-03-19 13:48 Subject: Re: About Summer of code idea -- better big-file support Message-ID: URL: https://gitlist.dev/e/AANLkTikHigPhaWkiozsurbxjvR5WkAGW9LUtFRT9tK8%3D%40mail.gmail.com In-Reply-To: ``` 2011/3/19 徐迪 : > So, how do you guys think about that we can test the file type, and then > decide to use zlib or not? We already have core.bigFileThreshold, which can be used to determine storage type (traditional one with zlib, or some fancy ones for big files). If that's not enough, .gitattributes can be used to mark certain files as 'big'. See below link for recent discussions regarding big file support: http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956 On the store of big files, bup (at github) model might fit in as well if large files change a small portion regularly. -- Duy ``` ## 徐迪, 2011-03-19 14:28 Subject: Re: About Summer of code idea -- better big-file support Message-ID: URL: https://gitlist.dev/e/AANLkTikp5GS0gnqLVgZY4E0wBDmQBp1iftOON3MSMHea%40mail.gmail.com In-Reply-To: ``` 2011/3/19, Nguyen Thai Ngoc Duy : > 2011/3/19 徐迪 : >> So, how do you guys think about that we can test the file type, and then >> decide to use zlib or not? > > We already have core.bigFileThreshold, which can be used to determine > storage type (traditional one with zlib, or some fancy ones for big > files). If that's not enough, .gitattributes can be used to mark > certain files as 'big'. > > See below link for recent discussions regarding big file support: > > http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956 > > On the store of big files, bup (at github) model might fit in as well > if large files change a small portion regularly. > -- > Duy > But, why those features haven't merged into upstream developement? Sorry for this question, but I'm fresh here. ``` ## Nguyen Thai Ngoc Duy, 2011-03-19 15:13 Subject: Re: About Summer of code idea -- better big-file support Message-ID: URL: https://gitlist.dev/e/AANLkTimH9pgO7JxN3vOcN%3DKb3QGpbM7OO4bMpL2eAs%3D3%40mail.gmail.com In-Reply-To: ``` 2011/3/19 徐迪 : > 2011/3/19, Nguyen Thai Ngoc Duy : >> See below link for recent discussions regarding big file support: >> >> http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956 >> >> On the store of big files, bup (at github) model might fit in as well >> if large files change a small portion regularly. >> > But, why those features haven't merged into upstream developement? > Sorry for this question, but I'm fresh here. This is just my guess. For specific apps, you should pull their authors in. All apps experiment different ways of dealing with big files and work in certain situations. None is generic enough to be merged upstream. Take bup as an example, it aims at big files _only_. A file is treated as a directory of many smaller pieces. In order to get it merged, we need to separate those big files from the rest because it's just too different in handling them. -- Duy ``` ## 徐迪, 2011-03-20 12:33 Subject: Re: About Summer of code idea -- better big-file support Message-ID: URL: https://gitlist.dev/e/AANLkTimTWLqkaFkiruqZ8FU8y3YX9C4ujPOPTTJzdRtk%40mail.gmail.com In-Reply-To: ``` 2011/3/19 Nguyen Thai Ngoc Duy : > 2011/3/19 徐迪 : >> 2011/3/19, Nguyen Thai Ngoc Duy : >>> See below link for recent discussions regarding big file support: >>> >>> http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956 >>> >>> On the store of big files, bup (at github) model might fit in as well >>> if large files change a small portion regularly. >>> >> But, why those features haven't merged into upstream developement? >> Sorry for this question, but I'm fresh here. > > This is just my guess. For specific apps, you should pull their authors in. > > All apps experiment different ways of dealing with big files and work > in certain situations. None is generic enough to be merged upstream. > Take bup as an example, it aims at big files _only_. A file is treated > as a directory of many smaller pieces. In order to get it merged, we > need to separate those big files from the rest because it's just too > different in handling them. > -- > Duy > Thanks for that timely link, I think I should do some more research now :P ```