threads / discuss / 12378

Will git have a baseline feature or something alike?

Subject: Will git have a baseline feature or something alike?

## tl;dr

16 messages between Feb 29, 2008 and Mar 2, 2008.

replies: 15people: 11as markdown or json

eric miao· Feb 29, 2008, 09:23 UTC · lore
All,
I kept a mirror of
http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
by a crontab task fetching the updated commits at midnight everyday.

Yet I found the repository now grows to be 1.2G without checking out anything. The checked out working tree of this is about 1.5G.

I tried "git prune" and "git repack" but it still remains so large. The trend of the kernel is still going to be enlarged. Thus I'm thinking of the possibility of a baseline feature. One can totally forget about the history before that baseline, and start the development there after.

E.g.
1. user downloads a released tarball
2. and build a repository
3. and "git fetch" will find the current repository is identical to
a baseline in the remote, and fetches only commits after
that baseline
4. continue the development work

The above steps with current git will generate a totally different hash value for the files in the downloaded tarball, thus making it failed to fetch commits thereafter.

I know the history is usually mixed with multiple branches, which makes this baseline feature a bit difficult to implement. It should be a nice feature, though.

-- 
Cheers
- eric
Sean· Feb 29, 2008, 09:56 UTC · re: eric miao · lore

Re: Will git have a baseline feature or something alike?

On Fri, 29 Feb 2008 17:23:37 +0800 "eric miao" <eric.y.miao@gmail.com> wrote:

Hi Eric,
Show 8 quoted lines
> I kept a mirror of
> 
> http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> 
> by a crontab task fetching the updated commits at midnight everyday.
> 
> Yet I found the repository now grows to be 1.2G without checking out
> anything. The checked out working tree of this is about 1.5G.

There's something wrong in your setup, the entire kernel history should take less than 200M.

Show 5 quoted lines
> I tried "git prune" and "git repack" but it still remains so large. The
> trend of the kernel is still going to be enlarged. Thus I'm thinking
> of the possibility of a baseline feature. One can totally forget about
> the history before that baseline, and start the development there
> after.

Git provides "shallow" clones which essentially give you that ability today via the --depth option. While it won't automatically determine a baseline, it avoids the need for you to download a tarball first.

Sean
Karl Hasselström· Feb 29, 2008, 10:38 UTC · re: Sean · lore

Re: Will git have a baseline feature or something alike?

On 2008-02-29 04:56:09 -0500, Sean wrote:
Show 9 quoted lines
> On Fri, 29 Feb 2008 17:23:37 +0800
>
> "eric miao" <eric.y.miao@gmail.com> wrote:
>
> > Yet I found the repository now grows to be 1.2G without checking
> > out anything. The checked out working tree of this is about 1.5G.
>
> There's something wrong in your setup, the entire kernel history
> should take less than 200M.
Try running "git gc".
-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle
Jakub Narebski· Feb 29, 2008, 13:21 UTC · re: eric miao · lore

Re: Will git have a baseline feature or something alike?

"eric miao" <eric.y.miao@gmail.com> writes:
Show 8 quoted lines
> I kept a mirror of
> 
> http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> 
> by a crontab task fetching the updated commits at midnight everyday.
> 
> Yet I found the repository now grows to be 1.2G without checking out
> anything. The checked out working tree of this is about 1.5G.

Did you (re)packed this repository, running "git gc", or "git repack"? Currently git either downloads small packs, or loose objects; it needs to repack to make repository size smaller.

BTW. the largest git repository is 1.6G OpenOffice.org conversion,
with > 2G checkout, and some large binary files under version
control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
So kernel should be quite smaller.
Show 5 quoted lines
> I tried "git prune" and "git repack" but it still remains so large. The
> trend of the kernel is still going to be enlarged. Thus I'm thinking
> of the possibility of a baseline feature. One can totally forget about
> the history before that baseline, and start the development there
> after.

There is so called "shallow clone" feature, which allows to clone only part of history. Currently it dupports only --depth, i.e. number of commits from tips; it could I guess support providing tag as delimiter. (You are welcome to implement it ;-).

-- 
Jakub Narebski
Poland
ShadeHawk on #git
Sam Vilain· Mar 1, 2008, 07:04 UTC · re: Jakub Narebski · lore

Re: Will git have a baseline feature or something alike?

Jakub Narebski wrote:
> BTW. the largest git repository is 1.6G OpenOffice.org conversion,
> with > 2G checkout, and some large binary files under version
> control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
> So kernel should be quite smaller.
I have an 8GB git-svn import of the KDE repository :-)
Sam.
Jakub Narebski· Mar 1, 2008, 12:39 UTC · re: Sam Vilain · lore

Re: Will git have a baseline feature or something alike?

On Sat, 1 Mar 2008, Sam Vilain wrote:
Show 8 quoted lines
> Jakub Narebski wrote:
>
>> BTW. the largest git repository is 1.6G OpenOffice.org conversion,
>> with > 2G checkout, and some large binary files under version
>> control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
>> So kernel should be quite smaller.
> 
> I have an 8GB git-svn import of the KDE repository :-)

First, how large full checkout is? And how large Subversion repo? Second, is this repository tightly packed (large window, big delta chain)?

And last, KDE repos should most probably be split into submodules.
-- 
Jakub Narebski
Poland
Sam Vilain· Mar 1, 2008, 13:20 UTC · re: Jakub Narebski · lore

Re: Will git have a baseline feature or something alike?

Jakub Narebski wrote:
Show 10 quoted lines
> On Sat, 1 Mar 2008, Sam Vilain wrote:
>> Jakub Narebski wrote:
>>
>>> BTW. the largest git repository is 1.6G OpenOffice.org conversion,
>>> with > 2G checkout, and some large binary files under version
>>> control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
>>> So kernel should be quite smaller.
>> I have an 8GB git-svn import of the KDE repository :-)
> 
> First, how large full checkout is? 
Heh, haven't tried tbh.
> And how large Subversion repo?

48GB apparently. This repository is approx. 90% of the subversion repository - I didn't prepare it.

> Second, is this repository tightly packed (large window, big delta
> chain)?
Yes, I don't have the details, though.
> And last, KDE repos should most probably be split into submodules.

Mmm. Everyone always says that; what it really needs I think is someone to really take the conversion on board and come up with a workable plan on this front. I think the counter-argument to this was "but you always want to have 70% of the repository checked out for development". Counter-counter argument is "yes but they don't always need to be deep clones". Anyway, it's not my baby, just thought I'd let you know about it :-)

Sam.
Jan Hudec· Mar 2, 2008, 19:38 UTC · re: Sam Vilain · lore

Re: Will git have a baseline feature or something alike?

On Sun, Mar 02, 2008 at 02:20:35 +1300, Sam Vilain wrote:
Show 7 quoted lines
> Jakub Narebski wrote:
> > And last, KDE repos should most probably be split into submodules.
> 
> Mmm.  Everyone always says that; what it really needs I think is someone
> to really take the conversion on board and come up with a workable plan
> on this front.  I think the counter-argument to this was "but you always
> want to have 70% of the repository checked out for development".

KDE should not be the case. The source for Debian packages is split in several independent source packages (which are compilable on their own). You should rarely need more than 1 or 2 (the one you hack + base) checked out.

> Counter-counter argument is "yes but they don't always need to be deep
> clones".  Anyway, it's not my baby, just thought I'd let you know about
> it :-)
-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>
Martin Langhoff· Mar 2, 2008, 21:29 UTC · re: Sam Vilain · lore

Re: Will git have a baseline feature or something alike?

On Sun, Mar 2, 2008 at 2:20 AM, Sam Vilain <sam@vilain.net> wrote:
Show 9 quoted lines
>  > And last, KDE repos should most probably be split into submodules.
>
>  Mmm.  Everyone always says that; what it really needs I think is someone
>  to really take the conversion on board and come up with a workable plan
>  on this front.  I think the counter-argument to this was "but you always
>  want to have 70% of the repository checked out for development".
>  Counter-counter argument is "yes but they don't always need to be deep
>  clones".  Anyway, it's not my baby, just thought I'd let you know about
>  it :-)

That sounds a lot like the x.org submodule split. AFAIU, it did involve some relatively minor changes to configure/build scripts to handle the submodules being built somewhere other than in a subdir of the main project. Nothing earth shattering, just small changes to how the build works, and I think after the initial shakeout everyone's happy and the development pace is quite a bit faster & more adventurous.

All of this from bits I read in the x.org dev lists, and some developer blogs - IANAX.OD

cheers,
m
eric miao· Mar 1, 2008, 14:10 UTC · re: Jakub Narebski · lore

Re: Will git have a baseline feature or something alike?

On Fri, Feb 29, 2008 at 9:21 PM, Jakub Narebski <jnareb@gmail.com> wrote:
Show 32 quoted lines
> "eric miao" <eric.y.miao@gmail.com> writes:
>
>  > I kept a mirror of
>  >
>  > http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>  >
>  > by a crontab task fetching the updated commits at midnight everyday.
>  >
>  > Yet I found the repository now grows to be 1.2G without checking out
>  > anything. The checked out working tree of this is about 1.5G.
>
>  Did you (re)packed this repository, running "git gc", or "git repack"?
>  Currently git either downloads small packs, or loose objects; it needs
>  to repack to make repository size smaller.
>
>  BTW. the largest git repository is 1.6G OpenOffice.org conversion,
>  with > 2G checkout, and some large binary files under version
>  control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
>  So kernel should be quite smaller.
>
>
>  > I tried "git prune" and "git repack" but it still remains so large. The
>  > trend of the kernel is still going to be enlarged. Thus I'm thinking
>  > of the possibility of a baseline feature. One can totally forget about
>  > the history before that baseline, and start the development there
>  > after.
>
>  There is so called "shallow clone" feature, which allows to clone only
>  part of history. Currently it dupports only --depth, i.e. number of
>  commits from tips; it could I guess support providing tag as
>  delimiter. (You are welcome to implement it ;-).
>

I haven't ever used the shallow clone, but it looks still a bit different from what I thought originally, say, if I download linux-2.6.24.tar.bz2 from kernel.org, that's about 40MB and should be a fair amount. I then unpack and "git init", I expect it to recognize it's a v2.6.24, and I can thereafter use "git fetch" to fetch those commits after v2.6.24 from git.kernel.org. Is this possible?

Show 5 quoted lines
>  --
>  Jakub Narebski
>  Poland
>  ShadeHawk on #git
>
-- 
Cheers
- eric
Nguyen Thai Ngoc Duy· Mar 1, 2008, 14:29 UTC · re: eric miao · lore

Re: Will git have a baseline feature or something alike?

On Sat, Mar 1, 2008 at 9:10 PM, eric miao <eric.y.miao@gmail.com> wrote:
Show 40 quoted lines
> On Fri, Feb 29, 2008 at 9:21 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>  > "eric miao" <eric.y.miao@gmail.com> writes:
>  >
>  >  > I kept a mirror of
>  >  >
>  >  > http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>  >  >
>  >  > by a crontab task fetching the updated commits at midnight everyday.
>  >  >
>  >  > Yet I found the repository now grows to be 1.2G without checking out
>  >  > anything. The checked out working tree of this is about 1.5G.
>  >
>  >  Did you (re)packed this repository, running "git gc", or "git repack"?
>  >  Currently git either downloads small packs, or loose objects; it needs
>  >  to repack to make repository size smaller.
>  >
>  >  BTW. the largest git repository is 1.6G OpenOffice.org conversion,
>  >  with > 2G checkout, and some large binary files under version
>  >  control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
>  >  So kernel should be quite smaller.
>  >
>  >
>  >  > I tried "git prune" and "git repack" but it still remains so large. The
>  >  > trend of the kernel is still going to be enlarged. Thus I'm thinking
>  >  > of the possibility of a baseline feature. One can totally forget about
>  >  > the history before that baseline, and start the development there
>  >  > after.
>  >
>  >  There is so called "shallow clone" feature, which allows to clone only
>  >  part of history. Currently it dupports only --depth, i.e. number of
>  >  commits from tips; it could I guess support providing tag as
>  >  delimiter. (You are welcome to implement it ;-).
>  >
>
>  I haven't ever used the shallow clone, but it looks still a bit different
>  from what I thought originally, say, if I download linux-2.6.24.tar.bz2
>  from kernel.org, that's about 40MB and should be a fair amount.
>  I then unpack and "git init", I expect it to recognize it's a v2.6.24,
>  and I can thereafter use "git fetch" to fetch those commits after
>  v2.6.24 from git.kernel.org. Is this possible?

I tried shallow clone (depth 1) with a fairly old linux-2.6 repo and the pack was 68MB. A bit bigger than 40MB but still acceptable IMO. The tarball+git-init way, I don't think it work. Maybe kernel.org could release shallow git bundles in addition to tarballs so users like you can download a bundle, make a repo from it and keep up with "git fetch".

-- 
Duy
Jakub Narebski· Mar 1, 2008, 15:41 UTC · re: eric miao · lore

Re: Will git have a baseline feature or something alike?

On Sat, 1 Mar 2008, eric miao wrote:
> On Fri, Feb 29, 2008 at 9:21 PM, Jakub Narebski <jnareb@gmail.com> wrote:
[could you please remove irrelevant parts of the reply? TIA]
Show 12 quoted lines
>> There is so called "shallow clone" feature, which allows to clone only
>> part of history. Currently it dupports only --depth, i.e. number of
>> commits from tips; it could I guess support providing tag as
>> delimiter. (You are welcome to implement it ;-).
>>
> 
> I haven't ever used the shallow clone, but it looks still a bit different
> from what I thought originally, say, if I download linux-2.6.24.tar.bz2
> from kernel.org, that's about 40MB and should be a fair amount.
> I then unpack and "git init", I expect it to recognize it's a v2.6.24,
> and I can thereafter use "git fetch" to fetch those commits after
> v2.6.24 from git.kernel.org. Is this possible?

No, this doesn't work and couldn't work. The tarfile contains only _contents_ of the working directory, and perhaps commit-id, but it doesn't contain even shred of history. Git has no information of where this content is in linux kernel git history.

You have to do "git clone --depth=1 <url>"; surrently there is no way to specify "git clone --from=<tag> <url>". I guess you can try to get info browsing gitweb, and fudge with grafts.

-- 
Jakub Narebski
Poland
Eyvind Bernhardsen· Mar 1, 2008, 17:30 UTC · re: Jakub Narebski · lore

Re: Will git have a baseline feature or something alike?

On 1. mars. 2008, at 16.41, Jakub Narebski wrote:
Show 16 quoted lines
> On Sat, 1 Mar 2008, eric miao wrote:
>
>> I haven't ever used the shallow clone, but it looks still a bit  
>> different
>> from what I thought originally, say, if I download  
>> linux-2.6.24.tar.bz2
>> from kernel.org, that's about 40MB and should be a fair amount.
>> I then unpack and "git init", I expect it to recognize it's a  
>> v2.6.24,
>> and I can thereafter use "git fetch" to fetch those commits after
>> v2.6.24 from git.kernel.org. Is this possible?
>
> No, this doesn't work and couldn't work. The tarfile contains only
> _contents_ of the working directory, and perhaps commit-id, but it
> doesn't contain even shred of history. Git has no information of
> where this content is in linux kernel git history.

Okay, as a git n00b I'm probably on completely the wrong track, but if you made a git repository out of a kernel tarball (cd linux-2.6.24 && git init && git add .) and then did a shallow fetch from kernel.org into that repository, wouldn't the blobs you added get reused (assuming the tarball you downloaded was fairly recent), thus reducing the amount of data fetch has to transfer?

I'm sure you'd end up transferring more data than just a straight shallow clone, but it would be cool if that worked, and might even be useful. I'm downloading 2.6.25-rc3 now to try it out :)

Eyvind Bernhardsen
Jakub Narebski· Mar 1, 2008, 18:00 UTC · re: Eyvind Bernhardsen · lore

Re: Will git have a baseline feature or something alike?

Eyvind Bernhardsen wrote:
Show 7 quoted lines
>
> Okay, as a git n00b I'm probably on completely the wrong track, but if  
> you made a git repository out of a kernel tarball (cd linux-2.6.24 &&  
> git init && git add .) and then did a shallow fetch from kernel.org  
> into that repository, wouldn't the blobs you added get reused  
> (assuming the tarball you downloaded was fairly recent), thus reducing  
> the amount of data fetch has to transfer?

I think it wouldn't. If I understand it correctly, the fetching engine deals only with commits. If you have commit, it assumes that you have tree, blobs, and ancestors. If you don't have commit, it assumes that you don't have tree and blobs.

-- 
Jakub Narebski
Poland
Shawn O. Pearce· Mar 2, 2008, 14:04 UTC · re: Jakub Narebski · lore

Re: Will git have a baseline feature or something alike?

Jakub Narebski <jnareb@gmail.com> wrote:
Show 13 quoted lines
> Eyvind Bernhardsen wrote:
> >
> > Okay, as a git n00b I'm probably on completely the wrong track, but if  
> > you made a git repository out of a kernel tarball (cd linux-2.6.24 &&  
> > git init && git add .) and then did a shallow fetch from kernel.org  
> > into that repository, wouldn't the blobs you added get reused  
> > (assuming the tarball you downloaded was fairly recent), thus reducing  
> > the amount of data fetch has to transfer?
> 
> I think it wouldn't. If I understand it correctly, the fetching engine
> deals only with commits. If you have commit, it assumes that you have 
> tree, blobs, and ancestors. If you don't have commit, it assumes that 
> you don't have tree and blobs.
Correct.

I was thinking about this just yesterday. I think that if we embedded inside of a tarball created by git-archive the raw sources of all files, plus the commit SHA-1 and the raw body of that commit, it should be possible to convert that into a shallow clone.

Unfortunately I think it is possible for git-archive to edit a file in-place during export, e.g. to edit an RPM spec file and insert the revision. That would damage the tree, as the blobs would no longer hash to the same value as they should be in that commit.

-- 
Shawn.
David Brown· Mar 1, 2008, 20:43 UTC · re: Eyvind Bernhardsen · lore

Re: Will git have a baseline feature or something alike?

On Sat, Mar 01, 2008 at 06:30:13PM +0100, Eyvind Bernhardsen wrote:
Show 6 quoted lines
> Okay, as a git n00b I'm probably on completely the wrong track, but if you 
> made a git repository out of a kernel tarball (cd linux-2.6.24 && git init 
> && git add .) and then did a shallow fetch from kernel.org into that 
> repository, wouldn't the blobs you added get reused (assuming the tarball 
> you downloaded was fairly recent), thus reducing the amount of data fetch 
> has to transfer?

Git uses the commit history to determine what objects you might already have. For normal use cases, this works quite well, however in this instance it doesn't help at all. You'll ending up transferring everything, even for objects you already have. Git will detect that you already have the object only after the transfer.

Think about it, though. In order to do this generally, the client would have to send the hash of every object it has. Perhaps this would be a useful thing to do when git detects that there are no common commits, but it would only really help the case of pulling from multiple repos that manage the same files with separate histories.

There are some cases where this happens, such as when changes go through another revision control system, but probably not normal usage.

David

← back to recent threads