threads / rfc / 32904

RFC patchclone: suggest index version 4 when the index is bigger than 512 KiB

Subject: [PATCH/RFC] clone: suggest index version 4 when the index is bigger than 512 KiB

## tl;dr

4 messages between Feb 14, 2013 and Feb 19, 2013. Diffs are folded; open one to read it.

replies: 3people: 3as markdown or json

Nguyễn Thái Ngọc Duy· Feb 14, 2013, 13:34 UTC · lore

I just realized that many of my big repos are still on index v2 while v4 should reduce its size significantly (3.8M -> 2.9M for linux-2.6 and 25M -> 14M for webkit, for example). I wanted to propose index v4 as the new default version, because I guess that not many people outside git@vger are aware of it, but perhaps this approach is less drastic. It only suggest index v4 when the index size after clone hits 512K limit.

I have 170 git repositories (most of them Gnome projects) and only big ones exceed the limit: webkit, linux-2.6, libreoffice-core, wine (530K), gentoo-x86. Gimp and banshee are close (510K and 321K). The rest barely hits 256K. So this hint won't show up often. On second thought, maybe we should lower it down to 300K to show more often and raise awareness on index v4 :)

Something I should have done in this patch too, is state that index v4 is only supported since vXXX, that git clients older than that will not work. But maybe just put that in update-index man page and refer there.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/clone.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

Show changes to builtin/clone.c +10 −0

diff --git a/builtin/clone.c b/builtin/clone.c
index e0aaf13..7cd1b60 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -543,6 +543,7 @@ static int checkout(void)
 	struct tree *tree;
 	struct tree_desc t;
 	int err = 0, fd;
+	struct stat st;
 
 	if (option_no_checkout)
 		return 0;
@@ -591,6 +592,15 @@ static int checkout(void)
 	if (!err && option_recursive)
 		err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);
 
+	if (!err &&
+	    !stat(git_path("index"), &st) &&
+	    st.st_size > 512 * 1024)
+		advise(_("Your index is quite large (%d KiB).\n"
+			 "You may want to update to index version 4 to reduce its size,\n"
+			"as large index files may affect performance, using the command:\n"
+			 "  git update-index --index-version 4"),
+		       st.st_size / 1024);
+
 	return err;
 }

-- 
1.8.1.2.536.gf441e6d

Thomas Ackermann· Feb 19, 2013, 10:45 UTC · re: Nguyễn Thái Ngọc Duy · lore

Re: [PATCH/RFC] clone: suggest index version 4 when the index is bigger than 512 KiB

Nguyễn Thái Ngọc Duy <pclouds <at> gmail.com> writes:

Show 9 quoted lines

> 
> I just realized that many of my big repos are still on index v2 while
> v4 should reduce its size significantly (3.8M -> 2.9M for linux-2.6
> and 25M -> 14M for webkit, for example). I wanted to propose index v4
> as the new default version, because I guess that not many people
> outside git <at> vger are aware of it, but perhaps this approach is less
> drastic. It only suggest index v4 when the index size after clone hits
> 512K limit.
>

Is V4 really recommended for general use? Default seems to be V3 and all I found in the docs is the coresponding release not describing V4 as "experimental". (I have repos with index files > 40MiB (!) which will shrink to approx. 20MiB with V4 so using V4 would be an interessting option for me.)

--- Thomas

Duy Nguyen· Feb 19, 2013, 10:54 UTC · re: Thomas Ackermann · lore

Re: [PATCH/RFC] clone: suggest index version 4 when the index is bigger than 512 KiB

On Tue, Feb 19, 2013 at 5:45 PM, Thomas Ackermann <th.acker@arcor.de> wrote:

Show 17 quoted lines

> Nguyễn Thái Ngọc Duy <pclouds <at> gmail.com> writes:
>
>>
>> I just realized that many of my big repos are still on index v2 while
>> v4 should reduce its size significantly (3.8M -> 2.9M for linux-2.6
>> and 25M -> 14M for webkit, for example). I wanted to propose index v4
>> as the new default version, because I guess that not many people
>> outside git <at> vger are aware of it, but perhaps this approach is less
>> drastic. It only suggest index v4 when the index size after clone hits
>> 512K limit.
>>
>
> Is V4 really recommended for general use?
> Default seems to be V3 and all I found in the docs is
> the coresponding release not describing V4 as "experimental".
> (I have repos with index files > 40MiB (!) which will shrink to
> approx. 20MiB with V4 so using V4 would be an interessting option for me.)

The default could be either v2 or v3, depending on whether you use some extra features. v4 is basically v3 with pathname compression. Yes, I think it's ok for general use.

-- 
Duy

Junio C Hamano· Feb 19, 2013, 18:59 UTC · re: Thomas Ackermann · lore

Re: [PATCH/RFC] clone: suggest index version 4 when the index is bigger than 512 KiB

Thomas Ackermann <th.acker@arcor.de> writes:

> Is V4 really recommended for general use?

If you stick to C-git, I do not think there is any reason to avoid it. It is already a mature technology, the difference between 2 and 4 are so trivial that it is very unlikely for a latent bug to be hiding in corner cases.

It is a different matter if various reimplementations already support the format. You can try it out, and if you find that other tools based on reimplementations of Git you use do not understand the format, you can safely go back with

	git update-index --index-version 2

after you upgraded to version 4.