threads / discuss / 43088

What's in git.git

Subject: What's in git.git

## tl;dr

20 messages between Nov 8, 2006 and Nov 13, 2006.

replies: 19people: 8as markdown or json

Junio C Hamano· Nov 8, 2006, 03:21 UTC · lore
Executive Summary.
[maint]
  I might do a 1.4.3.5 with the accumulated stuff, but the
  stablization cycle for v1.4.4 has started tonight, so it may
  not be worth the effort, unless something more pressing comes
  along.
[master]
  Three topics that have been cooking in 'next' have been
  merged, I've tagged the tip as v1.4.4-rc1.
   - Nico and Shawn's keep-pack work;
   - Loosening of bogusly overstrict 'a working tree file will
     be overwritten by the merge' check;
   - git-pickaxe.
[next]
  This now is almost empty, and I'd like to keep it that way
  until v1.4.5 final.  IOW, I'd be happier if people sent in
  only obviously correct fixes to 'master' than seeing the next
  greatest feature ;-).
  By the way, do people mind if I start to rewind and rebase
  'next' after every feature release (i.e. tagged release is
  made after 'master')?  I do not feel a strong need for it, and
  'git log --no-merges master..next' will show emptiness
  eventually, but being able to restart from clean slate after a
  release would be somewhat nice.
[pu]
  Johannes's shallow clone work now should rebase cleanly on top
  of 'master' although I haven't done so yet.  As he said
  himself the series is waiting for people who have needs for
  such a feature to raise hands.
  The part #2 of git-diff/git-apply change has a slight backward
  compatibility issue, and until everybody who is affected is
  upgraded to v1.4.3 (which has already prepared us for this
  change) we cannot push it out to 'master'.  It adjusts the
  diff output header for files with SP in their names to what
  GNU patch accepts.
----------------------------------------------------------------
* The 'maint' branch has these fixes since the last announcement.
   Alex Riesen (1):
      merge-recursive implicitely depends on trust_executable_bit
   Andy Parkins (2):
      Minor grammar fixes for git-diff-index.txt
      git-clone documentation didn't mention --origin as equivalent of -o
   Jakub Narebski (1):
      Documentation: Transplanting branch with git-rebase --onto
   Jeff King (1):
      Fix git-runstatus for repositories containing a file named HEAD
   Johannes Schindelin (1):
      link_temp_to_file: call adjust_shared_perm() only when we created the directory
   Junio C Hamano (2):
      apply: handle "traditional" creation/deletion diff correctly.
      adjust_shared_perm: chmod() only when needed.
   Shawn O. Pearce (3):
      Use ULONG_MAX rather than implicit cast of -1.
      Remove SIMPLE_PROGRAMS and make git-daemon a normal program.
      Remove unsupported C99 style struct initializers in git-archive.
   Tero Roponen (1):
      remove an unneeded test
* The 'master' branch has these since the last announcement.
   Alex Riesen (1):
      merge-recursive implicitely depends on trust_executable_bit
   Alexandre Julliard (5):
      pack-refs: Store the full name of the ref even when packing only tags.
      git.el: Added functions for moving to the next/prev unmerged file.
      git.el: Added a function to open the current file in another window.
      git.el: Move point after the log message header when entering log-edit mode.
      git.el: Include MERGE_MSG in the log-edit buffer even when not committing a merge.
   Andy Parkins (3):
      Remove uneccessarily similar printf() from print_ref_list() in builtin-branch
      Minor grammar fixes for git-diff-index.txt
      git-clone documentation didn't mention --origin as equivalent of -o
   Aneesh Kumar K.V (1):
      gitweb: Remove extra "/" in path names for git_get_project_list
   Eric Wong (2):
      git-svn: avoid printing filenames of files we're not tracking
      git-svn: don't die on rebuild when --upgrade is specified
   Jakub Narebski (4):
      gitweb: Use git-for-each-ref to generate list of heads and/or tags
      gitweb: Output also empty patches in "commitdiff" view
      gitweb: Better support for non-CSS aware web browsers
      Documentation: Transplanting branch with git-rebase --onto
   Jeff King (2):
      git-pickaxe: work properly in a subdirectory.
      Fix git-runstatus for repositories containing a file named HEAD
   Johannes Schindelin (1):
      link_temp_to_file: call adjust_shared_perm() only when we created the directory
   Junio C Hamano (40):
      git-pickaxe: blame rewritten.
      git-pickaxe -M: blame line movements within a file.
      git-pickaxe -C: blame cut-and-pasted lines.
      git-pickaxe: pagenate output by default.
      git-pickaxe: fix nth_line()
      git-pickaxe: improve "best match" heuristics
      git-pickaxe: introduce heuristics to avoid "trivial" chunks
      git-pickaxe: do not keep commit buffer.
      git-pickaxe: do not confuse two origins that are the same.
      git-pickaxe: get rid of wasteful find_origin().
      git-pickaxe: swap comparison loop used for -C
      merge: loosen overcautious "working file will be lost" check.
      merge-recursive: use abbreviated commit object name.
      merge-recursive: make a few functions static.
      merge-recursive: adjust to loosened "working file clobbered" check
      t6022: ignoring untracked files by merge-recursive when they do not matter
      send-pack --keep: do not explode into loose objects on the receiving end.
      git-pickaxe: WIP to refcount origin structure.
      git-pickaxe: allow -Ln,m as well as -L n,m
      git-pickaxe: refcount origin correctly in find_copy_in_parent()
      git-pickaxe: tighten sanity checks.
      Revert "send-pack --keep: do not explode into loose objects on the receiving end."
      git-pickaxe: split find_origin() into find_rename() and find_origin().
      git-pickaxe: cache one already found path per commit.
      Introduce a new revision set operator <rev>^!
      for-each-ref: "creator" and "creatordate" fields
      apply: handle "traditional" creation/deletion diff correctly.
      git-pickaxe: rename detection optimization
      git-pickaxe: simplify Octopus merges further
      git-pickaxe: re-scan the blob after making progress with -M
      git-pickaxe: re-scan the blob after making progress with -C
      git-pickaxe: fix origin refcounting
      cherry is built-in, do not ship git-cherry.sh
      git-blame: add internal statistics to count read blobs.
      git-pickaxe: optimize by avoiding repeated read_sha1_file().
      adjust_shared_perm: chmod() only when needed.
      Document git-pack-refs and link it to git(7).
      git-pickaxe: -L /regexp/,/regexp/
      git-pickaxe: allow "-L <something>,+N"
      GIT 1.4.3-rc1
   Linus Torvalds (2):
      Allow '-' in config variable names
      git push: add verbose flag and allow overriding of default target repository
   Nicolas Pitre (14):
      enable index-pack streaming capability
      make index-pack able to complete thin packs.
      add progress status to index-pack
      mimic unpack-objects when --stdin is used with index-pack
      enhance clone and fetch -k experience
      index-pack: minor fixes to comment and function name
      missing small substitution
      make git-push a bit more verbose
      Allow pack header preprocessing before unpack-objects/index-pack.
      git-fetch can use both --thin and --keep with fetch-pack now
      improve fetch-pack's handling of kept packs
      have index-pack create .keep file more carefully
      remove .keep pack lock files when done with refs update
      git-pack-objects progress flag documentation and cleanup
   Petr Baudis (1):
      gitweb: Support for 'forks'
   Sean Estabrooks (1):
      Add --global option to git-repo-config.
   Shawn O. Pearce (11):
      Added completion support for git-branch.exe.
      Added bash completion support for git-reset.
      Use ULONG_MAX rather than implicit cast of -1.
      Remove SIMPLE_PROGRAMS and make git-daemon a normal program.
      Remove unsupported C99 style struct initializers in git-archive.
      Added missing completions for show-branch and merge-base.
      Only load .exe suffix'd completions on Cygwin.
      Bash completion support for remotes in .git/config.
      Take --git-dir into consideration during bash completion.
      Support bash completion on symmetric difference operator.
      Remove more sed invocations from within bash completion.
   Shawn Pearce (5):
      Allow short pack names to git-pack-objects --unpacked=.
      Only repack active packs by skipping over kept packs.
      Teach git-index-pack how to keep a pack file.
      Remove unused variable in receive-pack.
      Teach receive-pack how to keep pack files based on object count.
   Tero Roponen (1):
      remove an unneeded test
* The 'next' branch, in addition, has these.
   Junio C Hamano:
      upload-pack: stop the other side when they have more roots than we do.
* The 'pu' branch, in addition, has these.
   Johannes Schindelin (6):
      Build in shortlog
      upload-pack: no longer call rev-list
      support fetching into a shallow repository
      allow cloning a repository "shallowly"
      allow deepening of a shallow repository
      add tests for shallow stuff
   Junio C Hamano (6):
      git-branch -a: show both local and remote tracking branches.
      git-commit: show --summary after successful commit.
      para-walk: walk n trees, index and working tree in parallel
      git-diff/git-apply: make diff output a bit friendlier to GNU patch (part 2)
      rev-list --left-right
      blame and pickaxe: --show-stats for easier optimization work.
David Lang· Nov 8, 2006, 04:13 UTC · re: Junio C Hamano · lore

Re: What's in git.git

On Tue, 7 Nov 2006, Junio C Hamano wrote:
Show 6 quoted lines
> [pu]
>
>  Johannes's shallow clone work now should rebase cleanly on top
>  of 'master' although I haven't done so yet.  As he said
>  himself the series is waiting for people who have needs for
>  such a feature to raise hands.

I haven't been watching this recently, but if this is what I understand it to be (the ability to get a partial repository from upstream and work normally from there with the result of data-mineing tools sometimes reporting 'that's part of the truncated history' if they hit the cutoff) consider my hand raised.

there are a number of cases where I would be interested in following a project as it moves forwards, but do not have the need to have the full history (even with the good compression that a git pack provides, it's still a significant amount of disk space and download time for large projects)

Aneesh Kumar K.V· Nov 8, 2006, 16:40 UTC · re: David Lang · lore

Shallow clone [Was Re: What's in git.git ]

David Lang wrote:
Show 21 quoted lines
> On Tue, 7 Nov 2006, Junio C Hamano wrote:
> 
>> [pu]
>>
>>  Johannes's shallow clone work now should rebase cleanly on top
>>  of 'master' although I haven't done so yet.  As he said
>>  himself the series is waiting for people who have needs for
>>  such a feature to raise hands.
> 
> I haven't been watching this recently, but if this is what I understand 
> it to be (the ability to get a partial repository from upstream and work 
> normally from there with the result of data-mineing tools sometimes 
> reporting 'that's part of the truncated history' if they hit the cutoff) 
> consider my hand raised.
> 
> there are a number of cases where I would be interested in following a 
> project as it moves forwards, but do not have the need to have the full 
> history (even with the good compression that a git pack provides, it's 
> still a significant amount of disk space and download time for large 
> projects)
> 

I am trying to test this feature. Is there a documentation .git/shallow some where. Atleast what those entries mean ? I know in the mail johannes mentioned only core git will touch this file. But it should be ok to be descriptive like other files. (FETCH_HEAD)

-aneesh 
Aneesh Kumar K.V· Nov 8, 2006, 17:59 UTC · re: Aneesh Kumar K.V · lore

Re: Shallow clone [Was Re: What's in git.git ]

Aneesh Kumar K.V wrote:
Show 28 quoted lines
> David Lang wrote:
>> On Tue, 7 Nov 2006, Junio C Hamano wrote:
>>
>>> [pu]
>>>
>>>  Johannes's shallow clone work now should rebase cleanly on top
>>>  of 'master' although I haven't done so yet.  As he said
>>>  himself the series is waiting for people who have needs for
>>>  such a feature to raise hands.
>>
>> I haven't been watching this recently, but if this is what I 
>> understand it to be (the ability to get a partial repository from 
>> upstream and work normally from there with the result of data-mineing 
>> tools sometimes reporting 'that's part of the truncated history' if 
>> they hit the cutoff) consider my hand raised.
>>
>> there are a number of cases where I would be interested in following a 
>> project as it moves forwards, but do not have the need to have the 
>> full history (even with the good compression that a git pack provides, 
>> it's still a significant amount of disk space and download time for 
>> large projects)
>>
> 
> I am trying to test this feature. Is there a documentation .git/shallow 
> some where. Atleast what those entries
> mean ? I know in the mail johannes mentioned only core git will touch 
> this file. But it should be ok to be descriptive like other files. 
> (FETCH_HEAD)
How about this 
-aneesh 
diff --git a/Documentation/repository-layout.txt b/Documentation/repository-layout.txt
index 275d18b..03a6f77 100644
--- a/Documentation/repository-layout.txt
+++ b/Documentation/repository-layout.txt
@@ -141,3 +141,9 @@ logs/refs/heads/`name`::
 
 logs/refs/tags/`name`::
 	Records all changes made to the tag named `name`.
+
+shallow::
+	Records the sha1 of the commits which is marked to have no
+	parents to represent a shallow repository.The commit object
+	will have the parent information present. It carry one
+	record per line.
Junio C Hamano· Nov 9, 2006, 04:04 UTC · re: Aneesh Kumar K.V · lore

Re: Shallow clone

"Aneesh Kumar K.V" <aneesh.kumar@gmail.com> writes:
Show 20 quoted lines
>> I am trying to test this feature. Is there a documentation
>> .git/shallow some where. Atleast what those entries
>> mean ? I know in the mail johannes mentioned only core git will
>> touch this file. But it should be ok to be descriptive like other
>> files. (FETCH_HEAD)
>
> diff --git a/Documentation/repository-layout.txt b/Documentation/repository-layout.txt
> index 275d18b..03a6f77 100644
> --- a/Documentation/repository-layout.txt
> +++ b/Documentation/repository-layout.txt
> @@ -141,3 +141,9 @@ logs/refs/heads/`name`::
>  
>  logs/refs/tags/`name`::
>  	Records all changes made to the tag named `name`.
> +
> +shallow::
> +	Records the sha1 of the commits which is marked to have no
> +	parents to represent a shallow repository.The commit object
> +	will have the parent information present. It carry one
> +	record per line.

I would drop the second sentence which is just confusing but otherwise it is correct, I think (I just started trying it out).

But it seems to need some more work. I just tried to clone git.git with --depth=1 and it cauterizes each branch with two commits (I think that is what depth=1 means -- the latest and one behind it), but it pulled almost the whole repository anyway, and it turns out that "git log v1.4.3-rc1" gives me the full history leading to it.

Subsequent "git fetch --depth 99999" makes the branches connected to the root commit, and I am reasonably sure we do not have that many commits, but .git/shallow did not become empty.

I haven't followed the code closely enough to tell if these are just minor details needing more polish, or something more fundamental in the design.

Aneesh Kumar· Nov 9, 2006, 04:17 UTC · re: Junio C Hamano · lore

Re: Shallow clone

On 11/9/06, Junio C Hamano <junkio@cox.net> wrote:
Show 14 quoted lines
> "Aneesh Kumar K.V" <aneesh.kumar@gmail.com> writes:
>
> >
> But it seems to need some more work.  I just tried to clone
> git.git with --depth=1 and it cauterizes each branch with two
> commits (I think that is what depth=1 means -- the latest and
> one behind it), but it pulled almost the whole repository
> anyway, and it turns out that "git log v1.4.3-rc1" gives me the
> full history leading to it.
>
> Subsequent "git fetch --depth 99999" makes the branches
> connected to the root commit, and I am reasonably sure we do not
> have that many commits, but .git/shallow did not become empty.
>

I tried a local clone with --depth 2 and it showed two commits in all the branches/DAG. But then doing a pull --depth 4 listed more commits than 4. I didn't look much into this.

Alexandre Julliard· Nov 11, 2006, 13:57 UTC · re: Junio C Hamano · lore

Re: Shallow clone

Junio C Hamano <junkio@cox.net> writes:
Show 6 quoted lines
> But it seems to need some more work.  I just tried to clone
> git.git with --depth=1 and it cauterizes each branch with two
> commits (I think that is what depth=1 means -- the latest and
> one behind it), but it pulled almost the whole repository
> anyway, and it turns out that "git log v1.4.3-rc1" gives me the
> full history leading to it.

That's apparently because tags are not considered when truncating the commit list. The patch below fixes it, and fetches the right number of commits for each tag. However the correct fix is probably to not fetch historical tags at all.

There's also a problem with the packing, a clone --depth 1 currently results in a pack that's about 3 times as large as it should be.

---
diff --git a/shallow.c b/shallow.c
index 58a7b20..2db1dc4 100644
--- a/shallow.c
+++ b/shallow.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "commit.h"
+#include "tag.h"
 
 static int is_shallow = -1;
 
@@ -54,7 +55,7 @@ struct commit_list *get_shallow_commits(
 		if (!commit) {
 			if (i < heads->nr) {
 				commit = (struct commit *)
-					heads->objects[i++].item;
+					deref_tag(heads->objects[i++].item, NULL, 0);
 				if (commit->object.type != OBJ_COMMIT) {
 					commit = NULL;
 					continue;
-- 
Alexandre Julliard
Junio C Hamano· Nov 12, 2006, 08:16 UTC · re: Alexandre Julliard · lore

Re: Shallow clone

Alexandre Julliard <julliard@winehq.org> writes:
> There's also a problem with the packing, a clone --depth 1 currently
> results in a pack that's about 3 times as large as it should be.
That's interesting.
  : gitster; git clone -n --depth 1 git://127.0.0.1/git.git victim-001
  remote: Generating pack...
  remote: Done counting 6246 objects.
  remote: Deltifying 6246 objects.
  remote:  100% (6246/6246) done
  Indexing 6246 objects.
  remote: Total 6246, written 6246 (delta 3106), reused 4313 (delta 3106)
   100% (6246/6246) done
  Resolving 3106 deltas.
   100% (3106/3106) done
  : gitster; cd victim-001
  : gitster; ls -lh .git/objects/pack/
  total 9.6M
  drwxrwsr-x 2 junio src 4.0K 2006-11-11 23:52 ./
  drwxrwsr-x 4 junio src 4.0K 2006-11-11 23:52 ../
  -r--r--r-- 1 junio src 148K 2006-11-11 23:52 pack-f5f88d83....idx
  -r--r--r-- 1 junio src 9.5M 2006-11-11 23:52 pack-f5f88d83....pack

Repacking immediately after cloning brings it down to what is expected.

  : gitster; git repack -a -d -f
  Generating pack...
  Done counting 6246 objects.
  Deltifying 6246 objects.
   100% (6246/6246) done
  Writing 6246 objects.
   100% (6246/6246) done
  Total 6246, written 6246 (delta 4815), reused 1407 (delta 0)
  Pack pack-f5f88d83524213e3ab05697ff75f245b1ef9081a created.
  : gitster; ls -lh .git/objects/pack/
  total 2.8M
  drwxrwsr-x 2 junio src 4.0K 2006-11-11 23:53 ./
  drwxrwsr-x 4 junio src 4.0K 2006-11-11 23:52 ../
  -rw-rw-r-- 1 junio src 148K 2006-11-11 23:53 pack-f5f88d83....idx
  -rw-rw-r-- 1 junio src 2.6M 2006-11-11 23:53 pack-f5f88d83....pack

In any case, after this "shallow" stuff, repeated "fetch --depth 99" seems to fetch 0 object and 3400 objects alternately, and the shallow file alternates between 900 bytes and 11000 bytes.

We would need to take a deeper look into what this series does, before moving it to 'next'.

Sergey Vlasov· Nov 12, 2006, 17:59 UTC · re: Junio C Hamano · lore

Re: Shallow clone

On Sun, 12 Nov 2006 00:16:40 -0800 Junio C Hamano wrote:
Show 8 quoted lines
> Alexandre Julliard <julliard@winehq.org> writes:
>
> > There's also a problem with the packing, a clone --depth 1 currently
> > results in a pack that's about 3 times as large as it should be.
>
> That's interesting.
>
>   : gitster; git clone -n --depth 1 git://127.0.0.1/git.git victim-001
[...]
Show 6 quoted lines
>   -r--r--r-- 1 junio src 9.5M 2006-11-11 23:52 pack-f5f88d83....pack
>
> Repacking immediately after cloning brings it down to what is
> expected.
>
>   : gitster; git repack -a -d -f
[...]
>   -rw-rw-r-- 1 junio src 2.6M 2006-11-11 23:53 pack-f5f88d83....pack
This is due to optimization in builtin-pack-objects.c:try_delta():
	/*
	 * We do not bother to try a delta that we discarded
	 * on an earlier try, but only when reusing delta data.
	 */
	if (!no_reuse_delta && trg_entry->in_pack &&
	    trg_entry->in_pack == src_entry->in_pack)
		return 0;

After removing this part the shallow pack after clone is 2.6M, as it should be.

The problem with this optimization is that it is only valid if we are repacking either the same set of objects as we did earlier, or its superset. But if we are packing a subset of objects, there will be some objects in that subset which were delta-compressed in the original pack, but base objects for that deltas are not included in our subset - therefore we will be unable to reuse existing deltas, and with that optimization we will never try to use delta compression for these objects. (The optimization assumes that if we will try to use delta compression, we will try mostly the same base objects as we have tried when we made the existing pack, and therefore will likely get the same result - which is close to the truth when we are doing "repack -a", but is badly wrong when we are doing "git-upload-pack" with a large number of common commits, and therefore are excluding a lot of objects.)

So any partial fetch (shallow or not) from a mostly packed repository currently results in a suboptimal pack. In fact, the fresh "repack -a -d -f" is probably the worst case for subsequent fetch (not initial clone) from that repository - objects for the most recent commit are most likely to be stored without delta compression, and even if deltas are used, they are likely in the wrong direction for someone who has an older version and wants to update it.

> In any case, after this "shallow" stuff, repeated "fetch --depth
> 99" seems to fetch 0 object and 3400 objects alternately, and
> the shallow file alternates between 900 bytes and 11000 bytes.
I confirm this - different numbers, but the same problem...
Junio C Hamano· Nov 12, 2006, 21:59 UTC · re: Sergey Vlasov · lore

Re: Shallow clone

Sergey Vlasov <vsu@altlinux.ru> writes:
Show 24 quoted lines
> This is due to optimization in builtin-pack-objects.c:try_delta():
>
> 	/*
> 	 * We do not bother to try a delta that we discarded
> 	 * on an earlier try, but only when reusing delta data.
> 	 */
> 	if (!no_reuse_delta && trg_entry->in_pack &&
> 	    trg_entry->in_pack == src_entry->in_pack)
> 		return 0;
>
> After removing this part the shallow pack after clone is 2.6M, as it
> should be.
>
> The problem with this optimization is that it is only valid if we are
> repacking either the same set of objects as we did earlier, or its
> superset.  But if we are packing a subset of objects, there will be some
> objects in that subset which were delta-compressed in the original pack,
> but base objects for that deltas are not included in our subset -
> therefore we will be unable to reuse existing deltas, and with that
> optimization we will never try to use delta compression for these
> objects.
> ...
> So any partial fetch (shallow or not) from a mostly packed repository
> currently results in a suboptimal pack.
That is correct.  How about something like this?

I think the determination of "repacking_superset" may need to be tweaked because existing packs may have overlaps, and the patch counts them once per pack.

diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 69e5dd3..fb25124 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -64,6 +64,7 @@ struct object_entry {
 static unsigned char object_list_sha1[20];
 static int non_empty;
 static int no_reuse_delta;
+static int repacking_superset;
 static int local;
 static int incremental;
 static int allow_ofs_delta;
@@ -1172,10 +1173,13 @@ static int try_delta(struct unpacked *tr
 		return -1;
 
 	/*
-	 * We do not bother to try a delta that we discarded
-	 * on an earlier try, but only when reusing delta data.
+	 * When we are packing the superset of objects we have already
+	 * packed, we do not bother to try a delta that we discarded
+	 * on an earlier try.  This heuristic of course should not
+	 * kick in when we are not reusing delta, or we know we are
+	 * sending a subset of objects from a repository.
 	 */
-	if (!no_reuse_delta && trg_entry->in_pack &&
+	if (!no_reuse_delta && repacking_superset && trg_entry->in_pack &&
 	    trg_entry->in_pack == src_entry->in_pack)
 		return 0;
 
@@ -1493,6 +1497,16 @@ static void get_object_list(int ac, cons
 	traverse_commit_list(&revs, show_commit, show_object);
 }
 
+static int count_packed_objects(void)
+{
+	struct packed_git *p;
+	int cnt = 0;
+
+	for (p = packed_git; p; p = p->next)
+		cnt += num_packed_objects(p);
+	return cnt;
+}
+
 int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 {
 	SHA_CTX ctx;
@@ -1631,6 +1645,8 @@ int cmd_pack_objects(int argc, const cha
 	if (non_empty && !nr_result)
 		return 0;
 
+	repacking_superset = count_packed_objects() < nr_result;
+
 	SHA1_Init(&ctx);
 	list = sorted_by_sha;
 	for (i = 0; i < nr_result; i++) {
Junio C Hamano· Nov 13, 2006, 05:29 UTC · re: Junio C Hamano · lore

Re: Shallow clone

Junio C Hamano <junkio@cox.net> writes:
Show 26 quoted lines
> Sergey Vlasov <vsu@altlinux.ru> writes:
>
>> This is due to optimization in builtin-pack-objects.c:try_delta():
>>
>> 	/*
>> 	 * We do not bother to try a delta that we discarded
>> 	 * on an earlier try, but only when reusing delta data.
>> 	 */
>> 	if (!no_reuse_delta && trg_entry->in_pack &&
>> 	    trg_entry->in_pack == src_entry->in_pack)
>> 		return 0;
>>
>> After removing this part the shallow pack after clone is 2.6M, as it
>> should be.
>>
>> The problem with this optimization is that it is only valid if we are
>> repacking either the same set of objects as we did earlier, or its
>> superset.  But if we are packing a subset of objects, there will be some
>> objects in that subset which were delta-compressed in the original pack,
>> but base objects for that deltas are not included in our subset -
>> therefore we will be unable to reuse existing deltas, and with that
>> optimization we will never try to use delta compression for these
>> objects.
>> ...
>> So any partial fetch (shallow or not) from a mostly packed repository
>> currently results in a suboptimal pack.

What we tried to avoid with the original heuristics in commit 51d1e83f was to avoid wasting time on undeltifiable blobs, and they would be stored as base in the original packs, so I think the following would fly better (the patch is for maint, for master we would also check for OBJ_REF_DELTA as well).

---
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 96c069a..84a8749 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -1101,7 +1101,8 @@ static int try_delta(struct unpacked *tr
 	 * on an earlier try, but only when reusing delta data.
 	 */
 	if (!no_reuse_delta && trg_entry->in_pack &&
-	    trg_entry->in_pack == src_entry->in_pack)
+	    trg_entry->in_pack == src_entry->in_pack &&
+	    trg_entry->in_pack_type != OBJ_DELTA)
 		return 0;
 
 	/*
Petr Baudis· Nov 12, 2006, 13:12 UTC · re: Aneesh Kumar K.V · lore

Re: Shallow clone [Was Re: What's in git.git ]

On Wed, Nov 08, 2006 at 06:59:05PM CET, Aneesh Kumar K.V wrote:
Show 10 quoted lines
> diff --git a/Documentation/repository-layout.txt b/Documentation/repository-layout.txt
> index 275d18b..03a6f77 100644
> --- a/Documentation/repository-layout.txt
> +++ b/Documentation/repository-layout.txt
> @@ -141,3 +141,9 @@ logs/refs/heads/`name`::
>  
>  logs/refs/tags/`name`::
>  	Records all changes made to the tag named `name`.
> +
> +shallow::
Hmm, shouldn't the shallow file rather belong to the info/ subdirectory?
-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
#!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1
Junio C Hamano· Nov 12, 2006, 20:04 UTC · re: Petr Baudis · lore

Re: Shallow clone

Petr Baudis <pasky@suse.cz> writes:
Show 13 quoted lines
> On Wed, Nov 08, 2006 at 06:59:05PM CET, Aneesh Kumar K.V wrote:
>> diff --git a/Documentation/repository-layout.txt b/Documentation/repository-layout.txt
>> index 275d18b..03a6f77 100644
>> --- a/Documentation/repository-layout.txt
>> +++ b/Documentation/repository-layout.txt
>> @@ -141,3 +141,9 @@ logs/refs/heads/`name`::
>>  
>>  logs/refs/tags/`name`::
>>  	Records all changes made to the tag named `name`.
>> +
>> +shallow::
>
> Hmm, shouldn't the shallow file rather belong to the info/ subdirectory?
I think that makes sense.
Johannes Schindelin· Nov 12, 2006, 22:25 UTC · re: David Lang · lore

Re: What's in git.git

Hi,
On Tue, 7 Nov 2006, David Lang wrote:
Show 14 quoted lines
> On Tue, 7 Nov 2006, Junio C Hamano wrote:
> 
> > [pu]
> > 
> >  Johannes's shallow clone work now should rebase cleanly on top
> >  of 'master' although I haven't done so yet.  As he said
> >  himself the series is waiting for people who have needs for
> >  such a feature to raise hands.
> 
> I haven't been watching this recently, but if this is what I understand it to
> be (the ability to get a partial repository from upstream and work normally
> from there with the result of data-mineing tools sometimes reporting 'that's
> part of the truncated history' if they hit the cutoff) consider my hand
> raised.

For now, it does not say "part of the truncated history". But yes, shallow clones are partial copies of remote repositories, by making some commits "shallow", i.e. grafting an empty set of parents onto them (thereby pretending that these commits are root commits).

Telling the user that a commit is shallow should not be too hard.

Ciao, Dscho

Jakub Narebski· Nov 8, 2006, 07:40 UTC · re: Junio C Hamano · lore

Re: What's in git.git

Junio C Hamano wrote:
>    Jakub Narebski (4):
[...]
>       gitweb: Output also empty patches in "commitdiff" view

I think that this patch is a bit premature. Without "new improved patchset view" the empty patches are just that: totally empty. It is new header and especially outputting extended header information which makes outputting "empty" patches (with no diff) in "commitdiff" view usefull. By "empty" patches I mean pure type change, pure rename, pure copy and type change and rename.

The "new improved patchset view" is prepared to send in two stages: updated filename quoting/unquoting series (two patches: unescape + minimal esc_path; esc_path with span.cntrl and backslash sequence quoting), and "new improved patchset view" (three patches originally, two remains: using "git diff" header and extended git diff headers; output and link also to patches with empty diff; improved formatting of chunk header).

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Junio C Hamano· Nov 8, 2006, 07:59 UTC · re: Jakub Narebski · lore

Re: What's in git.git

Jakub Narebski <jnareb@gmail.com> writes:
Show 14 quoted lines
> Junio C Hamano wrote:
>
>>    Jakub Narebski (4):
> [...]
>>       gitweb: Output also empty patches in "commitdiff" view
>
> I think that this patch is a bit premature. Without "new improved patchset
> view" the empty patches are just that: totally empty. It is new header and
> especially outputting extended header information which makes outputting
> "empty" patches (with no diff) in "commitdiff" view usefull. By "empty"
> patches I mean pure type change, pure rename, pure copy and type change and
> rename.
>
> The "new improved patchset view" is prepared to send in two stages...

At least this does not break the page even without these two follow-ups. If the follow-ups come in time and do not break the page, they are very welcome post -rc1 fixes to have before the final release. On the other hand, even if they don't, it's not the end of the world ;-).

Jakub Narebski· Nov 8, 2006, 07:58 UTC · re: Junio C Hamano · lore

Re: What's in git.git

Junio C Hamano wrote:
> [master]
> 
>   Three topics that have been cooking in 'next' have been
>   merged, I've tagged the tip as v1.4.4-rc1.

By the way, tag v1.4.4-rc1 has "GIT 1.4.4-rc1" as title, while the commit it points to, v1.4.4-rc1^{} has "GIT 1.4.3-rc1" as a title.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Junio C Hamano· Nov 8, 2006, 08:26 UTC · re: Jakub Narebski · lore

Re: What's in git.git

Jakub Narebski <jnareb@gmail.com> writes:
Show 9 quoted lines
> Junio C Hamano wrote:
>
>> [master]
>> 
>>   Three topics that have been cooking in 'next' have been
>>   merged, I've tagged the tip as v1.4.4-rc1.
>
> By the way, tag v1.4.4-rc1 has "GIT 1.4.4-rc1" as title, while the commit it
> points to, v1.4.4-rc1^{} has "GIT 1.4.3-rc1" as a title.
Yeah, sometimes I make typoes.  Not a news X-<.
Petr Baudis· Nov 8, 2006, 14:51 UTC · re: Junio C Hamano · lore

Re: What's in git.git

Show 6 quoted lines
>   By the way, do people mind if I start to rewind and rebase
>   'next' after every feature release (i.e. tagged release is
>   made after 'master')?  I do not feel a strong need for it, and
>   'git log --no-merges master..next' will show emptiness
>   eventually, but being able to restart from clean slate after a
>   release would be somewhat nice.
  It would be annoying for me since I can't mantain a long-term private
branch tracking next then (and I prefer next to master). I don't do that
right now but I've done that shortly in the past and it would be nice to
have that possibility in the future as well.
-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
#!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1
Junio C Hamano· Nov 9, 2006, 00:02 UTC · re: Junio C Hamano · lore

Re: What's in git.git

Sorry, I made a mistake of pushing out git-pickaxe before making it take over git-blame. So I will fix it up by renaming it to git-blame.

That is, unless there are too many people whose fingers have already been trained to type "git-pickaxe" from 'next' experience (you can obviously count me as one of these people). In which case I will keep git-pickaxe as a backward compatible synonym just like we still have git-annotate.

It might also make sense to eventually remove the other synonym git-annotate but that would be a longer term issue and should be handled separately.

← back to recent threads