git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 1/2 v8] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

From
Kirill Smelkov <kirr@nexedi.com>
Date
Sep 13, 2016, 07:50 UTC
Message-ID
<20160913075004.xlteg44fgtemkcwh@teco.navytux.spb.ru>
In-Reply-To
<xmqqd1k8wc6h.fsf@gitster.mtv.corp.google.com>
On Mon, Sep 12, 2016 at 11:23:18PM -0700, Junio C Hamano wrote:
Show 102 quoted lines
> Kirill Smelkov <kirr@nexedi.com> writes:
> 
> > +static int want_found_object(int exclude, struct packed_git *p)
> > +{
> > +	if (exclude)
> > +		return 1;
> > +	if (incremental)
> > +		return 0;
> > +
> > +	/*
> > +	 * When asked to do --local (do not include an object that appears in a
> > +	 * pack we borrow from elsewhere) or --honor-pack-keep (do not include
> > +	 * an object that appears in a pack marked with .keep), finding a pack
> > +	 * that matches the criteria is sufficient for us to decide to omit it.
> > +	 * However, even if this pack does not satisfy the criteria, we need to
> > +	 * make sure no copy of this object appears in _any_ pack that makes us
> > +	 * to omit the object, so we need to check all the packs.
> > +	 *
> > +	 * We can however first check whether these options can possible matter;
> > +	 * if they do not matter we know we want the object in generated pack.
> > +	 * Otherwise, we signal "-1" at the end to tell the caller that we do
> > +	 * not know either way, and it needs to check more packs.
> > +	 */
> > +	if (!ignore_packed_keep &&
> > +	    (!local || !have_non_local_packs))
> > +		return 1;
> > +
> > +	if (local && !p->pack_local)
> > +		return 0;
> > +	if (ignore_packed_keep && p->pack_local && p->pack_keep)
> > +		return 0;
> > +
> > +	/* we don't know yet; keep looking for more packs */
> > +	return -1;
> > +}
> 
> Moving this logic out to this helper made the main logic in the
> caller easier to grasp.
> 
> > @@ -958,15 +993,30 @@ static int want_object_in_pack(const unsigned char *sha1,
> >  			       off_t *found_offset)
> >  {
> >  	struct packed_git *p;
> > +	int want;
> >  
> >  	if (!exclude && local && has_loose_object_nonlocal(sha1))
> >  		return 0;
> >  
> > +	/*
> > +	 * If we already know the pack object lives in, start checks from that
> > +	 * pack - in the usual case when neither --local was given nor .keep files
> > +	 * are present we will determine the answer right now.
> > +	 */
> > +	if (*found_pack) {
> > +		want = want_found_object(exclude, *found_pack);
> > +		if (want != -1)
> > +			return want;
> > +	}
> >  
> >  	for (p = packed_git; p; p = p->next) {
> > +		off_t offset;
> > +
> > +		if (p == *found_pack)
> > +			offset = *found_offset;
> > +		else
> > +			offset = find_pack_entry_one(sha1, p);
> > +
> >  		if (offset) {
> >  			if (!*found_pack) {
> >  				if (!is_pack_valid(p))
> > @@ -974,31 +1024,9 @@ static int want_object_in_pack(const unsigned char *sha1,
> >  				*found_offset = offset;
> >  				*found_pack = p;
> >  			}
> > +			want = want_found_object(exclude, p);
> > +			if (want != -1)
> > +				return want;
> >  		}
> >  	}
> 
> As Peff noted in his earlier review, however, MRU code needed to be
> grafted in to the caller (an update to the MRU list was done in the
> code that was moved to the want_found_object() helper).  I think I
> did it correctly, which ended up looking like this:
> 
>                 want = want_found_object(exclude, p);
>                 if (!exclude && want > 0)
>                         mru_mark(packed_git_mru, entry);
>                 if (want != -1)
>                         return want;
> 
> I somewhat feel that it is ugly that the helper knows about exclude
> (i.e. in the original code, we immediately returned 1 without
> futzing with the MRU when we find an entry that is to be excluded,
> which now is done in the helper), and the caller also knows about
> exclude (i.e. the caller knows that the helper may return positive
> in two cases, it knows that MRU marking needs to happen only one of
> the two cases, and it also knows that "exclude" is what
> differentiates between the two cases) at the same time.
> 
> But probably the reason why I feel it ugly is only because I knew
> how the original looked like.  I dunno.

Junio, the code above is correct semantic merge of pack-mru and my topic, because in pack-mru if found and exclude=1, 1 was returned without marking found pack.

But I wonder: even if we exclude an object, we were still looking for it in packs, and when we found it, we found the corresponding pack too. So, that pack _was_ most-recently-used, and it is correct to mark it as MRU.

We can do the simplification in the follow-up patch after the merge, so merge does not change semantics and it is all bisectable, etc.

Jeff?
Previous: Junio C HamanoNext: Kirill Smelkov
Message 35 of 62 in “pack-objects: Use reachability bitmap index when generating non-stdout pack too”
  1. pack-objects: Use reachability bitmap index when generating non-stdout pack tooKirill Smelkov, Jul 7, 2016
  2. Jeff KingJul 7, 2016
  3. Kirill SmelkovJul 8, 2016
  4. Kirill SmelkovJul 12, 2016
  5. Jeff KingJul 13, 2016
  6. Kirill SmelkovJul 13, 2016
  7. Kirill SmelkovJul 17, 2016
  8. Jeff KingJul 19, 2016
  9. Kirill SmelkovJul 19, 2016
  10. Jeff KingJul 25, 2016
  11. Jeff KingJul 25, 2016
  12. Kirill SmelkovJul 27, 2016
  13. Junio C HamanoJul 27, 2016
  14. Kirill SmelkovJul 28, 2016
  15. Junio C HamanoJul 28, 2016
  16. Kirill SmelkovJul 29, 2016
  17. 1/2 pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incrementalKirill Smelkov, Jul 29, 2016
  18. Junio C HamanoAug 1, 2016
  19. Kirill SmelkovAug 8, 2016
  20. Jeff KingAug 8, 2016
  21. Jeff KingAug 8, 2016
  22. Junio C HamanoAug 8, 2016
  23. Kirill SmelkovAug 8, 2016
  24. pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incrementalKirill Smelkov, Aug 8, 2016
  25. Junio C HamanoAug 8, 2016
  26. Kirill SmelkovAug 9, 2016
  27. 1/2 pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in useKirill Smelkov, Aug 9, 2016
  28. Junio C HamanoAug 9, 2016
  29. Kirill SmelkovAug 9, 2016
  30. 1/2 pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in useKirill Smelkov, Aug 9, 2016
  31. Jeff KingAug 18, 2016
  32. Kirill SmelkovSep 10, 2016
  33. 1/2 pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in useKirill Smelkov, Sep 10, 2016
  34. Junio C HamanoSep 13, 2016
  35. Kirill SmelkovSep 13, 2016
  36. t/perf/run: Don't forget to copy config.mak.autogen & friends to build areaKirill Smelkov, Sep 10, 2016
  37. Junio C HamanoSep 12, 2016
  38. Junio C HamanoSep 12, 2016
  39. Junio C HamanoSep 12, 2016
  40. Kirill SmelkovSep 13, 2016
  41. Junio C HamanoSep 12, 2016
  42. 2/2 pack-objects: use reachability bitmap index when generating non-stdout packKirill Smelkov, Aug 9, 2016
  43. Jeff KingAug 18, 2016
  44. Kirill SmelkovSep 10, 2016
  45. 2/2 pack-objects: use reachability bitmap index when generating non-stdout packKirill Smelkov, Sep 10, 2016
  46. Junio C HamanoSep 12, 2016
  47. Junio C HamanoAug 9, 2016
  48. Junio C HamanoAug 8, 2016
  49. Jeff KingAug 8, 2016
  50. Junio C HamanoAug 8, 2016
  51. 2/2 pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack tooKirill Smelkov, Jul 29, 2016
  52. Jeff KingAug 8, 2016
  53. Kirill SmelkovAug 8, 2016
  54. Junio C HamanoAug 8, 2016
  55. Kirill SmelkovAug 8, 2016
  56. Junio C HamanoAug 8, 2016
  57. Kirill SmelkovAug 8, 2016
  58. pack-objects: teach it to use reachability bitmap index when generating non-stdout pack tooKirill Smelkov, Aug 8, 2016
  59. Junio C HamanoAug 8, 2016
  60. Kirill SmelkovAug 9, 2016
  61. 2/2 pack-objects: use reachability bitmap index when generating non-stdout packKirill Smelkov, Aug 9, 2016
  62. Jeff KingJul 13, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.