git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] sha1_file: remove recursion in packed_object_info

From
thomas <trast@student.ethz.ch>
Date
Mar 25, 2013, 09:27 UTC
Message-ID
<87620faky3.fsf@linux-k42r.v.cablecom.net>
In-Reply-To
<7vtxo6f27l.fsf@alter.siamese.dyndns.org>
Junio C Hamano <gitster@pobox.com> writes:
Show 23 quoted lines
> Thomas Rast <trast@student.ethz.ch> writes:
>
>> So here's a nonrecursive version.  Dijkstra is probably turning over
>> in his grave as we speak.
>>
>> I *think* I actually got it right.
>
> You seem to have lost the "if we cannot get delta base, this object
> is BAD" check where you measure the size of a deltified object,
> which would correspond to this check:
>
>> -static int packed_delta_info(struct packed_git *p,
>> -			     struct pack_window **w_curs,
>> -			     off_t curpos,
>> -			     enum object_type type,
>> -			     off_t obj_offset,
>> -			     unsigned long *sizep)
>> -{
>> -	off_t base_offset;
>> -
>> -	base_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);
>> -	if (!base_offset)
>> -		return OBJ_BAD;
True, I'll think about this.
Show 15 quoted lines
> The following comment is also lost but...
>
>> -	/* We choose to only get the type of the base object and
>> -	 * ignore potentially corrupt pack file that expects the delta
>> -	 * based on a base with a wrong size.  This saves tons of
>> -	 * inflate() calls.
>> -	 */
>> -	if (sizep) {
>> -		*sizep = get_size_from_delta(p, w_curs, curpos);
>> -		if (*sizep == 0)
>> -			type = OBJ_BAD;
>
> ... is this check correct?  There is an equivalent check at the
> beginning of the new packed_object_info() to error out a deltified
> result.  Why is an object whose size is 0 bad?
Cc'ing Nicolas, but I think there are several reasons:

If it's a delta, then according to docs[1] it starts with the SHA1 of the base object, plus the deflated data. So it is at least 20 bytes.

If it's not a delta, then it must start with '<type> <size>\0', which even after compression cannot possibly be 0 bytes.

Either way, get_size_from_delta() also uses 0 as the error return.
Show 7 quoted lines
> The stack/recursion is used _only_ for error recovery, no?  If we do
> not care about retrying with a different copy of an object we find
> in the delta chain, we can just update obj_offset with base_offset
> and keep digging.  It almost makes me wonder if a logical follow-up
> to this patch may be to do so, and rewrite the error recovery
> codepath to just mark the bad copy and jump back to the very top,
> retrying everything from scratch.

I totally agree. I'll try this again -- my last attempt just didn't work out...

[1]  Documentation/technical/pack-format.txt
-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Previous: Junio C HamanoNext: Thomas Rast
Message 36 of 46 in “regression in multi-threaded git-pack-index”
  1. Stefan ZagerMar 15, 2013
  2. Jeff KingMar 16, 2013
  3. Duy NguyenMar 16, 2013
  4. Thomas RastMar 19, 2013
  5. Jeff KingMar 19, 2013
  6. Jeff KingMar 19, 2013
  7. Jeff KingMar 19, 2013
  8. Jeff KingMar 19, 2013
  9. Thomas RastMar 19, 2013
  10. Jeff KingMar 19, 2013
  11. Thomas RastMar 19, 2013
  12. Jeff KingMar 19, 2013
  13. index-pack: always zero-initialize object_entry listJeff King, Mar 19, 2013
  14. Thomas RastMar 19, 2013
  15. Jeff KingMar 19, 2013
  16. Jeff KingMar 19, 2013
  17. index-pack: always zero-initialize object_entry listJeff King, Mar 19, 2013
  18. Thomas RastMar 19, 2013
  19. Junio C HamanoMar 19, 2013
  20. Eric SunshineMar 20, 2013
  21. Jeff KingMar 20, 2013
  22. Eric SunshineMar 20, 2013
  23. Duy NguyenMar 19, 2013
  24. index-pack: protect deepest_delta in multithread codeNguyễn Thái Ngọc Duy, Mar 19, 2013
  25. Jeff KingMar 19, 2013
  26. Thomas RastMar 19, 2013
  27. Duy NguyenMar 19, 2013
  28. index-pack: guard nr_resolved_deltas reads by lockThomas Rast, Mar 19, 2013
  29. Junio C HamanoMar 19, 2013
  30. Thomas RastMar 19, 2013
  31. Thomas RastMar 19, 2013
  32. Thomas RastMar 19, 2013
  33. Junio C HamanoMar 19, 2013
  34. sha1_file: remove recursion in packed_object_infoThomas Rast, Mar 19, 2013
  35. Junio C HamanoMar 20, 2013
  36. thomasMar 25, 2013
  37. 0/3 Recursion-free unpack_entry and packed_object_infoThomas Rast, Mar 25, 2013
  38. 1/3 sha1_file: remove recursion in packed_object_infoThomas Rast, Mar 25, 2013
  39. 2/3 Refactor parts of in_delta_base_cache/cache_or_unpack_entryThomas Rast, Mar 25, 2013
  40. Junio C HamanoMar 25, 2013
  41. thomasMar 26, 2013
  42. 3/3 sha1_file: remove recursion in unpack_entryThomas Rast, Mar 25, 2013
  43. Junio C HamanoMar 25, 2013
  44. Nicolas PitreMar 26, 2013
  45. Junio C HamanoMar 25, 2013
  46. Duy NguyenMar 20, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.