git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 00/21] np/pack-v4 updates

From
Nicolas Pitre <nico@fluxnic.net>
Date
Sep 12, 2013, 16:20 UTC
Message-ID
<alpine.LFD.2.03.1309121214550.20709@syhkavp.arg>
In-Reply-To
<CACsJy8B7WAz4eB88gJCWc3Z0fpSrop8f_Xex1tkmykYuSYUFUQ@mail.gmail.com>
On Thu, 12 Sep 2013, Duy Nguyen wrote:
Show 31 quoted lines
> On Wed, Sep 11, 2013 at 11:25 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > On Wed, 11 Sep 2013, Duy Nguyen wrote:
> >
> >> Nico, if you have time you may want to look into this. The result v4
> >> pack from pack-objects on git.git for me is 35MB (one branch) while
> >> packv4-create produces 30MB (v2 is 40MB). I don't know why there is
> >> such a big difference in size. I compared. Ident dict is identical.
> >> Tree dict is a bit different (some that have same hits are ordered
> >> differently). Delta chains do not differ much. Many groups of entries
> >> in the pack are displaced though. I guess I turned a wrong knob or
> >> something in pack-objects in v4 code..
> >
> > Will try to have a closer look.
> 
> Problem found. I encoded some trees as ref-delta instead of pv4-tree
> :( Something like this brings the size back to packv4-create output
> 
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index f604fa5..3d9ab0e 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -1490,7 +1490,8 @@ static void check_object(struct object_entry *entry)
>   * deltify other objects against, in order to avoid
>   * circular deltas.
>   */
> - entry->type = entry->in_pack_type;
> + if (pack_version < 4)
> + entry->type = entry->in_pack_type;
>   entry->delta = base_entry;
>   entry->delta_size = entry->size;
>   entry->delta_sibling = base_entry->delta_child;
Hmmm... I've folded this fix into your patch touching this area.

This code is becoming rather subtle and messy though. We'll have to find a way to better abstract things. Especially since object data reuse will work only for blobs and tags with packv4. Commits and trees will need adjustments to their indices.

Nicolas
Previous: Duy NguyenNext: Duy Nguyen
Message 34 of 36 in “[PULL REQUEST] initial pack v4 support”
  1. Nicolas PitreSep 9, 2013
  2. Junio C HamanoSep 9, 2013
  3. Junio C HamanoSep 10, 2013
  4. Nicolas PitreSep 10, 2013
  5. Junio C HamanoSep 10, 2013
  6. Nicolas PitreSep 10, 2013
  7. 00/21 np/pack-v4 updatesNguyễn Thái Ngọc Duy, Sep 11, 2013
  8. 01/21 fixup! pack-objects: prepare SHA-1 table in v4Nguyễn Thái Ngọc Duy, Sep 11, 2013
  9. 02/21 fixup! pack-objects: support writing pack v4Nguyễn Thái Ngọc Duy, Sep 11, 2013
  10. 03/21 fixup! pack v4: support "end-of-pack" indicator in index-pack and pack-objectsNguyễn Thái Ngọc Duy, Sep 11, 2013
  11. 04/21 fixup! index-pack: parse v4 header and dictionariesNguyễn Thái Ngọc Duy, Sep 11, 2013
  12. 05/21 fixup! index-pack: record all delta bases in v4 (tree and ref-delta)Nguyễn Thái Ngọc Duy, Sep 11, 2013
  13. 06/21 pack v4: lift dict size check in load_dict()Nguyễn Thái Ngọc Duy, Sep 11, 2013
  14. 07/21 pack v4: move pv4 objhdr parsing code to packv4-parse.cNguyễn Thái Ngọc Duy, Sep 11, 2013
  15. 08/21 pack-objects: respect compression level in v4Nguyễn Thái Ngọc Duy, Sep 11, 2013
  16. 09/21 pack-objects: recognize v4 as pack sourceNguyễn Thái Ngọc Duy, Sep 11, 2013
  17. 10/21 pack v4: add a note that streaming does not support OBJ_PV4_*Nguyễn Thái Ngọc Duy, Sep 11, 2013
  18. 11/21 unpack-objects: report missing object nameNguyễn Thái Ngọc Duy, Sep 11, 2013
  19. 12/21 unpack-objects: recognize end-of-pack in v4 thin packNguyễn Thái Ngọc Duy, Sep 11, 2013
  20. 13/21 unpack-objects: read v4 dictionariesNguyễn Thái Ngọc Duy, Sep 11, 2013
  21. 14/21 unpack-objects: decode v4 object headerNguyễn Thái Ngọc Duy, Sep 11, 2013
  22. 15/21 unpack-objects: decode v4 ref-deltaNguyễn Thái Ngọc Duy, Sep 11, 2013
  23. 16/21 unpack-objects: decode v4 commitsNguyễn Thái Ngọc Duy, Sep 11, 2013
  24. 17/21 unpack-objects: allow to save processed bytes to a bufferNguyễn Thái Ngọc Duy, Sep 11, 2013
  25. 18/21 unpack-objects: decode v4 treesNguyễn Thái Ngọc Duy, Sep 11, 2013
  26. 19/21 index-pack, pack-objects: allow creating .idx v2 with .pack v4Nguyễn Thái Ngọc Duy, Sep 11, 2013
  27. Nicolas PitreSep 11, 2013
  28. 20/21 show-index: acknowledge that it does not read .idx v3Nguyễn Thái Ngọc Duy, Sep 11, 2013
  29. Nicolas PitreSep 11, 2013
  30. 21/21 t1050, t5500: replace the use of "show-index|wc -l" with verify-packNguyễn Thái Ngọc Duy, Sep 11, 2013
  31. Duy NguyenSep 11, 2013
  32. Nicolas PitreSep 11, 2013
  33. Duy NguyenSep 12, 2013
  34. Nicolas PitreSep 12, 2013
  35. Duy NguyenSep 13, 2013
  36. Nicolas PitreSep 11, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.