[PATCH 00/11] Reduce pack-objects memory footprint
- From
Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
- Date
- Mar 1, 2018, 09:10 UTC
- Message-ID
- <20180301091052.32267-1-pclouds@gmail.com>
- In-Reply-To
- <20180228092722.GA25627@ash>
The array of object_entry in pack-objects can take a lot of memory when pack-objects is run in "pack everything" mode. On linux-2.6.git, this array alone takes roughly 800MB.
This series reorders some fields and reduces field size... to keep this struct smaller. Its size goes from 136 bytes to 96 bytes (29%) on 64-bit linux and saves 260MB on linux-2.6.git.
Now the bad side:
- the number of pack files pack-objects can handle is reduced to 4096 (previously unlimited) - max delta chain is also limited to 4096 (previously practically unlimited) - some patches are quite invasive (e.g. replacing pointer with uint32_t) and reduces readability a bit. - it may be tricker to add more data in object_entry in the future.
Nguyễn Thái Ngọc Duy (11): pack-objects: document holes in struct object_entry.h pack-objects: turn type and in_pack_type to bitfields pack-objects: use bitfield for object_entry::dfs_state pack-objects: use bitfield for object_entry::depth pack-objects: note about in_pack_header_size pack-objects: move in_pack_pos out of struct object_entry pack-objects: move in_pack out of struct object_entry pack-objects: faster reverse packed_git lookup pack-objects: refer to delta objects by index instead of pointer pack-objects: reorder 'hash' to pack struct object_entry pack-objects: increase pack file limit to 4096
builtin/pack-objects.c | 189 ++++++++++++++++++++++++++--------------- cache.h | 3 + object.h | 1 - pack-bitmap-write.c | 8 +- pack-bitmap.c | 2 +- pack-bitmap.h | 4 +- pack-objects.h | 70 ++++++++++----- 7 files changed, 180 insertions(+), 97 deletions(-)
-- 2.16.1.435.g8f24da2e1a