Re: [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs

From

IKIan Kumlien <pomac@vapor.com>

Date

Feb 25, 2012, 13:17 UTC

Message-ID

<20120225131708.GI9526@pomac.netswarm.net>

In-Reply-To

<CACsJy8C-8dvXpNTU=JpdupSpS8OuqqTpGvDs6s1ASeKdk9d5Dg@mail.gmail.com>

On Sat, Feb 25, 2012 at 08:49:55AM +0700, Nguyen Thai Ngoc Duy wrote:

> 2012/2/24 Ian Kumlien <pomac@vapor.com>:
> > Writing objects: 100% (1425/1425), 56.06 MiB | 4.62 MiB/s, done.
> > Total 1425 (delta 790), reused 1425 (delta 790)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > To ../test_data/
> >  ! [remote rejected] master -> master (missing necessary objects)
> >  ! [remote rejected] origin/HEAD -> origin/HEAD (missing necessary objects)
> >  ! [remote rejected] origin/master -> origin/master (missing necessary objects)
> > error: failed to push some refs to '../test_data/'
> >
> > So there are additional code paths to look at... =(
> 
> I can't say where that came from. Does this help? (Space damaged, may
> need manual application)

Anyway, with:

diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index 264e3ae..533081d 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c
@@ -183,7 +183,8 @@ static void show_object(struct object *obj,
        struct rev_list_info *info = cb_data;

        finish_object(obj, path, component, cb_data);
-       if (info->revs->verify_objects && !obj->parsed && obj->type != OBJ_COMMIT)
+       if (info->revs->verify_objects && !obj->parsed
+                       && obj->type != OBJ_COMMIT && obj->type != OBJ_BLOB)
                parse_object(obj->sha1);
        show_object_with_name(stdout, obj, path, component);
 }
---

I get:
../git/git push --mirror ../test_data/
Counting objects: 1425, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (617/617), done.
Writing objects: 100% (1425/1425), 56.06 MiB | 4.22 MiB/s, done.
Total 1425 (delta 790), reused 1425 (delta 790)
error: index-pack died of signal 11
error: unpack failed: index-pack abnormal exit
To ../test_data/
 ! [remote rejected] master -> master (n/a (unpacker error))
 ! [remote rejected] origin/HEAD -> origin/HEAD (n/a (unpacker error))
 ! [remote rejected] origin/master -> origin/master (n/a (unpacker error))
error: failed to push some refs to '../test_data/'

Which, to me, means that the installed git is now the problem - it can't verify 
the pack and say that it's all ok ;)

I'll have to look some more at this on monday, or during the weekend if i get too curious =)

For now, thank $deity that $company i work for allows VPN from Linux machines! It looks
really good, i wonder if there is further tests i should do - any clues?


> -- 
> Duy