{"thread":{"id":"14606","subject":"regression in 92392b4","startedAt":"2008-07-22T23:17:45Z","lastAt":"2008-07-23T15:30:12Z","messageCount":24,"participants":["Pierre Habouzit","Johannes Schindelin","Shawn O. Pearce","Björn Steinbrink"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"84397","messageId":"20080722231745.GD11831@artemis.madism.org","threadId":"14606","inReplyTo":null,"subject":"regression in 92392b4","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-22T23:17:45Z","receivedAt":"2008-07-22T23:17:45Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":" Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\ndid, since git in next cannot fetch on a regular basis for me. The\nculprit seems to be commit 92392b4:\n\n ┌─(1:11)──<~/dev/scm/git 92392b4...>──\n └[artemis] git fetch\n remote: Counting objects: 461, done.\n remote: Compressing objects: 100% (141/141), done.\n remote: Total 263 (delta 227), reused 155 (delta 121)\n Receiving objects: 100% (263/263), 95.55 KiB, done.\n fatal: Out of memory, malloc failed\n fatal: index-pack failed\n [2] 16674 abort (core dumped) git fetch\n\n ┌─(1:12)──<~/dev/scm/git 92392b4...>──\n └[artemis] git checkout -m HEAD~1; make git-index-pack\n Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n GIT_VERSION = 1.5.6.3.3.g03993\n\tCC index-pack.o\n\tLINK git-index-pack\n\n ┌─(1:12)──<~/dev/scm/git 03993e1...>──\n └[artemis] git fetch\n remote: Counting objects: 461, done.\n remote: Compressing objects: 100% (141/141), done.\n remote: Total 263 (delta 227), reused 155 (delta 121)\n Receiving objects: 100% (263/263), 95.55 KiB, done.\n Resolving deltas: 100% (227/227), completed with 153 local objects.\n From git://git.kernel.org/pub/scm/git/git\n 5ba2c22..0868a30 html -> origin/html\n 2857e17..abeeabe man -> origin/man\n 93310a4..95f8ebb master -> origin/master\n 559998f..e8bf351 next -> origin/next\n\nYou can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\nbroken, I've absolutely no clue about what happens.\n\nAll I can say is that at some point in get_data_from_pack, obj[1].idx\npoints to something that is *not* a sha so it's probably corrupted.\n(from index-pack.c).\n\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"84402","messageId":"alpine.DEB.1.00.0807230033000.8986@racer","threadId":"14606","inReplyTo":"20080722231745.GD11831@artemis.madism.org","subject":"Re: regression in 92392b4","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-22T23:34:03Z","receivedAt":"2008-07-22T23:34:03Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Jul 2008, Pierre Habouzit wrote:\n\n> Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> did, since git in next cannot fetch on a regular basis for me. The\n> culprit seems to be commit 92392b4:\n> \n> ┌─(1:11)──<~/dev/scm/git 92392b4...>──\n> └[artemis] git fetch\n> remote: Counting objects: 461, done.\n> remote: Compressing objects: 100% (141/141), done.\n> remote: Total 263 (delta 227), reused 155 (delta 121)\n> Receiving objects: 100% (263/263), 95.55 KiB, done.\n> fatal: Out of memory, malloc failed\n> fatal: index-pack failed\n> [2] 16674 abort (core dumped) git fetch\n> \n> ┌─(1:12)──<~/dev/scm/git 92392b4...>──\n> └[artemis] git checkout -m HEAD~1; make git-index-pack\n> Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n> HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n> GIT_VERSION = 1.5.6.3.3.g03993\n> \tCC index-pack.o\n> \tLINK git-index-pack\n> \n> ┌─(1:12)──<~/dev/scm/git 03993e1...>──\n> └[artemis] git fetch\n> remote: Counting objects: 461, done.\n> remote: Compressing objects: 100% (141/141), done.\n> remote: Total 263 (delta 227), reused 155 (delta 121)\n> Receiving objects: 100% (263/263), 95.55 KiB, done.\n> Resolving deltas: 100% (227/227), completed with 153 local objects.\n> From git://git.kernel.org/pub/scm/git/git\n> 5ba2c22..0868a30 html -> origin/html\n> 2857e17..abeeabe man -> origin/man\n> 93310a4..95f8ebb master -> origin/master\n> 559998f..e8bf351 next -> origin/next\n> \n> You can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\n> broken, I've absolutely no clue about what happens.\n> \n> All I can say is that at some point in get_data_from_pack, obj[1].idx\n> points to something that is *not* a sha so it's probably corrupted.\n> (from index-pack.c).\n\nJust a guess:\n\n-- snipsnap --\n[PATCH] index-pack: set base_data's data member to NULL after freeing it\n\nSigned-off-by: Johannes Schindelin \n\n---\n\n index-pack.c | 1 +\n 1 files changed, 1 insertions(+), 0 deletions(-)\n\ndiff --git a/index-pack.c b/index-pack.c\nindex ac20a46..19c39e5 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -257,6 +257,7 @@ static void unlink_base_data(struct base_data *c)\n \t\tbase_cache = NULL;\n \tif (c->data) {\n \t\tfree(c->data);\n+\t\tc->data = NULL;\n \t\tbase_cache_used -= c->size;\n \t}\n }"},{"id":"84404","messageId":"20080722233716.GE11831@artemis.madism.org","threadId":"14606","inReplyTo":"20080722231745.GD11831@artemis.madism.org","subject":"Re: regression in 92392b4","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-22T23:37:16Z","receivedAt":"2008-07-22T23:37:16Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Tue, Jul 22, 2008 at 11:17:45PM +0000, Pierre Habouzit wrote:\n> Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> did, since git in next cannot fetch on a regular basis for me. The\n> culprit seems to be commit 92392b4:\n> \n> ┌─(1:11)──<~/dev/scm/git 92392b4...>──\n> └[artemis] git fetch\n> remote: Counting objects: 461, done.\n> remote: Compressing objects: 100% (141/141), done.\n> remote: Total 263 (delta 227), reused 155 (delta 121)\n> Receiving objects: 100% (263/263), 95.55 KiB, done.\n> fatal: Out of memory, malloc failed\n> fatal: index-pack failed\n> [2] 16674 abort (core dumped) git fetch\n> \n> ┌─(1:12)──<~/dev/scm/git 92392b4...>──\n> └[artemis] git checkout -m HEAD~1; make git-index-pack\n> Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n> HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n> GIT_VERSION = 1.5.6.3.3.g03993\n> \tCC index-pack.o\n> \tLINK git-index-pack\n> \n> ┌─(1:12)──<~/dev/scm/git 03993e1...>──\n> └[artemis] git fetch\n> remote: Counting objects: 461, done.\n> remote: Compressing objects: 100% (141/141), done.\n> remote: Total 263 (delta 227), reused 155 (delta 121)\n> Receiving objects: 100% (263/263), 95.55 KiB, done.\n> Resolving deltas: 100% (227/227), completed with 153 local objects.\n> From git://git.kernel.org/pub/scm/git/git\n> 5ba2c22..0868a30 html -> origin/html\n> 2857e17..abeeabe man -> origin/man\n> 93310a4..95f8ebb master -> origin/master\n> 559998f..e8bf351 next -> origin/next\n> \n> You can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\n> broken, I've absolutely no clue about what happens.\n> \n> All I can say is that at some point in get_data_from_pack, obj[1].idx\n> points to something that is *not* a sha so it's probably corrupted.\n> (from index-pack.c).\n\n Though reading the code, we trust c->data NULL-iness to mean we have\nno data, and there is one code path that fails to reset it. The problem\nis I'm not able to reproduce the bug, because I foolishly didn't\nbackuped the git repository that exhibited the failure, so I cannot know\nif that can be the problem:\n\n--- snip ---\nFrom c3749f7bc50c642c5d437b2746d4ba589b7d9739 Mon Sep 17 00:00:00 2001\nFrom: Pierre Habouzit \nDate: Wed, 23 Jul 2008 01:35:11 +0200\nSubject: [PATCH] index-pack: missing pointer reset to NULL.\n\nSigned-off-by: Pierre Habouzit \n---\n index-pack.c | 1 +\n 1 files changed, 1 insertions(+), 0 deletions(-)\n\ndiff --git a/index-pack.c b/index-pack.c\nindex ac20a46..19c39e5 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -257,6 +257,7 @@ static void unlink_base_data(struct base_data *c)\n \t\tbase_cache = NULL;\n \tif (c->data) {\n \t\tfree(c->data);\n+\t\tc->data = NULL;\n \t\tbase_cache_used -= c->size;\n \t}\n }\n-- \n1.6.0.rc0.153.ge8bf3.dirty\n\n"},{"id":"84415","messageId":"20080723004108.GB14668@spearce.org","threadId":"14606","inReplyTo":"alpine.DEB.1.00.0807230033000.8986@racer","subject":"Re: regression in 92392b4","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-07-23T00:41:08Z","receivedAt":"2008-07-23T00:41:08Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin wrote:\n> On Wed, 23 Jul 2008, Pierre Habouzit wrote:\n> \n> > Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> > did, since git in next cannot fetch on a regular basis for me. The\n> > culprit seems to be commit 92392b4:\n> > \n> > ┌─(1:11)──<~/dev/scm/git 92392b4...>──\n> > └[artemis] git fetch\n> > remote: Counting objects: 461, done.\n> > remote: Compressing objects: 100% (141/141), done.\n> > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > fatal: Out of memory, malloc failed\n> > fatal: index-pack failed\n> > [2] 16674 abort (core dumped) git fetch\n...\n> \n> Just a guess:\n...\n> diff --git a/index-pack.c b/index-pack.c\n> index ac20a46..19c39e5 100644\n> --- a/index-pack.c\n> +++ b/index-pack.c\n> @@ -257,6 +257,7 @@ static void unlink_base_data(struct base_data *c)\n> \t\tbase_cache = NULL;\n> \tif (c->data) {\n> \t\tfree(c->data);\n> +\t\tc->data = NULL;\n> \t\tbase_cache_used -= c->size;\n> \t}\n> }\n\nOh. This is a pointless assignment. If you look at any call sites\nfor unlink_base_data() you will find that the struct passed in as\n\"c\" here is going out of scope after unlink_base_data() returns. In\nno such case does the value of c->data get tested once this free is\ncomplete.\n\nWe need the if (c->data) guard because we only want to decrement\nbase_cache_used if the memory is still allocated. It may have been\nreleased earlier, in which case base_cache_used has already been\ndecreased and we don't want to double-decrement it.\n\nThis patch makes the code more obvious, so Ack I guess, but it is\nnot a solution to Pierre's woes. Something else is wrong.\n\nReading above shows we got a \"fatal: Out of memory, malloc failed\"\nright before the segfault. What's odd is we segfaulted after we\nran out of memory and should have die'd.\n\nThere's at least two bugs in the above output:\n\na) index-pack ran out of memory on a small pull (95 KiB).\nb) fetch segfaulted when index-pack failed.\n\nAnd this patch will unfortunately address neither of them. :-|\n\nI've had a long past couple of days, and another one tomorrow.\nI'm not going to be able to debug this myself until perhaps Thursday\nor Friday. Sorry. If nobody beats me to it, I will put this on\nthe top of the pile and try to fix it once I get back online at my\nnew home.\n\n-- \nShawn.\n"},{"id":"84419","messageId":"alpine.DEB.1.00.0807230152580.8986@racer","threadId":"14606","inReplyTo":"20080723004108.GB14668@spearce.org","subject":"Re: regression in 92392b4","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-23T00:58:38Z","receivedAt":"2008-07-23T00:58:38Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 22 Jul 2008, Shawn O. Pearce wrote:\n\n> Reading above shows we got a \"fatal: Out of memory, malloc failed\" right \n> before the segfault. What's odd is we segfaulted after we ran out of \n> memory and should have die'd.\n> \n> There's at least two bugs in the above output:\n> \n> a) index-pack ran out of memory on a small pull (95 KiB).\n> b) fetch segfaulted when index-pack failed.\n> \n> And this patch will unfortunately address neither of them. :-|\n\nYeah, I thought I had something, but no matter what I tried, I could not \nfind a breakage. Not even valgrind complains.\n\nBut as I said: it was just a guess.\n\nMaybe it will be easier to reproduce by patching the code to malloc() \neverything that is available first, and then continuing.\n\nCiao,\nDscho\n"},{"id":"84422","messageId":"20080723010928.GG11831@artemis.madism.org","threadId":"14606","inReplyTo":"20080723004108.GB14668@spearce.org","subject":"Re: regression in 92392b4","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-23T01:09:28Z","receivedAt":"2008-07-23T01:09:28Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jul 23, 2008 at 12:41:08AM +0000, Shawn O. Pearce wrote:\n> Johannes Schindelin wrote:\n> > On Wed, 23 Jul 2008, Pierre Habouzit wrote:\n> > \n> > > Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> > > did, since git in next cannot fetch on a regular basis for me. The\n> > > culprit seems to be commit 92392b4:\n> > > \n> > > ┌─(1:11)──<~/dev/scm/git 92392b4....>──\n> > > └[artemis] git fetch\n> > > remote: Counting objects: 461, done.\n> > > remote: Compressing objects: 100% (141/141), done.\n> > > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > > fatal: Out of memory, malloc failed\n> > > fatal: index-pack failed\n> > > [2] 16674 abort (core dumped) git fetch\n> ....\n> > \n> > Just a guess:\n> ....\n> > diff --git a/index-pack.c b/index-pack.c\n> > index ac20a46..19c39e5 100644\n> > --- a/index-pack.c\n> > +++ b/index-pack.c\n> > @@ -257,6 +257,7 @@ static void unlink_base_data(struct base_data *c)\n> > \t\tbase_cache = NULL;\n> > \tif (c->data) {\n> > \t\tfree(c->data);\n> > +\t\tc->data = NULL;\n> > \t\tbase_cache_used -= c->size;\n> > \t}\n> > }\n> \n> Oh. This is a pointless assignment. If you look at any call sites\n> for unlink_base_data() you will find that the struct passed in as\n> \"c\" here is going out of scope after unlink_base_data() returns. In\n> no such case does the value of c->data get tested once this free is\n> complete.\n> \n> We need the if (c->data) guard because we only want to decrement\n> base_cache_used if the memory is still allocated. It may have been\n> released earlier, in which case base_cache_used has already been\n> decreased and we don't want to double-decrement it.\n> \n> This patch makes the code more obvious, so Ack I guess, but it is\n> not a solution to Pierre's woes. Something else is wrong.\n> \n> Reading above shows we got a \"fatal: Out of memory, malloc failed\"\n> right before the segfault. What's odd is we segfaulted after we\n> ran out of memory and should have die'd.\n> \n> There's at least two bugs in the above output:\n> \n> a) index-pack ran out of memory on a small pull (95 KiB).\n> b) fetch segfaulted when index-pack failed.\n> \n> And this patch will unfortunately address neither of them. :-|\n> \n> I've had a long past couple of days, and another one tomorrow.\n> I'm not going to be able to debug this myself until perhaps Thursday\n> or Friday. Sorry. If nobody beats me to it, I will put this on\n> the top of the pile and try to fix it once I get back online at my\n> new home.\n\n Like I said, I had a core that I stupidly lost, but I remember that\nthe broken malloc was:\n\n\n static void *get_data_from_pack(struct object_entry *obj)\n {\n\t off_t from = obj[0].idx.offset + obj[0].hdr_size;\n\t unsigned long len = obj[1].idx.offset - from;\n\t unsigned long rdy = 0;\n\t unsigned char *src, *data;\n\t z_stream stream;\n\t int st;\n\n\t src = xmalloc(len);\n ^^^^^^^^^^^^^^^^^^\n\n len was horribly big, and outputing obj[1].idx showed that `sha1` had\ntext in it. I mean something like \"could not\\r\\n han\" IIRC.\n\n I don't remember the rest of the backtrace, and have stupidly not kept\nany ways of reproducing it.\n"},{"id":"84427","messageId":"alpine.DEB.1.00.0807230219100.8986@racer","threadId":"14606","inReplyTo":"20080723010928.GG11831@artemis.madism.org","subject":"Re: regression in 92392b4","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-23T01:20:17Z","receivedAt":"2008-07-23T01:20:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Jul 2008, Pierre Habouzit wrote:\n\n> I had a core that I stupidly lost, but I remember that the broken malloc \n> was:\n> \n> \n> static void *get_data_from_pack(struct object_entry *obj)\n> {\n> \t off_t from = obj[0].idx.offset + obj[0].hdr_size;\n> \t unsigned long len = obj[1].idx.offset - from;\n> \t unsigned long rdy = 0;\n> \t unsigned char *src, *data;\n> \t z_stream stream;\n> \t int st;\n> \n> \t src = xmalloc(len);\n> ^^^^^^^^^^^^^^^^^^\n> \n> len was horribly big, and outputing obj[1].idx showed that `sha1` had\n> text in it. I mean something like \"could not\\r\\n han\" IIRC.\n> \n> I don't remember the rest of the backtrace, and have stupidly not kept\n> any ways of reproducing it.\n\nThat would not have helped. The memory corruption almost _certainly_ took \nplace way before that.\n\nCiao,\nDscho\n"},{"id":"84456","messageId":"20080723101415.GA23769@atjola.homenet","threadId":"14606","inReplyTo":"20080722231745.GD11831@artemis.madism.org","subject":"Re: regression in 92392b4","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-07-23T10:14:15Z","receivedAt":"2008-07-23T10:14:15Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.07.23 01:17:45 +0200, Pierre Habouzit wrote:\n> Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> did, since git in next cannot fetch on a regular basis for me. The\n> culprit seems to be commit 92392b4:\n> \n> ┌─(1:11)──<~/dev/scm/git 92392b4...>──\n> └[artemis] git fetch\n> remote: Counting objects: 461, done.\n> remote: Compressing objects: 100% (141/141), done.\n> remote: Total 263 (delta 227), reused 155 (delta 121)\n> Receiving objects: 100% (263/263), 95.55 KiB, done.\n> fatal: Out of memory, malloc failed\n> fatal: index-pack failed\n> [2] 16674 abort (core dumped) git fetch\n> \n> ┌─(1:12)──<~/dev/scm/git 92392b4...>──\n> └[artemis] git checkout -m HEAD~1; make git-index-pack\n> Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n> HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n> GIT_VERSION = 1.5.6.3.3.g03993\n> \tCC index-pack.o\n> \tLINK git-index-pack\n> \n> ┌─(1:12)──<~/dev/scm/git 03993e1...>──\n> └[artemis] git fetch\n> remote: Counting objects: 461, done.\n> remote: Compressing objects: 100% (141/141), done.\n> remote: Total 263 (delta 227), reused 155 (delta 121)\n> Receiving objects: 100% (263/263), 95.55 KiB, done.\n> Resolving deltas: 100% (227/227), completed with 153 local objects.\n> From git://git.kernel.org/pub/scm/git/git\n> 5ba2c22..0868a30 html -> origin/html\n> 2857e17..abeeabe man -> origin/man\n> 93310a4..95f8ebb master -> origin/master\n> 559998f..e8bf351 next -> origin/next\n> \n> You can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\n> broken, I've absolutely no clue about what happens.\n> \n> All I can say is that at some point in get_data_from_pack, obj[1].idx\n> points to something that is *not* a sha so it's probably corrupted.\n> (from index-pack.c).\n\nHere's how to reproduce:\n\n#!/bin/bash\n\n[ -d git-bug ] || \\\n\tgit clone git://git.kernel.org/pub/scm/git/git git-bug\ncd git-bug\n\ngit update-ref refs/remotes/origin/html 5ba2c22\ngit update-ref refs/remotes/origin/man 2857e17\ngit update-ref refs/remotes/origin/maint 2d9c572\ngit update-ref refs/remotes/origin/master 93310a4\ngit update-ref refs/remotes/origin/next 559998f\ngit update-ref refs/remotes/origin/pu 010581c8\n\ngit reset --hard origin/master\n\nsleep 1\n\ngit reflog expire --expire=0 --all\n\ngit repack -A -d -f --depth=10 --window=10\ngit prune\n\ngit config core.deltaBaseCacheLimit 100\n\ngit fetch\n\n\n\nThe values for html, man, master and next are taken from Pierre's\noutput, those for maint and pu are from a repo that works for\nreproducing the bug, just in case that future output of those branches\nmanage to break the reproducability.\n\nMight be, that any fetch will fail if deltaBaseCacheLimit is low enough,\nbut I'm too lazy to test that as well, now that I realized it. My head\nseems not to be working correctly anyway, can't even manage to get a\ncore dump... Stupid flu...\n\nHTH\nBjörn\n"},{"id":"84460","messageId":"20080723102239.GC15243@artemis.madism.org","threadId":"14606","inReplyTo":"20080723101415.GA23769@atjola.homenet","subject":"Re: regression in 92392b4","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-23T10:22:39Z","receivedAt":"2008-07-23T10:22:39Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jul 23, 2008 at 10:14:15AM +0000, Björn Steinbrink wrote:\n> On 2008.07.23 01:17:45 +0200, Pierre Habouzit wrote:\n> > Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> > did, since git in next cannot fetch on a regular basis for me. The\n> > culprit seems to be commit 92392b4:\n> > \n> > ┌─(1:11)──<~/dev/scm/git 92392b4...>──\n> > └[artemis] git fetch\n> > remote: Counting objects: 461, done.\n> > remote: Compressing objects: 100% (141/141), done.\n> > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > fatal: Out of memory, malloc failed\n> > fatal: index-pack failed\n> > [2] 16674 abort (core dumped) git fetch\n> > \n> > ┌─(1:12)──<~/dev/scm/git 92392b4...>──\n> > └[artemis] git checkout -m HEAD~1; make git-index-pack\n> > Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n> > HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n> > GIT_VERSION = 1.5.6.3.3.g03993\n> > \tCC index-pack.o\n> > \tLINK git-index-pack\n> > \n> > ┌─(1:12)──<~/dev/scm/git 03993e1...>──\n> > └[artemis] git fetch\n> > remote: Counting objects: 461, done.\n> > remote: Compressing objects: 100% (141/141), done.\n> > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > Resolving deltas: 100% (227/227), completed with 153 local objects.\n> > From git://git.kernel.org/pub/scm/git/git\n> > 5ba2c22..0868a30 html -> origin/html\n> > 2857e17..abeeabe man -> origin/man\n> > 93310a4..95f8ebb master -> origin/master\n> > 559998f..e8bf351 next -> origin/next\n> > \n> > You can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\n> > broken, I've absolutely no clue about what happens.\n> > \n> > All I can say is that at some point in get_data_from_pack, obj[1].idx\n> > points to something that is *not* a sha so it's probably corrupted.\n> > (from index-pack.c).\n> \n> Here's how to reproduce:\n> \n> #!/bin/bash\n> \n> [ -d git-bug ] || \\\n> \tgit clone git://git.kernel.org/pub/scm/git/git git-bug\n> cd git-bug\n> \n> git update-ref refs/remotes/origin/html 5ba2c22\n> git update-ref refs/remotes/origin/man 2857e17\n> git update-ref refs/remotes/origin/maint 2d9c572\n> git update-ref refs/remotes/origin/master 93310a4\n> git update-ref refs/remotes/origin/next 559998f\n> git update-ref refs/remotes/origin/pu 010581c8\n> \n> git reset --hard origin/master\n> \n> sleep 1\n> \n> git reflog expire --expire=0 --all\n> \n> git repack -A -d -f --depth=10 --window=10\n> git prune\n> \n> git config core.deltaBaseCacheLimit 100\n> \n> git fetch\n\n *THANKS* I was missing the \"git config core.deltaBaseCacheLimit\". I\ndid the same as you but it wasn't failing here. FWIW I don't have\ndeltaBaseCacheLimit set in my config, but oh well.\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"84462","messageId":"20080723103820.GE15243@artemis.madism.org","threadId":"14606","inReplyTo":"20080723101415.GA23769@atjola.homenet","subject":"Re: regression in 92392b4","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-23T10:38:20Z","receivedAt":"2008-07-23T10:38:20Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jul 23, 2008 at 10:14:15AM +0000, Björn Steinbrink wrote:\n> Here's how to reproduce:\n> \n> #!/bin/bash\n> \n> [ -d git-bug ] || \\\n> \tgit clone git://git.kernel.org/pub/scm/git/git git-bug\n> cd git-bug\n> \n> git update-ref refs/remotes/origin/html 5ba2c22\n> git update-ref refs/remotes/origin/man 2857e17\n> git update-ref refs/remotes/origin/maint 2d9c572\n> git update-ref refs/remotes/origin/master 93310a4\n> git update-ref refs/remotes/origin/next 559998f\n> git update-ref refs/remotes/origin/pu 010581c8\n> \n> git reset --hard origin/master\n> \n> sleep 1\n> \n> git reflog expire --expire=0 --all\n> \n> git repack -A -d -f --depth=10 --window=10\n> git prune\n> \n> git config core.deltaBaseCacheLimit 100\n> \n> git fetch\n\n interestingly, replacing git-index-pack with:\n\n#!/bin/sh\nexec valgrind --log-file=trace.$$ $(dirname $0)/git-index-pack2 \"$@\"\n\n it yields a different error in the console:\n remote: Counting objects: 461, done.\n remote: Compressing objects: 100% (141/141), done.\n remote: Total 263 (delta 227), reused 155 (delta 121)\n Receiving objects: 100% (263/263), 95.55 KiB | 71 KiB/s, done.\n fatal: serious inflate inconsistency\n fatal: index-pack failed\n git fetch 2,63s user 0,08s system 51% cpu 5,275 total\n\nthe valgrind log is attached, and my index-pack.c as well since I did\nthe c->data = NULL change which may mess up line counts.\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n\n\n#include \"cache.h\"\n#include \"delta.h\"\n#include \"pack.h\"\n#include \"csum-file.h\"\n#include \"blob.h\"\n#include \"commit.h\"\n#include \"tag.h\"\n#include \"tree.h\"\n#include \"progress.h\"\n#include \"fsck.h\"\n\nstatic const char index_pack_usage[] =\n\"git index-pack [-v] [-o ] [{ ---keep | --keep= }] [--strict] { | --stdin [--fix-thin] [] }\";\n\nstruct object_entry\n{\n\tstruct pack_idx_entry idx;\n\tunsigned long size;\n\tunsigned int hdr_size;\n\tenum object_type type;\n\tenum object_type real_type;\n};\n\nunion delta_base {\n\tunsigned char sha1[20];\n\toff_t offset;\n};\n\nstruct base_data {\n\tstruct base_data *base;\n\tstruct base_data *child;\n\tstruct object_entry *obj;\n\tvoid *data;\n\tunsigned long size;\n};\n\n/*\n * Even if sizeof(union delta_base) == 24 on 64-bit archs, we really want\n * to memcmp() only the first 20 bytes.\n */\n#define UNION_BASE_SZ\t20\n\n#define FLAG_LINK (1u<<20)\n#define FLAG_CHECKED (1u<<21)\n\nstruct delta_entry\n{\n\tunion delta_base base;\n\tint obj_no;\n};\n\nstatic struct object_entry *objects;\nstatic struct delta_entry *deltas;\nstatic struct base_data *base_cache;\nstatic size_t base_cache_used;\nstatic int nr_objects;\nstatic int nr_deltas;\nstatic int nr_resolved_deltas;\n\nstatic int from_stdin;\nstatic int strict;\nstatic int verbose;\n\nstatic struct progress *progress;\n\n/* We always read in 4kB chunks. */\nstatic unsigned char input_buffer[4096];\nstatic unsigned int input_offset, input_len;\nstatic off_t consumed_bytes;\nstatic SHA_CTX input_ctx;\nstatic uint32_t input_crc32;\nstatic int input_fd, output_fd, pack_fd;\n\nstatic int mark_link(struct object *obj, int type, void *data)\n{\n\tif (!obj)\n\t\treturn -1;\n\n\tif (type != OBJ_ANY && obj->type != type)\n\t\tdie(\"object type mismatch at %s\", sha1_to_hex(obj->sha1));\n\n\tobj->flags |= FLAG_LINK;\n\treturn 0;\n}\n\n/* The content of each linked object must have been checked\n or it must be already present in the object database */\nstatic void check_object(struct object *obj)\n{\n\tif (!obj)\n\t\treturn;\n\n\tif (!(obj->flags & FLAG_LINK))\n\t\treturn;\n\n\tif (!(obj->flags & FLAG_CHECKED)) {\n\t\tunsigned long size;\n\t\tint type = sha1_object_info(obj->sha1, &size);\n\t\tif (type != obj->type || type <= 0)\n\t\t\tdie(\"object of unexpected type\");\n\t\tobj->flags |= FLAG_CHECKED;\n\t\treturn;\n\t}\n}\n\nstatic void check_objects(void)\n{\n\tunsigned i, max;\n\n\tmax = get_max_object_index();\n\tfor (i = 0; i < max; i++)\n\t\tcheck_object(get_indexed_object(i));\n}\n\n\n/* Discard current buffer used content. */\nstatic void flush(void)\n{\n\tif (input_offset) {\n\t\tif (output_fd >= 0)\n\t\t\twrite_or_die(output_fd, input_buffer, input_offset);\n\t\tSHA1_Update(&input_ctx, input_buffer, input_offset);\n\t\tmemmove(input_buffer, input_buffer + input_offset, input_len);\n\t\tinput_offset = 0;\n\t}\n}\n\n/*\n * Make sure at least \"min\" bytes are available in the buffer, and\n * return the pointer to the buffer.\n */\nstatic void *fill(int min)\n{\n\tif (min <= input_len)\n\t\treturn input_buffer + input_offset;\n\tif (min > sizeof(input_buffer))\n\t\tdie(\"cannot fill %d bytes\", min);\n\tflush();\n\tdo {\n\t\tssize_t ret = xread(input_fd, input_buffer + input_len,\n\t\t\t\tsizeof(input_buffer) - input_len);\n\t\tif (ret <= 0) {\n\t\t\tif (!ret)\n\t\t\t\tdie(\"early EOF\");\n\t\t\tdie(\"read error on input: %s\", strerror(errno));\n\t\t}\n\t\tinput_len += ret;\n\t\tif (from_stdin)\n\t\t\tdisplay_throughput(progress, consumed_bytes + input_len);\n\t} while (input_len < min);\n\treturn input_buffer;\n}\n\nstatic void use(int bytes)\n{\n\tif (bytes > input_len)\n\t\tdie(\"used more bytes than were available\");\n\tinput_crc32 = crc32(input_crc32, input_buffer + input_offset, bytes);\n\tinput_len -= bytes;\n\tinput_offset += bytes;\n\n\t/* make sure off_t is sufficiently large not to wrap */\n\tif (consumed_bytes > consumed_bytes + bytes)\n\t\tdie(\"pack too large for current definition of off_t\");\n\tconsumed_bytes += bytes;\n}\n\nstatic char *open_pack_file(char *pack_name)\n{\n\tif (from_stdin) {\n\t\tinput_fd = 0;\n\t\tif (!pack_name) {\n\t\t\tstatic char tmpfile[PATH_MAX];\n\t\t\tsnprintf(tmpfile, sizeof(tmpfile),\n\t\t\t\t \"%s/tmp_pack_XXXXXX\", get_object_directory());\n\t\t\toutput_fd = xmkstemp(tmpfile);\n\t\t\tpack_name = xstrdup(tmpfile);\n\t\t} else\n\t\t\toutput_fd = open(pack_name, O_CREAT|O_EXCL|O_RDWR, 0600);\n\t\tif (output_fd < 0)\n\t\t\tdie(\"unable to create %s: %s\\n\", pack_name, strerror(errno));\n\t\tpack_fd = output_fd;\n\t} else {\n\t\tinput_fd = open(pack_name, O_RDONLY);\n\t\tif (input_fd < 0)\n\t\t\tdie(\"cannot open packfile '%s': %s\",\n\t\t\t pack_name, strerror(errno));\n\t\toutput_fd = -1;\n\t\tpack_fd = input_fd;\n\t}\n\tSHA1_Init(&input_ctx);\n\treturn pack_name;\n}\n\nstatic void parse_pack_header(void)\n{\n\tstruct pack_header *hdr = fill(sizeof(struct pack_header));\n\n\t/* Header consistency check */\n\tif (hdr->hdr_signature != htonl(PACK_SIGNATURE))\n\t\tdie(\"pack signature mismatch\");\n\tif (!pack_version_ok(hdr->hdr_version))\n\t\tdie(\"pack version %\"PRIu32\" unsupported\",\n\t\t\tntohl(hdr->hdr_version));\n\n\tnr_objects = ntohl(hdr->hdr_entries);\n\tuse(sizeof(struct pack_header));\n}\n\nstatic void bad_object(unsigned long offset, const char *format,\n\t\t ...) NORETURN __attribute__((format (printf, 2, 3)));\n\nstatic void bad_object(unsigned long offset, const char *format, ...)\n{\n\tva_list params;\n\tchar buf[1024];\n\n\tva_start(params, format);\n\tvsnprintf(buf, sizeof(buf), format, params);\n\tva_end(params);\n\tdie(\"pack has bad object at offset %lu: %s\", offset, buf);\n}\n\nstatic void prune_base_data(struct base_data *retain)\n{\n\tstruct base_data *b = base_cache;\n\tfor (b = base_cache;\n\t base_cache_used > delta_base_cache_limit && b;\n\t b = b->child) {\n\t\tif (b->data && b != retain) {\n\t\t\tfree(b->data);\n\t\t\tb->data = NULL;\n\t\t\tbase_cache_used -= b->size;\n\t\t}\n\t}\n}\n\nstatic void link_base_data(struct base_data *base, struct base_data *c)\n{\n\tif (base)\n\t\tbase->child = c;\n\telse\n\t\tbase_cache = c;\n\n\tc->base = base;\n\tc->child = NULL;\n\tbase_cache_used += c->size;\n\tprune_base_data(c);\n}\n\nstatic void unlink_base_data(struct base_data *c)\n{\n\tstruct base_data *base = c->base;\n\tif (base)\n\t\tbase->child = NULL;\n\telse\n\t\tbase_cache = NULL;\n\tif (c->data) {\n\t\tfree(c->data);\n\t\tc->data = NULL;\n\t\tbase_cache_used -= c->size;\n\t}\n}\n\nstatic void *unpack_entry_data(unsigned long offset, unsigned long size)\n{\n\tz_stream stream;\n\tvoid *buf = xmalloc(size);\n\n\tmemset(&stream, 0, sizeof(stream));\n\tstream.next_out = buf;\n\tstream.avail_out = size;\n\tstream.next_in = fill(1);\n\tstream.avail_in = input_len;\n\tinflateInit(&stream);\n\n\tfor (;;) {\n\t\tint ret = inflate(&stream, 0);\n\t\tuse(input_len - stream.avail_in);\n\t\tif (stream.total_out == size && ret == Z_STREAM_END)\n\t\t\tbreak;\n\t\tif (ret != Z_OK)\n\t\t\tbad_object(offset, \"inflate returned %d\", ret);\n\t\tstream.next_in = fill(1);\n\t\tstream.avail_in = input_len;\n\t}\n\tinflateEnd(&stream);\n\treturn buf;\n}\n\nstatic void *unpack_raw_entry(struct object_entry *obj, union delta_base *delta_base)\n{\n\tunsigned char *p, c;\n\tunsigned long size;\n\toff_t base_offset;\n\tunsigned shift;\n\tvoid *data;\n\n\tobj->idx.offset = consumed_bytes;\n\tinput_crc32 = crc32(0, Z_NULL, 0);\n\n\tp = fill(1);\n\tc = *p;\n\tuse(1);\n\tobj->type = (c >> 4) & 7;\n\tsize = (c & 15);\n\tshift = 4;\n\twhile (c & 0x80) {\n\t\tp = fill(1);\n\t\tc = *p;\n\t\tuse(1);\n\t\tsize += (c & 0x7fUL) << shift;\n\t\tshift += 7;\n\t}\n\tobj->size = size;\n\n\tswitch (obj->type) {\n\tcase OBJ_REF_DELTA:\n\t\thashcpy(delta_base->sha1, fill(20));\n\t\tuse(20);\n\t\tbreak;\n\tcase OBJ_OFS_DELTA:\n\t\tmemset(delta_base, 0, sizeof(*delta_base));\n\t\tp = fill(1);\n\t\tc = *p;\n\t\tuse(1);\n\t\tbase_offset = c & 127;\n\t\twhile (c & 128) {\n\t\t\tbase_offset += 1;\n\t\t\tif (!base_offset || MSB(base_offset, 7))\n\t\t\t\tbad_object(obj->idx.offset, \"offset value overflow for delta base object\");\n\t\t\tp = fill(1);\n\t\t\tc = *p;\n\t\t\tuse(1);\n\t\t\tbase_offset = (base_offset << 7) + (c & 127);\n\t\t}\n\t\tdelta_base->offset = obj->idx.offset - base_offset;\n\t\tif (delta_base->offset >= obj->idx.offset)\n\t\t\tbad_object(obj->idx.offset, \"delta base offset is out of bound\");\n\t\tbreak;\n\tcase OBJ_COMMIT:\n\tcase OBJ_TREE:\n\tcase OBJ_BLOB:\n\tcase OBJ_TAG:\n\t\tbreak;\n\tdefault:\n\t\tbad_object(obj->idx.offset, \"unknown object type %d\", obj->type);\n\t}\n\tobj->hdr_size = consumed_bytes - obj->idx.offset;\n\n\tdata = unpack_entry_data(obj->idx.offset, obj->size);\n\tobj->idx.crc32 = input_crc32;\n\treturn data;\n}\n\nstatic void *get_data_from_pack(struct object_entry *obj)\n{\n\toff_t from = obj[0].idx.offset + obj[0].hdr_size;\n\tunsigned long len = obj[1].idx.offset - from;\n\tunsigned long rdy = 0;\n\tunsigned char *src, *data;\n\tz_stream stream;\n\tint st;\n\n\tsrc = xmalloc(len);\n\tdata = src;\n\tdo {\n\t\tssize_t n = pread(pack_fd, data + rdy, len - rdy, from + rdy);\n\t\tif (n <= 0)\n\t\t\tdie(\"cannot pread pack file: %s\", strerror(errno));\n\t\trdy += n;\n\t} while (rdy < len);\n\tdata = xmalloc(obj->size);\n\tmemset(&stream, 0, sizeof(stream));\n\tstream.next_out = data;\n\tstream.avail_out = obj->size;\n\tstream.next_in = src;\n\tstream.avail_in = len;\n\tinflateInit(&stream);\n\twhile ((st = inflate(&stream, Z_FINISH)) == Z_OK);\n\tinflateEnd(&stream);\n\tif (st != Z_STREAM_END || stream.total_out != obj->size)\n\t\tdie(\"serious inflate inconsistency\");\n\tfree(src);\n\treturn data;\n}\n\nstatic int find_delta(const union delta_base *base)\n{\n\tint first = 0, last = nr_deltas;\n\n while (first < last) {\n int next = (first + last) / 2;\n struct delta_entry *delta = &deltas[next];\n int cmp;\n\n cmp = memcmp(base, &delta->base, UNION_BASE_SZ);\n if (!cmp)\n return next;\n if (cmp < 0) {\n last = next;\n continue;\n }\n first = next+1;\n }\n return -first-1;\n}\n\nstatic int find_delta_children(const union delta_base *base,\n\t\t\t int *first_index, int *last_index)\n{\n\tint first = find_delta(base);\n\tint last = first;\n\tint end = nr_deltas - 1;\n\n\tif (first < 0)\n\t\treturn -1;\n\twhile (first > 0 && !memcmp(&deltas[first - 1].base, base, UNION_BASE_SZ))\n\t\t--first;\n\twhile (last < end && !memcmp(&deltas[last + 1].base, base, UNION_BASE_SZ))\n\t\t++last;\n\t*first_index = first;\n\t*last_index = last;\n\treturn 0;\n}\n\nstatic void sha1_object(const void *data, unsigned long size,\n\t\t\tenum object_type type, unsigned char *sha1)\n{\n\thash_sha1_file(data, size, typename(type), sha1);\n\tif (has_sha1_file(sha1)) {\n\t\tvoid *has_data;\n\t\tenum object_type has_type;\n\t\tunsigned long has_size;\n\t\thas_data = read_sha1_file(sha1, &has_type, &has_size);\n\t\tif (!has_data)\n\t\t\tdie(\"cannot read existing object %s\", sha1_to_hex(sha1));\n\t\tif (size != has_size || type != has_type ||\n\t\t memcmp(data, has_data, size) != 0)\n\t\t\tdie(\"SHA1 COLLISION FOUND WITH %s !\", sha1_to_hex(sha1));\n\t\tfree(has_data);\n\t}\n\tif (strict) {\n\t\tif (type == OBJ_BLOB) {\n\t\t\tstruct blob *blob = lookup_blob(sha1);\n\t\t\tif (blob)\n\t\t\t\tblob->object.flags |= FLAG_CHECKED;\n\t\t\telse\n\t\t\t\tdie(\"invalid blob object %s\", sha1_to_hex(sha1));\n\t\t} else {\n\t\t\tstruct object *obj;\n\t\t\tint eaten;\n\t\t\tvoid *buf = (void *) data;\n\n\t\t\t/*\n\t\t\t * we do not need to free the memory here, as the\n\t\t\t * buf is deleted by the caller.\n\t\t\t */\n\t\t\tobj = parse_object_buffer(sha1, type, size, buf, &eaten);\n\t\t\tif (!obj)\n\t\t\t\tdie(\"invalid %s\", typename(type));\n\t\t\tif (fsck_object(obj, 1, fsck_error_function))\n\t\t\t\tdie(\"Error in object\");\n\t\t\tif (fsck_walk(obj, mark_link, 0))\n\t\t\t\tdie(\"Not all child objects of %s are reachable\", sha1_to_hex(obj->sha1));\n\n\t\t\tif (obj->type == OBJ_TREE) {\n\t\t\t\tstruct tree *item = (struct tree *) obj;\n\t\t\t\titem->buffer = NULL;\n\t\t\t}\n\t\t\tif (obj->type == OBJ_COMMIT) {\n\t\t\t\tstruct commit *commit = (struct commit *) obj;\n\t\t\t\tcommit->buffer = NULL;\n\t\t\t}\n\t\t\tobj->flags |= FLAG_CHECKED;\n\t\t}\n\t}\n}\n\nstatic void *get_base_data(struct base_data *c)\n{\n\tif (!c->data) {\n\t\tstruct object_entry *obj = c->obj;\n\n\t\tif (obj->type == OBJ_REF_DELTA || obj->type == OBJ_OFS_DELTA) {\n\t\t\tvoid *base = get_base_data(c->base);\n\t\t\tvoid *raw = get_data_from_pack(obj);\n\t\t\tc->data = patch_delta(\n\t\t\t\tbase, c->base->size,\n\t\t\t\traw, obj->size,\n\t\t\t\t&c->size);\n\t\t\tfree(raw);\n\t\t\tif (!c->data)\n\t\t\t\tbad_object(obj->idx.offset, \"failed to apply delta\");\n\t\t} else\n\t\t\tc->data = get_data_from_pack(obj);\n\n\t\tbase_cache_used += c->size;\n\t\tprune_base_data(c);\n\t}\n\treturn c->data;\n}\n\nstatic void resolve_delta(struct object_entry *delta_obj,\n\t\t\t struct base_data *base_obj, enum object_type type)\n{\n\tvoid *delta_data;\n\tunsigned long delta_size;\n\tunion delta_base delta_base;\n\tint j, first, last;\n\tstruct base_data result;\n\n\tdelta_obj->real_type = type;\n\tdelta_data = get_data_from_pack(delta_obj);\n\tdelta_size = delta_obj->size;\n\tresult.data = patch_delta(get_base_data(base_obj), base_obj->size,\n\t\t\t delta_data, delta_size,\n\t\t\t &result.size);\n\tfree(delta_data);\n\tif (!result.data)\n\t\tbad_object(delta_obj->idx.offset, \"failed to apply delta\");\n\tsha1_object(result.data, result.size, type, delta_obj->idx.sha1);\n\tnr_resolved_deltas++;\n\n\tresult.obj = delta_obj;\n\tlink_base_data(base_obj, &result);\n\n\thashcpy(delta_base.sha1, delta_obj->idx.sha1);\n\tif (!find_delta_children(&delta_base, &first, &last)) {\n\t\tfor (j = first; j <= last; j++) {\n\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n\t\t\tif (child->real_type == OBJ_REF_DELTA)\n\t\t\t\tresolve_delta(child, &result, type);\n\t\t}\n\t}\n\n\tmemset(&delta_base, 0, sizeof(delta_base));\n\tdelta_base.offset = delta_obj->idx.offset;\n\tif (!find_delta_children(&delta_base, &first, &last)) {\n\t\tfor (j = first; j <= last; j++) {\n\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n\t\t\tif (child->real_type == OBJ_OFS_DELTA)\n\t\t\t\tresolve_delta(child, &result, type);\n\t\t}\n\t}\n\n\tunlink_base_data(&result);\n}\n\nstatic int compare_delta_entry(const void *a, const void *b)\n{\n\tconst struct delta_entry *delta_a = a;\n\tconst struct delta_entry *delta_b = b;\n\treturn memcmp(&delta_a->base, &delta_b->base, UNION_BASE_SZ);\n}\n\n/* Parse all objects and return the pack content SHA1 hash */\nstatic void parse_pack_objects(unsigned char *sha1)\n{\n\tint i;\n\tstruct delta_entry *delta = deltas;\n\tstruct stat st;\n\n\t/*\n\t * First pass:\n\t * - find locations of all objects;\n\t * - calculate SHA1 of all non-delta objects;\n\t * - remember base (SHA1 or offset) for all deltas.\n\t */\n\tif (verbose)\n\t\tprogress = start_progress(\n\t\t\t\tfrom_stdin ? \"Receiving objects\" : \"Indexing objects\",\n\t\t\t\tnr_objects);\n\tfor (i = 0; i < nr_objects; i++) {\n\t\tstruct object_entry *obj = &objects[i];\n\t\tvoid *data = unpack_raw_entry(obj, &delta->base);\n\t\tobj->real_type = obj->type;\n\t\tif (obj->type == OBJ_REF_DELTA || obj->type == OBJ_OFS_DELTA) {\n\t\t\tnr_deltas++;\n\t\t\tdelta->obj_no = i;\n\t\t\tdelta++;\n\t\t} else\n\t\t\tsha1_object(data, obj->size, obj->type, obj->idx.sha1);\n\t\tfree(data);\n\t\tdisplay_progress(progress, i+1);\n\t}\n\tobjects[i].idx.offset = consumed_bytes;\n\tstop_progress(&progress);\n\n\t/* Check pack integrity */\n\tflush();\n\tSHA1_Final(sha1, &input_ctx);\n\tif (hashcmp(fill(20), sha1))\n\t\tdie(\"pack is corrupted (SHA1 mismatch)\");\n\tuse(20);\n\n\t/* If input_fd is a file, we should have reached its end now. */\n\tif (fstat(input_fd, &st))\n\t\tdie(\"cannot fstat packfile: %s\", strerror(errno));\n\tif (S_ISREG(st.st_mode) &&\n\t\t\tlseek(input_fd, 0, SEEK_CUR) - input_len != st.st_size)\n\t\tdie(\"pack has junk at the end\");\n\n\tif (!nr_deltas)\n\t\treturn;\n\n\t/* Sort deltas by base SHA1/offset for fast searching */\n\tqsort(deltas, nr_deltas, sizeof(struct delta_entry),\n\t compare_delta_entry);\n\n\t/*\n\t * Second pass:\n\t * - for all non-delta objects, look if it is used as a base for\n\t * deltas;\n\t * - if used as a base, uncompress the object and apply all deltas,\n\t * recursively checking if the resulting object is used as a base\n\t * for some more deltas.\n\t */\n\tif (verbose)\n\t\tprogress = start_progress(\"Resolving deltas\", nr_deltas);\n\tfor (i = 0; i < nr_objects; i++) {\n\t\tstruct object_entry *obj = &objects[i];\n\t\tunion delta_base base;\n\t\tint j, ref, ref_first, ref_last, ofs, ofs_first, ofs_last;\n\t\tstruct base_data base_obj;\n\n\t\tif (obj->type == OBJ_REF_DELTA || obj->type == OBJ_OFS_DELTA)\n\t\t\tcontinue;\n\t\thashcpy(base.sha1, obj->idx.sha1);\n\t\tref = !find_delta_children(&base, &ref_first, &ref_last);\n\t\tmemset(&base, 0, sizeof(base));\n\t\tbase.offset = obj->idx.offset;\n\t\tofs = !find_delta_children(&base, &ofs_first, &ofs_last);\n\t\tif (!ref && !ofs)\n\t\t\tcontinue;\n\t\tbase_obj.data = get_data_from_pack(obj);\n\t\tbase_obj.size = obj->size;\n\t\tbase_obj.obj = obj;\n\t\tlink_base_data(NULL, &base_obj);\n\n\t\tif (ref)\n\t\t\tfor (j = ref_first; j <= ref_last; j++) {\n\t\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n\t\t\t\tif (child->real_type == OBJ_REF_DELTA)\n\t\t\t\t\tresolve_delta(child, &base_obj, obj->type);\n\t\t\t}\n\t\tif (ofs)\n\t\t\tfor (j = ofs_first; j <= ofs_last; j++) {\n\t\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n\t\t\t\tif (child->real_type == OBJ_OFS_DELTA)\n\t\t\t\t\tresolve_delta(child, &base_obj, obj->type);\n\t\t\t}\n\t\tunlink_base_data(&base_obj);\n\t\tdisplay_progress(progress, nr_resolved_deltas);\n\t}\n}\n\nstatic int write_compressed(int fd, void *in, unsigned int size, uint32_t *obj_crc)\n{\n\tz_stream stream;\n\tunsigned long maxsize;\n\tvoid *out;\n\n\tmemset(&stream, 0, sizeof(stream));\n\tdeflateInit(&stream, zlib_compression_level);\n\tmaxsize = deflateBound(&stream, size);\n\tout = xmalloc(maxsize);\n\n\t/* Compress it */\n\tstream.next_in = in;\n\tstream.avail_in = size;\n\tstream.next_out = out;\n\tstream.avail_out = maxsize;\n\twhile (deflate(&stream, Z_FINISH) == Z_OK);\n\tdeflateEnd(&stream);\n\n\tsize = stream.total_out;\n\twrite_or_die(fd, out, size);\n\t*obj_crc = crc32(*obj_crc, out, size);\n\tfree(out);\n\treturn size;\n}\n\nstatic struct object_entry *append_obj_to_pack(\n\t\t\t const unsigned char *sha1, void *buf,\n\t\t\t unsigned long size, enum object_type type)\n{\n\tstruct object_entry *obj = &objects[nr_objects++];\n\tunsigned char header[10];\n\tunsigned long s = size;\n\tint n = 0;\n\tunsigned char c = (type << 4) | (s & 15);\n\ts >>= 4;\n\twhile (s) {\n\t\theader[n++] = c | 0x80;\n\t\tc = s & 0x7f;\n\t\ts >>= 7;\n\t}\n\theader[n++] = c;\n\twrite_or_die(output_fd, header, n);\n\tobj[0].idx.crc32 = crc32(0, Z_NULL, 0);\n\tobj[0].idx.crc32 = crc32(obj[0].idx.crc32, header, n);\n\tobj[1].idx.offset = obj[0].idx.offset + n;\n\tobj[1].idx.offset += write_compressed(output_fd, buf, size, &obj[0].idx.crc32);\n\thashcpy(obj->idx.sha1, sha1);\n\treturn obj;\n}\n\nstatic int delta_pos_compare(const void *_a, const void *_b)\n{\n\tstruct delta_entry *a = *(struct delta_entry **)_a;\n\tstruct delta_entry *b = *(struct delta_entry **)_b;\n\treturn a->obj_no - b->obj_no;\n}\n\nstatic void fix_unresolved_deltas(int nr_unresolved)\n{\n\tstruct delta_entry **sorted_by_pos;\n\tint i, n = 0;\n\n\t/*\n\t * Since many unresolved deltas may well be themselves base objects\n\t * for more unresolved deltas, we really want to include the\n\t * smallest number of base objects that would cover as much delta\n\t * as possible by picking the\n\t * trunc deltas first, allowing for other deltas to resolve without\n\t * additional base objects. Since most base objects are to be found\n\t * before deltas depending on them, a good heuristic is to start\n\t * resolving deltas in the same order as their position in the pack.\n\t */\n\tsorted_by_pos = xmalloc(nr_unresolved * sizeof(*sorted_by_pos));\n\tfor (i = 0; i < nr_deltas; i++) {\n\t\tif (objects[deltas[i].obj_no].real_type != OBJ_REF_DELTA)\n\t\t\tcontinue;\n\t\tsorted_by_pos[n++] = &deltas[i];\n\t}\n\tqsort(sorted_by_pos, n, sizeof(*sorted_by_pos), delta_pos_compare);\n\n\tfor (i = 0; i < n; i++) {\n\t\tstruct delta_entry *d = sorted_by_pos[i];\n\t\tenum object_type type;\n\t\tint j, first, last;\n\t\tstruct base_data base_obj;\n\n\t\tif (objects[d->obj_no].real_type != OBJ_REF_DELTA)\n\t\t\tcontinue;\n\t\tbase_obj.data = read_sha1_file(d->base.sha1, &type, &base_obj.size);\n\t\tif (!base_obj.data)\n\t\t\tcontinue;\n\n\t\tif (check_sha1_signature(d->base.sha1, base_obj.data,\n\t\t\t\tbase_obj.size, typename(type)))\n\t\t\tdie(\"local object %s is corrupt\", sha1_to_hex(d->base.sha1));\n\t\tbase_obj.obj = append_obj_to_pack(d->base.sha1, base_obj.data,\n\t\t\tbase_obj.size, type);\n\t\tlink_base_data(NULL, &base_obj);\n\n\t\tfind_delta_children(&d->base, &first, &last);\n\t\tfor (j = first; j <= last; j++) {\n\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n\t\t\tif (child->real_type == OBJ_REF_DELTA)\n\t\t\t\tresolve_delta(child, &base_obj, type);\n\t\t}\n\n\t\tunlink_base_data(&base_obj);\n\t\tdisplay_progress(progress, nr_resolved_deltas);\n\t}\n\tfree(sorted_by_pos);\n}\n\nstatic void final(const char *final_pack_name, const char *curr_pack_name,\n\t\t const char *final_index_name, const char *curr_index_name,\n\t\t const char *keep_name, const char *keep_msg,\n\t\t unsigned char *sha1)\n{\n\tconst char *report = \"pack\";\n\tchar name[PATH_MAX];\n\tint err;\n\n\tif (!from_stdin) {\n\t\tclose(input_fd);\n\t} else {\n\t\tfsync_or_die(output_fd, curr_pack_name);\n\t\terr = close(output_fd);\n\t\tif (err)\n\t\t\tdie(\"error while closing pack file: %s\", strerror(errno));\n\t\tchmod(curr_pack_name, 0444);\n\t}\n\n\tif (keep_msg) {\n\t\tint keep_fd, keep_msg_len = strlen(keep_msg);\n\t\tif (!keep_name) {\n\t\t\tsnprintf(name, sizeof(name), \"%s/pack/pack-%s.keep\",\n\t\t\t\t get_object_directory(), sha1_to_hex(sha1));\n\t\t\tkeep_name = name;\n\t\t}\n\t\tkeep_fd = open(keep_name, O_RDWR|O_CREAT|O_EXCL, 0600);\n\t\tif (keep_fd < 0) {\n\t\t\tif (errno != EEXIST)\n\t\t\t\tdie(\"cannot write keep file\");\n\t\t} else {\n\t\t\tif (keep_msg_len > 0) {\n\t\t\t\twrite_or_die(keep_fd, keep_msg, keep_msg_len);\n\t\t\t\twrite_or_die(keep_fd, \"\\n\", 1);\n\t\t\t}\n\t\t\tif (close(keep_fd) != 0)\n\t\t\t\tdie(\"cannot write keep file\");\n\t\t\treport = \"keep\";\n\t\t}\n\t}\n\n\tif (final_pack_name != curr_pack_name) {\n\t\tif (!final_pack_name) {\n\t\t\tsnprintf(name, sizeof(name), \"%s/pack/pack-%s.pack\",\n\t\t\t\t get_object_directory(), sha1_to_hex(sha1));\n\t\t\tfinal_pack_name = name;\n\t\t}\n\t\tif (move_temp_to_file(curr_pack_name, final_pack_name))\n\t\t\tdie(\"cannot store pack file\");\n\t}\n\n\tchmod(curr_index_name, 0444);\n\tif (final_index_name != curr_index_name) {\n\t\tif (!final_index_name) {\n\t\t\tsnprintf(name, sizeof(name), \"%s/pack/pack-%s.idx\",\n\t\t\t\t get_object_directory(), sha1_to_hex(sha1));\n\t\t\tfinal_index_name = name;\n\t\t}\n\t\tif (move_temp_to_file(curr_index_name, final_index_name))\n\t\t\tdie(\"cannot store index file\");\n\t}\n\n\tif (!from_stdin) {\n\t\tprintf(\"%s\\n\", sha1_to_hex(sha1));\n\t} else {\n\t\tchar buf[48];\n\t\tint len = snprintf(buf, sizeof(buf), \"%s\\t%s\\n\",\n\t\t\t\t report, sha1_to_hex(sha1));\n\t\twrite_or_die(1, buf, len);\n\n\t\t/*\n\t\t * Let's just mimic git-unpack-objects here and write\n\t\t * the last part of the input buffer to stdout.\n\t\t */\n\t\twhile (input_len) {\n\t\t\terr = xwrite(1, input_buffer + input_offset, input_len);\n\t\t\tif (err <= 0)\n\t\t\t\tbreak;\n\t\t\tinput_len -= err;\n\t\t\tinput_offset += err;\n\t\t}\n\t}\n}\n\nstatic int git_index_pack_config(const char *k, const char *v, void *cb)\n{\n\tif (!strcmp(k, \"pack.indexversion\")) {\n\t\tpack_idx_default_version = git_config_int(k, v);\n\t\tif (pack_idx_default_version > 2)\n\t\t\tdie(\"bad pack.indexversion=%\"PRIu32,\n\t\t\t\tpack_idx_default_version);\n\t\treturn 0;\n\t}\n\treturn git_default_config(k, v, cb);\n}\n\nint main(int argc, char **argv)\n{\n\tint i, fix_thin_pack = 0;\n\tchar *curr_pack, *pack_name = NULL;\n\tchar *curr_index, *index_name = NULL;\n\tconst char *keep_name = NULL, *keep_msg = NULL;\n\tchar *index_name_buf = NULL, *keep_name_buf = NULL;\n\tstruct pack_idx_entry **idx_objects;\n\tunsigned char sha1[20];\n\n\tgit_config(git_index_pack_config, NULL);\n\n\tfor (i = 1; i < argc; i++) {\n\t\tchar *arg = argv[i];\n\n\t\tif (*arg == '-') {\n\t\t\tif (!strcmp(arg, \"--stdin\")) {\n\t\t\t\tfrom_stdin = 1;\n\t\t\t} else if (!strcmp(arg, \"--fix-thin\")) {\n\t\t\t\tfix_thin_pack = 1;\n\t\t\t} else if (!strcmp(arg, \"--strict\")) {\n\t\t\t\tstrict = 1;\n\t\t\t} else if (!strcmp(arg, \"--keep\")) {\n\t\t\t\tkeep_msg = \"\";\n\t\t\t} else if (!prefixcmp(arg, \"--keep=\")) {\n\t\t\t\tkeep_msg = arg + 7;\n\t\t\t} else if (!prefixcmp(arg, \"--pack_header=\")) {\n\t\t\t\tstruct pack_header *hdr;\n\t\t\t\tchar *c;\n\n\t\t\t\thdr = (struct pack_header *)input_buffer;\n\t\t\t\thdr->hdr_signature = htonl(PACK_SIGNATURE);\n\t\t\t\thdr->hdr_version = htonl(strtoul(arg + 14, &c, 10));\n\t\t\t\tif (*c != ',')\n\t\t\t\t\tdie(\"bad %s\", arg);\n\t\t\t\thdr->hdr_entries = htonl(strtoul(c + 1, &c, 10));\n\t\t\t\tif (*c)\n\t\t\t\t\tdie(\"bad %s\", arg);\n\t\t\t\tinput_len = sizeof(*hdr);\n\t\t\t} else if (!strcmp(arg, \"-v\")) {\n\t\t\t\tverbose = 1;\n\t\t\t} else if (!strcmp(arg, \"-o\")) {\n\t\t\t\tif (index_name || (i+1) >= argc)\n\t\t\t\t\tusage(index_pack_usage);\n\t\t\t\tindex_name = argv[++i];\n\t\t\t} else if (!prefixcmp(arg, \"--index-version=\")) {\n\t\t\t\tchar *c;\n\t\t\t\tpack_idx_default_version = strtoul(arg + 16, &c, 10);\n\t\t\t\tif (pack_idx_default_version > 2)\n\t\t\t\t\tdie(\"bad %s\", arg);\n\t\t\t\tif (*c == ',')\n\t\t\t\t\tpack_idx_off32_limit = strtoul(c+1, &c, 0);\n\t\t\t\tif (*c || pack_idx_off32_limit & 0x80000000)\n\t\t\t\t\tdie(\"bad %s\", arg);\n\t\t\t} else\n\t\t\t\tusage(index_pack_usage);\n\t\t\tcontinue;\n\t\t}\n\n\t\tif (pack_name)\n\t\t\tusage(index_pack_usage);\n\t\tpack_name = arg;\n\t}\n\n\tif (!pack_name && !from_stdin)\n\t\tusage(index_pack_usage);\n\tif (fix_thin_pack && !from_stdin)\n\t\tdie(\"--fix-thin cannot be used without --stdin\");\n\tif (!index_name && pack_name) {\n\t\tint len = strlen(pack_name);\n\t\tif (!has_extension(pack_name, \".pack\"))\n\t\t\tdie(\"packfile name '%s' does not end with '.pack'\",\n\t\t\t pack_name);\n\t\tindex_name_buf = xmalloc(len);\n\t\tmemcpy(index_name_buf, pack_name, len - 5);\n\t\tstrcpy(index_name_buf + len - 5, \".idx\");\n\t\tindex_name = index_name_buf;\n\t}\n\tif (keep_msg && !keep_name && pack_name) {\n\t\tint len = strlen(pack_name);\n\t\tif (!has_extension(pack_name, \".pack\"))\n\t\t\tdie(\"packfile name '%s' does not end with '.pack'\",\n\t\t\t pack_name);\n\t\tkeep_name_buf = xmalloc(len);\n\t\tmemcpy(keep_name_buf, pack_name, len - 5);\n\t\tstrcpy(keep_name_buf + len - 5, \".keep\");\n\t\tkeep_name = keep_name_buf;\n\t}\n\n\tcurr_pack = open_pack_file(pack_name);\n\tparse_pack_header();\n\tobjects = xmalloc((nr_objects + 1) * sizeof(struct object_entry));\n\tdeltas = xmalloc(nr_objects * sizeof(struct delta_entry));\n\tparse_pack_objects(sha1);\n\tif (nr_deltas == nr_resolved_deltas) {\n\t\tstop_progress(&progress);\n\t\t/* Flush remaining pack final 20-byte SHA1. */\n\t\tflush();\n\t} else {\n\t\tif (fix_thin_pack) {\n\t\t\tchar msg[48];\n\t\t\tint nr_unresolved = nr_deltas - nr_resolved_deltas;\n\t\t\tint nr_objects_initial = nr_objects;\n\t\t\tif (nr_unresolved <= 0)\n\t\t\t\tdie(\"confusion beyond insanity\");\n\t\t\tobjects = xrealloc(objects,\n\t\t\t\t\t (nr_objects + nr_unresolved + 1)\n\t\t\t\t\t * sizeof(*objects));\n\t\t\tfix_unresolved_deltas(nr_unresolved);\n\t\t\tsprintf(msg, \"completed with %d local objects\",\n\t\t\t\tnr_objects - nr_objects_initial);\n\t\t\tstop_progress_msg(&progress, msg);\n\t\t\tfixup_pack_header_footer(output_fd, sha1,\n\t\t\t\t\t\t curr_pack, nr_objects);\n\t\t}\n\t\tif (nr_deltas != nr_resolved_deltas)\n\t\t\tdie(\"pack has %d unresolved deltas\",\n\t\t\t nr_deltas - nr_resolved_deltas);\n\t}\n\tfree(deltas);\n\tif (strict)\n\t\tcheck_objects();\n\n\tidx_objects = xmalloc((nr_objects) * sizeof(struct pack_idx_entry *));\n\tfor (i = 0; i < nr_objects; i++)\n\t\tidx_objects[i] = &objects[i].idx;\n\tcurr_index = write_idx_file(index_name, idx_objects, nr_objects, sha1);\n\tfree(idx_objects);\n\n\tfinal(pack_name, curr_pack,\n\t\tindex_name, curr_index,\n\t\tkeep_name, keep_msg,\n\t\tsha1);\n\tfree(objects);\n\tfree(index_name_buf);\n\tfree(keep_name_buf);\n\tif (pack_name == NULL)\n\t\tfree(curr_pack);\n\tif (index_name == NULL)\n\t\tfree(curr_index);\n\n\treturn 0;\n}\n\n\n==3429== Memcheck, a memory error detector.\n==3429== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.\n==3429== Using LibVEX rev 1854, a library for dynamic binary translation.\n==3429== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.\n==3429== Using valgrind-3.3.1-Debian, a dynamic binary instrumentation framework.\n==3429== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.\n==3429== For more details, rerun with: -v\n==3429== \n==3429== My PID = 3429, parent PID = 3421. Prog and args are:\n==3429== /home/madcoder/dev/scm/git/git-index-pack2\n==3429== --stdin\n==3429== -v\n==3429== --fix-thin\n==3429== --keep=fetch-pack 3421 on artemis\n==3429== --pack_header=2,263\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5068AF9: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5068B26: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5068B86: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5068B97: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5068BD9: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Use of uninitialised value of size 8\n==3429== at 0x5068C2A: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5068C8F: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5068C98: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Use of uninitialised value of size 8\n==3429== at 0x5068E21: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Use of uninitialised value of size 8\n==3429== at 0x5068E31: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x50689AC: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5066933: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x5068E57: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5068E64: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x5068E76: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x506B361: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x5068EAC: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Use of uninitialised value of size 8\n==3429== at 0x5069CDB: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506B7B0: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x5068EAC: (within /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x506728C: deflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x404BD9: main (index-pack.c:674)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x4035EE: get_base_data (index-pack.c:485)\n==3429== by 0x403687: get_base_data (index-pack.c:486)\n==3429== by 0x403711: resolve_delta (index-pack.c:516)\n==3429== by 0x4038A8: resolve_delta (index-pack.c:543)\n==3429== by 0x404CF5: main (index-pack.c:762)\n==3429== \n==3429== Syscall param pread64(count) contains uninitialised byte(s)\n==3429== at 0x5621353: __pread_nocancel (in /usr/lib/debug/libpthread-2.7.so)\n==3429== by 0x4034C9: get_data_from_pack (index-pack.c:368)\n==3429== by 0x4035FB: get_base_data (index-pack.c:496)\n==3429== by 0x403687: get_base_data (index-pack.c:486)\n==3429== by 0x403711: resolve_delta (index-pack.c:516)\n==3429== by 0x4038A8: resolve_delta (index-pack.c:543)\n==3429== by 0x404CF5: main (index-pack.c:762)\n==3429== \n==3429== Syscall param pread64(offset) contains uninitialised byte(s)\n==3429== at 0x5621353: __pread_nocancel (in /usr/lib/debug/libpthread-2.7.so)\n==3429== by 0x4034C9: get_data_from_pack (index-pack.c:368)\n==3429== by 0x4035FB: get_base_data (index-pack.c:496)\n==3429== by 0x403687: get_base_data (index-pack.c:486)\n==3429== by 0x403711: resolve_delta (index-pack.c:516)\n==3429== by 0x4038A8: resolve_delta (index-pack.c:543)\n==3429== by 0x404CF5: main (index-pack.c:762)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x4034D9: get_data_from_pack (index-pack.c:372)\n==3429== by 0x4035FB: get_base_data (index-pack.c:496)\n==3429== by 0x403687: get_base_data (index-pack.c:486)\n==3429== by 0x403711: resolve_delta (index-pack.c:516)\n==3429== by 0x4038A8: resolve_delta (index-pack.c:543)\n==3429== by 0x404CF5: main (index-pack.c:762)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x506D058: inflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x40352C: get_data_from_pack (index-pack.c:380)\n==3429== by 0x4035FB: get_base_data (index-pack.c:496)\n==3429== by 0x403687: get_base_data (index-pack.c:486)\n==3429== by 0x403711: resolve_delta (index-pack.c:516)\n==3429== by 0x4038A8: resolve_delta (index-pack.c:543)\n==3429== by 0x404CF5: main (index-pack.c:762)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x506C597: inflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x40352C: get_data_from_pack (index-pack.c:380)\n==3429== by 0x4035FB: get_base_data (index-pack.c:496)\n==3429== by 0x403687: get_base_data (index-pack.c:486)\n==3429== by 0x403711: resolve_delta (index-pack.c:516)\n==3429== by 0x4038A8: resolve_delta (index-pack.c:543)\n==3429== by 0x404CF5: main (index-pack.c:762)\n==3429== \n==3429== Conditional jump or move depends on uninitialised value(s)\n==3429== at 0x506C5E9: inflate (in /usr/lib/libz.so.1.2.3.3)\n==3429== by 0x40352C: get_data_from_pack (index-pack.c:380)\n==3429== by 0x4035FB: get_base_data (index-pack.c:496)\n==3429== by 0x403687: get_base_data (index-pack.c:486)\n==3429== by 0x403711: resolve_delta (index-pack.c:516)\n==3429== by 0x4038A8: resolve_delta (index-pack.c:543)\n==3429== by 0x404CF5: main (index-pack.c:762)\n==3429== \n==3429== ERROR SUMMARY: 146 errors from 23 contexts (suppressed: 8 from 1)\n==3429== malloc/free: in use at exit: 50,109 bytes in 17 blocks.\n==3429== malloc/free: 1,223 allocs, 1,206 frees, 18,574,785 bytes allocated.\n==3429== For counts of detected errors, rerun with: -v\n==3429== searching for pointers to 17 not-freed blocks.\n==3429== checked 618,664 bytes.\n==3429== \n==3429== LEAK SUMMARY:\n==3429== definitely lost: 0 bytes in 0 blocks.\n==3429== possibly lost: 8,100 bytes in 1 blocks.\n==3429== still reachable: 42,009 bytes in 16 blocks.\n==3429== suppressed: 0 bytes in 0 blocks.\n==3429== Rerun with --leak-check=full to see details of leaked memory.\n"},{"id":"84463","messageId":"alpine.DEB.1.00.0807231246560.2830@eeepc-johanness","threadId":"14606","inReplyTo":"20080723101415.GA23769@atjola.homenet","subject":"Re: regression in 92392b4","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-23T10:49:04Z","receivedAt":"2008-07-23T10:49:04Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Jul 2008, Björn Steinbrink wrote:\n\n> On 2008.07.23 01:17:45 +0200, Pierre Habouzit wrote:\n> > Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> > did, since git in next cannot fetch on a regular basis for me. The\n> > culprit seems to be commit 92392b4:\n> > \n> > ┌─(1:11)──<~/dev/scm/git 92392b4...>──\n> > └[artemis] git fetch\n> > remote: Counting objects: 461, done.\n> > remote: Compressing objects: 100% (141/141), done.\n> > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > fatal: Out of memory, malloc failed\n> > fatal: index-pack failed\n> > [2] 16674 abort (core dumped) git fetch\n> > \n> > ┌─(1:12)──<~/dev/scm/git 92392b4...>──\n> > └[artemis] git checkout -m HEAD~1; make git-index-pack\n> > Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n> > HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n> > GIT_VERSION = 1.5.6.3.3.g03993\n> > \tCC index-pack.o\n> > \tLINK git-index-pack\n> > \n> > ┌─(1:12)──<~/dev/scm/git 03993e1...>──\n> > └[artemis] git fetch\n> > remote: Counting objects: 461, done.\n> > remote: Compressing objects: 100% (141/141), done.\n> > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > Resolving deltas: 100% (227/227), completed with 153 local objects.\n> > From git://git.kernel.org/pub/scm/git/git\n> > 5ba2c22..0868a30 html -> origin/html\n> > 2857e17..abeeabe man -> origin/man\n> > 93310a4..95f8ebb master -> origin/master\n> > 559998f..e8bf351 next -> origin/next\n> > \n> > You can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\n> > broken, I've absolutely no clue about what happens.\n> > \n> > All I can say is that at some point in get_data_from_pack, obj[1].idx\n> > points to something that is *not* a sha so it's probably corrupted.\n> > (from index-pack.c).\n> \n> Here's how to reproduce:\n\nFunny. That does not reproduce the bug here at all.\n\nBut then, it is unsurprising, since both Pierre and me did something \nsimilar yesterday, fetching _just_ the pre-fetch refs into a freshly \ninitted Git repository, and then fetching from kernel.org.\n\nTested on x86_64.\n\nCiao,\nDscho\n"},{"id":"84464","messageId":"20080723105607.GB9888@atjola.homenet","threadId":"14606","inReplyTo":"alpine.DEB.1.00.0807231246560.2830@eeepc-johanness","subject":"Re: regression in 92392b4","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-07-23T10:56:07Z","receivedAt":"2008-07-23T10:56:07Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.07.23 12:49:04 +0200, Johannes Schindelin wrote:\n> Hi,\n> \n> On Wed, 23 Jul 2008, Björn Steinbrink wrote:\n> \n> > On 2008.07.23 01:17:45 +0200, Pierre Habouzit wrote:\n> > > Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> > > did, since git in next cannot fetch on a regular basis for me. The\n> > > culprit seems to be commit 92392b4:\n> > > \n> > > ┌─(1:11)──<~/dev/scm/git 92392b4...>──\n> > > └[artemis] git fetch\n> > > remote: Counting objects: 461, done.\n> > > remote: Compressing objects: 100% (141/141), done.\n> > > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > > fatal: Out of memory, malloc failed\n> > > fatal: index-pack failed\n> > > [2] 16674 abort (core dumped) git fetch\n> > > \n> > > ┌─(1:12)──<~/dev/scm/git 92392b4...>──\n> > > └[artemis] git checkout -m HEAD~1; make git-index-pack\n> > > Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n> > > HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n> > > GIT_VERSION = 1.5.6.3.3.g03993\n> > > \tCC index-pack.o\n> > > \tLINK git-index-pack\n> > > \n> > > ┌─(1:12)──<~/dev/scm/git 03993e1...>──\n> > > └[artemis] git fetch\n> > > remote: Counting objects: 461, done.\n> > > remote: Compressing objects: 100% (141/141), done.\n> > > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > > Resolving deltas: 100% (227/227), completed with 153 local objects.\n> > > From git://git.kernel.org/pub/scm/git/git\n> > > 5ba2c22..0868a30 html -> origin/html\n> > > 2857e17..abeeabe man -> origin/man\n> > > 93310a4..95f8ebb master -> origin/master\n> > > 559998f..e8bf351 next -> origin/next\n> > > \n> > > You can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\n> > > broken, I've absolutely no clue about what happens.\n> > > \n> > > All I can say is that at some point in get_data_from_pack, obj[1].idx\n> > > points to something that is *not* a sha so it's probably corrupted.\n> > > (from index-pack.c).\n> > \n> > Here's how to reproduce:\n> \n> Funny. That does not reproduce the bug here at all.\n> \n> But then, it is unsurprising, since both Pierre and me did something \n> similar yesterday, fetching _just_ the pre-fetch refs into a freshly \n> initted Git repository, and then fetching from kernel.org.\n> \n> Tested on x86_64.\n\nWeird. Same arch here. And it's 100% reproducable so far (around 25\ntimes now, I guess).\n\nBjörn\n"},{"id":"84467","messageId":"20080723111931.GF15243@artemis.madism.org","threadId":"14606","inReplyTo":"alpine.DEB.1.00.0807231246560.2830@eeepc-johanness","subject":"Re: regression in 92392b4","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-23T11:19:31Z","receivedAt":"2008-07-23T11:19:31Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jul 23, 2008 at 10:49:04AM +0000, Johannes Schindelin wrote:\n> Hi,\n> \n> On Wed, 23 Jul 2008, Björn Steinbrink wrote:\n> \n> > On 2008.07.23 01:17:45 +0200, Pierre Habouzit wrote:\n> > > Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> > > did, since git in next cannot fetch on a regular basis for me. The\n> > > culprit seems to be commit 92392b4:\n> > > \n> > > ┌─(1:11)──<~/dev/scm/git 92392b4....>──\n> > > └[artemis] git fetch\n> > > remote: Counting objects: 461, done.\n> > > remote: Compressing objects: 100% (141/141), done.\n> > > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > > fatal: Out of memory, malloc failed\n> > > fatal: index-pack failed\n> > > [2] 16674 abort (core dumped) git fetch\n> > > \n> > > ┌─(1:12)──<~/dev/scm/git 92392b4....>──\n> > > └[artemis] git checkout -m HEAD~1; make git-index-pack\n> > > Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n> > > HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n> > > GIT_VERSION = 1.5.6.3.3.g03993\n> > > \tCC index-pack.o\n> > > \tLINK git-index-pack\n> > > \n> > > ┌─(1:12)──<~/dev/scm/git 03993e1....>──\n> > > └[artemis] git fetch\n> > > remote: Counting objects: 461, done.\n> > > remote: Compressing objects: 100% (141/141), done.\n> > > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > > Resolving deltas: 100% (227/227), completed with 153 local objects.\n> > > From git://git.kernel.org/pub/scm/git/git\n> > > 5ba2c22..0868a30 html -> origin/html\n> > > 2857e17..abeeabe man -> origin/man\n> > > 93310a4..95f8ebb master -> origin/master\n> > > 559998f..e8bf351 next -> origin/next\n> > > \n> > > You can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\n> > > broken, I've absolutely no clue about what happens.\n> > > \n> > > All I can say is that at some point in get_data_from_pack, obj[1].idx\n> > > points to something that is *not* a sha so it's probably corrupted.\n> > > (from index-pack.c).\n> > \n> > Here's how to reproduce:\n> \n> Funny. That does not reproduce the bug here at all.\n> \n> But then, it is unsurprising, since both Pierre and me did something \n> similar yesterday, fetching _just_ the pre-fetch refs into a freshly \n> initted Git repository, and then fetching from kernel.org.\n> \n> Tested on x86_64.\n\nI can reproduce on x86_64 here. And I think I get the problem, and IMHO\nthe pruning thing is flawed. We need more than *one* base to be kept at\na time, pruning is too aggressive, and we still keep pointers to\nactually pruned data.\n\nThis patch makes the issue non reproducible for me. The rationale is\nthat get_base_data will already prune and is called as often, and in\nsafer places.\n\n=================================\ndiff --git a/index-pack.c b/index-pack.c\nindex ac20a46..5440e43 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -245,7 +245,6 @@ static void link_base_data(struct base_data *base, struct base_data *c)\n \tc->base = base;\n \tc->child = NULL;\n \tbase_cache_used += c->size;\n-\tprune_base_data(c);\n }\n \n static void unlink_base_data(struct base_data *c)\n=================================\n\n*But* I'm absolutely not sure it's enough. This should be written using\nreference counting instead of the \"retain\" hack, and prune should not\nfree() anything that isn't with a 0 reference counter. The current code\nis brittle, it makes my head explode when I try to understand if the\nget_base_data() we temporarily keep pointers too may be harmed or not.\n\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"84468","messageId":"alpine.DEB.1.00.0807231235150.8986@racer","threadId":"14606","inReplyTo":"20080723111931.GF15243@artemis.madism.org","subject":"Re: regression in 92392b4","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-23T11:37:00Z","receivedAt":"2008-07-23T11:37:00Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Jul 2008, Pierre Habouzit wrote:\n\n> On Wed, Jul 23, 2008 at 10:49:04AM +0000, Johannes Schindelin wrote:\n> \n> > On Wed, 23 Jul 2008, Björn Steinbrink wrote:\n> > \n> > > On 2008.07.23 01:17:45 +0200, Pierre Habouzit wrote:\n> > > > Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> > > > did, since git in next cannot fetch on a regular basis for me. The\n> > > > culprit seems to be commit 92392b4:\n> > > > \n> > > > ┌─(1:11)──<~/dev/scm/git 92392b4....>──\n> > > > └[artemis] git fetch\n> > > > remote: Counting objects: 461, done.\n> > > > remote: Compressing objects: 100% (141/141), done.\n> > > > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > > > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > > > fatal: Out of memory, malloc failed\n> > > > fatal: index-pack failed\n> > > > [2] 16674 abort (core dumped) git fetch\n> > > > \n> > > > ┌─(1:12)──<~/dev/scm/git 92392b4....>──\n> > > > └[artemis] git checkout -m HEAD~1; make git-index-pack\n> > > > Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n> > > > HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n> > > > GIT_VERSION = 1.5.6.3.3.g03993\n> > > > \tCC index-pack.o\n> > > > \tLINK git-index-pack\n> > > > \n> > > > ┌─(1:12)──<~/dev/scm/git 03993e1....>──\n> > > > └[artemis] git fetch\n> > > > remote: Counting objects: 461, done.\n> > > > remote: Compressing objects: 100% (141/141), done.\n> > > > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > > > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > > > Resolving deltas: 100% (227/227), completed with 153 local objects.\n> > > > From git://git.kernel.org/pub/scm/git/git\n> > > > 5ba2c22..0868a30 html -> origin/html\n> > > > 2857e17..abeeabe man -> origin/man\n> > > > 93310a4..95f8ebb master -> origin/master\n> > > > 559998f..e8bf351 next -> origin/next\n> > > > \n> > > > You can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\n> > > > broken, I've absolutely no clue about what happens.\n> > > > \n> > > > All I can say is that at some point in get_data_from_pack, obj[1].idx\n> > > > points to something that is *not* a sha so it's probably corrupted.\n> > > > (from index-pack.c).\n> > > \n> > > Here's how to reproduce:\n> > \n> > Funny. That does not reproduce the bug here at all.\n> > \n> > But then, it is unsurprising, since both Pierre and me did something \n> > similar yesterday, fetching _just_ the pre-fetch refs into a freshly \n> > initted Git repository, and then fetching from kernel.org.\n> > \n> > Tested on x86_64.\n> \n> I can reproduce on x86_64 here.\n\nWell, I cannot. However, I get some pread issue on i686. To be nice to \nkernel.org, I downloaded the pack in question:\n\n\thttp://pacific.mpi-cbg.de/git/thin-pack.pack\n\nYou should be able to reproduce the behavior by piping this into\n\ngit-index-pack --stdin -v --fix-thin --keep=fetch-pack --pack_header=2,263\n\nHth,\nDscho\n"},{"id":"84472","messageId":"20080723115026.GH15243@artemis.madism.org","threadId":"14606","inReplyTo":"alpine.DEB.1.00.0807231235150.8986@racer","subject":"Re: regression in 92392b4","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-23T11:50:26Z","receivedAt":"2008-07-23T11:50:26Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jul 23, 2008 at 11:37:00AM +0000, Johannes Schindelin wrote:\n> Hi,\n> \n> On Wed, 23 Jul 2008, Pierre Habouzit wrote:\n> \n> > On Wed, Jul 23, 2008 at 10:49:04AM +0000, Johannes Schindelin wrote:\n> > \n> > > On Wed, 23 Jul 2008, Björn Steinbrink wrote:\n> > > Funny. That does not reproduce the bug here at all.\n> > > \n> > > But then, it is unsurprising, since both Pierre and me did something \n> > > similar yesterday, fetching _just_ the pre-fetch refs into a freshly \n> > > initted Git repository, and then fetching from kernel.org.\n> > > \n> > > Tested on x86_64.\n> > \n> > I can reproduce on x86_64 here.\n> \n> Well, I cannot. However, I get some pread issue on i686. To be nice to \n> kernel.org, I downloaded the pack in question:\n> \n> \thttp://pacific.mpi-cbg.de/git/thin-pack.pack\n> \n> You should be able to reproduce the behavior by piping this into\n> \n> git-index-pack --stdin -v --fix-thin --keep=fetch-pack --pack_header=2,263\n\n I can reproduce. Funily enough, I just happened to see that I have\ncore.deltabasecachelimit = 0 in my git.git repository... which I\nprobably used meaning -1 but oh well. The bottom line is that the\npruning algorithm likely removes a pointer we still have a pointer to\nsomewhere...\n\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"84474","messageId":"20080723120045.GA21274@atjola.homenet","threadId":"14606","inReplyTo":"alpine.DEB.1.00.0807231235150.8986@racer","subject":"Re: regression in 92392b4","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-07-23T12:00:45Z","receivedAt":"2008-07-23T12:00:45Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.07.23 12:37:00 +0100, Johannes Schindelin wrote:\n> Hi,\n> \n> On Wed, 23 Jul 2008, Pierre Habouzit wrote:\n> \n> > On Wed, Jul 23, 2008 at 10:49:04AM +0000, Johannes Schindelin wrote:\n> > \n> > > On Wed, 23 Jul 2008, Björn Steinbrink wrote:\n> > > \n> > > > On 2008.07.23 01:17:45 +0200, Pierre Habouzit wrote:\n> > > > > Hi, here is a manual painful down-secting (opposed to a bisect ;P) I\n> > > > > did, since git in next cannot fetch on a regular basis for me. The\n> > > > > culprit seems to be commit 92392b4:\n> > > > > \n> > > > > ┌─(1:11)──<~/dev/scm/git 92392b4....>──\n> > > > > └[artemis] git fetch\n> > > > > remote: Counting objects: 461, done.\n> > > > > remote: Compressing objects: 100% (141/141), done.\n> > > > > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > > > > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > > > > fatal: Out of memory, malloc failed\n> > > > > fatal: index-pack failed\n> > > > > [2] 16674 abort (core dumped) git fetch\n> > > > > \n> > > > > ┌─(1:12)──<~/dev/scm/git 92392b4....>──\n> > > > > └[artemis] git checkout -m HEAD~1; make git-index-pack\n> > > > > Previous HEAD position was 92392b4... index-pack: Honor core.deltaBaseCacheLimit when resolving deltas\n> > > > > HEAD is now at 03993e1... index-pack: Track the object_entry that creates each base_data\n> > > > > GIT_VERSION = 1.5.6.3.3.g03993\n> > > > > \tCC index-pack.o\n> > > > > \tLINK git-index-pack\n> > > > > \n> > > > > ┌─(1:12)──<~/dev/scm/git 03993e1....>──\n> > > > > └[artemis] git fetch\n> > > > > remote: Counting objects: 461, done.\n> > > > > remote: Compressing objects: 100% (141/141), done.\n> > > > > remote: Total 263 (delta 227), reused 155 (delta 121)\n> > > > > Receiving objects: 100% (263/263), 95.55 KiB, done.\n> > > > > Resolving deltas: 100% (227/227), completed with 153 local objects.\n> > > > > From git://git.kernel.org/pub/scm/git/git\n> > > > > 5ba2c22..0868a30 html -> origin/html\n> > > > > 2857e17..abeeabe man -> origin/man\n> > > > > 93310a4..95f8ebb master -> origin/master\n> > > > > 559998f..e8bf351 next -> origin/next\n> > > > > \n> > > > > You can see the commit sha's in the prompt. 03993e1 is fine, 92392b4 is\n> > > > > broken, I've absolutely no clue about what happens.\n> > > > > \n> > > > > All I can say is that at some point in get_data_from_pack, obj[1].idx\n> > > > > points to something that is *not* a sha so it's probably corrupted.\n> > > > > (from index-pack.c).\n> > > > \n> > > > Here's how to reproduce:\n> > > \n> > > Funny. That does not reproduce the bug here at all.\n> > > \n> > > But then, it is unsurprising, since both Pierre and me did something \n> > > similar yesterday, fetching _just_ the pre-fetch refs into a freshly \n> > > initted Git repository, and then fetching from kernel.org.\n> > > \n> > > Tested on x86_64.\n> > \n> > I can reproduce on x86_64 here.\n> \n> Well, I cannot. However, I get some pread issue on i686. To be nice to \n> kernel.org, I downloaded the pack in question:\n> \n> \thttp://pacific.mpi-cbg.de/git/thin-pack.pack\n> \n> You should be able to reproduce the behavior by piping this into\n> \n> git-index-pack --stdin -v --fix-thin --keep=fetch-pack --pack_header=2,263\n\nOK, that gave me a seemingly sane backtrace. What seems to happen (AFA\nmy limited knowledge tells me):\n\nIn fix_unresolved_deltas, we read base_obj from an existing pack, other\nthan the one we're reading. We then link that object to the base cache. \n\nThen in resolve_delta, we create the \"result\" base_data object and link\nthat one, too. Now this triggers the pruning, and because the cache is\nso small, we prune the object that we read from the existing pack! Fast\nforward a few function calls, we end up in get_base_data trying to\nre-read the data for that object, but this time from the pack that we\ngot on stdin. And boom it goes.\n\nDoes that make any sense to you?\n\nBjörn\n"},{"id":"84476","messageId":"20080723121118.GA20614@artemis.madism.org","threadId":"14606","inReplyTo":"20080723120045.GA21274@atjola.homenet","subject":"[PATCH] index-pack: never prune base_cache.","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-23T12:11:18Z","receivedAt":"2008-07-23T12:11:18Z","isPatch":true,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"It may belong to something (stdin) that is consumed.\n\nSigned-off-by: Pierre Habouzit \n---\n\n On Wed, Jul 23, 2008 at 12:00:45PM +0000, Björn Steinbrink wrote:\n > On 2008.07.23 12:37:00 +0100, Johannes Schindelin wrote:\n > > Hi,\n > > \n > > Well, I cannot. However, I get some pread issue on i686. To be nice to \n > > kernel.org, I downloaded the pack in question:\n > > \n > > \thttp://pacific.mpi-cbg.de/git/thin-pack.pack\n > > \n > > You should be able to reproduce the behavior by piping this into\n > > \n > > git-index-pack --stdin -v --fix-thin --keep=fetch-pack --pack_header=2,263\n > \n > OK, that gave me a seemingly sane backtrace. What seems to happen (AFA\n > my limited knowledge tells me):\n > \n > In fix_unresolved_deltas, we read base_obj from an existing pack, other\n > than the one we're reading. We then link that object to the base cache. \n > \n > Then in resolve_delta, we create the \"result\" base_data object and link\n > that one, too. Now this triggers the pruning, and because the cache is\n > so small, we prune the object that we read from the existing pack! Fast\n > forward a few function calls, we end up in get_base_data trying to\n > re-read the data for that object, but this time from the pack that we\n > got on stdin. And boom it goes.\n > \n > Does that make any sense to you?\n\n Yes, that's obvious, the pack that we read from stdin is consumed, we\n should *NEVER* prune base_cache. And indeed that little patch works for\n me.\n\n index-pack.c | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/index-pack.c b/index-pack.c\nindex ac20a46..eb81ed4 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -227,7 +227,7 @@ static void prune_base_data(struct base_data *retain)\n \tfor (b = base_cache;\n \t base_cache_used > delta_base_cache_limit && b;\n \t b = b->child) {\n-\t\tif (b->data && b != retain) {\n+\t\tif (b != base_cache && b->data && b != retain) {\n \t\t\tfree(b->data);\n \t\t\tb->data = NULL;\n \t\t\tbase_cache_used -= b->size;\n-- \n1.6.0.rc0.155.ga0442.dirty\n"},{"id":"84478","messageId":"20080723125226.GA11679@atjola.homenet","threadId":"14606","inReplyTo":"20080723121118.GA20614@artemis.madism.org","subject":"Re: [PATCH] index-pack: never prune base_cache.","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-07-23T12:52:26Z","receivedAt":"2008-07-23T12:52:26Z","isPatch":true,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.07.23 14:11:18 +0200, Pierre Habouzit wrote:\n> It may belong to something (stdin) that is consumed.\n\nProbably thanks to me, babbling about stdin without having a clue what\nI'm talking about, that rationale is wrong.\n\nWe may not prune base_cache since that object might come from a\ndifferent pack than the one that we are processing. In such a case, we\nwould try to restore the data for that object from the pack we're\nprocessing and fail miserably.\n\n\nThe patch itself should be fine.\n\n> \n> Signed-off-by: Pierre Habouzit \n> ---\n> \n> On Wed, Jul 23, 2008 at 12:00:45PM +0000, Björn Steinbrink wrote:\n> > On 2008.07.23 12:37:00 +0100, Johannes Schindelin wrote:\n> > > Hi,\n> > > \n> > > Well, I cannot. However, I get some pread issue on i686. To be nice to \n> > > kernel.org, I downloaded the pack in question:\n> > > \n> > > \thttp://pacific.mpi-cbg.de/git/thin-pack.pack\n> > > \n> > > You should be able to reproduce the behavior by piping this into\n> > > \n> > > git-index-pack --stdin -v --fix-thin --keep=fetch-pack --pack_header=2,263\n> > \n> > OK, that gave me a seemingly sane backtrace. What seems to happen (AFA\n> > my limited knowledge tells me):\n> > \n> > In fix_unresolved_deltas, we read base_obj from an existing pack, other\n> > than the one we're reading. We then link that object to the base cache. \n> > \n> > Then in resolve_delta, we create the \"result\" base_data object and link\n> > that one, too. Now this triggers the pruning, and because the cache is\n> > so small, we prune the object that we read from the existing pack! Fast\n> > forward a few function calls, we end up in get_base_data trying to\n> > re-read the data for that object, but this time from the pack that we\n> > got on stdin. And boom it goes.\n> > \n> > Does that make any sense to you?\n> \n> Yes, that's obvious, the pack that we read from stdin is consumed, we\n> should *NEVER* prune base_cache. And indeed that little patch works for\n> me.\n> \n> index-pack.c | 2 +-\n> 1 files changed, 1 insertions(+), 1 deletions(-)\n> \n> diff --git a/index-pack.c b/index-pack.c\n> index ac20a46..eb81ed4 100644\n> --- a/index-pack.c\n> +++ b/index-pack.c\n> @@ -227,7 +227,7 @@ static void prune_base_data(struct base_data *retain)\n> \tfor (b = base_cache;\n> \t base_cache_used > delta_base_cache_limit && b;\n> \t b = b->child) {\n> -\t\tif (b->data && b != retain) {\n> +\t\tif (b != base_cache && b->data && b != retain) {\n> \t\t\tfree(b->data);\n> \t\t\tb->data = NULL;\n> \t\t\tbase_cache_used -= b->size;\n"},{"id":"84482","messageId":"alpine.DEB.1.00.0807231407040.8986@racer","threadId":"14606","inReplyTo":"20080723125226.GA11679@atjola.homenet","subject":"Re: [PATCH] index-pack: never prune base_cache.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-23T13:09:40Z","receivedAt":"2008-07-23T13:09:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Jul 2008, Björn Steinbrink wrote:\n\n> On 2008.07.23 14:11:18 +0200, Pierre Habouzit wrote:\n> > It may belong to something (stdin) that is consumed.\n> \n> Probably thanks to me, babbling about stdin without having a clue what \n> I'm talking about, that rationale is wrong.\n> \n> We may not prune base_cache since that object might come from a\n> different pack than the one that we are processing. In such a case, we\n> would try to restore the data for that object from the pack we're\n> processing and fail miserably.\n\nThen the proper fix would be to load the object from that pack again.\n\n> The patch itself should be fine.\n\nNo, since it opens the whole issue of memory explosion again, the same \nissue Shawn's original patch tried to fix.\n\nCiao,\nDscho\n\nP.S.: Could you please, please, please cull the part you are not \nresponding to? This mailing list is read by more than 50 people. If you \nsum up the time it takes them to realize that that quoted part was \nirrelevant, I am sure you will end up with a larger number of minutes than \nit would take you to just delete it.\n\nThanks.\n"},{"id":"84485","messageId":"20080723132031.GC20614@artemis.madism.org","threadId":"14606","inReplyTo":"alpine.DEB.1.00.0807231407040.8986@racer","subject":"Re: [PATCH] index-pack: never prune base_cache.","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-23T13:20:31Z","receivedAt":"2008-07-23T13:20:31Z","isPatch":true,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jul 23, 2008 at 01:09:40PM +0000, Johannes Schindelin wrote:\n> On Wed, 23 Jul 2008, Björn Steinbrink wrote:\n> > The patch itself should be fine.\n> \n> No, since it opens the whole issue of memory explosion again, the same \n> issue Shawn's original patch tried to fix.\n\n No it won't. Indeed the issue is with fix_unresolved_deltas that\nsometimes put at the root of the chain (in base_cache) something that\ncomes from our store, not the pack we are writing. Then starts a delta\nchain resolution.\n\n It won't explode in memory at all, we just keep the first data of a\ndelta chain in memory, that's all. It indeed consumes more memory, but\nwe talk about *one* single object per delta chain because we're too lazy\nto memorize where it comes from. It's probably not much of an explosion.\n\n We also waste that object even when it's from our own pack. Well, I'd\nsay \"too bad\".\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"84487","messageId":"20080723134448.GB11679@atjola.homenet","threadId":"14606","inReplyTo":"alpine.DEB.1.00.0807231407040.8986@racer","subject":"Re: [PATCH] index-pack: never prune base_cache.","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-07-23T13:44:48Z","receivedAt":"2008-07-23T13:44:48Z","isPatch":true,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.07.23 14:09:40 +0100, Johannes Schindelin wrote:\n> Hi,\n> \n> On Wed, 23 Jul 2008, Björn Steinbrink wrote:\n> \n> > On 2008.07.23 14:11:18 +0200, Pierre Habouzit wrote:\n> > > It may belong to something (stdin) that is consumed.\n> > \n> > Probably thanks to me, babbling about stdin without having a clue what \n> > I'm talking about, that rationale is wrong.\n> > \n> > We may not prune base_cache since that object might come from a\n> > different pack than the one that we are processing. In such a case, we\n> > would try to restore the data for that object from the pack we're\n> > processing and fail miserably.\n> \n> Then the proper fix would be to load the object from that pack again.\n\nActually, my analysis was total bullshit. Right after reading the object\nfrom the foreign pack, we also call append_obj_to_pack, so we are\nactually able to reread that object just fine. The real issue seems to\nbe that we just forget to initialize some fields.\n\nThis patch fixes the issue for me, but I guess it's not quite the right\nway to do it, pure guesswork.\n\nBjörn\n\n---\n\ndiff --git a/index-pack.c b/index-pack.c\nindex ac20a46..33ba8ef 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -699,6 +699,9 @@ static struct object_entry *append_obj_to_pack(\n \twrite_or_die(output_fd, header, n);\n \tobj[0].idx.crc32 = crc32(0, Z_NULL, 0);\n \tobj[0].idx.crc32 = crc32(obj[0].idx.crc32, header, n);\n+\tobj[0].hdr_size = n;\n+\tobj[0].type = type;\n+\tobj[0].size = size;\n \tobj[1].idx.offset = obj[0].idx.offset + n;\n \tobj[1].idx.offset += write_compressed(output_fd, buf, size, &obj[0].idx.crc32);\n \thashcpy(obj->idx.sha1, sha1);\n"},{"id":"84488","messageId":"alpine.DEB.1.00.0807231436440.8986@racer","threadId":"14606","inReplyTo":"20080723132031.GC20614@artemis.madism.org","subject":"Re: [PATCH] index-pack: never prune base_cache.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-23T13:46:38Z","receivedAt":"2008-07-23T13:46:38Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Jul 2008, Pierre Habouzit wrote:\n\n> On Wed, Jul 23, 2008 at 01:09:40PM +0000, Johannes Schindelin wrote:\n> > On Wed, 23 Jul 2008, Björn Steinbrink wrote:\n> > > The patch itself should be fine.\n> > \n> > No, since it opens the whole issue of memory explosion again, the same \n> > issue Shawn's original patch tried to fix.\n> \n> No it won't. Indeed the issue is with fix_unresolved_deltas that\n> sometimes put at the root of the chain (in base_cache) something that\n> comes from our store, not the pack we are writing. Then starts a delta\n> chain resolution.\n\nIf it comes from our store, we should have _no_ problem reconstructing the \nobject.\n\n> It won't explode in memory at all, we just keep the first data of a\n> delta chain in memory, that's all. It indeed consumes more memory, but\n> we talk about *one* single object per delta chain because we're too lazy\n> to memorize where it comes from. It's probably not much of an explosion.\n\n\"Probably\".\n\n> We also waste that object even when it's from our own pack. Well, I'd\n> say \"too bad\".\n\nAnd I say it's dirty, and since the pack code traditionally is one of the \ncleanest parts of Git, coming from Nico, let's not change that, okay?\n\nCiao,\nDscho\n"},{"id":"84500","messageId":"alpine.DEB.1.00.0807231537420.8986@racer","threadId":"14606","inReplyTo":"20080723134448.GB11679@atjola.homenet","subject":"Re: [PATCH] index-pack: never prune base_cache.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-23T14:41:14Z","receivedAt":"2008-07-23T14:41:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Jul 2008, Björn Steinbrink wrote:\n\n> diff --git a/index-pack.c b/index-pack.c\n> index ac20a46..33ba8ef 100644\n> --- a/index-pack.c\n> +++ b/index-pack.c\n> @@ -699,6 +699,9 @@ static struct object_entry *append_obj_to_pack(\n> \twrite_or_die(output_fd, header, n);\n> \tobj[0].idx.crc32 = crc32(0, Z_NULL, 0);\n> \tobj[0].idx.crc32 = crc32(obj[0].idx.crc32, header, n);\n> +\tobj[0].hdr_size = n;\n> +\tobj[0].type = type;\n> +\tobj[0].size = size;\n> \tobj[1].idx.offset = obj[0].idx.offset + n;\n> \tobj[1].idx.offset += write_compressed(output_fd, buf, size, &obj[0].idx.crc32);\n> \thashcpy(obj->idx.sha1, sha1);\n\nI confirm that the issues I saw went away with this patch, and it looks \nobviously like the correct approach.\n\nThe only things valgrind is still complaining about (apart from libz, \nwhich I will not bother commenting about) are uninitialized parts of the \ndata being written to disk, and a crc over them.\n\nJudging from the addresses, those are probably the bytes that are padded \nfor 4- or 8-byte alignment, so they are probably fine.\n\nThanks,\nDscho\n"},{"id":"84521","messageId":"20080723153012.GE20614@artemis.madism.org","threadId":"14606","inReplyTo":"alpine.DEB.1.00.0807231537420.8986@racer","subject":"Re: [PATCH] index-pack: never prune base_cache.","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-07-23T15:30:12Z","receivedAt":"2008-07-23T15:30:12Z","isPatch":true,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jul 23, 2008 at 02:41:14PM +0000, Johannes Schindelin wrote:\n> Hi,\n> \n> On Wed, 23 Jul 2008, Björn Steinbrink wrote:\n> \n> > diff --git a/index-pack.c b/index-pack.c\n> > index ac20a46..33ba8ef 100644\n> > --- a/index-pack.c\n> > +++ b/index-pack.c\n> > @@ -699,6 +699,9 @@ static struct object_entry *append_obj_to_pack(\n> > \twrite_or_die(output_fd, header, n);\n> > \tobj[0].idx.crc32 = crc32(0, Z_NULL, 0);\n> > \tobj[0].idx.crc32 = crc32(obj[0].idx.crc32, header, n);\n> > +\tobj[0].hdr_size = n;\n> > +\tobj[0].type = type;\n> > +\tobj[0].size = size;\n> > \tobj[1].idx.offset = obj[0].idx.offset + n;\n> > \tobj[1].idx.offset += write_compressed(output_fd, buf, size, &obj[0]..idx.crc32);\n> > \thashcpy(obj->idx.sha1, sha1);\n> \n> I confirm that the issues I saw went away with this patch, and it looks \n> obviously like the correct approach.\n> \n> The only things valgrind is still complaining about (apart from libz, \n> which I will not bother commenting about) are uninitialized parts of the \n> data being written to disk, and a crc over them.\n> \n> Judging from the addresses, those are probably the bytes that are padded \n> for 4- or 8-byte alignment, so they are probably fine.\n\n I confirm this analysis having done the same independently and reached\nthe same conclusions.\n"}]}