Re: git packs

From: bob <kranki@mac.com>
Date: Nov 12, 2007, 14:15 UTC
Message-ID: <904628F9-EE34-4C4B-A114-82159AFF0260@mac.com>
In-Reply-To: <alpine.LFD.0.9999.0711112307070.21255@xanadu.home>

On Nov 11, 2007, at 11:21 PM, Nicolas Pitre wrote:

Show 15 quoted lines

> On Sun, 11 Nov 2007, bob wrote:
>
>> I applied the patch and these commands:
>>
>> cd rmwHtmlOld
>> rm -fr .git
>> git init
>> git config core.compression 0
>> git add .
>
> Note that I did "git config core.compression 0" simply to disable
> zlib compression altogether when creating the test repo just so it  
> gets
> created faster.  even then, auto-generating and cloning a 8GB test
> repository isn't particularly quick.

	I wasn't sure why you used that, but figured that I should put it
	in, because you did.  And you are very correct that these tests
	can take a very long time.

>
>> I then got the same error as before, "Bus error".  Rats!
>
> Do you get that with a 32-bit or 64-bit build of Git?

	On that machine, I changed the config.mak to generate nothing
	but a 64-bit build.  And activity monitor shows it as 64-bit
	when I am running it. Also, off_t is 64 bit on regular
	MacOSX and as a 64-bit program supported by the
	latest MacOSX.

Show 14 quoted lines

>
>> Then I modified your script since I do not have seq or
>> your test-genrandom.
>
> test-genrandom is built with Git.  It is just not installed anywhere.
>
>> I substituted:
>>
>> dd count=XX  if=/dev/random of=file_$i
>>
>> where XX is adjusted to meet dd's requirements.  Also,
>
> Again I used test-genrandom instead of /dev/random or /dev/urandom
> simply because the former is much faster.

	I didn't know that it existed, since I am not that familiar with the
	internals and source directory for git.

Show 15 quoted lines

>
>> I found after searching for a while, that the following
>> works just like your seq command:
>>
>> xyzzy="1 2 3 4"
>> for i in $xyzzy
>> do
>> ...
>> done
>>
>> Your script then ran flawlessly.
>
> However 'seq -w 1 2 63' should be replaced  with "01 03 05 07 09 11 13
> 15" and so on up to 63, and 'seq -w 2 2 64' is "02 04 06 08 10 12 16"
> and so on.

	That I missed but the files added up to quite a bit over 8gigs,
	because my count calculation was off.  So, I thought that it
	was adequate for the testing.  Based on above, it was
	quite a few files less as well.

Show 9 quoted lines

>
>> I looked through index-pack.c some more, but it is
>> very hard to figure it out without doing a lot of research
>> since there doesn't seem to be anything that describes
>> the layout of a pack.  The link towards the end of the user's
>> manual doesn't work for me.
>
> Look at Documentation/technical/pack-format.txt in the Git source  
> tree.

Thanks.

Show 8 quoted lines

>
>> The difference between your test and my data is that
>> instead of having a few large files, I have 11,500 files
>> of varying sizes.  On average though, the file size is
>> about 370k.
>
> Are you saying that the test repo with big files works for you but not
> your own data set?

	That is correct, but you are probably correct that this is a separate
	problem, because it is only occurring on the 64-bit side.  The 32-bit
	side is running without error.

>
> Would you please recap what your problem is?

	With more testing last night, I believe that the problem on the 32-bit
	side has been fixed.  But the 'Bus error' on 64-bit side persists.  I
	agree that I was combining it and probably shouldn't have.

> With my one line patch you should not get the "serious inflate
> inconsistency" error anymore.  The bus error must be another issue.

I no longer do on the 32-bit side.

>
>
> Nicolas

I am going to wait on the 64-bit problem until MacOSX 10.5 (Leopard) settles down a bit. It has a few bugs in it. I will submit the crashreporter issue to Apple. Currently, I don't have the time to commit a lot to git right now.

As Apple comes out with new subreleases, I will retry my data. If it doesn't get resolved after a subrelease or two, then I dig deeper and hopefully have more time to devote to it at that point.

Thank you, I do appreciate your help and your patience.