[regression] Newer gits cannot clone any remote repos

*Any* git clone fails with:

fatal: premature end of pack file, 106 bytes missing fatal: index-pack failed

At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5 then, and it worked. Then I tried 1.7.12.2, but it fails the same way as 1.8.0. So I decided to git bisect.

b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit ``index-pack: support multithreaded delta resolving''

--------------------------------------------------------------------------------

b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit
commit b8a2486f1524947f232f657e9f2ebf44e3e7a243
Author: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date:   Sun May 6 19:31:55 2012 +0700

    index-pack: support multithreaded delta resolving

    This puts delta resolving on each base on a separate thread, one base
    cache per thread. Per-thread data is grouped in struct thread_local.
    When running with nr_threads == 1, no pthreads calls are made. The
    system essentially runs in non-thread mode.

    An experiment on a Xeon 24 core machine with git.git shows that
    performance does not increase proportional to the number of cores. So
    by default, we use maximum 3 cores. Some numbers with --threads from 1
    to 16:

    1..4
    real    0m8.003s  0m5.307s  0m4.321s  0m3.830s
    user    0m7.720s  0m8.009s  0m8.133s  0m8.305s
    sys     0m0.224s  0m0.372s  0m0.360s  0m0.360s

    5..8
    real    0m3.727s  0m3.604s  0m3.332s  0m3.369s
    user    0m9.361s  0m9.817s  0m9.525s  0m9.769s
    sys     0m0.584s  0m0.624s  0m0.540s  0m0.560s

    9..12
    real    0m3.036s  0m3.139s  0m3.177s  0m2.961s
    user    0m8.977s  0m10.205s 0m9.737s  0m10.073s
    sys     0m0.596s  0m0.680s  0m0.684s  0m0.680s

    13..16
    real    0m2.985s  0m2.894s  0m2.975s  0m2.971s
    user    0m9.825s  0m10.573s 0m10.833s 0m11.361s
    sys     0m0.788s  0m0.732s  0m0.904s  0m1.016s

    On an Intel dual core and linux-2.6.git

    1..4
    real    2m37.789s 2m7.963s  2m0.920s  1m58.213s
    user    2m28.415s 2m52.325s 2m50.176s 2m41.187s
    sys     0m7.808s  0m11.181s 0m11.224s 0m10.731s

    Thanks Ramsay Jones for troubleshooting and support on MinGW platform.

    Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

:040000 040000 e5597fe041cd7985648030c8cf1cc35ea57786bb 6e64f3e76b350c04c9d0c435e0a53b89e0821339 M Documentation :100644 100644 cf2c40b44f8383d002235400660d76ef7f6de33c e41955ff955693ddc78722c14fff2a6e98663a46 M Makefile :040000 040000 1e6b5ac5f11bf7ba3eff7b5be4410bad379feee6 85bc79414f00b7fc3698c069161ea10039641082 M builtin :040000 040000 04916545d959c8af772cd3a2de5e6b470529bea1 325c8c748dd41d469135db84c7621d8225cc1cdd M t

$ git bisect log
    git bisect start
    # bad: [d8cf053dacb4f78920c112d10c7be21e4f5a5817] Git 1.7.12.2
    git bisect bad d8cf053dacb4f78920c112d10c7be21e4f5a5817
    # good: [785ee4960c3d334cbc2b17ab74d2cebdf1b4db64] Git 1.7.10.5
    git bisect good 785ee4960c3d334cbc2b17ab74d2cebdf1b4db64
    # bad: [d692d34653f74be6b16add3e993e957f33fe049b] Merge branch
'rs/git-blame-mapcar-mapc'
    git bisect bad d692d34653f74be6b16add3e993e957f33fe049b
    # good: [73ff8cf784b6ee447072fad6c06fd0eef0e9c9f6] Merge branch
'lp/diffstat-with-graph'
    git bisect good 73ff8cf784b6ee447072fad6c06fd0eef0e9c9f6
    # bad: [7903e66a3e10e98f29fc9524c13274376bae5303] Merge branch
'mh/test-keep-prove-cache'
    git bisect bad 7903e66a3e10e98f29fc9524c13274376bae5303
    # good: [499e7b31509cfbb59dcb2a046f8e2fd1a3e73d6f] Merge branch
'jc/install-no-hardlinks'
    git bisect good 499e7b31509cfbb59dcb2a046f8e2fd1a3e73d6f
    # bad: [2e464dd5b220d4d2d8a16d5c43efe7af3c9adda9] Merge branch
'rs/xdiff-lose-emit-func'
    git bisect bad 2e464dd5b220d4d2d8a16d5c43efe7af3c9adda9
    # good: [8fbe0db4fce68ed8239742b14e9d77d45201870b] Merge branch
'master' of git://github.com/git-l10n/git-po
    git bisect good 8fbe0db4fce68ed8239742b14e9d77d45201870b
    # bad: [2b26b65f9abc77c4af87626452005a73edda0c8f] git-svn: clarify
the referent of dcommit's optional argument
    git bisect bad 2b26b65f9abc77c4af87626452005a73edda0c8f
    # bad: [cc13431a49800a6a1d2b7ff0b94f67da0fecdbab] Merge branch
'nd/threaded-index-pack'
    git bisect bad cc13431a49800a6a1d2b7ff0b94f67da0fecdbab
    # bad: [b8a2486f1524947f232f657e9f2ebf44e3e7a243] index-pack:
support multithreaded delta resolving
    git bisect bad b8a2486f1524947f232f657e9f2ebf44e3e7a243
    # good: [5272f75587abb4cb396059ecbf1a6130bb2e69d3] index-pack:
restructure pack processing into three main functions
    git bisect good 5272f75587abb4cb396059ecbf1a6130bb2e69d3

$ git bisect reset Previous HEAD position was 5272f75... index-pack: restructure pack processing into three main functions HEAD is now at d8cf053... Git 1.7.12.2

Maybe handy to say that you're on a Powerpc platform.

On Mon, Nov 12, 2012 at 5:32 PM, Douglas Mencken <dougmencken@gmail.com> wrote:

Oh, and yes, I'm on 2 x 2-core ("4-core") machine.

Douglas Mencken wrote:

This looks like the same problem I had on cygwin, which lead to commit c0f86547c ("index-pack: Disable threading on cygwin", 26-06-2012).

I didn't notice which platform you are on, but maybe you also have a thread-unsafe pread()? Could you try re-building git with the NO_THREAD_SAFE_PREAD build variable set?

HTH.

ATB, Ramsay Jones

Yeah! It works!!!

--- evil/Makefile +++ good/Makefile

By the way, it is the job of autotools to check such stuff, isn't it?

On Wed, Nov 14, 2012 at 2:55 AM, Douglas Mencken <dougmencken@gmail.com> wrote:

Sorry you had to figure that out the hard way. Could you make it a proper patch? I'm surprised that Linux pread does not behave the same way across platforms though. Or maybe it only happens with certain Linux versions. What version are you using?

Douglas Mencken wrote:

OK, good.

You didn't mention which platform you are on; from the above Makefile hunk, however, I can deduce that you are on *some* version of Linux.

Hmm, it doesn't seem too likely that your pread() is thread-unsafe (possible, just unlikely), so it could be a more fundamental problem with the threaded index-pack code (ie commit b8a2486f1 et. seq.).

However, as a first step could you try running the test program (given below) on your system to determine if your pread() is thread-safe or not. (gcc -I. -o test-pread test-pread.c; ./test-pread)

Also, what is the output of "uname -a".

ATB, Ramsay Jones

-- >8 -- #include "git-compat-util.h" #include "thread-utils.h"

#define DATA_FILE "junk.data" #define MAX_DATA 256 * 1024 #define NUM_THREADS 3 #define TRIALS 500000

struct thread_data {
	pthread_t t;
	int fd;
	int cnt;
	int fails;
	unsigned long n;
};

static struct thread_data t[NUM_THREADS+1];

int create_data_file(void)
{
	int i, fd = open(DATA_FILE, O_CREAT | O_TRUNC | O_WRONLY, 0600);
	if (fd < 0)
		return -1;
	for (i = 0; i < MAX_DATA; i++)
		if (write(fd, &i, sizeof(int)) < 0) {
			close(fd);
			unlink(DATA_FILE);
			return -1;
		}
	close(fd);
	return 0;
}

void *read_thread(void *data)
{
	struct thread_data *d = (struct thread_data *)data;
	int i, j, rd;
	for (i = 0; i < TRIALS; i += MAX_DATA) {
		for (j = 0; j < MAX_DATA; j++) {
			ssize_t sz = read(d->fd, &rd, sizeof(int));
			if (sz < 0 || rd != j)
				d->fails++;
			d->cnt++;
		}
		lseek(d->fd, 0, SEEK_SET);
	}
	return NULL;
}

void *pread_thread(void *data)
{
	struct thread_data *d = (struct thread_data *)data;
	int i, j, rd;
	for (i = 0; i < TRIALS; i++) {
		ssize_t sz;
		d->n = d->n * 1103515245 + 12345;
		j = d->n % MAX_DATA;
		sz = pread(d->fd, &rd, sizeof(int), j * sizeof(int));
		if (sz < 0 || rd != j)
			d->fails++;
		d->cnt++;
	}
	return NULL;
}

int main(int argc, char *argv[])
{
	int fd, i;

	if (create_data_file() < 0) {
		printf("can't create data file\n");
		return 1;
	}

	if ((fd = open(DATA_FILE, O_RDONLY)) < 0) {
		printf("can't open data file\n");
		unlink(DATA_FILE);
		return 1;
	}

	for (i = 0; i < NUM_THREADS+1; i++) {
		int ret;

		t[i].fd = fd;
		t[i].cnt = 0;
		t[i].fails = 0;
		t[i].n = i * 16381;
		ret = pthread_create(&t[i].t, NULL,
				(i == 0) ? read_thread : pread_thread,
				&t[i]);
		if (ret) {
			printf("can't create thread %d (%s)\n", i, strerror(ret));
			unlink(DATA_FILE);
			return 1;
		}
	}

	for (i = 0; i < NUM_THREADS+1; i++)
		pthread_join(t[i].t, NULL);
	close(fd);

	for (i = 0; i < NUM_THREADS+1; i++)
		printf("%2d: trials %d, failed %d\n", i, t[i].cnt, t[i].fails);

	unlink(DATA_FILE);
	return 0;
}

On 13.11.12 19:55, Ramsay Jones wrote:

This is interesting. I had the same problem on a PowerPC (Old PowerBook G4 running Linux).

Using NO_THREAD_SAFE_PREAD helped, thanks for the hint. (After recompiling without NO_THREAD_SAFE_PREAD I could clone from this machine again, so the problem is not really reproducable)

Are there more people running PowerPC (on the server side) ? /Torsten

Torsten Bögershausen <tboegi@web.de> writes:

I cannot reproduce the problem (on openSUSE 12.2).

Andreas.

You do need multiple CPU/multi-core machine, as I got it.

Douglas Mencken <dougmencken@gmail.com> writes:

Which is what I have.

Andreas.

Then try to build *vanilla* git 1.8.0, not OpenSuSE's one (with a lot of patches inside srcrpm).

Douglas Mencken <dougmencken@gmail.com> writes:

Which is what I did.

Which lot of patches?

Andreas.

Torsten Bögershausen wrote:

Yes, the failures would be intermittent (and often not easily reproducible). The threaded index-pack code did not fail for me on cygwin at all during development, including tests, but failed immediately I installed v1.7.11. On real repositories, it failed intermittently. On some repos it always failed, on some it never failed and on some others it would sometimes fail, sometimes not.

ATB, Ramsay Jones

Then why did you commit it? If it has so high random failure rate.

My host, build and target is: powerpc-linux-gnu