{"thread":{"id":"30908","subject":"[PATCH v2 0/1] v1.7.11 index-pack failures on Cygwin","startedAt":"2012-06-26T18:04:38Z","lastAt":"2012-06-26T18:04:38Z","messageCount":1,"participants":["Ramsay Jones"],"isPatch":true,"patchVersion":2,"patchTotal":1},"messages":[{"id":"194300","messageId":"4FE9F9B6.4080805@ramsay1.demon.co.uk","threadId":"30908","inReplyTo":null,"subject":"[PATCH v2 0/1] v1.7.11 index-pack failures on Cygwin","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsay1.demon.co.uk","sentAt":"2012-06-26T18:04:38Z","receivedAt":"2012-06-26T18:04:38Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"[This is exactly the same as last time, except I remembered to post\nit to the list this time! The patch is different, of course :-) ]\n\nHi Junio,\n\nHaving just installed v1.7.11 on cygwin, I discover that git is\nfailing intermittently while fetching:\n\n    ramsay $ cd ../ffmpeg/\n    ramsay $ git fetch origin\n    remote: Counting objects: 569, done.\n    remote: Compressing objects: 100% (363/363), done.\n    remote: Total 363 (delta 294), reused 0 (delta 0)Receiving objects:   9% (33/363\n    ), 12.00 KiB | 5 KiB/s\n    Receiving objects: 100% (363/363), 69.27 KiB | 4 KiB/s, done.\n    error: inflate: data stream error (incorrect header check)\n    fatal: serious inflate inconsistency\n    fatal: index-pack failed\n    ramsay $ git fetch origin\n    remote: Counting objects: 569, done.\n    remote: Compressing objects: 100% (363/363), done.\n    Receiving objects:   9% (33/363), 12.00 KiB | 5 KiB/s   remote: Total 363 (delta\n    Receiving objects: 100% (363/363), 69.27 KiB | 5 KiB/s, done.\n    fatal: premature end of pack file, 1111 bytes missing\n    fatal: serious inflate inconsistency\n    fatal: index-pack failed\n    ramsay $ \n\nI immediately suspected the new threaded index-pack. In particular, the\nthread-safely (or otherwise) of pread(); ie exactly the same problem\nreported by Johannes on MinGW, which lead to commit b038a61 (\"index-pack:\ndisable threading if NO_PREAD is defined\", 06-05-2012).\n\nNote that I did not see any test failures on MinGW or Cygwin (still don't!)\nrelating to index-pack. However, I had a look at the cygwin source and\nfound that the cygwin pread() implementation for disk files is essentially\nthe same as compat/pread.c (look in ../winsup/cygwin/fhandler_disk_file.cc).\n\nAlso, I hacked up a small test (attached below) to confirm that pread() is\nnot thread-safe:\n\n    ramsay $ gcc -I. -o test-pread test-pread.c\n    ramsay $ ./test-pread.exe\n     0: trials 524288, failed 524283\n     1: trials 500000, failed 191184\n     2: trials 500000, failed 195607\n     3: trials 500000, failed 192931\n    ramsay $\n\nI had a \"fix pread() on MinGW\" item low down on my todo list; I guess I need\nto address cygwin too! :D\n\nAs a short term fix, I've created a patch to disable threading in index-pack\non cygwin (ala commit b038a61).\n\nHTH\n\nATB,\nRamsay Jones\n\n\n-- >8 --\n#include \"git-compat-util.h\"\n#include \"thread-utils.h\"\n\n#define DATA_FILE \"junk.data\"\n#define MAX_DATA 256 * 1024\n#define NUM_THREADS 3\n#define TRIALS 500000\n\nstruct thread_data {\n\tpthread_t t;\n\tint fd;\n\tint cnt;\n\tint fails;\n\tunsigned long n;\n};\n\nstatic struct thread_data t[NUM_THREADS+1];\n\nint create_data_file(void)\n{\n\tint i, fd = open(DATA_FILE, O_CREAT | O_TRUNC | O_WRONLY, 0600);\n\tif (fd < 0)\n\t\treturn -1;\n\tfor (i = 0; i < MAX_DATA; i++)\n\t\tif (write(fd, &i, sizeof(int)) < 0) {\n\t\t\tclose(fd);\n\t\t\tunlink(DATA_FILE);\n\t\t\treturn -1;\n\t\t}\n\tclose(fd);\n\treturn 0;\n}\n\nvoid *read_thread(void *data)\n{\n\tstruct thread_data *d = (struct thread_data *)data;\n\tint i, j, rd;\n\tfor (i = 0; i < TRIALS; i += MAX_DATA) {\n\t\tfor (j = 0; j < MAX_DATA; j++) {\n\t\t\tssize_t sz = read(d->fd, &rd, sizeof(int));\n\t\t\tif (sz < 0 || rd != j)\n\t\t\t\td->fails++;\n\t\t\td->cnt++;\n\t\t}\n\t\tlseek(d->fd, 0, SEEK_SET);\n\t}\n\treturn NULL;\n}\n\nvoid *pread_thread(void *data)\n{\n\tstruct thread_data *d = (struct thread_data *)data;\n\tint i, j, rd;\n\tfor (i = 0; i < TRIALS; i++) {\n\t\tssize_t sz;\n\t\td->n = d->n * 1103515245 + 12345;\n\t\tj = d->n % MAX_DATA;\n\t\tsz = pread(d->fd, &rd, sizeof(int), j * sizeof(int));\n\t\tif (sz < 0 || rd != j)\n\t\t\td->fails++;\n\t\td->cnt++;\n\t}\n\treturn NULL;\n}\n\nint main(int argc, char *argv[])\n{\n\tint fd, i;\n\n\tif (create_data_file() < 0) {\n\t\tprintf(\"can't create data file\\n\");\n\t\treturn 1;\n\t}\n\n\tif ((fd = open(DATA_FILE, O_RDONLY)) < 0) {\n\t\tprintf(\"can't open data file\\n\");\n\t\tunlink(DATA_FILE);\n\t\treturn 1;\n\t}\n\n\tfor (i = 0; i < NUM_THREADS+1; i++) {\n\t\tint ret;\n\n\t\tt[i].fd = fd;\n\t\tt[i].cnt = 0;\n\t\tt[i].fails = 0;\n\t\tt[i].n = i * 16381;\n\t\tret = pthread_create(&t[i].t, NULL,\n\t\t\t\t(i == 0) ? read_thread : pread_thread,\n\t\t\t\t&t[i]);\n\t\tif (ret) {\n\t\t\tprintf(\"can't create thread %d (%s)\\n\", i, strerror(ret));\n\t\t\tunlink(DATA_FILE);\n\t\t\treturn 1;\n\t\t}\n\t}\n\n\tfor (i = 0; i < NUM_THREADS+1; i++)\n\t\tpthread_join(t[i].t, NULL);\n\tclose(fd);\n\n\tfor (i = 0; i < NUM_THREADS+1; i++)\n\t\tprintf(\"%2d: trials %d, failed %d\\n\", i, t[i].cnt, t[i].fails);\n\n\tunlink(DATA_FILE);\n\treturn 0;\n}\n"}]}