{"thread":{"id":"18151","subject":"[PATCH 0/2] 2GB fixes for Windows","startedAt":"2009-03-05T16:05:06Z","lastAt":"2009-03-06T09:49:25Z","messageCount":10,"participants":["Johannes Schindelin","Jeff King","Johannes Sixt","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"107073","messageId":"cover.1236268730u.git.johannes.schindelin@gmx.de","threadId":"18151","inReplyTo":null,"subject":"[PATCH 0/2] 2GB fixes for Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-05T16:05:06Z","receivedAt":"2009-03-05T16:05:06Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"On Windows, we can actually have files larger than 2 gigabyte, just not \nusing off_t, but off64_t instead.\n\nThis came up as issue 194 on the msysGit tracker, and as I first brushed \nthe people off saying that it is not an msysGit issue, by way of an \napology, I started working on this myself.\n\nThe first patch adds a test for cloning repositories larger than 2 \ngigabyte, which is disabled by default, since it is quite expensive (both \nin terms of time and in terms of space), and since it must fail when the \nunderlying filesystem does not allow files larger than 2 gigabyte.\n\nThe second patch convinces msysGit (AKA the MinGW port of Git) to make use \nof the 64-bit file offsets.\n\nJohannes Schindelin (2):\n Add an (optional, since expensive) test for >2g clones\n MinGW: 64-bit file offsets\n\n compat/mingw.c | 8 +++++---\n compat/mingw.h | 5 ++++-\n t/t5705-clone-2gb.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 54 insertions(+), 4 deletions(-)\n create mode 100755 t/t5705-clone-2gb.sh\n"},{"id":"107074","messageId":"96a26f3a883890b3e56c867e8183618784837d4d.1236268730u.git.johannes.schindelin@gmx.de","threadId":"18151","inReplyTo":"cover.1236268730u.git.johannes.schindelin@gmx.de","subject":"[PATCH 1/2] Add an (optional, since expensive) test for >2g clones","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-05T16:05:09Z","receivedAt":"2009-03-05T16:05:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Define GIT_TEST_CLONE_2GB=t if you want the test not to be skipped.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t5705-clone-2gb.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 45 insertions(+), 0 deletions(-)\n create mode 100755 t/t5705-clone-2gb.sh\n\ndiff --git a/t/t5705-clone-2gb.sh b/t/t5705-clone-2gb.sh\nnew file mode 100755\nindex 0000000..dd4a8ec\n--- /dev/null\n+++ b/t/t5705-clone-2gb.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+test_description='Test cloning a repository larger than 2 gigabyte'\n+. ./test-lib.sh\n+\n+test -z \"$GIT_TEST_CLONE_2GB\" &&\n+say \"Skipping expensive 2GB clone test; enable it with GIT_TEST_CLONE_2GB=t\" &&\n+test_done &&\n+exit\n+\n+test_expect_success 'setup' '\n+\n+\tgit config pack.compression 0 &&\n+\tgit config pack.depth 0 &&\n+\tblobsize=$((20*1024*1024))\n+\tblobcount=$((2*1024*1024*1024/$blobsize+1))\n+\ti=1\n+\t(while test $i -le $blobcount\n+\t do\n+\t\tprintf \"Generating blob $i/$blobcount\\r\" >&2 &&\n+\t\tprintf \"blob\\nmark :$i\\ndata $blobsize\\n\" &&\n+\t\t#test-genrandom $i $blobsize &&\n+\t\tprintf \"%-${blobsize}s\" $i &&\n+\t\techo \"M 100644 :$i $i\" >> commit\n+\t\ti=$(($i+1)) ||\n+\t\techo $? > exit-status\n+\t done &&\n+\t echo \"commit refs/heads/master\" &&\n+\t echo \"author A U Thor 123456789 +0000\" &&\n+\t echo \"committer C O Mitter 123456789 +0000\" &&\n+\t echo \"data 5\" &&\n+\t echo \">2gb\" &&\n+\t cat commit) |\n+\tgit fast-import &&\n+\ttest ! -f exit-status\n+\n+'\n+\n+test_expect_success 'clone' '\n+\n+\tgit clone --bare --no-hardlinks . clone\n+\n+'\n+\n+test_done\n-- \n1.6.2.rc1.493.g27ad8\n"},{"id":"107075","messageId":"ae01b6dfcd430694dad008bbf201ee1490b071a1.1236268730u.git.johannes.schindelin@gmx.de","threadId":"18151","inReplyTo":"cover.1236268730u.git.johannes.schindelin@gmx.de","subject":"[PATCH 2/2] MinGW: 64-bit file offsets","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-05T16:05:12Z","receivedAt":"2009-03-05T16:05:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"The type 'off_t' should be used everywhere so that the bit-depth of that\ntype can be adjusted in the standard C library, and you just need to\nrecompile your program to benefit from the extended precision.\n\nOnly that it was not done that way in the MS runtime library.\n\nThis patch reroutes off_t to off64_t and provides the other necessary\nchanges so that finally, clones larger than 2 gigabyte work on Windows\n(provided you are on a file system that allows files larger than 2gb).\n\nInitial patch by Sickboy .\n\nSigned-off-by: Johannes Schindelin \n---\n compat/mingw.c | 8 +++++---\n compat/mingw.h | 5 ++++-\n 2 files changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex 3dbe6a7..27bcf3f 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -46,7 +46,8 @@ static int do_lstat(const char *file_name, struct stat *buf)\n \t\tbuf->st_uid = 0;\n \t\tbuf->st_nlink = 1;\n \t\tbuf->st_mode = file_attr_to_st_mode(fdata.dwFileAttributes);\n-\t\tbuf->st_size = fdata.nFileSizeLow; /* Can't use nFileSizeHigh, since it's not a stat64 */\n+\t\tbuf->st_size = fdata.nFileSizeLow |\n+\t\t\t(((off_t)fdata.nFileSizeHigh)<<32);\n \t\tbuf->st_dev = buf->st_rdev = 0; /* not used by Git */\n \t\tbuf->st_atime = filetime_to_time_t(&(fdata.ftLastAccessTime));\n \t\tbuf->st_mtime = filetime_to_time_t(&(fdata.ftLastWriteTime));\n@@ -101,7 +102,7 @@ int mingw_fstat(int fd, struct stat *buf)\n \t}\n \t/* direct non-file handles to MS's fstat() */\n \tif (GetFileType(fh) != FILE_TYPE_DISK)\n-\t\treturn fstat(fd, buf);\n+\t\treturn _fstati64(fd, buf);\n \n \tif (GetFileInformationByHandle(fh, &fdata)) {\n \t\tbuf->st_ino = 0;\n@@ -109,7 +110,8 @@ int mingw_fstat(int fd, struct stat *buf)\n \t\tbuf->st_uid = 0;\n \t\tbuf->st_nlink = 1;\n \t\tbuf->st_mode = file_attr_to_st_mode(fdata.dwFileAttributes);\n-\t\tbuf->st_size = fdata.nFileSizeLow; /* Can't use nFileSizeHigh, since it's not a stat64 */\n+\t\tbuf->st_size = fdata.nFileSizeLow |\n+\t\t\t(((off_t)fdata.nFileSizeHigh)<<32);\n \t\tbuf->st_dev = buf->st_rdev = 0; /* not used by Git */\n \t\tbuf->st_atime = filetime_to_time_t(&(fdata.ftLastAccessTime));\n \t\tbuf->st_mtime = filetime_to_time_t(&(fdata.ftLastWriteTime));\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex a255898..cb9c4d4 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -163,11 +163,14 @@ int mingw_rename(const char*, const char*);\n /* Use mingw_lstat() instead of lstat()/stat() and\n * mingw_fstat() instead of fstat() on Windows.\n */\n+#define off_t off64_t\n+#define stat _stati64\n+#define lseek _lseeki64\n int mingw_lstat(const char *file_name, struct stat *buf);\n int mingw_fstat(int fd, struct stat *buf);\n #define fstat mingw_fstat\n #define lstat mingw_lstat\n-#define stat(x,y) mingw_lstat(x,y)\n+#define stat64(x,y) mingw_lstat(x,y)\n \n int mingw_utime(const char *file_name, const struct utimbuf *times);\n #define utime mingw_utime\n-- \n1.6.2.rc1.493.g27ad8\n"},{"id":"107079","messageId":"20090305164153.GD4213@coredump.intra.peff.net","threadId":"18151","inReplyTo":"96a26f3a883890b3e56c867e8183618784837d4d.1236268730u.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 1/2] Add an (optional, since expensive) test for >2g clones","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-03-05T16:41:54Z","receivedAt":"2009-03-05T16:41:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 05, 2009 at 05:05:09PM +0100, Johannes Schindelin wrote:\n\n> +\t(while test $i -le $blobcount\n> +\t do\n> +\t\tprintf \"Generating blob $i/$blobcount\\r\" >&2 &&\n> +\t\tprintf \"blob\\nmark :$i\\ndata $blobsize\\n\" &&\n> +\t\t#test-genrandom $i $blobsize &&\n> +\t\tprintf \"%-${blobsize}s\" $i &&\n\nLeftover cruft using genrandom? (I'm guessing you tried random at first\nto avoid compression, but I think your pack.compression=0 technique is\nmore sensible).\n\n-Peff\n"},{"id":"107081","messageId":"alpine.DEB.1.00.0903051754260.6524@intel-tinevez-2-302","threadId":"18151","inReplyTo":"20090305164153.GD4213@coredump.intra.peff.net","subject":"Re: [PATCH 1/2] Add an (optional, since expensive) test for >2g clones","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-05T16:58:23Z","receivedAt":"2009-03-05T16:58:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 5 Mar 2009, Jeff King wrote:\n\n> On Thu, Mar 05, 2009 at 05:05:09PM +0100, Johannes Schindelin wrote:\n> \n> > +\t(while test $i -le $blobcount\n> > +\t do\n> > +\t\tprintf \"Generating blob $i/$blobcount\\r\" >&2 &&\n> > +\t\tprintf \"blob\\nmark :$i\\ndata $blobsize\\n\" &&\n> > +\t\t#test-genrandom $i $blobsize &&\n> > +\t\tprintf \"%-${blobsize}s\" $i &&\n> \n> Leftover cruft using genrandom? (I'm guessing you tried random at first\n> to avoid compression, but I think your pack.compression=0 technique is\n> more sensible).\n\nActually, I left it in on purpose. Yes, it happens to work right now, as \nthe packs are built with 0 compression and with 0 deltification.\n\nHowever, there might be a day when we cannot guarantee anymore that a \nsingle number padded with spaces to a certain width really makes the pack \ngrow by that many bytes. Then we would need something like test-genrandom \n(which is substantially slower, though).\n\nInteresting random note: it took me quite a while to figure out that both \npack.compression and pack.depth need to be set to 0. In hindsight, it is \nobvious, but that does not really help the time I lost due to Windows' \nslowness.\n\nCiao,\nDscho\n"},{"id":"107086","messageId":"20090305185004.GB19244@coredump.intra.peff.net","threadId":"18151","inReplyTo":"alpine.DEB.1.00.0903051754260.6524@intel-tinevez-2-302","subject":"Re: [PATCH 1/2] Add an (optional, since expensive) test for >2g clones","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-03-05T18:50:04Z","receivedAt":"2009-03-05T18:50:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 05, 2009 at 05:58:23PM +0100, Johannes Schindelin wrote:\n\n> > Leftover cruft using genrandom? (I'm guessing you tried random at first\n> > to avoid compression, but I think your pack.compression=0 technique is\n> > more sensible).\n> \n> Actually, I left it in on purpose. Yes, it happens to work right now, as \n> the packs are built with 0 compression and with 0 deltification.\n\nFair enough. A comment in the commit message or in the code might have\nmade that more clear, though...\n\n-Peff\n"},{"id":"107102","messageId":"200903052153.45398.j6t@kdbg.org","threadId":"18151","inReplyTo":"ae01b6dfcd430694dad008bbf201ee1490b071a1.1236268730u.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 2/2] MinGW: 64-bit file offsets","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2009-03-05T20:53:45Z","receivedAt":"2009-03-05T20:53:45Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Donnerstag, 5. März 2009, Johannes Schindelin wrote:\n> The type 'off_t' should be used everywhere so that the bit-depth of that\n> type can be adjusted in the standard C library, and you just need to\n> recompile your program to benefit from the extended precision.\n>\n> Only that it was not done that way in the MS runtime library.\n>\n> This patch reroutes off_t to off64_t and provides the other necessary\n> changes so that finally, clones larger than 2 gigabyte work on Windows\n> (provided you are on a file system that allows files larger than 2gb).\n>\n> Initial patch by Sickboy .\n>\n> Signed-off-by: Johannes Schindelin \n\nAcked-by: Johannes Sixt \n\n-- Hannes\n"},{"id":"107118","messageId":"7vfxhry3da.fsf@gitster.siamese.dyndns.org","threadId":"18151","inReplyTo":"96a26f3a883890b3e56c867e8183618784837d4d.1236268730u.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 1/2] Add an (optional, since expensive) test for >2g clones","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-03-05T23:08:33Z","receivedAt":"2009-03-05T23:08:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> +test_expect_success 'setup' '\n> +\n> +\tgit config pack.compression 0 &&\n> +\tgit config pack.depth 0 &&\n> +\tblobsize=$((20*1024*1024))\n> +\tblobcount=$((2*1024*1024*1024/$blobsize+1))\n> +\ti=1\n\nWhat happened to the && chain?\n"},{"id":"107173","messageId":"alpine.DEB.1.00.0903061048220.10279@pacific.mpi-cbg.de","threadId":"18151","inReplyTo":"7vfxhry3da.fsf@gitster.siamese.dyndns.org","subject":"[PATCH v2 1/2] Add an (optional, since expensive) test for >2gb clones","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-06T09:48:53Z","receivedAt":"2009-03-06T09:48:53Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"\nDefine GIT_TEST_CLONE_2GB=t if you want the test not to be skipped.\n\nThe test works by constructing a repository larger than 2gb, and then\ncloning it.\n\nThe repository is forced larger than 2gb by setting compression and\ndelta depth to zero, and then adding just enough unique objects of\na given size.\n\nThe objects consist of a running decimal number in ASCII, padded by\nspaces. Should that break in the future, e.g. when pack v4 becomes\ndefault, there is a commented-out call to test-genrandom which can be\nsubstituted, but that uses more cycles than the current method.\n\nSigned-off-by: Johannes Schindelin \n---\n\n\tChanges since v1: fixed && chain, better commit message.\n\n t/t5705-clone-2gb.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 45 insertions(+), 0 deletions(-)\n create mode 100755 t/t5705-clone-2gb.sh\n\ndiff --git a/t/t5705-clone-2gb.sh b/t/t5705-clone-2gb.sh\nnew file mode 100755\nindex 0000000..9f52154\n--- /dev/null\n+++ b/t/t5705-clone-2gb.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+test_description='Test cloning a repository larger than 2 gigabyte'\n+. ./test-lib.sh\n+\n+test -z \"$GIT_TEST_CLONE_2GB\" &&\n+say \"Skipping expensive 2GB clone test; enable it with GIT_TEST_CLONE_2GB=t\" &&\n+test_done &&\n+exit\n+\n+test_expect_success 'setup' '\n+\n+\tgit config pack.compression 0 &&\n+\tgit config pack.depth 0 &&\n+\tblobsize=$((20*1024*1024)) &&\n+\tblobcount=$((2*1024*1024*1024/$blobsize+1)) &&\n+\ti=1 &&\n+\t(while test $i -le $blobcount\n+\t do\n+\t\tprintf \"Generating blob $i/$blobcount\\r\" >&2 &&\n+\t\tprintf \"blob\\nmark :$i\\ndata $blobsize\\n\" &&\n+\t\t#test-genrandom $i $blobsize &&\n+\t\tprintf \"%-${blobsize}s\" $i &&\n+\t\techo \"M 100644 :$i $i\" >> commit\n+\t\ti=$(($i+1)) ||\n+\t\techo $? > exit-status\n+\t done &&\n+\t echo \"commit refs/heads/master\" &&\n+\t echo \"author A U Thor 123456789 +0000\" &&\n+\t echo \"committer C O Mitter 123456789 +0000\" &&\n+\t echo \"data 5\" &&\n+\t echo \">2gb\" &&\n+\t cat commit) |\n+\tgit fast-import &&\n+\ttest ! -f exit-status\n+\n+'\n+\n+test_expect_success 'clone' '\n+\n+\tgit clone --bare --no-hardlinks . clone\n+\n+'\n+\n+test_done\n-- \n1.6.2.240.g23c7\n"},{"id":"107174","messageId":"alpine.DEB.1.00.0903061049090.10279@pacific.mpi-cbg.de","threadId":"18151","inReplyTo":"alpine.DEB.1.00.0903061048220.10279@pacific.mpi-cbg.de","subject":"[PATCH v2 2/2] MinGW: 64-bit file offsets","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-06T09:49:25Z","receivedAt":"2009-03-06T09:49:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"\nThe type 'off_t' should be used everywhere so that the bit-depth of that\ntype can be adjusted in the standard C library, and you just need to\nrecompile your program to benefit from the extended precision.\n\nOnly that it was not done that way in the MS runtime library.\n\nThis patch reroutes off_t to off64_t and provides the other necessary\nchanges so that finally, clones larger than 2 gigabyte work on Windows\n(provided you are on a file system that allows files larger than 2gb).\n\nInitial patch by Sickboy .\n\nSigned-off-by: Johannes Schindelin \nAcked-by: Johannes Sixt \n---\n\n\tUnchanged since v1 (therefore I dared adding Hannes' ACK).\n\n compat/mingw.c | 8 +++++---\n compat/mingw.h | 5 ++++-\n 2 files changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex 3dbe6a7..27bcf3f 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -46,7 +46,8 @@ static int do_lstat(const char *file_name, struct stat *buf)\n \t\tbuf->st_uid = 0;\n \t\tbuf->st_nlink = 1;\n \t\tbuf->st_mode = file_attr_to_st_mode(fdata.dwFileAttributes);\n-\t\tbuf->st_size = fdata.nFileSizeLow; /* Can't use nFileSizeHigh, since it's not a stat64 */\n+\t\tbuf->st_size = fdata.nFileSizeLow |\n+\t\t\t(((off_t)fdata.nFileSizeHigh)<<32);\n \t\tbuf->st_dev = buf->st_rdev = 0; /* not used by Git */\n \t\tbuf->st_atime = filetime_to_time_t(&(fdata.ftLastAccessTime));\n \t\tbuf->st_mtime = filetime_to_time_t(&(fdata.ftLastWriteTime));\n@@ -101,7 +102,7 @@ int mingw_fstat(int fd, struct stat *buf)\n \t}\n \t/* direct non-file handles to MS's fstat() */\n \tif (GetFileType(fh) != FILE_TYPE_DISK)\n-\t\treturn fstat(fd, buf);\n+\t\treturn _fstati64(fd, buf);\n \n \tif (GetFileInformationByHandle(fh, &fdata)) {\n \t\tbuf->st_ino = 0;\n@@ -109,7 +110,8 @@ int mingw_fstat(int fd, struct stat *buf)\n \t\tbuf->st_uid = 0;\n \t\tbuf->st_nlink = 1;\n \t\tbuf->st_mode = file_attr_to_st_mode(fdata.dwFileAttributes);\n-\t\tbuf->st_size = fdata.nFileSizeLow; /* Can't use nFileSizeHigh, since it's not a stat64 */\n+\t\tbuf->st_size = fdata.nFileSizeLow |\n+\t\t\t(((off_t)fdata.nFileSizeHigh)<<32);\n \t\tbuf->st_dev = buf->st_rdev = 0; /* not used by Git */\n \t\tbuf->st_atime = filetime_to_time_t(&(fdata.ftLastAccessTime));\n \t\tbuf->st_mtime = filetime_to_time_t(&(fdata.ftLastWriteTime));\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex a255898..cb9c4d4 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -163,11 +163,14 @@ int mingw_rename(const char*, const char*);\n /* Use mingw_lstat() instead of lstat()/stat() and\n * mingw_fstat() instead of fstat() on Windows.\n */\n+#define off_t off64_t\n+#define stat _stati64\n+#define lseek _lseeki64\n int mingw_lstat(const char *file_name, struct stat *buf);\n int mingw_fstat(int fd, struct stat *buf);\n #define fstat mingw_fstat\n #define lstat mingw_lstat\n-#define stat(x,y) mingw_lstat(x,y)\n+#define stat64(x,y) mingw_lstat(x,y)\n \n int mingw_utime(const char *file_name, const struct utimbuf *times);\n #define utime mingw_utime\n-- \n1.6.2.240.g23c7\n"}]}