{"thread":{"id":"18368","subject":"[PATCH] MinGW: implement mmap","startedAt":"2009-03-18T07:36:02Z","lastAt":"2009-03-18T11:18:37Z","messageCount":2,"participants":["Johannes Sixt","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"108349","messageId":"49C0A462.3060902@kdbg.org","threadId":"18368","inReplyTo":null,"subject":"[PATCH] MinGW: implement mmap","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2009-03-18T07:36:02Z","receivedAt":"2009-03-18T07:36:02Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Janos Laube <janos.dev@gmail.com>\nDate: Fri, 13 Mar 2009 16:50:45 +0100\n\nAdd USE_WIN32_MMAP which triggers the use of windows' native\nfile memory mapping functionality in git_mmap()/git_munmap() functions.\n\nAs git functions currently use mmap with MAP_PRIVATE set only, this\nimplementation supports only that mode for now.\n\nOn Windows, offsets for memory mapped files need to match the allocation\ngranularity. Take this into account when calculating the packed git-\nwindowsize and file offsets. At the moment, the only function which makes\nuse of offsets in conjunction with mmap is use_pack() in sha1-file.c.\n\nGit fast-import's code path tries to map a portion of the temporary\npackfile that exceeds the current filesize, i.e. offset+length is\ngreater than the filesize. The NO_MMAP code worked with that since pread()\njust reads the file content until EOF and returns gracefully, while\nMapViewOfFile() aborts the mapping and returns 'Access Denied'.\nWorking around that by determining the filesize and adjusting the length\nparameter.\n\nSigned-off-by: Janos Laube <janos.dev@gmail.com>\nSigned-off-by: Johannes Sixt <j6t@kdbg.org>\n---\n  Makefile           |    7 +++++-\n  compat/mingw.h     |    5 ++++\n  compat/win32mmap.c |   53 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n  git-compat-util.h  |   12 ++++++++--\n  4 files changed, 73 insertions(+), 4 deletions(-)\n  create mode 100644 compat/win32mmap.c\n\ndiff --git a/Makefile b/Makefile\nindex 1087884..38e75e5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -776,7 +776,6 @@ ifneq (,$(findstring CYGWIN,$(uname_S)))\n  \tCOMPAT_OBJS += compat/cygwin.o\n  endif\n  ifneq (,$(findstring MINGW,$(uname_S)))\n-\tNO_MMAP = YesPlease\n  \tNO_PREAD = YesPlease\n  \tNO_OPENSSL = YesPlease\n  \tNO_CURL = YesPlease\n@@ -799,6 +798,7 @@ ifneq (,$(findstring MINGW,$(uname_S)))\n  \tRUNTIME_PREFIX = YesPlease\n  \tNO_POSIX_ONLY_PROGRAMS = YesPlease\n  \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\n+\tUSE_WIN32_MMAP = YesPlease\n  \tCOMPAT_CFLAGS += -D__USE_MINGW_ACCESS -DNOGDI -Icompat -Icompat/regex -Icompat/fnmatch\n  \tCOMPAT_CFLAGS += -DSNPRINTF_SIZE_CORR=1\n  \tCOMPAT_CFLAGS += -DSTRIP_EXTENSION=\\\".exe\\\"\n@@ -967,6 +967,11 @@ endif\n  ifdef NO_MMAP\n  \tCOMPAT_CFLAGS += -DNO_MMAP\n  \tCOMPAT_OBJS += compat/mmap.o\n+else\n+\tifdef USE_WIN32_MMAP\n+\t\tCOMPAT_CFLAGS += -DUSE_WIN32_MMAP\n+\t\tCOMPAT_OBJS += compat/win32mmap.o\n+\tendif\n  endif\n  ifdef NO_PREAD\n  \tCOMPAT_CFLAGS += -DNO_PREAD\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 6e24686..f5da647 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -160,6 +160,11 @@ int mingw_connect(int sockfd, struct sockaddr *sa, size_t sz);\n  int mingw_rename(const char*, const char*);\n  #define rename mingw_rename\n\n+#ifdef USE_WIN32_MMAP\n+int mingw_getpagesize(void);\n+#define getpagesize mingw_getpagesize\n+#endif\n+\n  /* Use mingw_lstat() instead of lstat()/stat() and\n   * mingw_fstat() instead of fstat() on Windows.\n   */\ndiff --git a/compat/win32mmap.c b/compat/win32mmap.c\nnew file mode 100644\nindex 0000000..66314b8\n--- /dev/null\n+++ b/compat/win32mmap.c\n@@ -0,0 +1,53 @@\n+#include \"../git-compat-util.h\"\n+\n+/* Note that this doesn't return the actual pagesize, but\n+ * the allocation granularity. If future Windows specific git code\n+ * needs the real getpagesize function, we need to find another solution.\n+ */\n+int mingw_getpagesize(void)\n+{\n+\tSYSTEM_INFO si;\n+\tGetSystemInfo(&si);\n+\treturn si.dwAllocationGranularity;\n+}\n+\n+void *git_mmap\n+(void *start, size_t length, int prot, int flags, int fd, off_t offset)\n+{\n+\tHANDLE hmap;\n+\tvoid *temp;\n+\tsize_t len;\n+\tstruct stat st;\n+\tuint64_t o = offset;\n+\tuint32_t l = o & 0xFFFFFFFF;\n+\tuint32_t h = (o >> 32) & 0xFFFFFFFF;\n+\n+\tif (!fstat(fd, &st))\n+\t\tlen = xsize_t(st.st_size);\n+\telse\n+\t\tdie(\"mmap: could not determine filesize\");\n+\n+\tif ((length + offset) > len)\n+\t\tlength = len - offset;\n+\n+\tif (!(flags & MAP_PRIVATE))\n+\t\tdie(\"Invalid usage of mmap when built with USE_WIN32_MMAP\");\n+\n+\thmap = CreateFileMapping((HANDLE)_get_osfhandle(fd), 0, PAGE_WRITECOPY,\n+\t\t0, 0, 0);\n+\n+\tif (!hmap)\n+\t\treturn MAP_FAILED;\n+\n+\ttemp = MapViewOfFileEx(hmap, FILE_MAP_COPY, h, l, length, start);\n+\n+\tif (!CloseHandle(hmap))\n+\t\twarning(\"unable to close file mapping handle\\n\");\n+\n+\treturn temp ? temp : MAP_FAILED;\n+}\n+\n+int git_munmap(void *start, size_t length)\n+{\n+\treturn !UnmapViewOfFile(start);\n+}\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 878d83d..1eef4eb 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -166,7 +166,7 @@ static inline const char *skip_prefix(const char *str, const char *prefix)\n  \treturn strncmp(str, prefix, len) ? NULL : str + len;\n  }\n\n-#ifdef NO_MMAP\n+#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n\n  #ifndef PROT_READ\n  #define PROT_READ 1\n@@ -180,13 +180,19 @@ static inline const char *skip_prefix(const char *str, const char *prefix)\n  extern void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n  extern int git_munmap(void *start, size_t length);\n\n+#else /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#include <sys/mman.h>\n+\n+#endif /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#ifdef NO_MMAP\n+\n  /* This value must be multiple of (pagesize * 2) */\n  #define DEFAULT_PACKED_GIT_WINDOW_SIZE (1 * 1024 * 1024)\n\n  #else /* NO_MMAP */\n\n-#include <sys/mman.h>\n-\n  /* This value must be multiple of (pagesize * 2) */\n  #define DEFAULT_PACKED_GIT_WINDOW_SIZE \\\n  \t(sizeof(void*) >= 8 \\\n-- \n1.6.2.rc2.971.g14d5\n"},{"id":"108366","messageId":"alpine.DEB.1.00.0903181213050.10279@pacific.mpi-cbg.de","threadId":"18368","inReplyTo":"49C0A462.3060902@kdbg.org","subject":"Re: [PATCH] MinGW: implement mmap","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-18T11:18:37Z","receivedAt":"2009-03-18T11:18:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 18 Mar 2009, Johannes Sixt wrote:\n\n> From: Janos Laube <janos.dev@gmail.com>\n> Date: Fri, 13 Mar 2009 16:50:45 +0100\n> \n> Add USE_WIN32_MMAP which triggers the use of windows' native\n> file memory mapping functionality in git_mmap()/git_munmap() functions.\n> \n> As git functions currently use mmap with MAP_PRIVATE set only, this\n> implementation supports only that mode for now.\n> \n> On Windows, offsets for memory mapped files need to match the allocation\n> granularity. Take this into account when calculating the packed git-\n> windowsize and file offsets. At the moment, the only function which makes\n> use of offsets in conjunction with mmap is use_pack() in sha1-file.c.\n> \n> Git fast-import's code path tries to map a portion of the temporary\n> packfile that exceeds the current filesize, i.e. offset+length is\n> greater than the filesize. The NO_MMAP code worked with that since pread()\n> just reads the file content until EOF and returns gracefully, while\n> MapViewOfFile() aborts the mapping and returns 'Access Denied'.\n> Working around that by determining the filesize and adjusting the length\n> parameter.\n> \n> Signed-off-by: Janos Laube <janos.dev@gmail.com>\n> Signed-off-by: Johannes Sixt <j6t@kdbg.org>\n> ---\n\nIn case anybody wonders why this is not integrated into mingw.[ch]: I \nasked for it, as it is Windows-specific and could benefit Cygwin (or MSVC, \nshould someone attempt that), too.\n\nAnd I think I missed this style in the patch:\n\n> +void *git_mmap\n> +(void *start, size_t length, int prot, int flags, int fd, off_t offset)\n> +{\n\n... which I would like to see fixed, of course... :-)\n\nUnfortunately, there is also a long line, which is corrupted due to \nwrapping:\n\n> +\thmap = CreateFileMapping((HANDLE)_get_osfhandle(fd), 0,\n> PAGE_WRITECOPY,\n> +\t\t0, 0, 0);\n\nIt is easily fixed by moving the PAGE_WRITECOPY to the next line.\n\nThese are only style issues, though.\n\nThe more important part is: how much does this buy us (or is it more \nexpensive, as we saw one MacOSX at one stage).  Janos was nice enough to \nperform some benchmark tests, and saw a ~40% improvement on rev-list \n--objects --all on a sizeable project (GCC 'twas, IIRC).  Funnily, the \nnumbers got better when reducing the window, up until 1MB, and they got \nworse when enlarging the window.\n\nSo I guess there will be a follow-up patch at some stage which changes the \ndefault pack window size on Windows to something smaller than 64MB.\n\nCiao,\nDscho\n"}]}