{"thread":{"id":"6088","subject":"[PATCH 0/17] Sliding window mmap for packfiles.","startedAt":"2006-12-23T07:33:17Z","lastAt":"2007-01-02T15:28:33Z","messageCount":9,"participants":["Shawn O. Pearce","Junio C Hamano","Shawn Pearce","Francis Moreau","Linus Torvalds","Andy Whitcroft"],"isPatch":true,"patchVersion":1,"patchTotal":17},"messages":[{"id":"30163","messageId":"20061223073317.GA9837@spearce.org","threadId":"6088","inReplyTo":null,"subject":"[PATCH 0/17] Sliding window mmap for packfiles.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:33:17Z","receivedAt":"2006-12-23T07:33:17Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This 17 patch series implements my much discussed, but never produced\n(until now), 'mmap sliding window' for packfile data access.\n\nThe key idea behind this topic is to mmap large non-contiguous\nsegments of a packfile rather than the entire file. If available\nvirtual memory is getting low we unmap the least recently used\npackfile segment to free up address space for the currently needed\nsegment.\n\nThis series also permits accessing packfiles up to 4 GiB in size,\neven on systems which permit only 2 GiB of virtual memory within\na single process (e.g. Windows and some older UNIXes). Of course\n4 GiB is still the upper limit on packfile size due to the current\nformat of the index file.\n\nThis series is 'pu' ready, but it may be too early to bring it\ndirectly into 'next'.\n\nPatch summary\n-------------\n\n 1 - Replace unpack_entry_gently with unpack_entry.\n 2 - Introduce new config option for mmap limit.\n 3 - Refactor packed_git to prepare for sliding mmap windows.\n 4 - Use off_t for index and pack file lengths.\n 5 - Create read_or_die utility routine.\n 6 - Refactor how we open pack files to prepare for multiple windows.\n\n Most of the above changes are incremental refactorings to help\n get the code in a state where we can start to implement and make\n use of the struct pack_window concept this series introduces.\n\n 7 - Replace use_packed_git with window cursors.\n 8 - Loop over pack_windows when inflating/accessing data.\n 9 - Document why header parsing won't exceed a window.\n 10 - Unmap individual windows rather than entire files.\n 11 - Fully activate the sliding window pack access.\n\n These commits actually implement the core of the mmap sliding\n window implementation and the necessary garbage collection to\n support unmapping the least recently used window.\n\n 12 - Load core configuration in git-verify-pack.\n 13 - Ensure core.packedGitWindowSize cannot be less than 2 pages.\n 14 - Improve error message when packfile mmap fails.\n 15 - Support unmapping windows on 'temporary' packfiles.\n\n The above sequence of commits are bug fixes on top of the initial\n commits. I did not fold these back into the earlier commits\n as I felt the bug fix commit messages provided useful details.\n\n 16 - Create pack_report() as a debugging aid.\n 17 - Test suite for sliding window mmap implementation.\n\n These provide debugging and testing tools.\n"},{"id":"30189","messageId":"7vlkkzrm0w.fsf@assigned-by-dhcp.cox.net","threadId":"6088","inReplyTo":"20061223073317.GA9837@spearce.org","subject":"Re: [PATCH 0/17] Sliding window mmap for packfiles.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-23T09:37:03Z","receivedAt":"2006-12-23T09:37:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I have to say i am very much impressed (I've taken a look at\nonly the first half up to #11, though). How much has this been\nused in real projects?\n\nA couple of comments:\n\n[3/17]\n\nI think losing \"p->next = NULL\" does not matter with the callers\nwe have right now, but somehow this part makes me feel uneasy.\n\n[5/17]\n\nI think it makes sense to exit(0) for the existing write_or_die\nupon EPIPE because that indicates we are the upstream of the\npipe and the reading process has exit (i.e. user said 'q' to\nless while we still have more to say). \n\nI suspect the symmetry would not hold for read_or_die; when we\nare reading, EPIPE is not any different from any other errors\n(except for EAGAIN or EINTR which we already take care of in\nxread()) and the net effect is that we could not read what we\nwanted to.\n"},{"id":"30190","messageId":"20061223094249.GC9396@spearce.org","threadId":"6088","inReplyTo":"7vlkkzrm0w.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 0/17] Sliding window mmap for packfiles.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T09:42:49Z","receivedAt":"2006-12-23T09:42:49Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> I have to say i am very much impressed (I've taken a look at\n> only the first half up to #11, though). How much has this been\n> used in real projects?\n\nNone yet. I just finished remerging it onto current git.git code.\n\n> A couple of comments:\n> \n> [3/17]\n> \n> I think losing \"p->next = NULL\" does not matter with the callers\n> we have right now, but somehow this part makes me feel uneasy.\n\nThat's a bug in that patch. I removed it by mistake. Good catch.\n \n> [5/17]\n> \n> I think it makes sense to exit(0) for the existing write_or_die\n> upon EPIPE because that indicates we are the upstream of the\n> pipe and the reading process has exit (i.e. user said 'q' to\n> less while we still have more to say). \n> \n> I suspect the symmetry would not hold for read_or_die; when we\n> are reading, EPIPE is not any different from any other errors\n> (except for EAGAIN or EINTR which we already take care of in\n> xread()) and the net effect is that we could not read what we\n> wanted to.\n\nOh, good point.\n\n-- \nShawn.\n"},{"id":"30230","messageId":"38b2ab8a0612240056k152344ael891e9b0b9f8cbc47@mail.gmail.com","threadId":"6088","inReplyTo":"20061223073317.GA9837@spearce.org","subject":"Re: [PATCH 0/17] Sliding window mmap for packfiles.","fromName":"Francis Moreau","fromEmail":"francis.moro@gmail.com","sentAt":"2006-12-24T08:56:59Z","receivedAt":"2006-12-24T08:56:59Z","isPatch":true,"sender":{"key":"francis.moro@gmail.com","avatar":null},"body":"Hi,\n\nOn 12/23/06, Shawn O. Pearce wrote:\n> This 17 patch series implements my much discussed, but never produced\n[snip]\n>\n> This series also permits accessing packfiles up to 4 GiB in size,\n> even on systems which permit only 2 GiB of virtual memory within\n> a single process (e.g. Windows and some older UNIXes). Of course\n\nJust out of curiosity, do you mean that there are some OS running on\n32 bits machines which allow 4GiB size of virtual memory within a\nsingle process ? If so, could you give an example of such OS ?\n\nthanks\n-- \nFrancis\n"},{"id":"30231","messageId":"20061224090508.GF7443@spearce.org","threadId":"6088","inReplyTo":"38b2ab8a0612240056k152344ael891e9b0b9f8cbc47@mail.gmail.com","subject":"Re: [PATCH 0/17] Sliding window mmap for packfiles.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-24T09:05:08Z","receivedAt":"2006-12-24T09:05:08Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Francis Moreau wrote:\n> On 12/23/06, Shawn O. Pearce wrote:\n> >This 17 patch series implements my much discussed, but never produced\n> [snip]\n> >\n> >This series also permits accessing packfiles up to 4 GiB in size,\n> >even on systems which permit only 2 GiB of virtual memory within\n> >a single process (e.g. Windows and some older UNIXes). Of course\n> \n> Just out of curiosity, do you mean that there are some OS running on\n> 32 bits machines which allow 4GiB size of virtual memory within a\n> single process ? If so, could you give an example of such OS ?\n\nNo. What I meant was the Git packfile/index format currently\nsupports up to 4 GiB of data in a single packfile. But *no*\nOS using 32 bit virtual address space would permit us to access\nthat packfile prior to this series as we would have *no* memory\nleft for a stack, let alone for parsing commits, etc., as *all*\nof the address space would have been dedicated to the packfile.\n\nHowever with this series even a 32 bit OS which only permits\nprocesses to have at most 2 GiB of address space (2 GiB split\nbetween kernel space and userspace) can access packfiles up\nto 4 GiB in size. That seems to be the split most OSes wind\nup using, if they didn't push it out to 3.2 GiB like Linux\nand Solaris have done.\n\nThis series is a good change because Git can now really make\nfull use of the space allowed by a single packfile. :-)\n\n-- \nShawn.\n"},{"id":"30232","messageId":"Pine.LNX.4.64.0612240126380.3671@woody.osdl.org","threadId":"6088","inReplyTo":"38b2ab8a0612240056k152344ael891e9b0b9f8cbc47@mail.gmail.com","subject":"Re: [PATCH 0/17] Sliding window mmap for packfiles.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-24T09:29:29Z","receivedAt":"2006-12-24T09:29:29Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sun, 24 Dec 2006, Francis Moreau wrote:\n> \n> Just out of curiosity, do you mean that there are some OS running on\n> 32 bits machines which allow 4GiB size of virtual memory within a\n> single process ? If so, could you give an example of such OS ?\n\nActually, Linux will do it on certain architectures (some architectures \nhave separate \"address spaces\" for kernel and user). And even on x86, if \nyou apply the (insane) 4GB patches, user space will actually have almost \nall of the 4GB, because there's only a _tiny_ trampoline thing that \nswitches the whole page table around that is kernel-mapped and takes away \nfrom the 4GB thing.\n\nIn practice, though, most 32-bit architectures will have between 1-3GB of \nuser virtual memory. And obviously stack space, binaries, heap etc take up \nspace, so you often end up with with just ~0.5 GB of actual dependable \ncontiguous virtual memory.\n\n\t\t\tLinus\n"},{"id":"30233","messageId":"38b2ab8a0612240136r559376d4s14af3123f762a45d@mail.gmail.com","threadId":"6088","inReplyTo":"20061224090508.GF7443@spearce.org","subject":"Re: [PATCH 0/17] Sliding window mmap for packfiles.","fromName":"Francis Moreau","fromEmail":"francis.moro@gmail.com","sentAt":"2006-12-24T09:36:16Z","receivedAt":"2006-12-24T09:36:16Z","isPatch":true,"sender":{"key":"francis.moro@gmail.com","avatar":null},"body":"On 12/24/06, Shawn Pearce wrote:\n> Francis Moreau wrote:\n> > On 12/23/06, Shawn O. Pearce wrote:\n> > >This 17 patch series implements my much discussed, but never produced\n> > [snip]\n> > >\n> > >This series also permits accessing packfiles up to 4 GiB in size,\n> > >even on systems which permit only 2 GiB of virtual memory within\n> > >a single process (e.g. Windows and some older UNIXes). Of course\n> >\n> > Just out of curiosity, do you mean that there are some OS running on\n> > 32 bits machines which allow 4GiB size of virtual memory within a\n> > single process ? If so, could you give an example of such OS ?\n>\n> No. What I meant was the Git packfile/index format currently\n> supports up to 4 GiB of data in a single packfile. But *no*\n> OS using 32 bit virtual address space would permit us to access\n> that packfile prior to this series as we would have *no* memory\n> left for a stack, let alone for parsing commits, etc., as *all*\n> of the address space would have been dedicated to the packfile.\n>\n\nok.\n\n> However with this series even a 32 bit OS which only permits\n> processes to have at most 2 GiB of address space (2 GiB split\n> between kernel space and userspace) can access packfiles up\n> to 4 GiB in size. That seems to be the split most OSes wind\n> up using, if they didn't push it out to 3.2 GiB like Linux\n> and Solaris have done.\n>\n\nDoes it still needed for 64 bit OS ?\n\nif not, can the overhead (if there is a significant one) implied by\nyour rework be avoid for such cases ?\n\n> This series is a good change because Git can now really make\n> full use of the space allowed by a single packfile. :-)\n>\n\nYes I agree with you.\n\n-- \nFrancis\n"},{"id":"30234","messageId":"20061224094959.GA7814@spearce.org","threadId":"6088","inReplyTo":"38b2ab8a0612240136r559376d4s14af3123f762a45d@mail.gmail.com","subject":"Re: [PATCH 0/17] Sliding window mmap for packfiles.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-24T09:49:59Z","receivedAt":"2006-12-24T09:49:59Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Francis Moreau wrote:\n> On 12/24/06, Shawn Pearce wrote:\n> >However with this series even a 32 bit OS which only permits\n> >processes to have at most 2 GiB of address space (2 GiB split\n> >between kernel space and userspace) can access packfiles up\n> >to 4 GiB in size. That seems to be the split most OSes wind\n> >up using, if they didn't push it out to 3.2 GiB like Linux\n> >and Solaris have done.\n> >\n> \n> Does it still needed for 64 bit OS ?\n\nNot really. Almost any reasonable 64 bit OS which is also running\na Git compiled for 64 bit userspace would be able to mmap multiple\n4 GiB packfiles without this series.\n \n> if not, can the overhead (if there is a significant one) implied by\n> your rework be avoid for such cases ?\n\nThe overhead is rather low. I did try hard to make it only a handful\nof machine instructions worth of additional work, and even then I\ntried to ammortize those over relatively large blocks of data to\nreduce the impact. But yes, there is an overhead over the current\nshipping version of Git.\n\nHowever at least some of the overhead can be avoided by setting\ncore.packedGitWindowSize and core.packedGitLimit to higher values.\nThis will allow the implementation to mmap() larger windows of the\npackfiles and retain a greater number of windows in memory at once.\n\nIf core.packedGitWindowSize is larger than your largest packfile\nthen most of the code will just \"shutoff\" and won't get in the way.\nIts default is 32 MiB (see Documentation/config.txt).\n\nI think the additional overhead added by this series is neglible\nand worth the more graceful degredation it allows when virtual\naddress space becomes limited.\n\n-- \nShawn.\n"},{"id":"30673","messageId":"459A7A21.4060901@shadowen.org","threadId":"6088","inReplyTo":"20061224094959.GA7814@spearce.org","subject":"Re: [PATCH 0/17] Sliding window mmap for packfiles.","fromName":"Andy Whitcroft","fromEmail":"apw@shadowen.org","sentAt":"2007-01-02T15:28:33Z","receivedAt":"2007-01-02T15:28:33Z","isPatch":true,"sender":{"key":"apw@shadowen.org","avatar":"https://gravatar.com/avatar/d3088262854661a913ef35cc40fedcc270142d4461791142bc1ea0b2a4e2e147?d=mp&s=160"},"body":"Shawn Pearce wrote:\n> Francis Moreau wrote:\n>> On 12/24/06, Shawn Pearce wrote:\n>>> However with this series even a 32 bit OS which only permits\n>>> processes to have at most 2 GiB of address space (2 GiB split\n>>> between kernel space and userspace) can access packfiles up\n>>> to 4 GiB in size. That seems to be the split most OSes wind\n>>> up using, if they didn't push it out to 3.2 GiB like Linux\n>>> and Solaris have done.\n>>>\n>> Does it still needed for 64 bit OS ?\n> \n> Not really. Almost any reasonable 64 bit OS which is also running\n> a Git compiled for 64 bit userspace would be able to mmap multiple\n> 4 GiB packfiles without this series.\n> \n>> if not, can the overhead (if there is a significant one) implied by\n>> your rework be avoid for such cases ?\n> \n> The overhead is rather low. I did try hard to make it only a handful\n> of machine instructions worth of additional work, and even then I\n> tried to ammortize those over relatively large blocks of data to\n> reduce the impact. But yes, there is an overhead over the current\n> shipping version of Git.\n> \n> However at least some of the overhead can be avoided by setting\n> core.packedGitWindowSize and core.packedGitLimit to higher values.\n> This will allow the implementation to mmap() larger windows of the\n> packfiles and retain a greater number of windows in memory at once.\n> \n> If core.packedGitWindowSize is larger than your largest packfile\n> then most of the code will just \"shutoff\" and won't get in the way.\n> Its default is 32 MiB (see Documentation/config.txt).\n> \n> I think the additional overhead added by this series is neglible\n> and worth the more graceful degredation it allows when virtual\n> address space becomes limited.\n\nYou now change the default size based on NO_MMAP, could you not just\nbump the window size to 4GiB on 64 bit?\n\n-apw\n"}]}