Re: [PATCH] macOS: queue for munmap operations

>
> On Mon, Oct 20, 2025 at 10:35:02PM +0000, Koji Nakamaru via GitGitGadget wrote:
>
> > From: Koji Nakamaru <koji.nakamaru@gree.net>
> >
> > Executing many mmap/munmap calls alternately can cause a huge load on
> > macOS. In order to reduce it, we should temporarily store munmap
> > operations in a queue and process them all at once when the queue is
> > filled. When the program terminates, we can discard any remaining munmap
> > operations as corresponding mmaped regions are automatically reclaimed.
> >
> > Add a queue for munmap operations to perform them all at once.
> >
> > Here are some example timings. On the Linux kernel repository that
> > requires about 1700 mmap/munmap calls:
> >
> >   time git ls-tree -r -l --full-tree 211ddde > /dev/null
>
> Why is it doing so many mmap calls? Do you have a ton of loose objects?
> We have to mmap loose objects individually (because they're all in
> separate files), but each pack only gets a single map (well, there's a
> window parameter, but it's 1GB on 64-bit systems, so you should get a
> handful of maps at most).
>
> If you run "git gc", how does the resulting ls-tree perform? I have only
> 27 mmap() calls on my system.
>
> I know that running "git gc" is relatively expensive, but it is also
> bringing other optimizations (like the fact that we don't have to open()
> and map each of those files in the first place!).
>
> > On a private repository that requires about 943000 mmap/munmap calls:
> >
> >   time git ls-tree -r -l --full-tree xxxxxxx > /dev/null
>
> Ditto here. I'd be curious how well packed the repo is, and how it does
> after a repack. If it has a very large packfile, you might also try:
>
>   git config core.packedGitWindowSize 4G
>
> or similar (though for just an ls-tree, we should only be looking at
> tree objects, which in general I'd expect to be in a confined area of
> the packfile; so the 1GB window is probably plenty).

> > +int git_munmap(void *start, size_t length)
> > +{
> > +     static pthread_mutex_t mutex;
> > +     static struct munmap_queue *queue;
> > +     static int count;
> > +     int i;
> > +
> > +     pthread_mutex_lock(&mutex);
> > +     if (!queue)
> > +             queue = xmalloc(COUNT_MAX * sizeof(struct munmap_queue));
> > +     queue[count].start = start;
> > +     queue[count].length = length;
> > +     if (++count == COUNT_MAX) {
> > +             for (i = 0; i < COUNT_MAX; i++)
> > +                     munmap(queue[i].start, queue[i].length);
> > +             count = 0;
> > +     }
> > +     pthread_mutex_unlock(&mutex);
> > +     return 0;
> > +}
>
> Does batching those unmaps actually make them faster? Or is it just that
> the commands you showed did not fill the queue, so we essentially just
> leaked all of those maps until the program exited?
>
> If the latter, then I'd wonder:
>
>   1. Does this increase memory pressure, since the OS has no idea we're
>      not actually interested in those maps anymore? Some of them can be
>      quite large, if the command is looking at blobs.
>
>   2. How does it perform on a command that actually fills the queue? I
>      guess something like "git log --raw" might do it (though if my
>      guesses above are right, you'd need on the order of 64,000 loose
>      trees).