# large files and low memory 29 messages from 2010-10-04 to 2010-10-05. Participants: Enrico Weigelt, Shawn Pearce, Joshua Jensen, Jonathan Nieder, Ævar Arnfjörð Bjarmason, Matthieu Moy, Nguyen Thai Ngoc Duy, Alex Riesen, Junio C Hamano, Nicolas Pitre. Thread: https://gitlist.dev/t/25332 ## Enrico Weigelt, 2010-10-04 09:20 Subject: large files and low memory Message-ID: <20101004092046.GA4382@nibiru.local> URL: https://gitlist.dev/e/20101004092046.GA4382%40nibiru.local ``` Hi folks, when adding files which are larger than available physical memory, git performs very slow. Perhaps it has to do with git's mmap()ing the whole file. Is there any way to do it w/o mmap (hoping that might perform a bit better) ? cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ---------------------------------------------------------------------- ``` ## Shawn Pearce, 2010-10-04 18:05 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/AANLkTimbdrAqoWMxiteT5zNYmwHp8M698BEv1FLuiAxx%40mail.gmail.com In-Reply-To: <20101004092046.GA4382@nibiru.local> ``` On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt wrote: > > when adding files which are larger than available physical memory, > git performs very slow. Perhaps it has to do with git's mmap()ing > the whole file. Is there any way to do it w/o mmap (hoping that > might perform a bit better) ? The mmap() isn't the problem. Its the allocation of a buffer that is larger than the file in order to hold the result of deflating the file before it gets written to disk. When the file is bigger than physical memory, the kernel has to page in parts of the file as well as swap in and out parts of that allocated buffer to hold the deflated file. This is a known area in Git where big files aren't handled well. -- Shawn. ``` ## Joshua Jensen, 2010-10-04 18:24 Subject: Re: large files and low memory Message-ID: <4CAA1BEB.3050908@workspacewhiz.com> URL: https://gitlist.dev/e/4CAA1BEB.3050908%40workspacewhiz.com In-Reply-To: ``` ----- Original Message ----- From: Shawn Pearce Date: 10/4/2010 12:05 PM > On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt wrote: >> when adding files which are larger than available physical memory, >> git performs very slow. Perhaps it has to do with git's mmap()ing >> the whole file. Is there any way to do it w/o mmap (hoping that >> might perform a bit better) ? > The mmap() isn't the problem. Its the allocation of a buffer that is > larger than the file in order to hold the result of deflating the file > before it gets written to disk. When the file is bigger than physical > memory, the kernel has to page in parts of the file as well as swap in > and out parts of that allocated buffer to hold the deflated file. > > This is a known area in Git where big files aren't handled well. As a curiosity, I've always done streaming decompression with zlib using minimal buffer sizes (64k, perhaps). I'm sure there is good reason why Git doesn't do this (delta application?). Do you know what it is? Josh ``` ## Shawn Pearce, 2010-10-04 18:57 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/AANLkTi%3DnzyRgiBL07f2oeyjdUc8XnaKbJEJ%2Bk8_g9rQ1%40mail.gmail.com In-Reply-To: <4CAA1BEB.3050908@workspacewhiz.com> ``` On Mon, Oct 4, 2010 at 11:24 AM, Joshua Jensen wrote: >> On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt  wrote: >>> >>> when adding files which are larger than available physical memory, >>> git performs very slow. >> >> The mmap() isn't the problem.  Its the allocation of a buffer that is >> larger than the file in order to hold the result of deflating the file >> before it gets written to disk. ... >> This is a known area in Git where big files aren't handled well. > > As a curiosity, I've always done streaming decompression with zlib using > minimal buffer sizes (64k, perhaps).  I'm sure there is good reason why Git > doesn't do this (delta application?).  Do you know what it is? Laziness. Git originally assumed it would only be used for smaller source files written by humans. Its easier to write the code as a single malloc'd buffer than to stream it. We'd like to fix it, but its harder than it sounds. Today we copy the file into a buffer before we deflate and compute the SHA-1 as this prevents us from getting into a consistency error when the file is modified between these two stages. -- Shawn. ``` ## Jonathan Nieder, 2010-10-04 18:58 Subject: Re: large files and low memory Message-ID: <20101004185854.GA6466@burratino> URL: https://gitlist.dev/e/20101004185854.GA6466%40burratino In-Reply-To: ``` Shawn Pearce wrote: > The mmap() isn't the problem. Its the allocation of a buffer that is > larger than the file in order to hold the result of deflating the file > before it gets written to disk. Wasn't this already fixed, at least in some cases? commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1) Author: Nicolas Pitre Date: Sat Feb 20 23:27:31 2010 -0500 sha1_file: don't malloc the whole compressed result when writing out objects There is no real advantage to malloc the whole output buffer and deflate the data in a single pass when writing loose objects. That is like only 1% faster while using more memory, especially with large files where memory usage is far more. It is best to deflate and write the data out in small chunks reusing the same memory instead. For example, using 'git add' on a few large files averaging 40 MB ... Before: 21.45user 1.10system 0:22.57elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+828040outputs (0major+142640minor)pagefaults 0swaps After: 21.50user 1.25system 0:22.76elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+828040outputs (0major+104408minor)pagefaults 0swaps While the runtime stayed relatively the same, the number of minor page faults went down significantly. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano ``` ## Shawn Pearce, 2010-10-04 19:11 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/AANLkTin-mbzt93DWtp71vYBEUcDLHgo%3DG-6zynT5NC_r%40mail.gmail.com In-Reply-To: <20101004185854.GA6466@burratino> ``` On Mon, Oct 4, 2010 at 11:58 AM, Jonathan Nieder wrote: > Shawn Pearce wrote: > >> The mmap() isn't the problem.  Its the allocation of a buffer that is >> larger than the file in order to hold the result of deflating the file >> before it gets written to disk. > > Wasn't this already fixed, at least in some cases? > > commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1) > Author: Nicolas Pitre > Date:   Sat Feb 20 23:27:31 2010 -0500 > >    sha1_file: don't malloc the whole compressed result when writing out objects This change only removes the deflate copy. But due to the SHA-1 consistency issue I alluded to earlier, I think we're still making a full copy of the file in memory before we SHA-1 it or deflate it. So Nico halved the memory usage, but we're still using 1x the size of the file rather than ~2x. -- Shawn. ``` ## Jonathan Nieder, 2010-10-04 19:16 Subject: Re: large files and low memory Message-ID: <20101004191657.GC6466@burratino> URL: https://gitlist.dev/e/20101004191657.GC6466%40burratino In-Reply-To: ``` Shawn Pearce wrote: > This change only removes the deflate copy. But due to the SHA-1 > consistency issue I alluded to earlier, I think we're still making a > full copy of the file in memory before we SHA-1 it or deflate it. Hmm, I _think_ we still use mmap for that (which is why 748af44c needs to compare the sha1 before and after). But 1) a one-pass calculation would presumably be a little (5%?) faster 2) if there are smudge/clean filters or autocrlf involved, the cleaned-up file is backed by swap and this all becomes moot. ``` ## Enrico Weigelt, 2010-10-05 00:50 Subject: Re: large files and low memory Message-ID: <20101005005003.GA2768@nibiru.local> URL: https://gitlist.dev/e/20101005005003.GA2768%40nibiru.local In-Reply-To: ``` * Shawn Pearce wrote: > The mmap() isn't the problem. Its the allocation of a buffer that is > larger than the file in order to hold the result of deflating the file > before it gets written to disk. > When the file is bigger than physical memory, the kernel has to > page in parts of the file as well as swap in and out parts of > that allocated buffer to hold the deflated file. What are the access pattern of these memory areas ? Perhaps madvise() could help ? cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ---------------------------------------------------------------------- ``` ## Enrico Weigelt, 2010-10-05 00:57 Subject: Re: large files and low memory Message-ID: <20101005005715.GB2768@nibiru.local> URL: https://gitlist.dev/e/20101005005715.GB2768%40nibiru.local In-Reply-To: <20101004185854.GA6466@burratino> ``` * Jonathan Nieder wrote: > Shawn Pearce wrote: > > > The mmap() isn't the problem. Its the allocation of a buffer that is > > larger than the file in order to hold the result of deflating the file > > before it gets written to disk. > > Wasn't this already fixed, at least in some cases? > > commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1) > Author: Nicolas Pitre > Date: Sat Feb 20 23:27:31 2010 -0500 I guess I'll have to do a update. But: latest tag (1.7.3.1) doesnt build: CC read-cache.o read-cache.c: In function `fill_stat_cache_info': read-cache.c:73: structure has no member named `st_ctim' read-cache.c:74: structure has no member named `st_mtim' read-cache.c: In function `read_index_from': read-cache.c:1334: structure has no member named `st_mtim' read-cache.c: In function `write_index': read-cache.c:1614: structure has no member named `st_mtim' make: *** [read-cache.o] Fehler 1 Is my libc too old ? cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ---------------------------------------------------------------------- ``` ## Enrico Weigelt, 2010-10-05 00:59 Subject: Re: large files and low memory Message-ID: <20101005005902.GC2768@nibiru.local> URL: https://gitlist.dev/e/20101005005902.GC2768%40nibiru.local In-Reply-To: ``` * Shawn Pearce wrote: > Laziness. Git originally assumed it would only be used for smaller > source files written by humans. Its easier to write the code as a > single malloc'd buffer than to stream it. We'd like to fix it, but > its harder than it sounds. Today we copy the file into a buffer > before we deflate and compute the SHA-1 as this prevents us from > getting into a consistency error when the file is modified between > these two stages. hmm, perhaps copy it to a temporary file, if it's too large ? cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ---------------------------------------------------------------------- ``` ## Ævar Arnfjörð Bjarmason, 2010-10-05 01:07 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/AANLkTinF5-WoYKDOSwMQudnK9SP-EUx3wWhUMi4z1kv3%40mail.gmail.com In-Reply-To: <20101005005715.GB2768@nibiru.local> ``` On Tue, Oct 5, 2010 at 00:57, Enrico Weigelt wrote: > * Jonathan Nieder wrote: >> Shawn Pearce wrote: >> >> > The mmap() isn't the problem.  Its the allocation of a buffer that is >> > larger than the file in order to hold the result of deflating the file >> > before it gets written to disk. >> >> Wasn't this already fixed, at least in some cases? >> >> commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1) >> Author: Nicolas Pitre >> Date:   Sat Feb 20 23:27:31 2010 -0500 > > I guess I'll have to do a update. > > But: latest tag (1.7.3.1) doesnt build: > > >    CC read-cache.o >    read-cache.c: In function `fill_stat_cache_info': >    read-cache.c:73: structure has no member named `st_ctim' >    read-cache.c:74: structure has no member named `st_mtim' >    read-cache.c: In function `read_index_from': >    read-cache.c:1334: structure has no member named `st_mtim' >    read-cache.c: In function `write_index': >    read-cache.c:1614: structure has no member named `st_mtim' >    make: *** [read-cache.o] Fehler 1 > > Is my libc too old ? Those lines are accessing members called st_ctime, i.e. with an "e" at the end, but your errors just report "st_ctim". What gives? ``` ## Jonathan Nieder, 2010-10-05 01:10 Subject: Re: large files and low memory Message-ID: <20101005011009.GD9994@burratino> URL: https://gitlist.dev/e/20101005011009.GD9994%40burratino In-Reply-To: <20101005005715.GB2768@nibiru.local> ``` Enrico Weigelt wrote: > CC read-cache.o > read-cache.c: In function `fill_stat_cache_info': > read-cache.c:73: structure has no member named `st_ctim' > read-cache.c:74: structure has no member named `st_mtim' > read-cache.c: In function `read_index_from': > read-cache.c:1334: structure has no member named `st_mtim' > read-cache.c: In function `write_index': > read-cache.c:1614: structure has no member named `st_mtim' > make: *** [read-cache.o] Fehler 1 > > Is my libc too old ? What platform are you on? You probably need USE_ST_TIMESPEC; if so, please send a makefile patch so the next person trying it doesn't need to worry about it. Also, please don't destroy the cc lists; it makes it hard for people with some mail setups (e.g., mine) to notice when you've replied to them. ``` ## Enrico Weigelt, 2010-10-05 07:35 Subject: Re: large files and low memory Message-ID: <20101005073505.GA14821@nibiru.local> URL: https://gitlist.dev/e/20101005073505.GA14821%40nibiru.local In-Reply-To: <20101005011009.GD9994@burratino> ``` * Jonathan Nieder wrote: > What platform are you on? GNU/Linux. glibc-2.25 cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ---------------------------------------------------------------------- ``` ## Enrico Weigelt, 2010-10-05 07:41 Subject: Re: large files and low memory Message-ID: <20101005074144.GA22564@nibiru.local> URL: https://gitlist.dev/e/20101005074144.GA22564%40nibiru.local In-Reply-To: <20101005005902.GC2768@nibiru.local> ``` * Enrico Weigelt wrote: Found another possible bottleneck: git-commit seems to scan through a lot of files. Shouldnt it just create a commit object from the current index and update the head ? cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ---------------------------------------------------------------------- ``` ## Matthieu Moy, 2010-10-05 08:01 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/vpqpqvpf4qm.fsf%40bauges.imag.fr In-Reply-To: <20101005074144.GA22564@nibiru.local> ``` Enrico Weigelt writes: > * Enrico Weigelt wrote: > > > > Found another possible bottleneck: git-commit seems to scan through > a lot of files. Shouldnt it just create a commit object from the > current index and update the head ? git commit will show what's being commited (the output of "git commit --dry-run") in your editor, hence it needs to compute that. -- Matthieu Moy http://www-verimag.imag.fr/~moy/ ``` ## Enrico Weigelt, 2010-10-05 08:17 Subject: Re: large files and low memory Message-ID: <20101005081700.GB22564@nibiru.local> URL: https://gitlist.dev/e/20101005081700.GB22564%40nibiru.local In-Reply-To: ``` * Matthieu Moy wrote: > git commit will show what's being commited (the output of "git commit > --dry-run") in your editor, hence it needs to compute that. hmm, is there any way to get around this ? cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ---------------------------------------------------------------------- ``` ## Nguyen Thai Ngoc Duy, 2010-10-05 10:13 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/AANLkTimHKyQhAzuh4Hsh7%3DbEet1T0mNRSMgiZkhWQwSN%40mail.gmail.com In-Reply-To: <20101005074144.GA22564@nibiru.local> ``` On Tue, Oct 5, 2010 at 2:41 PM, Enrico Weigelt wrote: > * Enrico Weigelt wrote: > > > > Found another possible bottleneck: git-commit seems to scan through > a lot of files. Shouldnt it just create a commit object from the > current index and update the head ? You mean a lot of stat()? There is no way to avoid that unless you set assume-unchanged bits. Or you could use write-tree/commit-tree/update-ref directly. -- Duy ``` ## Nguyen Thai Ngoc Duy, 2010-10-05 10:59 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/AANLkTik1kzzzOKFaJCWtwiYf5Vz8oNjZXD%2B-SAq-ZvAa%40mail.gmail.com In-Reply-To: <20101004191657.GC6466@burratino> ``` On Tue, Oct 5, 2010 at 2:16 AM, Jonathan Nieder wrote: > Shawn Pearce wrote: > >> This change only removes the deflate copy.  But due to the SHA-1 >> consistency issue I alluded to earlier, I think we're still making a >> full copy of the file in memory before we SHA-1 it or deflate it. > > Hmm, I _think_ we still use mmap for that (which is why 748af44c needs > to compare the sha1 before and after). Just tried valgrind massif on a 200MB file with master. It used ~270kb heap. I haven't tested but I believe git-checkout will keep the whole inflated copy in memory. So git-add alone does not help much. -- Duy ``` ## Alex Riesen, 2010-10-05 11:29 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/AANLkTin%2Bi7VZApDRmairgpMth4MzPoDF%3DoHt-UHCt98E%40mail.gmail.com In-Reply-To: <20101005081700.GB22564@nibiru.local> ``` On Tue, Oct 5, 2010 at 10:17, Enrico Weigelt wrote: > * Matthieu Moy wrote: > >> git commit will show what's being commited (the output of "git commit >> --dry-run") in your editor, hence it needs to compute that. > > hmm, is there any way to get around this ? > Try "git commit -q -uno". This should skip creation of summary in the commit message and lookup for untracked files. This will somewhat speed things up ``` ## Matthieu Moy, 2010-10-05 11:38 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/vpqbp78x418.fsf%40bauges.imag.fr In-Reply-To: ``` Alex Riesen writes: > On Tue, Oct 5, 2010 at 10:17, Enrico Weigelt wrote: >> * Matthieu Moy wrote: >> >>> git commit will show what's being commited (the output of "git commit >>> --dry-run") in your editor, hence it needs to compute that. >> >> hmm, is there any way to get around this ? >> > > Try "git commit -q -uno". This should skip creation of summary in the > commit message and lookup for untracked files. To avoid including the summary, the option would be --no-status (-q makes commit less verbose in stdout, not in COMMIT_EDITMSG). But strace -fe lstat64 git commit -uno --no-status -q still shows lstat64 for each tracked file in my working tree (even when using -m to avoid launching the editor). I don't know if this is intended, or just that nobody cared enough to optimize this. -- Matthieu Moy http://www-verimag.imag.fr/~moy/ ``` ## Nguyen Thai Ngoc Duy, 2010-10-05 11:55 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/AANLkTi%3Dni_21uP7oPrmB3obMscg0o9Z%3DuawkM2PoxWxT%40mail.gmail.com In-Reply-To: ``` On Tue, Oct 5, 2010 at 6:38 PM, Matthieu Moy wrote: > Alex Riesen writes: > >> On Tue, Oct 5, 2010 at 10:17, Enrico Weigelt wrote: >>> * Matthieu Moy wrote: >>> >>>> git commit will show what's being commited (the output of "git commit >>>> --dry-run") in your editor, hence it needs to compute that. >>> >>> hmm, is there any way to get around this ? >>> >> >> Try "git commit -q -uno". This should skip creation of summary in the >> commit message and lookup for untracked files. > > To avoid including the summary, the option would be --no-status (-q > makes commit less verbose in stdout, not in COMMIT_EDITMSG). > > But > > strace -fe lstat64 git commit -uno --no-status -q > > still shows lstat64 for each tracked file in my working tree (even > when using -m to avoid launching the editor). I don't know if this is > intended, or just that nobody cared enough to optimize this. I assume you do git-commit with no index modification at all. The index refresh part dated back in 2888605 (builtin-commit: fix partial-commit support - 2007-11-18). Commit message does not tell why index refresh is needed (for summary stat maybe?). If so, then we can skip refreshing if -q is given. -- Duy ``` ## Jonathan Nieder, 2010-10-05 13:47 Subject: Re: large files and low memory Message-ID: <20101005134744.GB12797@burratino> URL: https://gitlist.dev/e/20101005134744.GB12797%40burratino In-Reply-To: <20101005073505.GA14821@nibiru.local> ``` Enrico Weigelt wrote: > * Jonathan Nieder wrote: >> What platform are you on? > > GNU/Linux. glibc-2.25 Hmm, I've heard of glib 2.25 but never glibc 2.25. :) $ /lib/libc.so.6 | head -1 GNU C Library (Debian EGLIBC 2.11.2-6) stable release version 2.11.2, by Roland McGrath et al. ``` ## Junio C Hamano, 2010-10-05 16:42 Subject: Re: large files and low memory Message-ID: <7vfwwk7fqz.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vfwwk7fqz.fsf%40alter.siamese.dyndns.org In-Reply-To: ``` Nguyen Thai Ngoc Duy writes: > I assume you do git-commit with no index modification at all. The > index refresh part dated back in 2888605 (builtin-commit: fix > partial-commit support - 2007-11-18). Commit message does not tell why > index refresh is needed (for summary stat maybe?). If so, then we can > skip refreshing if -q is given. Most likely to give a clean index to post-commit hook. ``` ## Nicolas Pitre, 2010-10-05 19:06 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/alpine.LFD.2.00.1010051504530.3107%40xanadu.home In-Reply-To: <20101005005003.GA2768@nibiru.local> ``` On Tue, 5 Oct 2010, Enrico Weigelt wrote: > * Shawn Pearce wrote: > > > The mmap() isn't the problem. Its the allocation of a buffer that is > > larger than the file in order to hold the result of deflating the file > > before it gets written to disk. > > When the file is bigger than physical memory, the kernel has to > > page in parts of the file as well as swap in and out parts of > > that allocated buffer to hold the deflated file. > > What are the access pattern of these memory areas ? Perfectly linear. > Perhaps madvise() could help ? Perhaps. Nicolas ``` ## Nicolas Pitre, 2010-10-05 19:12 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/alpine.LFD.2.00.1010051510160.3107%40xanadu.home In-Reply-To: ``` On Tue, 5 Oct 2010, Nguyen Thai Ngoc Duy wrote: > On Tue, Oct 5, 2010 at 2:41 PM, Enrico Weigelt wrote: > > * Enrico Weigelt wrote: > > > > > > > > Found another possible bottleneck: git-commit seems to scan through > > a lot of files. Shouldnt it just create a commit object from the > > current index and update the head ? > > You mean a lot of stat()? There is no way to avoid that unless you set > assume-unchanged bits. Or you could use > write-tree/commit-tree/update-ref directly. Avoiding memory exhaustion is also going to help a lot as the stat() information will remain cached instead of requiring disk access. Just a guess given $subject. Nicolas ``` ## Nicolas Pitre, 2010-10-05 20:17 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/alpine.LFD.2.00.1010051518570.3107%40xanadu.home In-Reply-To: <20101004191657.GC6466@burratino> ``` On Mon, 4 Oct 2010, Jonathan Nieder wrote: > Shawn Pearce wrote: > > > This change only removes the deflate copy. But due to the SHA-1 > > consistency issue I alluded to earlier, I think we're still making a > > full copy of the file in memory before we SHA-1 it or deflate it. > > Hmm, I _think_ we still use mmap for that (which is why 748af44c needs > to compare the sha1 before and after). > > But > > 1) a one-pass calculation would presumably be a little (5%?) faster You can't do a one-pass calculation. The first one is required to compute the SHA1 of the file being added, and if that corresponds to an object that we already have then the operation stops right there as there is actually nothing to do. The second pass is to deflate the data, and recompute the SHA1 to make sure what we deflated and written out is still the same data. In the case of big files, what we need to do is to stream the file data in, compute the SHA1 and deflate it, in order to stream it out into a temporary file, then rename it according to the final SHA1. This would allow Git to work with big files, but of course it won't be possible to know if the object corresponding to the file is already known until all the work has been done, possibly just to throw it away. But normally big files are the minority. Nicolas ``` ## Jonathan Nieder, 2010-10-05 20:34 Subject: Re: large files and low memory Message-ID: <20101005203450.GA2096@burratino> URL: https://gitlist.dev/e/20101005203450.GA2096%40burratino In-Reply-To: ``` Nicolas Pitre wrote: > You can't do a one-pass calculation. The first one is required to > compute the SHA1 of the file being added, and if that corresponds to an > object that we already have then the operation stops right there as > there is actually nothing to do. Ah. Thanks for a reminder. > In the case of big files, what we need to do is to stream the file data > in, compute the SHA1 and deflate it, in order to stream it out into a > temporary file, then rename it according to the final SHA1. This would > allow Git to work with big files, but of course it won't be possible to > know if the object corresponding to the file is already known until all > the work has been done, possibly just to throw it away. To make sure I understand correctly: are you suggesting that for big files we should skip the first pass? I suppose that makes sense: for small files, using a patch application tool to reach a postimage that matches an existing object is something git historically needed to expect, but for typical big files: - once you've computed the SHA1, you've already invested a noticeable amount of time. - emailing patches around is difficult, making "git am" etc less important - hopefully git or zlib can notice when files are uncompressible, making the deflate not cost so much in that case. ``` ## Nicolas Pitre, 2010-10-05 21:11 Subject: Re: large files and low memory Message-ID: URL: https://gitlist.dev/e/alpine.LFD.2.00.1010051657440.3107%40xanadu.home In-Reply-To: <20101005203450.GA2096@burratino> ``` On Tue, 5 Oct 2010, Jonathan Nieder wrote: > Nicolas Pitre wrote: > > > You can't do a one-pass calculation. The first one is required to > > compute the SHA1 of the file being added, and if that corresponds to an > > object that we already have then the operation stops right there as > > there is actually nothing to do. > > Ah. Thanks for a reminder. > > > In the case of big files, what we need to do is to stream the file data > > in, compute the SHA1 and deflate it, in order to stream it out into a > > temporary file, then rename it according to the final SHA1. This would > > allow Git to work with big files, but of course it won't be possible to > > know if the object corresponding to the file is already known until all > > the work has been done, possibly just to throw it away. > > To make sure I understand correctly: are you suggesting that for big > files we should skip the first pass? For big files we need a totally separate code path to process the file data in small chunks at 'git add' time, using a loop containing read()+SHA1sum()+deflate()+write(). Then, if the SHA1 matches an existing object we delete the temporary output file, otherwise we rename it as a valid object. No CRLF, no smudge filters, no diff, no deltas, just plain storage of huge objects, based on the value of core.bigFileThreshold config option. Same thing on the checkout path: a simple loop to read()+inflate()+write() in small chunks. That's the only sane way to kinda support big files with Git. > I suppose that makes sense: for small files, using a patch application > tool to reach a postimage that matches an existing object is something > git historically needed to expect, but for typical big files: > > - once you've computed the SHA1, you've already invested a noticeable > amount of time. > - emailing patches around is difficult, making "git am" etc less important > - hopefully git or zlib can notice when files are uncompressible, > making the deflate not cost so much in that case. Emailing is out of the question. We're talking file sizes in the hundreds of megabytes and above here. So yes, simply computing the SHA1 is a significant cost, given that you are going to trash your page cache in the process already, so better pay the price of deflating it at the same time even if it turns out to be unnecessary. Nicolas ``` ## Enrico Weigelt, 2010-10-05 22:51 Subject: Re: large files and low memory Message-ID: <20101005225150.GA27633@nibiru.local> URL: https://gitlist.dev/e/20101005225150.GA27633%40nibiru.local In-Reply-To: ``` * Nicolas Pitre wrote: > > > The mmap() isn't the problem. Its the allocation of a buffer that is > > > larger than the file in order to hold the result of deflating the file > > > before it gets written to disk. > > > When the file is bigger than physical memory, the kernel has to > > > page in parts of the file as well as swap in and out parts of > > > that allocated buffer to hold the deflated file. > > > > What are the access pattern of these memory areas ? > > Perfectly linear. In this case, I wonder why my machine goes into thrashing so easily (P3 w/ 256MB ram). Seems the mmu/paging code doesnt recognize that the previously-used pages can be kicked-off quickly ;-o Perhaps I should talk to the kernel folks. > > Perhaps madvise() could help ? > > Perhaps. hmm, so we should try it ;-p where'd be the right place to add it ? cu -- ---------------------------------------------------------------------- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weigelt@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 ---------------------------------------------------------------------- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme ---------------------------------------------------------------------- ```