{"thread":{"id":"25332","subject":"large files and low memory","startedAt":"2010-10-04T09:20:47Z","lastAt":"2010-10-05T22:51:50Z","messageCount":29,"participants":["Enrico Weigelt","Shawn Pearce","Joshua Jensen","Jonathan Nieder","Ævar Arnfjörð Bjarmason","Matthieu Moy","Nguyen Thai Ngoc Duy","Alex Riesen","Junio C Hamano","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"152498","messageId":"20101004092046.GA4382@nibiru.local","threadId":"25332","inReplyTo":null,"subject":"large files and low memory","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-04T09:20:47Z","receivedAt":"2010-10-04T09:20:47Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"\nHi folks,\n\n\nwhen adding files which are larger than available physical memory,\ngit performs very slow. Perhaps it has to do with git's mmap()ing\nthe whole file. Is there any way to do it w/o mmap (hoping that\nmight perform a bit better) ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"152557","messageId":"AANLkTimbdrAqoWMxiteT5zNYmwHp8M698BEv1FLuiAxx@mail.gmail.com","threadId":"25332","inReplyTo":"20101004092046.GA4382@nibiru.local","subject":"Re: large files and low memory","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-10-04T18:05:54Z","receivedAt":"2010-10-04T18:05:54Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt wrote:\n>\n> when adding files which are larger than available physical memory,\n> git performs very slow. Perhaps it has to do with git's mmap()ing\n> the whole file. Is there any way to do it w/o mmap (hoping that\n> might perform a bit better) ?\n\nThe mmap() isn't the problem. Its the allocation of a buffer that is\nlarger than the file in order to hold the result of deflating the file\nbefore it gets written to disk. When the file is bigger than physical\nmemory, the kernel has to page in parts of the file as well as swap in\nand out parts of that allocated buffer to hold the deflated file.\n\nThis is a known area in Git where big files aren't handled well.\n\n-- \nShawn.\n"},{"id":"152563","messageId":"4CAA1BEB.3050908@workspacewhiz.com","threadId":"25332","inReplyTo":"AANLkTimbdrAqoWMxiteT5zNYmwHp8M698BEv1FLuiAxx@mail.gmail.com","subject":"Re: large files and low memory","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2010-10-04T18:24:43Z","receivedAt":"2010-10-04T18:24:43Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":" ----- Original Message -----\nFrom: Shawn Pearce\nDate: 10/4/2010 12:05 PM\n> On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt wrote:\n>> when adding files which are larger than available physical memory,\n>> git performs very slow. Perhaps it has to do with git's mmap()ing\n>> the whole file. Is there any way to do it w/o mmap (hoping that\n>> might perform a bit better) ?\n> The mmap() isn't the problem. Its the allocation of a buffer that is\n> larger than the file in order to hold the result of deflating the file\n> before it gets written to disk. When the file is bigger than physical\n> memory, the kernel has to page in parts of the file as well as swap in\n> and out parts of that allocated buffer to hold the deflated file.\n>\n> This is a known area in Git where big files aren't handled well.\nAs a curiosity, I've always done streaming decompression with zlib using \nminimal buffer sizes (64k, perhaps). I'm sure there is good reason why \nGit doesn't do this (delta application?). Do you know what it is?\n\nJosh\n"},{"id":"152571","messageId":"AANLkTi=nzyRgiBL07f2oeyjdUc8XnaKbJEJ+k8_g9rQ1@mail.gmail.com","threadId":"25332","inReplyTo":"4CAA1BEB.3050908@workspacewhiz.com","subject":"Re: large files and low memory","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-10-04T18:57:34Z","receivedAt":"2010-10-04T18:57:34Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Mon, Oct 4, 2010 at 11:24 AM, Joshua Jensen\n wrote:\n>> On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt  wrote:\n>>>\n>>> when adding files which are larger than available physical memory,\n>>> git performs very slow.\n>>\n>> The mmap() isn't the problem.  Its the allocation of a buffer that is\n>> larger than the file in order to hold the result of deflating the file\n>> before it gets written to disk.\n...\n>> This is a known area in Git where big files aren't handled well.\n>\n> As a curiosity, I've always done streaming decompression with zlib using\n> minimal buffer sizes (64k, perhaps).  I'm sure there is good reason why Git\n> doesn't do this (delta application?).  Do you know what it is?\n\nLaziness. Git originally assumed it would only be used for smaller\nsource files written by humans. Its easier to write the code as a\nsingle malloc'd buffer than to stream it. We'd like to fix it, but\nits harder than it sounds. Today we copy the file into a buffer\nbefore we deflate and compute the SHA-1 as this prevents us from\ngetting into a consistency error when the file is modified between\nthese two stages.\n\n-- \nShawn.\n"},{"id":"152572","messageId":"20101004185854.GA6466@burratino","threadId":"25332","inReplyTo":"AANLkTimbdrAqoWMxiteT5zNYmwHp8M698BEv1FLuiAxx@mail.gmail.com","subject":"Re: large files and low memory","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-04T18:58:54Z","receivedAt":"2010-10-04T18:58:54Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Shawn Pearce wrote:\n\n> The mmap() isn't the problem. Its the allocation of a buffer that is\n> larger than the file in order to hold the result of deflating the file\n> before it gets written to disk.\n\nWasn't this already fixed, at least in some cases?\n\ncommit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1)\nAuthor: Nicolas Pitre \nDate: Sat Feb 20 23:27:31 2010 -0500\n\n sha1_file: don't malloc the whole compressed result when writing out objects\n\n There is no real advantage to malloc the whole output buffer and\n deflate the data in a single pass when writing loose objects. That is\n like only 1% faster while using more memory, especially with large\n files where memory usage is far more. It is best to deflate and write\n the data out in small chunks reusing the same memory instead.\n\n For example, using 'git add' on a few large files averaging 40 MB ...\n\n Before:\n 21.45user 1.10system 0:22.57elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n 0inputs+828040outputs (0major+142640minor)pagefaults 0swaps\n\n After:\n 21.50user 1.25system 0:22.76elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n 0inputs+828040outputs (0major+104408minor)pagefaults 0swaps\n\n While the runtime stayed relatively the same, the number of minor page\n faults went down significantly.\n\n Signed-off-by: Nicolas Pitre \n Signed-off-by: Junio C Hamano \n"},{"id":"152576","messageId":"AANLkTin-mbzt93DWtp71vYBEUcDLHgo=G-6zynT5NC_r@mail.gmail.com","threadId":"25332","inReplyTo":"20101004185854.GA6466@burratino","subject":"Re: large files and low memory","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-10-04T19:11:22Z","receivedAt":"2010-10-04T19:11:22Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Mon, Oct 4, 2010 at 11:58 AM, Jonathan Nieder wrote:\n> Shawn Pearce wrote:\n>\n>> The mmap() isn't the problem.  Its the allocation of a buffer that is\n>> larger than the file in order to hold the result of deflating the file\n>> before it gets written to disk.\n>\n> Wasn't this already fixed, at least in some cases?\n>\n> commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1)\n> Author: Nicolas Pitre \n> Date:   Sat Feb 20 23:27:31 2010 -0500\n>\n>    sha1_file: don't malloc the whole compressed result when writing out objects\n\nThis change only removes the deflate copy. But due to the SHA-1\nconsistency issue I alluded to earlier, I think we're still making a\nfull copy of the file in memory before we SHA-1 it or deflate it. So\nNico halved the memory usage, but we're still using 1x the size of the\nfile rather than ~2x.\n\n\n-- \nShawn.\n"},{"id":"152579","messageId":"20101004191657.GC6466@burratino","threadId":"25332","inReplyTo":"AANLkTin-mbzt93DWtp71vYBEUcDLHgo=G-6zynT5NC_r@mail.gmail.com","subject":"Re: large files and low memory","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-04T19:16:57Z","receivedAt":"2010-10-04T19:16:57Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Shawn Pearce wrote:\n\n> This change only removes the deflate copy. But due to the SHA-1\n> consistency issue I alluded to earlier, I think we're still making a\n> full copy of the file in memory before we SHA-1 it or deflate it.\n\nHmm, I _think_ we still use mmap for that (which is why 748af44c needs\nto compare the sha1 before and after).\n\nBut\n\n 1) a one-pass calculation would presumably be a little (5%?) faster\n 2) if there are smudge/clean filters or autocrlf involved, the\n cleaned-up file is backed by swap and this all becomes moot.\n"},{"id":"152625","messageId":"20101005005003.GA2768@nibiru.local","threadId":"25332","inReplyTo":"AANLkTimbdrAqoWMxiteT5zNYmwHp8M698BEv1FLuiAxx@mail.gmail.com","subject":"Re: large files and low memory","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-05T00:50:04Z","receivedAt":"2010-10-05T00:50:04Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Shawn Pearce wrote:\n\n> The mmap() isn't the problem. Its the allocation of a buffer that is\n> larger than the file in order to hold the result of deflating the file\n> before it gets written to disk. \n> When the file is bigger than physical memory, the kernel has to\n> page in parts of the file as well as swap in and out parts of\n> that allocated buffer to hold the deflated file.\n\nWhat are the access pattern of these memory areas ?\nPerhaps madvise() could help ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"152626","messageId":"20101005005715.GB2768@nibiru.local","threadId":"25332","inReplyTo":"20101004185854.GA6466@burratino","subject":"Re: large files and low memory","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-05T00:57:15Z","receivedAt":"2010-10-05T00:57:15Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Jonathan Nieder wrote:\n> Shawn Pearce wrote:\n> \n> > The mmap() isn't the problem. Its the allocation of a buffer that is\n> > larger than the file in order to hold the result of deflating the file\n> > before it gets written to disk.\n> \n> Wasn't this already fixed, at least in some cases?\n> \n> commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1)\n> Author: Nicolas Pitre \n> Date: Sat Feb 20 23:27:31 2010 -0500\n\nI guess I'll have to do a update.\n\nBut: latest tag (1.7.3.1) doesnt build:\n\n\n CC read-cache.o\n read-cache.c: In function `fill_stat_cache_info':\n read-cache.c:73: structure has no member named `st_ctim'\n read-cache.c:74: structure has no member named `st_mtim'\n read-cache.c: In function `read_index_from':\n read-cache.c:1334: structure has no member named `st_mtim'\n read-cache.c: In function `write_index':\n read-cache.c:1614: structure has no member named `st_mtim'\n make: *** [read-cache.o] Fehler 1\n \nIs my libc too old ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"152627","messageId":"20101005005902.GC2768@nibiru.local","threadId":"25332","inReplyTo":"AANLkTi=nzyRgiBL07f2oeyjdUc8XnaKbJEJ+k8_g9rQ1@mail.gmail.com","subject":"Re: large files and low memory","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-05T00:59:02Z","receivedAt":"2010-10-05T00:59:02Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Shawn Pearce wrote:\n\n> Laziness. Git originally assumed it would only be used for smaller\n> source files written by humans. Its easier to write the code as a\n> single malloc'd buffer than to stream it. We'd like to fix it, but\n> its harder than it sounds. Today we copy the file into a buffer\n> before we deflate and compute the SHA-1 as this prevents us from\n> getting into a consistency error when the file is modified between\n> these two stages.\n\nhmm, perhaps copy it to a temporary file, if it's too large ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"152628","messageId":"AANLkTinF5-WoYKDOSwMQudnK9SP-EUx3wWhUMi4z1kv3@mail.gmail.com","threadId":"25332","inReplyTo":"20101005005715.GB2768@nibiru.local","subject":"Re: large files and low memory","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-10-05T01:07:04Z","receivedAt":"2010-10-05T01:07:04Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Tue, Oct 5, 2010 at 00:57, Enrico Weigelt wrote:\n> * Jonathan Nieder wrote:\n>> Shawn Pearce wrote:\n>>\n>> > The mmap() isn't the problem.  Its the allocation of a buffer that is\n>> > larger than the file in order to hold the result of deflating the file\n>> > before it gets written to disk.\n>>\n>> Wasn't this already fixed, at least in some cases?\n>>\n>> commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1)\n>> Author: Nicolas Pitre \n>> Date:   Sat Feb 20 23:27:31 2010 -0500\n>\n> I guess I'll have to do a update.\n>\n> But: latest tag (1.7.3.1) doesnt build:\n>\n>\n>    CC read-cache.o\n>    read-cache.c: In function `fill_stat_cache_info':\n>    read-cache.c:73: structure has no member named `st_ctim'\n>    read-cache.c:74: structure has no member named `st_mtim'\n>    read-cache.c: In function `read_index_from':\n>    read-cache.c:1334: structure has no member named `st_mtim'\n>    read-cache.c: In function `write_index':\n>    read-cache.c:1614: structure has no member named `st_mtim'\n>    make: *** [read-cache.o] Fehler 1\n>\n> Is my libc too old ?\n\nThose lines are accessing members called st_ctime, i.e. with an \"e\" at\nthe end, but your errors just report \"st_ctim\". What gives?\n"},{"id":"152632","messageId":"20101005011009.GD9994@burratino","threadId":"25332","inReplyTo":"20101005005715.GB2768@nibiru.local","subject":"Re: large files and low memory","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-05T01:10:09Z","receivedAt":"2010-10-05T01:10:09Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Enrico Weigelt wrote:\n\n> CC read-cache.o\n> read-cache.c: In function `fill_stat_cache_info':\n> read-cache.c:73: structure has no member named `st_ctim'\n> read-cache.c:74: structure has no member named `st_mtim'\n> read-cache.c: In function `read_index_from':\n> read-cache.c:1334: structure has no member named `st_mtim'\n> read-cache.c: In function `write_index':\n> read-cache.c:1614: structure has no member named `st_mtim'\n> make: *** [read-cache.o] Fehler 1\n> \n> Is my libc too old ?\n\nWhat platform are you on? You probably need USE_ST_TIMESPEC;\nif so, please send a makefile patch so the next person trying\nit doesn't need to worry about it.\n\nAlso, please don't destroy the cc lists; it makes it hard for\npeople with some mail setups (e.g., mine) to notice when you've\nreplied to them.\n"},{"id":"152649","messageId":"20101005073505.GA14821@nibiru.local","threadId":"25332","inReplyTo":"20101005011009.GD9994@burratino","subject":"Re: large files and low memory","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-05T07:35:06Z","receivedAt":"2010-10-05T07:35:06Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Jonathan Nieder wrote:\n\n> What platform are you on? \n\nGNU/Linux. glibc-2.25\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"152650","messageId":"20101005074144.GA22564@nibiru.local","threadId":"25332","inReplyTo":"20101005005902.GC2768@nibiru.local","subject":"Re: large files and low memory","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-05T07:41:45Z","receivedAt":"2010-10-05T07:41:45Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Enrico Weigelt wrote:\n\n\n\nFound another possible bottleneck: git-commit seems to scan through\na lot of files. Shouldnt it just create a commit object from the\ncurrent index and update the head ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"152653","messageId":"vpqpqvpf4qm.fsf@bauges.imag.fr","threadId":"25332","inReplyTo":"20101005074144.GA22564@nibiru.local","subject":"Re: large files and low memory","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2010-10-05T08:01:05Z","receivedAt":"2010-10-05T08:01:05Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Enrico Weigelt writes:\n\n> * Enrico Weigelt wrote:\n>\n> \n>\n> Found another possible bottleneck: git-commit seems to scan through\n> a lot of files. Shouldnt it just create a commit object from the\n> current index and update the head ?\n\ngit commit will show what's being commited (the output of \"git commit\n--dry-run\") in your editor, hence it needs to compute that.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"152655","messageId":"20101005081700.GB22564@nibiru.local","threadId":"25332","inReplyTo":"vpqpqvpf4qm.fsf@bauges.imag.fr","subject":"Re: large files and low memory","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-05T08:17:00Z","receivedAt":"2010-10-05T08:17:00Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Matthieu Moy wrote:\n\n> git commit will show what's being commited (the output of \"git commit\n> --dry-run\") in your editor, hence it needs to compute that.\n\nhmm, is there any way to get around this ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"152660","messageId":"AANLkTimHKyQhAzuh4Hsh7=bEet1T0mNRSMgiZkhWQwSN@mail.gmail.com","threadId":"25332","inReplyTo":"20101005074144.GA22564@nibiru.local","subject":"Re: large files and low memory","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-10-05T10:13:35Z","receivedAt":"2010-10-05T10:13:35Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Oct 5, 2010 at 2:41 PM, Enrico Weigelt wrote:\n> * Enrico Weigelt wrote:\n>\n> \n>\n> Found another possible bottleneck: git-commit seems to scan through\n> a lot of files. Shouldnt it just create a commit object from the\n> current index and update the head ?\n\nYou mean a lot of stat()? There is no way to avoid that unless you set\nassume-unchanged bits. Or you could use\nwrite-tree/commit-tree/update-ref directly.\n-- \nDuy\n"},{"id":"152662","messageId":"AANLkTik1kzzzOKFaJCWtwiYf5Vz8oNjZXD+-SAq-ZvAa@mail.gmail.com","threadId":"25332","inReplyTo":"20101004191657.GC6466@burratino","subject":"Re: large files and low memory","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-10-05T10:59:34Z","receivedAt":"2010-10-05T10:59:34Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Oct 5, 2010 at 2:16 AM, Jonathan Nieder wrote:\n> Shawn Pearce wrote:\n>\n>> This change only removes the deflate copy.  But due to the SHA-1\n>> consistency issue I alluded to earlier, I think we're still making a\n>> full copy of the file in memory before we SHA-1 it or deflate it.\n>\n> Hmm, I _think_ we still use mmap for that (which is why 748af44c needs\n> to compare the sha1 before and after).\n\nJust tried valgrind massif on a 200MB file with master. It used ~270kb\nheap. I haven't tested but I believe git-checkout will keep the whole\ninflated copy in memory. So git-add alone does not help much.\n-- \nDuy\n"},{"id":"152664","messageId":"AANLkTin+i7VZApDRmairgpMth4MzPoDF=oHt-UHCt98E@mail.gmail.com","threadId":"25332","inReplyTo":"20101005081700.GB22564@nibiru.local","subject":"Re: large files and low memory","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2010-10-05T11:29:11Z","receivedAt":"2010-10-05T11:29:11Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On Tue, Oct 5, 2010 at 10:17, Enrico Weigelt wrote:\n> * Matthieu Moy wrote:\n>\n>> git commit will show what's being commited (the output of \"git commit\n>> --dry-run\") in your editor, hence it needs to compute that.\n>\n> hmm, is there any way to get around this ?\n>\n\nTry \"git commit -q -uno\". This should skip creation of summary in the\ncommit message and lookup for untracked files.\nThis will somewhat speed things up\n"},{"id":"152665","messageId":"vpqbp78x418.fsf@bauges.imag.fr","threadId":"25332","inReplyTo":"AANLkTin+i7VZApDRmairgpMth4MzPoDF=oHt-UHCt98E@mail.gmail.com","subject":"Re: large files and low memory","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2010-10-05T11:38:59Z","receivedAt":"2010-10-05T11:38:59Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Alex Riesen writes:\n\n> On Tue, Oct 5, 2010 at 10:17, Enrico Weigelt wrote:\n>> * Matthieu Moy wrote:\n>>\n>>> git commit will show what's being commited (the output of \"git commit\n>>> --dry-run\") in your editor, hence it needs to compute that.\n>>\n>> hmm, is there any way to get around this ?\n>>\n>\n> Try \"git commit -q -uno\". This should skip creation of summary in the\n> commit message and lookup for untracked files.\n\nTo avoid including the summary, the option would be --no-status (-q\nmakes commit less verbose in stdout, not in COMMIT_EDITMSG).\n\nBut\n\nstrace -fe lstat64 git commit -uno --no-status -q\n\nstill shows lstat64 for each tracked file in my working tree (even\nwhen using -m to avoid launching the editor). I don't know if this is\nintended, or just that nobody cared enough to optimize this.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"152668","messageId":"AANLkTi=ni_21uP7oPrmB3obMscg0o9Z=uawkM2PoxWxT@mail.gmail.com","threadId":"25332","inReplyTo":"vpqbp78x418.fsf@bauges.imag.fr","subject":"Re: large files and low memory","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-10-05T11:55:53Z","receivedAt":"2010-10-05T11:55:53Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Oct 5, 2010 at 6:38 PM, Matthieu Moy\n wrote:\n> Alex Riesen writes:\n>\n>> On Tue, Oct 5, 2010 at 10:17, Enrico Weigelt wrote:\n>>> * Matthieu Moy wrote:\n>>>\n>>>> git commit will show what's being commited (the output of \"git commit\n>>>> --dry-run\") in your editor, hence it needs to compute that.\n>>>\n>>> hmm, is there any way to get around this ?\n>>>\n>>\n>> Try \"git commit -q -uno\". This should skip creation of summary in the\n>> commit message and lookup for untracked files.\n>\n> To avoid including the summary, the option would be --no-status (-q\n> makes commit less verbose in stdout, not in COMMIT_EDITMSG).\n>\n> But\n>\n> strace -fe lstat64 git commit -uno --no-status -q\n>\n> still shows lstat64 for each tracked file in my working tree (even\n> when using -m to avoid launching the editor). I don't know if this is\n> intended, or just that nobody cared enough to optimize this.\n\nI assume you do git-commit with no index modification at all. The\nindex refresh part dated back in 2888605 (builtin-commit: fix\npartial-commit support - 2007-11-18). Commit message does not tell why\nindex refresh is needed (for summary stat maybe?). If so, then we can\nskip refreshing if -q is given.\n-- \nDuy\n"},{"id":"152675","messageId":"20101005134744.GB12797@burratino","threadId":"25332","inReplyTo":"20101005073505.GA14821@nibiru.local","subject":"Re: large files and low memory","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-05T13:47:44Z","receivedAt":"2010-10-05T13:47:44Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Enrico Weigelt wrote:\n> * Jonathan Nieder wrote:\n\n>> What platform are you on? \n>\n> GNU/Linux. glibc-2.25\n\nHmm, I've heard of glib 2.25 but never glibc 2.25. :)\n\n $ /lib/libc.so.6 | head -1\n GNU C Library (Debian EGLIBC 2.11.2-6) stable release version 2.11.2, by Roland McGrath et al.\n"},{"id":"152691","messageId":"7vfwwk7fqz.fsf@alter.siamese.dyndns.org","threadId":"25332","inReplyTo":"AANLkTi=ni_21uP7oPrmB3obMscg0o9Z=uawkM2PoxWxT@mail.gmail.com","subject":"Re: large files and low memory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-10-05T16:42:44Z","receivedAt":"2010-10-05T16:42:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy writes:\n\n> I assume you do git-commit with no index modification at all. The\n> index refresh part dated back in 2888605 (builtin-commit: fix\n> partial-commit support - 2007-11-18). Commit message does not tell why\n> index refresh is needed (for summary stat maybe?). If so, then we can\n> skip refreshing if -q is given.\n\nMost likely to give a clean index to post-commit hook.\n"},{"id":"152701","messageId":"alpine.LFD.2.00.1010051504530.3107@xanadu.home","threadId":"25332","inReplyTo":"20101005005003.GA2768@nibiru.local","subject":"Re: large files and low memory","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-10-05T19:06:05Z","receivedAt":"2010-10-05T19:06:05Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 5 Oct 2010, Enrico Weigelt wrote:\n\n> * Shawn Pearce wrote:\n> \n> > The mmap() isn't the problem. Its the allocation of a buffer that is\n> > larger than the file in order to hold the result of deflating the file\n> > before it gets written to disk. \n> > When the file is bigger than physical memory, the kernel has to\n> > page in parts of the file as well as swap in and out parts of\n> > that allocated buffer to hold the deflated file.\n> \n> What are the access pattern of these memory areas ?\n\nPerfectly linear.\n\n> Perhaps madvise() could help ?\n\nPerhaps.\n\n\nNicolas\n"},{"id":"152702","messageId":"alpine.LFD.2.00.1010051510160.3107@xanadu.home","threadId":"25332","inReplyTo":"AANLkTimHKyQhAzuh4Hsh7=bEet1T0mNRSMgiZkhWQwSN@mail.gmail.com","subject":"Re: large files and low memory","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-10-05T19:12:29Z","receivedAt":"2010-10-05T19:12:29Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 5 Oct 2010, Nguyen Thai Ngoc Duy wrote:\n\n> On Tue, Oct 5, 2010 at 2:41 PM, Enrico Weigelt wrote:\n> > * Enrico Weigelt wrote:\n> >\n> > \n> >\n> > Found another possible bottleneck: git-commit seems to scan through\n> > a lot of files. Shouldnt it just create a commit object from the\n> > current index and update the head ?\n> \n> You mean a lot of stat()? There is no way to avoid that unless you set\n> assume-unchanged bits. Or you could use\n> write-tree/commit-tree/update-ref directly.\n\nAvoiding memory exhaustion is also going to help a lot as the stat() \ninformation will remain cached instead of requiring disk access. Just a \nguess given $subject.\n\n\nNicolas\n"},{"id":"152709","messageId":"alpine.LFD.2.00.1010051518570.3107@xanadu.home","threadId":"25332","inReplyTo":"20101004191657.GC6466@burratino","subject":"Re: large files and low memory","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-10-05T20:17:47Z","receivedAt":"2010-10-05T20:17:47Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 4 Oct 2010, Jonathan Nieder wrote:\n\n> Shawn Pearce wrote:\n> \n> > This change only removes the deflate copy. But due to the SHA-1\n> > consistency issue I alluded to earlier, I think we're still making a\n> > full copy of the file in memory before we SHA-1 it or deflate it.\n> \n> Hmm, I _think_ we still use mmap for that (which is why 748af44c needs\n> to compare the sha1 before and after).\n> \n> But\n> \n> 1) a one-pass calculation would presumably be a little (5%?) faster\n\nYou can't do a one-pass calculation. The first one is required to \ncompute the SHA1 of the file being added, and if that corresponds to an \nobject that we already have then the operation stops right there as \nthere is actually nothing to do. The second pass is to deflate the \ndata, and recompute the SHA1 to make sure what we deflated and written \nout is still the same data.\n\nIn the case of big files, what we need to do is to stream the file data \nin, compute the SHA1 and deflate it, in order to stream it out into a \ntemporary file, then rename it according to the final SHA1. This would \nallow Git to work with big files, but of course it won't be possible to \nknow if the object corresponding to the file is already known until all \nthe work has been done, possibly just to throw it away. But normally \nbig files are the minority.\n\n\nNicolas\n"},{"id":"152713","messageId":"20101005203450.GA2096@burratino","threadId":"25332","inReplyTo":"alpine.LFD.2.00.1010051518570.3107@xanadu.home","subject":"Re: large files and low memory","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-05T20:34:50Z","receivedAt":"2010-10-05T20:34:50Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Nicolas Pitre wrote:\n\n> You can't do a one-pass calculation. The first one is required to \n> compute the SHA1 of the file being added, and if that corresponds to an \n> object that we already have then the operation stops right there as \n> there is actually nothing to do.\n\nAh. Thanks for a reminder.\n\n> In the case of big files, what we need to do is to stream the file data \n> in, compute the SHA1 and deflate it, in order to stream it out into a \n> temporary file, then rename it according to the final SHA1. This would \n> allow Git to work with big files, but of course it won't be possible to \n> know if the object corresponding to the file is already known until all \n> the work has been done, possibly just to throw it away.\n\nTo make sure I understand correctly: are you suggesting that for big\nfiles we should skip the first pass?\n\nI suppose that makes sense: for small files, using a patch application\ntool to reach a postimage that matches an existing object is something\ngit historically needed to expect, but for typical big files:\n\n - once you've computed the SHA1, you've already invested a noticeable\n amount of time.\n - emailing patches around is difficult, making \"git am\" etc less important\n - hopefully git or zlib can notice when files are uncompressible,\n making the deflate not cost so much in that case.\n"},{"id":"152726","messageId":"alpine.LFD.2.00.1010051657440.3107@xanadu.home","threadId":"25332","inReplyTo":"20101005203450.GA2096@burratino","subject":"Re: large files and low memory","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-10-05T21:11:45Z","receivedAt":"2010-10-05T21:11:45Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 5 Oct 2010, Jonathan Nieder wrote:\n\n> Nicolas Pitre wrote:\n> \n> > You can't do a one-pass calculation. The first one is required to \n> > compute the SHA1 of the file being added, and if that corresponds to an \n> > object that we already have then the operation stops right there as \n> > there is actually nothing to do.\n> \n> Ah. Thanks for a reminder.\n> \n> > In the case of big files, what we need to do is to stream the file data \n> > in, compute the SHA1 and deflate it, in order to stream it out into a \n> > temporary file, then rename it according to the final SHA1. This would \n> > allow Git to work with big files, but of course it won't be possible to \n> > know if the object corresponding to the file is already known until all \n> > the work has been done, possibly just to throw it away.\n> \n> To make sure I understand correctly: are you suggesting that for big\n> files we should skip the first pass?\n\nFor big files we need a totally separate code path to process the file \ndata in small chunks at 'git add' time, using a loop containing \nread()+SHA1sum()+deflate()+write(). Then, if the SHA1 matches an \nexisting object we delete the temporary output file, otherwise we rename \nit as a valid object. No CRLF, no smudge filters, no diff, no deltas, \njust plain storage of huge objects, based on the value of \ncore.bigFileThreshold config option.\n\nSame thing on the checkout path: a simple loop to \nread()+inflate()+write() in small chunks.\n\nThat's the only sane way to kinda support big files with Git.\n\n> I suppose that makes sense: for small files, using a patch application\n> tool to reach a postimage that matches an existing object is something\n> git historically needed to expect, but for typical big files:\n> \n> - once you've computed the SHA1, you've already invested a noticeable\n> amount of time.\n> - emailing patches around is difficult, making \"git am\" etc less important\n> - hopefully git or zlib can notice when files are uncompressible,\n> making the deflate not cost so much in that case.\n\nEmailing is out of the question. We're talking file sizes in the \nhundreds of megabytes and above here. So yes, simply computing the SHA1 \nis a significant cost, given that you are going to trash your page cache \nin the process already, so better pay the price of deflating it at the \nsame time even if it turns out to be unnecessary.\n\n\nNicolas\n"},{"id":"152735","messageId":"20101005225150.GA27633@nibiru.local","threadId":"25332","inReplyTo":"alpine.LFD.2.00.1010051504530.3107@xanadu.home","subject":"Re: large files and low memory","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-05T22:51:50Z","receivedAt":"2010-10-05T22:51:50Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Nicolas Pitre wrote:\n\n> > > The mmap() isn't the problem. Its the allocation of a buffer that is\n> > > larger than the file in order to hold the result of deflating the file\n> > > before it gets written to disk. \n> > > When the file is bigger than physical memory, the kernel has to\n> > > page in parts of the file as well as swap in and out parts of\n> > > that allocated buffer to hold the deflated file.\n> > \n> > What are the access pattern of these memory areas ?\n> \n> Perfectly linear.\n\nIn this case, I wonder why my machine goes into thrashing so easily\n(P3 w/ 256MB ram). Seems the mmu/paging code doesnt recognize that\nthe previously-used pages can be kicked-off quickly ;-o\nPerhaps I should talk to the kernel folks.\n\n> > Perhaps madvise() could help ?\n> \n> Perhaps.\n\nhmm, so we should try it ;-p\nwhere'd be the right place to add it ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"}]}