{"thread":{"id":"3816","subject":"Re: Cygwin can't handle huge packfiles?","startedAt":"2006-04-06T20:57:24Z","lastAt":"2006-04-07T03:05:48Z","messageCount":3,"participants":["linux@horizon.com","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"18432","messageId":"20060406205724.12216.qmail@science.horizon.com","threadId":"3816","inReplyTo":null,"subject":"Re: Cygwin can't handle huge packfiles?","fromName":"","fromEmail":"linux@horizon.com","sentAt":"2006-04-06T20:57:24Z","receivedAt":"2006-04-06T20:57:24Z","isPatch":false,"sender":{"key":"linux@horizon.com","avatar":null},"body":"> Right now we LRU the pack files and evict older ones when we\n> mmap too many, but the unit of eviction is the whole file, so it\n> would not help the case like yours at all.  It might be possible\n> to mmap only part of a packfile, but it would involve fairly\n> major surgery to sha1_file.c.\n\nThe simplest solution seems to be to limit pack file size to a reasonable\nfraction of a 32-bit address space.  Say, 0.5 G.\n\nThat should be a fairly straightforward hack to git-pack-objects.\nIt already emits two files; just make it emit more.\n\nYou can tweak the heurisitics to try to find a good break point: start\nthinking about splitting the pack when you get to one size, but don't\nforce a break until you hit a harder limit as long as the deltas are\nworking well.\n\nThis can all be adjustable with a command line and/or config file option\nto allow for the eventual demise of 32-bit systems.\n"},{"id":"18438","messageId":"7vk6a2uupy.fsf@assigned-by-dhcp.cox.net","threadId":"3816","inReplyTo":"20060406205724.12216.qmail@science.horizon.com","subject":"Re: Cygwin can't handle huge packfiles?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-04-06T23:53:29Z","receivedAt":"2006-04-06T23:53:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"linux@horizon.com writes:\n\n>> Right now we LRU the pack files and evict older ones when we\n>> mmap too many, but the unit of eviction is the whole file, so it\n>> would not help the case like yours at all.  It might be possible\n>> to mmap only part of a packfile, but it would involve fairly\n>> major surgery to sha1_file.c.\n>\n> The simplest solution seems to be to limit pack file size to a reasonable\n> fraction of a 32-bit address space.  Say, 0.5 G.\n\nI do not think that would help the original poster's situation\nwhere only 5 revs result in a 1.5G pack.  I would _almost_ say\n\"do not pack such a repository\", but there is the initial\ncloning over git-aware transports which always results in a\nrepository with a single pack.\n"},{"id":"18441","messageId":"20060407030548.22399.qmail@science.horizon.com","threadId":"3816","inReplyTo":"7vk6a2uupy.fsf@assigned-by-dhcp.cox.net","subject":"Re: Cygwin can't handle huge packfiles?","fromName":"","fromEmail":"linux@horizon.com","sentAt":"2006-04-07T03:05:48Z","receivedAt":"2006-04-07T03:05:48Z","isPatch":false,"sender":{"key":"linux@horizon.com","avatar":null},"body":"> I do not think that would help the original poster's situation\n> where only 5 revs result in a 1.5G pack.  I would _almost_ say\n> \"do not pack such a repository\", but there is the initial\n> cloning over git-aware transports which always results in a\n> repository with a single pack.\n\nHuh?  Why not?  That repository has a lot of files.  For compression,\nyou want all versions of a file in one pack, and with few versions that\nmakes it easier to split up, not harder.\n\nAs for network transport of packs, I haven't studied the details,\nbut if you allow \"thin packs\" that have deltas relative to\nobjects not in the pack, then breaking up the pack anywhere\nshould be legal.\n\nOr, if necessary, you can stuff an arbitrarily large file through\ngit-unpack-objects, which reads a stream from stdin without\nattempting to mmap it.\n\n\n(Speaking of unpack-objects.c, what's that \"static unsigned long eof\"\nvariable in there?  It never seems to be set to a non-zero value.)\n"}]}