{"thread":{"id":"5326","subject":"git clone dies (large git repository)","startedAt":"2006-08-18T22:42:06Z","lastAt":"2006-08-22T00:42:53Z","messageCount":6,"participants":["Troy Telford","Jakub Narebski","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"25593","messageId":"op.teh30gmyies9li@rygel.lnxi.com","threadId":"5326","inReplyTo":null,"subject":"git clone dies (large git repository)","fromName":"Troy Telford","fromEmail":"ttelford@linuxnetworx.com","sentAt":"2006-08-18T22:42:06Z","receivedAt":"2006-08-18T22:42:06Z","isPatch":false,"sender":{"key":"ttelford@linuxnetworx.com","avatar":null},"body":"I've got a git repository I use to manage a set of RPMs.  It's got history  \nstretching back for years, and imported nicely into git.  Since it's used  \nto create RPMs, the repository has a structure similar to this:\n.\n|--README\n|-- foo\n|    |--SOURCES\n|    |  |--foo.tar.bz2\n|    |  `--foo-build.patch\n|    `--SPECS\n|       `--foo.spec\n`-- bar\n      |--SOURCES\n      |  |--bar.tar.bz2\n      |  `--bar-build.patch\n      `--SPECS\n         `--bar.spec\n\nThe source tarballs are updated when there's a new version of the  \nsoftware; I don't need to worry about changes that are /inside/ the  \ntarball-- just that the tarball itself has changed.  As you can imagine, a  \nfair amount of the 'stuff' in the repository are these binary tarballs.\n\nThe total repository size (ie. the '.git' folder):  4GB\n\nI have only one complaint (and I can work around it anyway):  I can't 'git  \nclone' the repository.\n\nif I run:\ngit clone git://my.server.net/git/rpms\nI get the following output:\n\nremote: Generating pack...\nremote: Done counting 20971 objects.\nremote: Deltifying 20971 objects.\nremote:  100% (20971/20971) done\n3707.885MB  (21657 kB/s)\n\nremote: Total 20971, written 20971 (delta 9604), reused 20971 (delta 9604)\nerror: git-fetch-pack: unable to read from git-index-pack\nerror: git-index-pack died of signal 11\nfetch-pack from 'git://my.server.net/git/rpms' failed.\n\nIt's interesting to note that during the pack file transfer, it stops  \nincrementing at ~3700 MB; the pack file is 4.0 GB.  So either 300MB isn't  \nbeing transferred, or it's just not updating the display for the last few  \nhundred megs.\n\nMy workaround is to just use 'rsync' to copy the data (although scp works  \ntoo), then checkout the working copy.  After that, fetch/pull and push  \nwork fine.\n\nThe behavior is consistent with git v1.4.1 and v1.4.2, on SLES 9, SLES 10,  \nRHEL 4, and Gentoo.\n\nIt is also consistent if I clone via the git daemon, or the ssh protocol  \n('git clone server:/path/to/repo')\n\nI originally had everything as loose objects.  I then ran 'git-repack -d'  \non occasion, so I had a combination of a large pack file, smaller pack  \nfiles, and loose objects.  Finally, I tried 'git repack -a -d' and  \nconsolidated it all into a single 4GB pack file.  It didn't seem to make  \nmuch difference in the output.\n\nAm I bumping some sort of limitation within git, or have I uncovered a bug?\n-- \nTroy Telford\n"},{"id":"25606","messageId":"ec6qsa$aal$2@sea.gmane.org","threadId":"5326","inReplyTo":"op.teh30gmyies9li@rygel.lnxi.com","subject":"Re: git clone dies (large git repository)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-08-19T10:58:22Z","receivedAt":"2006-08-19T10:58:22Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Troy Telford wrote:\n\n> I originally had everything as loose objects.  I then ran 'git-repack -d'  \n> on occasion, so I had a combination of a large pack file, smaller pack  \n> files, and loose objects.  Finally, I tried 'git repack -a -d' and  \n> consolidated it all into a single 4GB pack file.  It didn't seem to make  \n> much difference in the output.\n> \n> Am I bumping some sort of limitation within git, or have I uncovered\n> a bug? \n\nYou _might_ have bumped into filesystem limit on file size, or system limit\non mmap size.\n\nIIRC it was to be addressed (splitting packs into manageable hunks).\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"25626","messageId":"7vfyfs313t.fsf@assigned-by-dhcp.cox.net","threadId":"5326","inReplyTo":"op.teh30gmyies9li@rygel.lnxi.com","subject":"Re: git clone dies (large git repository)","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-08-19T20:46:30Z","receivedAt":"2006-08-19T20:46:30Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Troy Telford\" <ttelford@linuxnetworx.com> writes:\n\n> I originally had everything as loose objects.  I then ran 'git-repack\n> -d' on occasion, so I had a combination of a large pack file, smaller\n> pack  files, and loose objects.  Finally, I tried 'git repack -a -d'\n> and  consolidated it all into a single 4GB pack file.  It didn't seem\n> to make  much difference in the output.\n>\n> Am I bumping some sort of limitation within git, or have I uncovered a bug?\n\nThe former.  Unfortunately this comes from an old design\ndecision.\n\nFortunately this design decision is not something irreversible\n(see Chapter 1 of Documentation/ManagementStyle in the kernel\nrepository ;-).\n\nThe packfile is a dual-use format.  When used for network\ntransfer, we only send the .pack file and have the recipient\nreconstruct the corresponding .idx file.  When used locally, we\nneed both .pack and .idx file; .pack contains the meat of the\ndata, and .idx allows us random access to the objects stored in\nthe corresponding .pack file.\n\nWhat is interesting is that .pack format does not have (as far\nas I know) inherent size limitation.  However, .idx file has\nhardcoded 32-bit offsets into .pack -- hence, in practice, you\ncannot use a .pack that is over 4GB locally.\n\nOne crude workaround that would work _today_ for your situation\nwithout changing file formats would be to use git-fetch into an\nempty repository (and do ref cloning by hand) instead of using\ngit-clone.  git-fetch gets .pack data over the wire and explode\nthe objects contained in the stream into individual objects (as\nopposed to git-clone gets .pack data, stores it as a .pack and\ntries to create corresponding .idx which in your case would bust\nthe 32-bit limit and fail).\n\nThis is from a private note I sent to Linus on Jun 26 2005 when\npack & idx pairs were initially introduced.\n\n - Design decision.  As before, you have assumption that nothing\n   is longer than 2^32 bytes.  I am not unhappy with that\n   restriction with individual objects (even their uncompressed\n   size limited below 4GB or even 2GB is fine --- after all we\n   are talking about a source control system).  I am however\n   wondering if we would regret it later to have a packed file\n   also limited to 4GB by having object_entry.offset \"unsigned\n   long\" (and fwrite htonl'ed 4 bytes).  I personally do not\n   have problem with this, but I can easily see HPA frowning on\n   us.  He didn't like it when I said \"in GIT world, file sizes\n   and offsets are of type 'unsigned long'\" some time ago.\n\nI do not have a copy of a response from Linus to this point, but\nif I recall things correctly, since then, the plan always has\nbeen (1) to limit the size of individual packfiles to fit within\nthe idx limit and/or (2) extend the idx format to be able to\nexpress offset over 2^32.  The latter is possible because idx\nfile is a local matter, used only for local accesses and does\nnot get set over the wire.\n\nHowever, even if we revise the .idx file format, we have another\npractical problem to solve.  Currently we assume that we can mmap\none packfile as a whole and do a random access into it.  This\nneeds to be changed so that we (perhaps optionally, only when\ndealing with a huge packfile) mmap part of a .pack at a time.\n\nI recall more recently (as opposed to the heated discussion\nimmediately after packfile was introduced June last year) we had\nanother discussion about people not being able to mmap huge\npackfiles, and partial mmapping was one of the things that were\ndiscussed there.\n"},{"id":"25719","messageId":"op.tenp9fv1ies9li@rygel.lnxi.com","threadId":"5326","inReplyTo":"7vfyfs313t.fsf@assigned-by-dhcp.cox.net","subject":"Re: git clone dies (large git repository)","fromName":"Troy Telford","fromEmail":"ttelford@linuxnetworx.com","sentAt":"2006-08-21T23:30:41Z","receivedAt":"2006-08-21T23:30:41Z","isPatch":false,"sender":{"key":"ttelford@linuxnetworx.com","avatar":null},"body":"On Sat, 19 Aug 2006 14:46:30 -0600, Junio C Hamano <junkio@cox.net> wrote:\n\n> What is interesting is that .pack format does not have (as far\n> as I know) inherent size limitation.  However, .idx file has\n> hardcoded 32-bit offsets into .pack -- hence, in practice, you\n> cannot use a .pack that is over 4GB locally.\n\nConfessing my (complete, total, frightening) ignorance about git:  Is it  \neven possible to take a large pack file and split it into smaller packs?\n\nI'm thinking of it as an option for git-repack-- that the user can set the  \nmaximum size of any individual pack, and after that limit is reached, a  \nnew pack file is started.  (ie. --max-size 2GB) and will end up with two  \npacks, each 2GB in size.\n\nThat being said -- I've been able to work around it (although I haven't  \ntried your suggestion yet); it's not a 'critical' problem.  I'm now just  \ncurious if my fantasy (above) makes sense.\n-- \nTroy Telford\n"},{"id":"25720","messageId":"7v3bbpty7j.fsf@assigned-by-dhcp.cox.net","threadId":"5326","inReplyTo":"op.tenp9fv1ies9li@rygel.lnxi.com","subject":"Re: git clone dies (large git repository)","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-08-22T00:23:44Z","receivedAt":"2006-08-22T00:23:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Troy Telford\" <ttelford@linuxnetworx.com> writes:\n\n> I'm thinking of it as an option for git-repack-- that the user can set\n> the maximum size of any individual pack, and after that limit is\n> reached, a  new pack file is started.  (ie. --max-size 2GB) and will\n> end up with two  packs, each 2GB in size.\n\nThe way I would suggest you do it is not by size but by distance\nfrom the latest.  If you want to split the kernel history for\nexample, you repack up to 2.6.14 for example, and then repack\nthe remainder.  That way, you can optimize for size for older\n(presumably less frequently used) data while optimizing for\nspeed for more reent stuff.\n\nThere is no wrapper support for the above splitting in\ngit-repack.  The low-level plumbing tools can be used this way\nfor example:\n\n\tname=`\n                git rev-list --objects $list_old_tags_here |\n                git pack-objects --window=50 --depth=50 --non-empty .tmp-pack\n        ` &&\n        mv -f .tmp-pack-$name.{pack,idx} .git/objects/pack/\n\n\tname=`\n\t\tgit revlist --objects --all --not $list_old_tags_here |\n\t\tgit pack-objects --non-empty .tmp-pack\n\t` &&\n        mv -f .tmp-pack-$name.{pack,idx} .git/objects/pack/\n\nIf you are splitting into more than two, you would instead have\nmore than one $list_old_tags_here list, and iterate them\nthrough, something like:\n\n\tpack_between () {\n        \talready_done=\"$1\"\n                do_this_time=\"$2\"\n                w=${3-10}\n\t\tname=`\n\t\t\tgit rev-list --objects \\\n                        \t$do_this_time \\\n                                --not $already_done |\n\t\t\tgit pack-objects --window=$w --depth=$w \\\n                        \t--non-empty .tmp-pack\n                ` &&\n                mv -f .tmp-pack-$name.{pack,idx} .git/objects/pack/\n\t}\n\n\tpack_between \"\" \"$prehistoric_tag_list\" 100\n\tpack_between \"$prehistoric_tag_list\" \"$more_recent_tag_list\" 50\n\tpack_between \"$more_recent_tag_list\" --all\n\nAll untested, of course, so do not play with it in your precious\nrepository you do not have any other copy, but hopefully you get\nan idea ;-).\n"},{"id":"25721","messageId":"ecdjub$bed$1@sea.gmane.org","threadId":"5326","inReplyTo":"7v3bbpty7j.fsf@assigned-by-dhcp.cox.net","subject":"Re: git clone dies (large git repository)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-08-22T00:42:53Z","receivedAt":"2006-08-22T00:42:53Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> \"Troy Telford\" <ttelford@linuxnetworx.com> writes:\n> \n>> I'm thinking of it as an option for git-repack-- that the user can set\n>> the maximum size of any individual pack, and after that limit is\n>> reached, a  new pack file is started.  (ie. --max-size 2GB) and will\n>> end up with two  packs, each 2GB in size.\n> \n> The way I would suggest you do it is not by size but by distance\n> from the latest.  If you want to split the kernel history for\n> example, you repack up to 2.6.14 for example, and then repack\n> the remainder.  That way, you can optimize for size for older\n> (presumably less frequently used) data while optimizing for\n> speed for more reent stuff.\n\nIf there would be some enhancement to pack files allowing either limiting\nthe size of pack (e.g. filesystem limits, mmap limits) and/or mmaping only\nfragment or fragments of pack file, the maximal size of the pack, or\nmaximal size of the mmapped fragment should be configurable per repository,\nnot olny as an option to some command (git-repack for example).\n\nProbably would need some enhancement to git-fetch/git-clone too...\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"}]}