{"thread":{"id":"8273","subject":"[PATCH] Prevent megablobs from gunking up git packs","startedAt":"2007-05-22T06:14:32Z","lastAt":"2007-05-25T05:44:58Z","messageCount":24,"participants":["Dana How","Shawn O. Pearce","Junio C Hamano","Jakub Narebski","Nicolas Pitre","Johannes Schindelin","david@lang.hm","Geert Bosch"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"42948","messageId":"46528A48.9050903@gmail.com","threadId":"8273","inReplyTo":null,"subject":"[PATCH] Prevent megablobs from gunking up git packs","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-22T06:14:32Z","receivedAt":"2007-05-22T06:14:32Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"\nUsing fast-import and repack with the max-pack-size patch,\n3628 commits were imported from Perforce comprising\n100.35GB (uncompressed) in 38829 blobs, and saved in\n7 packfiles of 12.5GB total (--window=0 and --depth=0 were\nused due to runtime limits). When using these packfiles,\nseveral git commands showed very large process sizes,\nand some slowdowns (compared to comparable operations\non the linux kernel repo) were also apparent.\n\ngit stores data in loose blobs or in packfiles. The former\nhas essentially now become an exception mechanism, to store\nexceptionally *young* blobs. Why not use this to store\nexceptionally *large* blobs as well? This allows us to\nre-use all the \"exception\" machinery with only a small change.\n\nRepacking the entire repository with a max-blob-size of 256KB\nresulted in a single 13.1MB packfile, as well as 2853 loose\nobjects totaling 15.4GB compressed and 100.08GB uncompressed,\n11 files per objects/xx directory on average. All was created\nin half the runtime of the previous yet with standard\n--window=10 and --depth=50 parameters. The data in the\npackfile was 270MB uncompressed in 35976 blobs. Operations\nsuch as \"git-log --pretty=oneline\" were about 30X faster\non a cold cache and 2 to 3X faster otherwise. Process sizes\nremained reasonable.\n\nThis patch implements the following:\n1. git pack-objects takes a new --max-blob-size=N flag,\n with the effect that only blobs less than N KB are written\n to the packfiles(s). If a blob was in a pack but violates\n this limit (perhaps the packs were created by fast-import\n or max-blob-size was reduced), then a new loose object\n is written out if needed so the data is not lost.\n2. git repack inspects repack.maxblobsize . If set, its\n value is passed to git pack-objects on the command line.\n The user should change repack.maxblobsize , NOT specify\n --max-blob-size=N .\n3. No other caller of git pack-objects supplies this new flag,\n so other callers see no change.\n\nThis patch is on top of the earlier max-pack-size patch,\nbecause I thought I needed some behavior it supplied,\nbut could be rebased on master if desired.\n\nSigned-off-by: Dana L. How \n---\n builtin-pack-objects.c | 33 +++++++++++++++++++++++++++++++--\n cache.h | 2 ++\n git-repack.sh | 4 ++++\n sha1_file.c | 17 +++++++++++++++--\n 4 files changed, 52 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 930b57a..e88f6b7 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -16,7 +16,7 @@\n \n static const char pack_usage[] = \"\\\n git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n-\t[--local] [--incremental] [--window=N] [--depth=N] \\n\\\n+\t[--local] [--incremental] [--window=N] [--depth=N] [--max-blob-size=N]\\n\\\n \t[--no-reuse-delta] [--no-reuse-object] [--delta-base-offset] \\n\\\n \t[--non-empty] [--revs [--unpacked | --all]*] [--reflog] \\n\\\n \t[--stdout | base-name] [size >= max_blob_size) {\n+\t\tif (e->in_pack) {\n+\t\t\t/* rewrite as loose object so git-repack doesn't lose data */\n+\t\t\tvoid *buf;\n+\t\t\tenum object_type type;\n+\t\t\tunsigned long size;\n+\t\t\tbuf = read_sha1_file(e->sha1, &type, &size);\n+\t\t\tif (!buf)\n+\t\t\t\tdie(\"unable to read %s\", sha1_to_hex(e->sha1));\n+\t\t\tif (write_sha1_file_ignore_packs(buf, size, typename(type), NULL) < 0)\n+\t\t\t\tdie(\"failed to write object\");\n+\t\t\tfree(buf);\n+\t\t}\n+\t\twritten++;\n+\t\treturn offset;\n+\t}\n+\n \te->offset = offset;\n \tsize = write_object(f, e, offset);\n \tif (!size) {\n@@ -1391,13 +1410,16 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n+\tsrc_size = src_entry->size;\n+\t/* prevent use if later dropped from packfile */\n+\tif (max_blob_size && (trg_size >= max_blob_size || src_size >= max_blob_size))\n+\t\treturn 0;\n \tmax_size = trg_size/2 - 20;\n \tmax_size = max_size * (max_depth - src_entry->depth) / max_depth;\n \tif (max_size == 0)\n \t\treturn 0;\n \tif (trg_entry->delta && trg_entry->delta_size <= max_size)\n \t\tmax_size = trg_entry->delta_size-1;\n-\tsrc_size = src_entry->size;\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -1701,6 +1723,13 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\tincremental = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!prefixcmp(arg, \"--max-blob-size=\")) {\n+\t\t\tchar *end;\n+\t\t\tmax_blob_size = strtoul(arg+16, &end, 0) * 1024;\n+\t\t\tif (!arg[16] || *end)\n+\t\t\t\tusage(pack_usage);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!prefixcmp(arg, \"--compression=\")) {\n \t\t\tchar *end;\n \t\t\tint level = strtoul(arg+14, &end, 0);\ndiff --git a/cache.h b/cache.h\nindex 3143853..ad39f67 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -343,6 +343,8 @@ extern int sha1_object_info(const unsigned char *, unsigned long *);\n extern void * read_sha1_file(const unsigned char *sha1, enum object_type *type, unsigned long *size);\n extern int hash_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *sha1);\n extern int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *return_sha1);\n+extern int write_sha1_file_ignore_packs(void *buf, unsigned long len, const char *type,\n+\t\t\t\t\tunsigned char *return_sha1);\n extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned char *);\n \n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\ndiff --git a/git-repack.sh b/git-repack.sh\nindex 4ea6e5b..c2b2112 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -35,6 +35,10 @@ true)\n \textra=\"$extra --delta-base-offset\" ;;\n esac\n \n+# handle blob limiting\n+mbs=\"`git config --int repack.maxblobsize`\"\n+[ -n \"$mbs\" ] && extra=\"$extra --max-blob-size=$mbs\"\n+\n PACKDIR=\"$GIT_OBJECT_DIRECTORY/pack\"\n PACKTMP=\"$GIT_OBJECT_DIRECTORY/.tmp-$$-pack\"\n rm -f \"$PACKTMP\"-*\ndiff --git a/sha1_file.c b/sha1_file.c\nindex e715527..8786af7 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1979,7 +1979,8 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n \treturn 0;\n }\n \n-int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *returnsha1)\n+static int write_sha1_file_core(void *buf, unsigned long len, const char *type,\n+\t\t\t\tint checkpacks, unsigned char *returnsha1)\n {\n \tint size, ret;\n \tunsigned char *compressed;\n@@ -1997,7 +1998,7 @@ int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned cha\n \tfilename = sha1_file_name(sha1);\n \tif (returnsha1)\n \t\thashcpy(returnsha1, sha1);\n-\tif (has_sha1_file(sha1))\n+\tif (checkpacks && has_sha1_file(sha1))\n \t\treturn 0;\n \tfd = open(filename, O_RDONLY);\n \tif (fd >= 0) {\n@@ -2062,6 +2063,18 @@ int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned cha\n \treturn move_temp_to_file(tmpfile, filename);\n }\n \n+int write_sha1_file(void *buf, unsigned long len, const char *type,\n+\t\t unsigned char *returnsha1)\n+{\n+\treturn write_sha1_file_core(buf, len, type, 1, returnsha1);\n+}\n+\n+int write_sha1_file_ignore_packs(void *buf, unsigned long len, const char *type,\n+\t\t\t\t unsigned char *returnsha1)\n+{\n+\treturn write_sha1_file_core(buf, len, type, 0, returnsha1);\n+}\n+\n /*\n * We need to unpack and recompress the object for writing\n * it out to a different file.\n-- \n1.5.2.rc3.726.g279d-dirty\n"},{"id":"42951","messageId":"20070522063050.GD11636@spearce.org","threadId":"8273","inReplyTo":"46528A48.9050903@gmail.com","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-05-22T06:30:50Z","receivedAt":"2007-05-22T06:30:50Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Dana How wrote:\n...\n> 7 packfiles of 12.5GB total (--window=0 and --depth=0 were\n> used due to runtime limits). When using these packfiles,\n...\n> Repacking the entire repository with a max-blob-size of 256KB\n> resulted in a single 13.1MB packfile, as well as 2853 loose\n> objects totaling 15.4GB compressed and 100.08GB uncompressed,\n> 11 files per objects/xx directory on average. All was created\n> in half the runtime of the previous yet with standard\n> --window=10 and --depth=50 parameters. The data in the\n> packfile was 270MB uncompressed in 35976 blobs. Operations\n> such as \"git-log --pretty=oneline\" were about 30X faster\n> on a cold cache and 2 to 3X faster otherwise. Process sizes\n> remained reasonable.\n\nCan you give me details about your system? Is this a 64 bit binary?\nWhat is your core.packedGitWindowSize and core.packedGitLimit set to?\n\nIt sounds like the packed version was almost 3 GiB smaller, but\nwas slower because we were mmap'ing far too much data at startup\nand that was making your OS page in things that you didn't really\nneed to have.\n\nMind trying git-log with a smaller core.packedGitWindow{Size,Limit}?\nPerhaps its just as simple as our defaults are far far too high for\nyour workload...\n\n-- \nShawn.\n"},{"id":"42955","messageId":"7vtzu58i4c.fsf@assigned-by-dhcp.cox.net","threadId":"8273","inReplyTo":"46528A48.9050903@gmail.com","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-22T06:52:51Z","receivedAt":"2007-05-22T06:52:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Dana How writes:\n\n> Using fast-import and repack with the max-pack-size patch,\n> 3628 commits were imported from Perforce comprising\n> 100.35GB (uncompressed) in 38829 blobs, and saved in\n> 7 packfiles of 12.5GB total (--window=0 and --depth=0 were\n> used due to runtime limits). When using these packfiles,\n> several git commands showed very large process sizes,\n> and some slowdowns (compared to comparable operations\n> on the linux kernel repo) were also apparent.\n>\n> git stores data in loose blobs or in packfiles. The former\n> has essentially now become an exception mechanism, to store\n> exceptionally *young* blobs. Why not use this to store\n> exceptionally *large* blobs as well? This allows us to\n> re-use all the \"exception\" machinery with only a small change.\n\nWell, I had an impression that mmapping a single loose object\n(and then munmapping it after done) would be more expensive than\nmmapping a whole pack and accessing that object through window,\nas long as you touch the same set of objects and the object in\nthe pack is not deltified.\n\n> Repacking the entire repository with a max-blob-size of 256KB\n> resulted in a single 13.1MB packfile, as well as 2853 loose\n> objects totaling 15.4GB compressed and 100.08GB uncompressed,\n> 11 files per objects/xx directory on average. All was created\n> in half the runtime of the previous yet with standard\n> --window=10 and --depth=50 parameters. The data in the\n> packfile was 270MB uncompressed in 35976 blobs. Operations\n> such as \"git-log --pretty=oneline\" were about 30X faster\n> on a cold cache and 2 to 3X faster otherwise. Process sizes\n> remained reasonable.\n\nI think more reasonable comparison to figure out what is really\ngoing on would be to create such a pack with the same 0/0 window\nand depth (i.e. \"keeping the huge objects out of the pack\" would\nbe the only difference with the \"horrible\" case). With huge\npacks, I wouldn't be surprised if seeking to extract base object\nfrom a far away part of a packfile takes a lot longer than\nreading delta and applying the delta to base object that is kept\nin the in-core delta base cache.\n\nAlso if you mean by \"process size\" the total VM size, not RSS, I\nthink it is a wrong measure. As long as you do not touch the\nrest of the pack, even if you mmap a huge packfile, you would\nnot bring that much data actually into your main memory, would\nyou? Well, assuming that your mmap() implementation and virtual\nmemory subsystem does a descent job... maybe we are spoiled by\nLinux here...\n"},{"id":"42958","messageId":"56b7f5510705220033p43d20aabnbf86f1f6959d611a@mail.gmail.com","threadId":"8273","inReplyTo":"20070522063050.GD11636@spearce.org","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-22T07:33:23Z","receivedAt":"2007-05-22T07:33:23Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 5/21/07, Shawn O. Pearce wrote:\n> Dana How wrote:\n> > ... Operations\n> > such as \"git-log --pretty=oneline\" were about 30X faster\n> > on a cold cache and 2 to 3X faster otherwise. Process sizes\n> > remained reasonable.\n>\n> Can you give me details about your system? Is this a 64 bit binary?\nRHEL4/Nahant on an Opteron. Yes.\n\n> What is your core.packedGitWindowSize and core.packedGitLimit set to?\nI didn't change the default.\n\n> It sounds like the packed version was almost 3 GiB smaller, but\n> was slower because we were mmap'ing far too much data at startup\n> and that was making your OS page in things that you didn't really\n> need to have.\nThe difference in size is because of the \"Custom compression levels\"\npatch -- now the loose objects use Z_BEST_SPEED, whereas the packs\nuse Z_DEFAULT_COMPRESSION.\n\n> Mind trying git-log with a smaller core.packedGitWindow{Size,Limit}?\n> Perhaps its just as simple as our defaults are far far too high for\n> your workload...\nI think that's a good idea and it should be easy to try tomorrow.\nIt will improve the cold cache case definitely.\n\nBut we need to consider both *read* and *creation* performance.\nThe portion of the repo I imported to git grows at about 500MB/week\n(compressed). Should I repack -a every week? Every month? In any case,\nshould I use default window/depth, or 0/0? If default, run-times are\nprohibitive (in fact, I've always killed each attempt so the machine\ncould be used for \"real\" work), and if 0/0, then I lose deltification\non all objects.\n\nThese megablobs really are outliers and stress the \"one size fits\nall\" approach of packing in git. As a thought experiment,\nlet's (1) pretend git-repack takes --max-blob-size= and --max-pack-size= ,\n(2) pretend the patch doesn't add the repack.maxblobsize variable,\nand (3) do the following:\n % git-repack -a -d --max-blob-size=256\n % git-repack --max-pack-size=2047 --window=0 --depth=0\nThe first step makes a digestible 13MB packfile, and the second\nputs all the megablobs in 6+ 2GB packfiles. Is there really any\nadvantage to carrying out the second step? If I'm processing\na 100MB+ blob, do I really care about an extra open(2) call?\n\nThanks,\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"42959","messageId":"56b7f5510705220100h77e91196r1784b33772911660@mail.gmail.com","threadId":"8273","inReplyTo":"7vtzu58i4c.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-22T08:00:06Z","receivedAt":"2007-05-22T08:00:06Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 5/21/07, Junio C Hamano wrote:\n> Dana How writes:\n> > git stores data in loose blobs or in packfiles. The former\n> > has essentially now become an exception mechanism, to store\n> > exceptionally *young* blobs. Why not use this to store\n> > exceptionally *large* blobs as well? This allows us to\n> > re-use all the \"exception\" machinery with only a small change.\n> Well, I had an impression that mmapping a single loose object\n> (and then munmapping it after done) would be more expensive than\n> mmapping a whole pack and accessing that object through window,\n> as long as you touch the same set of objects and the object in\n> the pack is not deltified.\nI agree with your comparison. However, if I'm processing a 100MB+\nblob, I doubt the extra open/mmap/munmap/close calls are going\nto matter to me. What I think _helped_ me was that, with the megablobs\npushed out of the pack, git-log etc could play around inside a\n\"tiny\" 13MB packfile very quickly. This packfile contained all the\ncommits, all the trees, and all the blobs < 256KB.\n\n> > Repacking the entire repository with a max-blob-size of 256KB\n> > resulted in a single 13.1MB packfile, as well as 2853 loose\n> > objects totaling 15.4GB compressed and 100.08GB uncompressed,\n> > 11 files per objects/xx directory on average. All was created\n> > in half the runtime of the previous yet with standard\n> > --window=10 and --depth=50 parameters. The data in the\n> > packfile was 270MB uncompressed in 35976 blobs. Operations\n> > such as \"git-log --pretty=oneline\" were about 30X faster\n> > on a cold cache and 2 to 3X faster otherwise. Process sizes\n> > remained reasonable.\n>\n> I think more reasonable comparison to figure out what is really\n> going on would be to create such a pack with the same 0/0 window\n> and depth (i.e. \"keeping the huge objects out of the pack\" would\n> be the only difference with the \"horrible\" case). With huge\n> packs, I wouldn't be surprised if seeking to extract base object\n> from a far away part of a packfile takes a lot longer than\n> reading delta and applying the delta to base object that is kept\n> in the in-core delta base cache.\nYes, changing only one variable at a time would be better.\nI will do that experiment. However, the huge pack _did_ have\n0/0, and the small pack had default/default, which I think is the\nreverse of what you concluded above?, so the experiment should\nmake things no better for the huge pack case.\n\n> Also if you mean by \"process size\" the total VM size, not RSS, I\n> think it is a wrong measure. As long as you do not touch the\n> rest of the pack, even if you mmap a huge packfile, you would\n> not bring that much data actually into your main memory, would\n> you? Well, assuming that your mmap() implementation and virtual\n> memory subsystem does a descent job... maybe we are spoiled by\n> Linux here...\nYou are right that the VM number was more shocking, but both\nwere too high. But let's compare using 12GB+ of packfiles versus 13MB.\nIn the former case, I'm depending on the sliding mmap windows doing\nthe right thing in an operating regime no one uses (which is why\nShawn was asking about my packedGitLimit settings etc), and in the\nlatter case, the packfile is <10% of the linux2.6 packfile but I have\nto endure an extra open/mmap/munmap/close sequence when accessing\nenormouse files. The small extra cost of the latter is more attractive\nto me than an unknown amount of tuning to get the former right,\nand in the former case I still have to figure out how to *create*\nthe packfiles efficiently.\n\nThere's actually an even more extreme example from my day job.\nThe software team has a project whose files/revisions would be\nsimilar to those in the linux kernel (larger commits, I'm sure).\nBut they have *ONE* 500MB file they check in because it takes\n2 or 3 days to generate and different people use different versions of it.\nI'm sure it has 50+ revisions now. If they converted to git and included\nthese blobs in their packfile, that's a 25GB uncompressed increase!\n*Every* git operation must wade through 10X -- 100X more packfile.\nOr it could be kept in 50+ loose objects in objects/xx ,\nrequiring a few extra syscalls by each user to get a new version.\n\nThanks,\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"42966","messageId":"f2uigr$ufj$1@sea.gmane.org","threadId":"8273","inReplyTo":"56b7f5510705220100h77e91196r1784b33772911660@mail.gmail.com","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-05-22T11:05:40Z","receivedAt":"2007-05-22T11:05:40Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dana How wrote:\n\n> There's actually an even more extreme example from my day job.\n> The software team has a project whose files/revisions would be\n> similar to those in the linux kernel (larger commits, I'm sure).\n> But they have *ONE* 500MB file they check in because it takes\n> 2 or 3 days to generate and different people use different versions of it.\n> I'm sure it has 50+ revisions now.  If they converted to git and included\n> these blobs in their packfile, that's a 25GB uncompressed increase!\n> *Every* git operation must wade through 10X -- 100X more packfile.\n> Or it could be kept in 50+ loose objects in objects/xx ,\n> requiring a few extra syscalls by each user to get a new version.\n\nOr keeping those large objects in separate, _kept_ packfile, containing\nonly those objects (which can delta well, even if they are large).\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"42985","messageId":"56b7f5510705220959x1b37a4adk537cc0cba1a27530@mail.gmail.com","threadId":"8273","inReplyTo":"f2uigr$ufj$1@sea.gmane.org","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-22T16:59:29Z","receivedAt":"2007-05-22T16:59:29Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 5/22/07, Jakub Narebski wrote:\n> Dana How wrote:\n> > There's actually an even more extreme example from my day job.\n> > The software team has a project whose files/revisions would be\n> > similar to those in the linux kernel (larger commits, I'm sure).\n> > But they have *ONE* 500MB file they check in because it takes\n> > 2 or 3 days to generate and different people use different versions of it.\n> > I'm sure it has 50+ revisions now. If they converted to git and included\n> > these blobs in their packfile, that's a 25GB uncompressed increase!\n> > *Every* git operation must wade through 10X -- 100X more packfile.\n> > Or it could be kept in 50+ loose objects in objects/xx ,\n> > requiring a few extra syscalls by each user to get a new version.\n> Or keeping those large objects in separate, _kept_ packfile, containing\n> only those objects (which can delta well, even if they are large).\n\nYes, I experimented with various changes to git-repack and\nhaving it create .keep files just before coming up with the maxblobsize\napproach. The problem with a 12GB+ repo is not only the large\nrepack time, but the fact that the repack time keeps growing with\nthe repo size. So, with split packs, I had repack create .keep\nfiles for all new packs except the last (fragmentary) one. The next\nrepack would then only repack new stuff plus the single fragmentary\npack, keeping repack time from growing (until you deleted the .keep\nfiles [just the ones with \"repack\" in them] to start over from scratch).\nBut this approach is not going to distribute commits and trees all that well.\n\nLast night before signing off Junio proposed some partitioning ideas.\nHe presented them as ordering things *within* one pack; what I had\ntried was making repack operate in 2 passes: the first one would create\npack(s) containing commits+trees+tags, the 2nd would create\npack(s) containing only blobs. Of course the first group would contain\nonly 1 tiny pack, and the latter 6 or 7 enormous packs. I also combined\nthis with the previous paragraph, putting .keep files on all but the last\npack in each group. Then the metadata always got repacked,\nand the blob data only got its \"tail\" repacked.\n\nLet's just stipulate that you've convinced me that putting everything\nin packs, and not ejecting megablobs, is better or equivalent on\nthe \"central\" git repository which will replace (part of) our Perforce\nrepository. What about the users' repositories?\n\nEach person at my day job has his own workstation. They are all\non a grid and are constantly running jobs in the background.\nEach person would have at least one personal repo. What should the\npacking strategy be there?\n\n(1) If we must put everything in packs, then we could:\n(1a) Repack everything in local repos, incurring large local runtimes.\n This extra work then denies the CPU cycles to the grid,\n which WILL be noticed and cause much whining.\n So the response will be to reduce window and/or turn\n on nodelta for some group of objects, worsening packing\n and failing to squash the whining. This happens across\n 20 to 30 workstations. Or we reduce the frequency of\n repacking and stagger it across the network. Since daily\n pull/fetch/checkout (\"sync\" in p4 parlance) grabs 400+ new\n revisions each day, if we make repacking weekly we have\n a policy that results in 400*5/2=1000 extra loose blobs on average,\n and there will still be whining. Why not just set maxblobsize\n to some size resulting in ~1000 loose blobs, leave window/depth\n at default, and enjoy <1hr repacking?\n(1b) Repack everything ONLY in the central repo, and have the users' repos\n point to it as an alternate. Now we have enormous network traffic.\n However, this is better than (1a), and was what I thought I'd be\n stuck with. We still do have the possible problem of excessive\n packing time on the central repo, but it's easier to solve/hide\n in just one place.\n(2) We repack everything but leave megablobs loose. Now packfiles\n are 13MB, repack time with default window/depth is <1hr, and we\n can repack each users' repository from his own cron job. This will\n be noticed, but it won't cause too much complaining. Most git\n operations by users will be against their local repos, but the\n server's db will still be an alternate to fetch at least megablobs.\n This is not a problem compared to Perforce, which stores *NO*\n repository state locally at all.\n\nI really think megablob ejection from packs makes a lot of sense for local\nrepos on a network of workstations. It lets me keep almost all repo\nstate locally very cheaply. It is just another consequence of the tendency\nthat an adequate solution that operates principally on only 13MB of data\ndoesn't have to work as hard or as carefully as something\noperating on the full 12GB -- three orders of magnitude larger.\n\nIf there's interest, I could submit my other alterations to git-repack.\nThey still have bugs which would take a while to work out since\neach run operates on 12GB of data. With quicker runtimes,\nmaxblobsize was much quicker to debug even though I made\nmore stupid mistakes at first ;-)\n\nThanks,\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"42986","messageId":"alpine.LFD.0.99.0705221329420.3366@xanadu.home","threadId":"8273","inReplyTo":"46528A48.9050903@gmail.com","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-22T17:38:03Z","receivedAt":"2007-05-22T17:38:03Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 21 May 2007, Dana How wrote:\n\n> \n> Using fast-import and repack with the max-pack-size patch,\n> 3628 commits were imported from Perforce comprising\n> 100.35GB (uncompressed) in 38829 blobs, and saved in\n> 7 packfiles of 12.5GB total (--window=0 and --depth=0 were\n> used due to runtime limits). When using these packfiles,\n> several git commands showed very large process sizes,\n> and some slowdowns (compared to comparable operations\n> on the linux kernel repo) were also apparent.\n> \n> git stores data in loose blobs or in packfiles. The former\n> has essentially now become an exception mechanism, to store\n> exceptionally *young* blobs. Why not use this to store\n> exceptionally *large* blobs as well? This allows us to\n> re-use all the \"exception\" machinery with only a small change.\n> \n> Repacking the entire repository with a max-blob-size of 256KB\n> resulted in a single 13.1MB packfile, as well as 2853 loose\n> objects totaling 15.4GB compressed and 100.08GB uncompressed,\n> 11 files per objects/xx directory on average. All was created\n> in half the runtime of the previous yet with standard\n> --window=10 and --depth=50 parameters. The data in the\n> packfile was 270MB uncompressed in 35976 blobs. Operations\n> such as \"git-log --pretty=oneline\" were about 30X faster\n> on a cold cache and 2 to 3X faster otherwise. Process sizes\n> remained reasonable.\n> \n> This patch implements the following:\n> 1. git pack-objects takes a new --max-blob-size=N flag,\n> with the effect that only blobs less than N KB are written\n> to the packfiles(s). If a blob was in a pack but violates\n> this limit (perhaps the packs were created by fast-import\n> or max-blob-size was reduced), then a new loose object\n> is written out if needed so the data is not lost.\n> 2. git repack inspects repack.maxblobsize . If set, its\n> value is passed to git pack-objects on the command line.\n> The user should change repack.maxblobsize , NOT specify\n> --max-blob-size=N .\n> 3. No other caller of git pack-objects supplies this new flag,\n> so other callers see no change.\n> \n> This patch is on top of the earlier max-pack-size patch,\n> because I thought I needed some behavior it supplied,\n> but could be rebased on master if desired.\n\nI think what this patch is missing is a test after all options have been \nparsed to prevent --stdout and --max-blob-size to be used together.\n\n\nNicolas\n"},{"id":"42987","messageId":"56b7f5510705221107m535fa7d2ob0dad6ebb25b9443@mail.gmail.com","threadId":"8273","inReplyTo":"alpine.LFD.0.99.0705221329420.3366@xanadu.home","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-22T18:07:32Z","receivedAt":"2007-05-22T18:07:32Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 5/22/07, Nicolas Pitre wrote:\n> On Mon, 21 May 2007, Dana How wrote:\n> > This patch implements the following:\n> > 1. git pack-objects takes a new --max-blob-size=N flag,\n> > with the effect that only blobs less than N KB are written\n> > to the packfiles(s). If a blob was in a pack but violates\n> > this limit (perhaps the packs were created by fast-import\n> > or max-blob-size was reduced), then a new loose object\n> > is written out if needed so the data is not lost.\n> > 2. git repack inspects repack.maxblobsize . If set, its\n> > value is passed to git pack-objects on the command line.\n> > The user should change repack.maxblobsize , NOT specify\n> > --max-blob-size=N .\n> > 3. No other caller of git pack-objects supplies this new flag,\n> > so other callers see no change.\n> >\n> > This patch is on top of the earlier max-pack-size patch,\n> > because I thought I needed some behavior it supplied,\n> > but could be rebased on master if desired.\n>\n> I think what this patch is missing is a test after all options have been\n> parsed to prevent --stdout and --max-blob-size to be used together.\nYes. I will also update the documentation.\n\nThanks,\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"43007","messageId":"200705230144.38290.jnareb@gmail.com","threadId":"8273","inReplyTo":"56b7f5510705220959x1b37a4adk537cc0cba1a27530@mail.gmail.com","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-05-22T23:44:37Z","receivedAt":"2007-05-22T23:44:37Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dana How wrote:\n> On 5/22/07, Jakub Narebski wrote:\n>> Dana How wrote:\n\n>>> There's actually an even more extreme example from my day job.\n>>> The software team has a project whose files/revisions would be\n>>> similar to those in the linux kernel (larger commits, I'm sure).\n>>> But they have *ONE* 500MB file they check in because it takes\n>>> 2 or 3 days to generate and different people use different versions of it.\n>>> I'm sure it has 50+ revisions now. If they converted to git and included\n>>> these blobs in their packfile, that's a 25GB uncompressed increase!\n>>> *Every* git operation must wade through 10X -- 100X more packfile.\n>>> Or it could be kept in 50+ loose objects in objects/xx ,\n>>> requiring a few extra syscalls by each user to get a new version.\n>>\n>> Or keeping those large objects in separate, _kept_ packfile, containing\n>> only those objects (which can delta well, even if they are large).\n> \n> Yes, I experimented with various changes to git-repack and\n> having it create .keep files just before coming up with the maxblobsize\n> approach. The problem with a 12GB+ repo is not only the large\n> repack time, but the fact that the repack time keeps growing with\n> the repo size. So, with split packs, I had repack create .keep\n> files for all new packs except the last (fragmentary) one. The next\n> repack would then only repack new stuff plus the single fragmentary\n> pack, keeping repack time from growing (until you deleted the .keep\n> files [just the ones with \"repack\" in them] to start over from scratch).\n> But this approach is not going to distribute commits and trees all that well.\n\nNo, I was thinking about separate _kept_ pack (so it would be not \nrepacked unless -f option is given) containing _only_ the large blobs.\nThe only difference between this and your proposal is that megablobs\nwould be in their mergablobs pack, but not loose.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"43010","messageId":"7v3b1o758u.fsf@assigned-by-dhcp.cox.net","threadId":"8273","inReplyTo":"200705230144.38290.jnareb@gmail.com","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-23T00:28:33Z","receivedAt":"2007-05-23T00:28:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski writes:\n\n> No, I was thinking about separate _kept_ pack (so it would be not \n> repacked unless -f option is given) containing _only_ the large blobs.\n> The only difference between this and your proposal is that megablobs\n> would be in their mergablobs pack, but not loose.\n\nI am not sure about the \"unless -f option is given\" part, but a\nsingle .kept pack that contains only problematic blobs would be\nan interesting experiment.\n\n (0) prepare object names of problematic blobs, in huge.txt, one\n object name per line;\n\n (1) prepare a single pack that has them:\n\n $ N=$(git-pack-object --depth=0 --window=0 pack pack-$N.keep\n $ mv pack-$N.* .git/object/pack/.\n\n (2) repack the remainder, with the default depth/window:\n\n $ git repack -a -d\n $ git prune\n"},{"id":"43012","messageId":"alpine.LFD.0.99.0705222154560.3366@xanadu.home","threadId":"8273","inReplyTo":"7v3b1o758u.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-23T01:58:34Z","receivedAt":"2007-05-23T01:58:34Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 22 May 2007, Junio C Hamano wrote:\n\n> Jakub Narebski writes:\n> \n> > No, I was thinking about separate _kept_ pack (so it would be not \n> > repacked unless -f option is given) containing _only_ the large blobs.\n> > The only difference between this and your proposal is that megablobs\n> > would be in their mergablobs pack, but not loose.\n> \n> I am not sure about the \"unless -f option is given\" part, but a\n> single .kept pack that contains only problematic blobs would be\n> an interesting experiment.\n> \n> (0) prepare object names of problematic blobs, in huge.txt, one\n> object name per line;\n> \n> (1) prepare a single pack that has them:\n> \n> $ N=$(git-pack-object --depth=0 --window=0 pack $ echo 'Huge blobs -- do not repack' >pack-$N.keep\n> $ mv pack-$N.* .git/object/pack/.\n\nIf you're going to keep this pack, I think it might be worth attempting \ndeltas between those blobs anyway. If they ever deltify you'll gain in \ndisk space. And if they don't, well, you wasted the CPU cycles only \nonce. Unless you know for sure they're unlikely to deltify well.\n\n\nNicolas\n"},{"id":"43058","messageId":"7v7iqz19d2.fsf@assigned-by-dhcp.cox.net","threadId":"8273","inReplyTo":"46528A48.9050903@gmail.com","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-23T22:08:25Z","receivedAt":"2007-05-23T22:08:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Dana How writes:\n\n> This patch implements the following:\n> 1. git pack-objects takes a new --max-blob-size=N flag,\n> with the effect that only blobs less than N KB are written\n> to the packfiles(s). If a blob was in a pack but violates\n> this limit (perhaps the packs were created by fast-import\n> or max-blob-size was reduced), then a new loose object\n> is written out if needed so the data is not lost.\n\nWhy?\n\nI really do not like that \"write a new loose object\" part\nwithout proper justification. From your description, I thought\nthe most natural way to do this is to pretend you did not hear\nabout large objects at all, by rejecting them early, perhaps\ninside add_object_entry() or inside get_object_details() --\neither case you would do sha1_object_info() early instead of\ndoing it in check_object().\n\nBy the way, is there fundamental reason that this needs to be\n\"blob size\" limit? Wouldn't \"max-object-size\" be more clean in\ntheory, and work the same way in practice?\n\n> 2. git repack inspects repack.maxblobsize . If set, its\n> value is passed to git pack-objects on the command line.\n> The user should change repack.maxblobsize , NOT specify\n> --max-blob-size=N .\n\nWhy not?\n\n> This patch is on top of the earlier max-pack-size patch,\n> because I thought I needed some behavior it supplied,\n> but could be rebased on master if desired.\n\nYour earlier \"split according to max-pack-size\" will hopefully be\non master shortly.\n"},{"id":"43086","messageId":"56b7f5510705231655o589de801w88adc1aa6c18162b@mail.gmail.com","threadId":"8273","inReplyTo":"7v7iqz19d2.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-23T23:55:27Z","receivedAt":"2007-05-23T23:55:27Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 5/23/07, Junio C Hamano wrote:\n> Dana How writes:\n> > This patch implements the following:\n> > 1. git pack-objects takes a new --max-blob-size=N flag,\n> > with the effect that only blobs less than N KB are written\n> > to the packfiles(s). If a blob was in a pack but violates\n> > this limit (perhaps the packs were created by fast-import\n> > or max-blob-size was reduced), then a new loose object\n> > is written out if needed so the data is not lost.\n>\n> Why?\n>\n> I really do not like that \"write a new loose object\" part\n> without proper justification. From your description, I thought\n> the most natural way to do this is to pretend you did not hear\n> about large objects at all, by rejecting them early, perhaps\n> inside add_object_entry() or inside get_object_details() --\n> either case you would do sha1_object_info() early instead of\n> doing it in check_object().\n>\n> By the way, is there fundamental reason that this needs to be\n> \"blob size\" limit? Wouldn't \"max-object-size\" be more clean in\n> theory, and work the same way in practice?\n\nI agree with your sentiments. Some nasty details pushed\nme to implement it this way. Let me explain them and perhaps\nyou can come up with a different combination of solutions.\n\nEach object to be packed can be in one of 4 states:\n{loose or packed} X {small or too big}.\nRegardless of the {loose or packed} selection,\na \"small\" object can always be placed in the pack.\nA loose object which is too big does not require\nany special action either -- if we drop it from the pack,\nwe don't lose anything since it already exists in objects/xx .\n\nThe packed X too big combination is the problem. As the\ncommit message says, this could happen if the packs\ncame from fast-import, or if the max-blob-size were reduced.\nWe have three options in this case:\n(1) Drop the object (do not put it in the new pack(s)).\n(2) Pass the object into the new pack(s).\n(3) Write out the object as a new loose object.\n\nOption (1) is unacceptable. When you call git-repack -a,\nit blindly deletes all the non-kept packs at the end. So\nthe megablobs would be lost.\n\nLet's suppose we always used Option (2), and Option (3)\nwere never available. Then once a large object got into\na pack, the only way to get it out would be to explode\nthe pack using git-unpack-objects -- but hey, that doesn't\nwork because they all already exist in this repository packed.\nSo we make a temporary repository to explode the pack,\nthen copy over the loose objects, then delete the pack,\nthen repack with the correct/updated max-blob-size specification.\nI would claim this is even more ugly, and more importantly,\nmore error-prone than supporting Option (3) in at least some cases.\n\nThe way large objects get into a pack is that you realize after\nthe fact that you need a max-blob-size, or that you need to\ndecrease it since your pack files have become unwieldy.\n\nI think I've shown we need Option (3) in at least some cases.\nTo better address _which_ cases, let's move on to your\nsecond point:: why did I implement --max-blob-size instead\nof --max-object-size? I take this to mean that I should use\nthe blob size if undeltified, and the delta size if previously deltified?\nThat's actually what I do internally, but the total effect\nover many repackings is what one would expect from --max-blob-size.\n\nIn normal use on a personal repository starting from scratch,\nall blobs are first packed from loose objects. When I am\ndeciding if I want to include the loose object or not, I want\nto use the object's direct size, not its deltified size. If I use\nthe latter, then I have to deltify all the megablobs to see\nif their deltas are small enough. This takes a long time\nand is one of the things this patch aims to eliminate.\nSo I take the list of loose objects, throw out the megablobs,\ntry to deltify the rest, and write out the pack.\n\nWhen I repack, any deltas are smaller than the original\nobjects, so in any sequence of packings from loose\nobjects and repackings of packs, in which all packs\nwere created with a max-blob-size limit which stayed the\nsame or increased over time, there is no difference\nbetween max-blob-size or max-object-size: checking loose\nobject size or in-pack size gives the same pack contents.\n\nNow let's say you decrease max-blob-size and repack,\nor repack from some packs that had no max-blob-size\n(e.g. from fast-import). Now there is some difficulty in\na clean definition when you follow Option (3). You might\nhave a base object B in an old pack, and a deltified object\nD based on it, where B is now larger than the newly-decreased\nmax-blob-size while D's deltified size is still smaller.\nSo you write out B, but D can no longer be stored deltified\nand may become bigger as well.\n\nBut in this case you should do the following:\nSince you are going to decrease max-blob-size when you decide\nyour repository packs have become \"gunked up\" with megablobs,\nI would propose git-repack -a -f (git-pack-objects --no-object-reuse)\nbe used. This means all deltas are recomputed\nfrom the new, smaller universe of blobs, which makes sense,\nand since no deltas are reused, all blob-size limits can only\nbe based on the actual blob size by the arguments 3 paragraphs back.\n(Note: I made sure the current patch loses no data if you decrease\n max-blob-size and forget to specify -f -- you just get a few large\n packed objects resulting from \"lost\" deltifications.)\n\nIn actuality, the object_entry.size field is checked, so objects\n_are_ being filtered by delta size if deltified. But I went through\nthe (suggested) usage scenarios above to show that the *result* is that\nyou get a pack limited in raw object size. --max-object-size\nis what I implemented because it's easier, but in use it acts\nlike --max-blob-size so I named it that.\n\nNow, concerning Option (2) vs Option (3): I need to write\nout loose objects in *some* cases. Right now I always write\nout a large object as a loose one. It would be reasonable to\nchange this so that it only happens with --no-object-reuse\n(i.e. git-repack -f). This is what you should specify when you're\ndecreasing max-blob-size. Shall I change it so that large objects\nare passed to the result pack without --no-object-reuse,\nand written out as loose objects with --no-object-reuse?\nI did not do this because it's a strange interaction --\njust look how long it took me to build up to explaining it!\n\n> > 2. git repack inspects repack.maxblobsize . If set, its\n> > value is passed to git pack-objects on the command line.\n> > The user should change repack.maxblobsize , NOT specify\n> > --max-blob-size=N .\n> Why not?\nWell, indeed. That's why [PATCH *v2*] lets you specify\n--max-blob-size to git-repack, and gives an example of\nwhy you might want to do that [I don't].\n\n> > This patch is on top of the earlier max-pack-size patch,\n> > because I thought I needed some behavior it supplied,\n> > but could be rebased on master if desired.\n>\n> Your earlier \"split according to max-pack-size\" will hopefully be\n> on master shortly.\nThanks!\n\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"43089","messageId":"7vps4ryp02.fsf@assigned-by-dhcp.cox.net","threadId":"8273","inReplyTo":"56b7f5510705231655o589de801w88adc1aa6c18162b@mail.gmail.com","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-24T01:44:13Z","receivedAt":"2007-05-24T01:44:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Dana How\" writes:\n\n> The packed X too big combination is the problem. As the\n> commit message says, this could happen if the packs\n> came from fast-import,...\n> We have three options in this case:\n> (1) Drop the object (do not put it in the new pack(s)).\n> (2) Pass the object into the new pack(s).\n> (3) Write out the object as a new loose object.\n>\n> Option (1) is unacceptable. When you call git-repack -a,\n> it blindly deletes all the non-kept packs at the end. So\n> the megablobs would be lost.\n\nOk, I can buy that -- (1) nor (2) are unacceptable and (3) is\nthe only sane thing to do for a previously packed objects that\nexceed the size limit.\n\nSince you have to handle that case _anyway_, I think it makes\nsense to always say \"Ok, we will write it out if there is no\nloose representation already available\".\n\nThat is, unless somebody smarter than me, like Nico or Shawn,\ncome up with better ideas to do this ;-).\n\n> ... why did I implement --max-blob-size instead\n> of --max-object-size? I take this to mean that I should use\n> the blob size if undeltified, and the delta size if previously deltified?\n\nNo, I think the only sensible way for the end user to specify\nthe size is uncompressed size of the object. For a blob, that\nis the size of checked-out file. IOW:\n\n\t$ git cat-file $type $sha | wc -c\n\nNothing else would make any sense.\n"},{"id":"43102","messageId":"20070524071235.GL28023@spearce.org","threadId":"8273","inReplyTo":"7vps4ryp02.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-05-24T07:12:36Z","receivedAt":"2007-05-24T07:12:36Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Dana How\" writes:\n> \n> > The packed X too big combination is the problem. As the\n> > commit message says, this could happen if the packs\n> > came from fast-import,...\n> > We have three options in this case:\n> > (1) Drop the object (do not put it in the new pack(s)).\n> > (2) Pass the object into the new pack(s).\n> > (3) Write out the object as a new loose object.\n> >\n> > Option (1) is unacceptable. When you call git-repack -a,\n> > it blindly deletes all the non-kept packs at the end. So\n> > the megablobs would be lost.\n> \n> Ok, I can buy that -- (1) nor (2) are unacceptable and (3) is\n> the only sane thing to do for a previously packed objects that\n> exceed the size limit.\n\nI still don't buy the idea that these megablobs shouldn't be packed.\nI understand Dana's pain here (at least a little bit, my problems\naren't as bad as his are), but I also hate to see us run away from\npackfiles for these really sick cases just because we have some\nissues in our current packfile handling.\n\nPackfiles give us a lot of benefits:\n\n 1) less inode usage;\n 2) transport can write directly to local disk;\n 3) transport can (quickly) copy from local disk;\n 4) testing for existance is *much* faster;\n 5) deltafication is possible;\n\nNow #3 is actually really important here. Don't forget that we\n*just* disabled the fancy \"new loose object format\". It doesn't\nexist. We can read the packfile-like loose objects, but we cannot\nwrite them anymore. So lets say we explode a megablob into a loose\nobject, and its 800 MiB by itself. Now we have to send that object\nto a client. Yes, that's right, we must *RECOMPRESS* 800 MiB for\nno reason. Not the best choice. Maybe we shouldn't have deleted\nthat packfile formatted loose object writer...\n\nNow one argument to work around that recompression problem would\nbe to NFS share out the loose objects directory, and let clients\nmount that volume and add it to their .git/objects/info/alternates\nlist. But this doesn't work in the very general distributed case,\nsuch as me getting huge files from kernel.org. Last I checked,\nthe kernel.org admins did not offer up NSF mounts. Besides, the\nround-trip latency between me and kernel.org is too large for it\nto be useful anyway over NFS. :)\n\nSo I think this \"explode out megablobs\" is a bad idea. Its violating\nother things that make us fast, like #3's being able to reuse large\nparts of an existing packfile during transfer.\n\n\nDana pointed out the megablobs make access slower because their\npackfile indexes must still be searched to locate a commit; but if\nthe megablob packfile(s) contain only blobs then there is no value\nin looking at those packfiles.\n\nWe might be able to fix this by altering the sort_pack function\nin sha1_file.c to not only order by mtime, but also by the ratio\nof the size of the .pack to the number of objects stored in it.\nAny packfile with a high size/object ratio is likely to be what\nDana has been calling a \"metadata\" pack, holding things like tags,\ncommits, trees and small blobs. Its these packfiles that we want\nto search first, as they are the most likely to be accessed.\n\nBy pushing the megablob packs to the end of our packed_git search\nlist we won't tend to scan their indexes, as most of our objects\nwill be found earlier in the search list. Hence we will generally\navoid any costs associated with their indexes.\n\n\nHuge packfiles probably should be scheduled for keeping with a .keep\nautomatically. We probably should teach pack-objects to generate a\n.keep file if the resulting .pack was over a certain size threshold\n(say 1.5 GiB by default) and teach git-repack to rename the .keep\nfile as it also renames the .idx and .pack.\n\nBetter that we degrade gracefully when faced with massive inputs\nthan we do something stupid by default and make the poor user pay\nfor their mistake of not throughly reading plumbing documentation\nbefore use.\n\n\nNow I would agree that we should punt on deltification of anything\nthat is just too large, and let the user decide what too large means,\nand default it around 500 or 1024 MiB. But I would still stuff it\ninto a packfile.\n\nMaybe it still makes sense to have a limit on the maximum size of a\nloose object to pack, but I think that's only to avoid the sick case\nof a very simple no-argument \"git repack\" taking a long while because\nthere's 8 loose objects and 6 of them are 900 MiB image files.\n\nOnce in a packfile, I'd keep it there, even if the user decreases\nthe threshold, as the advantages of it being in the packfile outweigh\nthe disadvantages of it being in the packfile. And there's like no\nadvantage to being loose once packed.\n\n\nAll of that is actually a very minor set of changes to the system,\nand doesn't create odd corner cases. It should also degrade better\nout of the box.\n\n> > ... why did I implement --max-blob-size instead\n> > of --max-object-size? I take this to mean that I should use\n> > the blob size if undeltified, and the delta size if previously deltified?\n> \n> No, I think the only sensible way for the end user to specify\n> the size is uncompressed size of the object. For a blob, that\n> is the size of checked-out file. IOW:\n> \n> \t$ git cat-file $type $sha | wc -c\n> \n> Nothing else would make any sense.\n\nI agree. And when you combine it with what I'm saying above about\nonly applying this to loose objects, its really quite easy to fetch\nthat value from the header and perform the test.\n\n-- \nShawn.\n"},{"id":"43114","messageId":"Pine.LNX.4.64.0705241038030.4648@racer.site","threadId":"8273","inReplyTo":"20070524071235.GL28023@spearce.org","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-05-24T09:38:46Z","receivedAt":"2007-05-24T09:38:46Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 24 May 2007, Shawn O. Pearce wrote:\n\n> Junio C Hamano wrote:\n> > \"Dana How\" writes:\n> > \n> > > The packed X too big combination is the problem. As the\n> > > commit message says, this could happen if the packs\n> > > came from fast-import,...\n> > > We have three options in this case:\n> > > (1) Drop the object (do not put it in the new pack(s)).\n> > > (2) Pass the object into the new pack(s).\n> > > (3) Write out the object as a new loose object.\n> > >\n> > > Option (1) is unacceptable. When you call git-repack -a,\n> > > it blindly deletes all the non-kept packs at the end. So\n> > > the megablobs would be lost.\n> > \n> > Ok, I can buy that -- (1) nor (2) are unacceptable and (3) is\n> > the only sane thing to do for a previously packed objects that\n> > exceed the size limit.\n> \n> I still don't buy the idea that these megablobs shouldn't be packed.\n> I understand Dana's pain here (at least a little bit, my problems\n> aren't as bad as his are), but I also hate to see us run away from\n> packfiles for these really sick cases just because we have some\n> issues in our current packfile handling.\n\nIsn't this issue helpable by the \"-delta\" attribute?\n\nCiao,\nDscho\n"},{"id":"43152","messageId":"Pine.LNX.4.64.0705241020450.21766@asgard.lang.hm","threadId":"8273","inReplyTo":"20070524071235.GL28023@spearce.org","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"","fromEmail":"david@lang.hm","sentAt":"2007-05-24T17:23:39Z","receivedAt":"2007-05-24T17:23:39Z","isPatch":true,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Thu, 24 May 2007, Shawn O. Pearce wrote:\n\n> Now #3 is actually really important here. Don't forget that we\n> *just* disabled the fancy \"new loose object format\". It doesn't\n> exist. We can read the packfile-like loose objects, but we cannot\n> write them anymore. So lets say we explode a megablob into a loose\n> object, and its 800 MiB by itself. Now we have to send that object\n> to a client. Yes, that's right, we must *RECOMPRESS* 800 MiB for\n> no reason. Not the best choice. Maybe we shouldn't have deleted\n> that packfile formatted loose object writer...\n\nwhen did the object store get changed so that loose objects aren't \ncompressed?\n\nif the problem is that the codepath for fetching does an uncompress \nfollowed by a compress then it would seem that this is a fairly easy \nproblem to fix (how hard would it be to add the headers around the \ncompressed object to make it look to the receiver like it's a pack with \nonly one thing in it)\n\nDavid Lang\n"},{"id":"43153","messageId":"Pine.LNX.4.64.0705241828160.4648@racer.site","threadId":"8273","inReplyTo":"Pine.LNX.4.64.0705241020450.21766@asgard.lang.hm","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-05-24T17:29:47Z","receivedAt":"2007-05-24T17:29:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 24 May 2007, david@lang.hm wrote:\n\n> On Thu, 24 May 2007, Shawn O. Pearce wrote:\n> \n> > Now #3 is actually really important here. Don't forget that we\n> > *just* disabled the fancy \"new loose object format\". It doesn't\n> > exist. We can read the packfile-like loose objects, but we cannot\n> > write them anymore. So lets say we explode a megablob into a loose\n> > object, and its 800 MiB by itself. Now we have to send that object\n> > to a client. Yes, that's right, we must *RECOMPRESS* 800 MiB for\n> > no reason. Not the best choice. Maybe we shouldn't have deleted\n> > that packfile formatted loose object writer...\n> \n> when did the object store get changed so that loose objects aren't\n> compressed?\n\nThat never happened. But we had a different file format for loose objects, \nwhich was meant to make it easier to copy as-is into a pack. That file \nformat went away, since it was not as useful as we hoped.\n\nCiao,\nDscho\n"},{"id":"43179","messageId":"C35F50C5-28E8-45B6-8A4E-02233CE4B4B9@adacore.com","threadId":"8273","inReplyTo":"20070524071235.GL28023@spearce.org","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2007-05-24T20:43:52Z","receivedAt":"2007-05-24T20:43:52Z","isPatch":true,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"[resent because of malformed headers causing rejection]\nOn May 24, 2007, at 03:12, Shawn O. Pearce wrote:\n> I still don't buy the idea that these megablobs shouldn't be packed.\n> I understand Dana's pain here (at least a little bit, my problems\n> aren't as bad as his are), but I also hate to see us run away from\n> packfiles for these really sick cases just because we have some\n> issues in our current packfile handling.\n>\n> Packfiles give us a lot of benefits:\n>\n> 1) less inode usage;\nUsing 1 inode per huge blob can never be an issue\n> 2) transport can write directly to local disk;\n> 3) transport can (quickly) copy from local disk;\nCan do these by re-enabling the new loose object format\n> 4) testing for existance is *much* faster;\n> 5) deltafication is possible;\n\nLook at it the other way. If we have huge objects (say >1GB),\nwe should put them in a pack of their own anyway. What's better:\nhaving a pack with a separate index file or just a loose object?\nWhile the one object per file model is awful for many small files\nwith lots of similarity, it is really quite efficient for large\nobjects, and the most reasonable model for huge objects.\n\nSuch blobs are just too large to do anything useful with.\nThe only operations done on them will be to check them in\nor check them out. Ideally, we should never try to have them\nin memory at all, but just stream them to/from disk while\ncompressing or decompressing.\n\nTrying to deltify huge objects just takes too much time.\nSimilarly, we don't want to read 100MB to then apply a delta\nand maybe throw out half of the data we read in the first place.\nIt's just too inefficient. If we'd even read the huge blobs once\nduring \"git repack\", we'll waste so much time that we're unlikely\nto ever gain it back in any real world scenario.\n\n -Geert\n"},{"id":"43192","messageId":"56b7f5510705241629n192a41adi4c0d63c53cf8472b@mail.gmail.com","threadId":"8273","inReplyTo":"20070524071235.GL28023@spearce.org","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-24T23:29:20Z","receivedAt":"2007-05-24T23:29:20Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 5/24/07, Shawn O. Pearce wrote:\n> Junio C Hamano wrote:\n> > \"Dana How\" writes:\n> > > We have three options in this case:\n> > > (1) Drop the object (do not put it in the new pack(s)).\n> > > (2) Pass the object into the new pack(s).\n> > > (3) Write out the object as a new loose object.\n> > > Option (1) is unacceptable. When you call git-repack -a,\n> > > it blindly deletes all the non-kept packs at the end. So\n> > > the megablobs would be lost.\n> > Ok, I can buy that -- (1) nor (2) are unacceptable and (3) is\n> > the only sane thing to do for a previously packed objects that\n> > exceed the size limit.\n>\n> I still don't buy the idea that these megablobs shouldn't be packed.\n> I understand Dana's pain here (at least a little bit, my problems\n> aren't as bad as his are), but I also hate to see us run away from\n> packfiles for these really sick cases just because we have some\n> issues in our current packfile handling.\n>\n> Packfiles give us a lot of benefits:\n>\n> 1) less inode usage;\nI agree with Geert that blowing an inode on a 100MB+ object\nis no big deal.\n> 2) transport can write directly to local disk;\n> 3) transport can (quickly) copy from local disk;\nFor (2) and (3) see comments on next para plus NFS discussion.\n> 4) testing for existance is *much* faster;\nThis is true. But I don't care about this cost if it\nis only incurred on large objects which are leaf nodes\nin the git \"data relationship tree\" (tags->commits->trees->blobs) anyway.\n> 5) deltafication is possible;\nAgain Geert made a good argument that didn't occur to me that\nyou definitely DON'T want to do deltification on such large objects.\nJunio recently added delta/nodelta attribute; this would be useful\nto me, but unfortunately I have several continua of files, each with\nthe same suffix, but with largely varying sizes, so attributes won't\nhelp me unless the name globs in .gitattributes are expanded to full\nexpressions similar to find(1) [i.e. include testing based on size,\nperms, type], which I think would be insane.\n\n> Now #3 is actually really important here. Don't forget that we\n> *just* disabled the fancy \"new loose object format\". It doesn't\n> exist. We can read the packfile-like loose objects, but we cannot\n> write them anymore. So lets say we explode a megablob into a loose\n> object, and its 800 MiB by itself. Now we have to send that object\n> to a client. Yes, that's right, we must *RECOMPRESS* 800 MiB for\n> no reason. Not the best choice. Maybe we shouldn't have deleted\n> that packfile formatted loose object writer...\nI completely agree with your argument. I do not suggest that repositories\nthat communicate with others via packs should use this feature.\nOur repositories will communicate via alternates/NFS in one direction\nand probably packs in the other. In the latter case the packs would\nbe generated with maxblobsize=0. See comments on next para.\n\n> Now one argument to work around that recompression problem would\n> be to NFS share out the loose objects directory, and let clients\n> mount that volume and add it to their .git/objects/info/alternates\n> list. But this doesn't work in the very general distributed case,\n> such as me getting huge files from kernel.org. Last I checked,\n> the kernel.org admins did not offer up NSF mounts. Besides, the\n> round-trip latency between me and kernel.org is too large for it\n> to be useful anyway over NFS. :)\nThe NFS case is exactly what I want to use. I want each repo to\nhave their own packfiles to reduce load on the central alternate,\nbut these local repos would not include megablobs. I do not have\nas strong a feeling about whether the central alternate should\npack its megablobs or not [but I don't want to do it if it costs me\ndeltification for everybody], but I need a way to exclude megablobs\nfrom getting into local packs. WIth such exclusion, git-gc/repack\nis extremely quick. There is NO WAY this can be true if\nseveral GB have to be copied around, which again comes from Geert.\n\nI think this conversation does suggest one alteration to my patch:\nas submitted it writes out a loose object if the object is packed\nand has no loose object. It should really do this only if the\nobject is packed LOCALLY and has no loose object.\n\n> So I think this \"explode out megablobs\" is a bad idea. Its violating\n> other things that make us fast, like #3's being able to reuse large\n> parts of an existing packfile during transfer.\nAs I said, your true argument doesn't apply in my case.\n\n> Dana pointed out the megablobs make access slower because their\n> packfile indexes must still be searched to locate a commit; but if\n> the megablob packfile(s) contain only blobs then there is no value\n> in looking at those packfiles.\n>\n> We might be able to fix this by altering the sort_pack function\n> in sha1_file.c to not only order by mtime, but also by the ratio\n> of the size of the .pack to the number of objects stored in it.\n> Any packfile with a high size/object ratio is likely to be what\n> Dana has been calling a \"metadata\" pack, holding things like tags,\n> commits, trees and small blobs. Its these packfiles that we want\n> to search first, as they are the most likely to be accessed.\n>\n> By pushing the megablob packs to the end of our packed_git search\n> list we won't tend to scan their indexes, as most of our objects\n> will be found earlier in the search list. Hence we will generally\n> avoid any costs associated with their indexes.\nGood argument and I submitted a patch to do this.\nLet's see who chokes on the floating arithmetic ;-)\n\n> Huge packfiles probably should be scheduled for keeping with a .keep\n> automatically. We probably should teach pack-objects to generate a\n> .keep file if the resulting .pack was over a certain size threshold\n> (say 1.5 GiB by default) and teach git-repack to rename the .keep\n> file as it also renames the .idx and .pack.\nI have experimented with this, and Jakub Narebski made related\nsuggestions. I find this quite hokey, but even if I do it in my central\nalternate, I still do not want to be packing megablobs in individual user's\nrepos EVER, and need some way to exclude them.\n\n> Better that we degrade gracefully when faced with massive inputs\n> than we do something stupid by default and make the poor user pay\n> for their mistake of not throughly reading plumbing documentation\n> before use.\nUnnecessary copying of several GB is not degrading gracefully in my view.\nIn fact having repack.maxblobsize = 2000 (K) in the default \"config\"\nstrikes me as degrading much more gracefully than what the code\nwould currently do.\n\nThis silly patch took my packfile sets from 12GB+ to 13MB,\nand it's difficult to describe how relieved I now feel.\n\nIt's also difficult for me to believe that a setup that treats 12GB\n(almost) equally is going to be as efficient as one which concentrates on 13MB.\nThat's three orders of magnitude, a point I've made before.\n\nBut I think you have an understandable motivation:\nyou want packfiles to be as good as possible, and any escape\nmechanism from them decreases the motivation to \"fix\" packing.\nNow I agree with this, which is why I just submitted some other patches,\nbut I don't share your goal of the universality of all packfiles --\njust the ones used for transport. Don't your packv4 plans introduce\nmods which won't be used for transport as well?\n\n> Now I would agree that we should punt on deltification of anything\n> that is just too large, and let the user decide what too large means,\n> and default it around 500 or 1024 MiB. But I would still stuff it\n> into a packfile.\n>\n> Maybe it still makes sense to have a limit on the maximum size of a\n> loose object to pack, but I think that's only to avoid the sick case\n> of a very simple no-argument \"git repack\" taking a long while because\n> there's 8 loose objects and 6 of them are 900 MiB image files.\nPerhaps we _will_ make progress if we all agree to describe\nmy situation as \"sick\" ;-) . In this paragraph you seem to agree that\nthere is some argument for keeping megablobs from _entering_ packs?\n\nOne reason I like my patch is because I do view megablobs\nas perverting the system, and just keeping them out of the optimized\npackfile system is a big step forward.\n\n> Once in a packfile, I'd keep it there, even if the user decreases\n> the threshold, as the advantages of it being in the packfile outweigh\n> the disadvantages of it being in the packfile. And there's like no\n> advantage to being loose once packed.\nTo (almost) follow this suggestion I would need git-fast-import to respect\nrepack.maxblobsize as well. Is that OK with you?\n\nI previously offered to Junio that the \"write loose object\" thing\ncould be restricted: it would only happen if -f were supplied to\ngit-repack, otherwise the bad blob would pass through to the new pack.\nDoes this \"reduction in strength\" make this feature more palatable to you?\n\nIf the stats on a repo change significantly, \"write loose object\"\nbecomes more important if you have to make a significant reduction\nto repack.maxblobsize (or specify it for the first time).\n\nI don't agree that once in a packfile, a blob should stay there.\nIts presence is degrading access to \"normal\" blobs co-habiting with it.\nSo you will want to repack to separate them in different packs\n(the various .keep-related ideas) or just write them out loose.\n\nTo conclude:\nthe patch wrote out a new loose object when it was previously\npacked and is larger then repack.maxblobsize.\nI could change this to only happen when the object is\n(1) packed AND\n(2) locally packed AND\n(3) -f/--no-object-reuse was specified to git-repack/git-pack-objects.\nThe previous behavior that a megablob never _enters_ the pack\nwould remain unchanged.\n\nThis more restrictive behavior would be sufficient for me,\nand I think I *need* it at least in the users' repositories\nin an NFS/alternates setup.\n\nWhat do you think?\n\nThanks,\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"43195","messageId":"20070525005507.GR28023@spearce.org","threadId":"8273","inReplyTo":"Pine.LNX.4.64.0705241828160.4648@racer.site","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-05-25T00:55:07Z","receivedAt":"2007-05-25T00:55:07Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin wrote:\n> On Thu, 24 May 2007, david@lang.hm wrote:\n> > On Thu, 24 May 2007, Shawn O. Pearce wrote:\n> > \n> > > Now #3 is actually really important here. Don't forget that we\n> > > *just* disabled the fancy \"new loose object format\". It doesn't\n> > > exist. We can read the packfile-like loose objects, but we cannot\n> > > write them anymore. So lets say we explode a megablob into a loose\n> > > object, and its 800 MiB by itself. Now we have to send that object\n> > > to a client. Yes, that's right, we must *RECOMPRESS* 800 MiB for\n> > > no reason. Not the best choice. Maybe we shouldn't have deleted\n> > > that packfile formatted loose object writer...\n> > \n> > when did the object store get changed so that loose objects aren't\n> > compressed?\n> \n> That never happened. But we had a different file format for loose objects, \n> which was meant to make it easier to copy as-is into a pack. That file \n> format went away, since it was not as useful as we hoped.\n\nThat \"different file format\" thing was added exactly for this type\nof problem. Someone added a bunch of large blobs to their repository\nand then spent a lot of time decompressing and recompressing them\nduring their next repack.\n\nThe reason that recompress must happen is the deflate stream in a\nstandard (aka legacy) loose object contains both the Git object\nheader and the raw data; in a packfile the Git object header is\nstored external from the deflate stream. The \"different file format\"\nused the packfile format, allowing us to store the Git object header\nexternal from the deflate stream. That meant we could just copy\nthe raw bytes as-is from the loose object into the packfile.\n\nSo we still store loose objects compressed, its just that we can\nno longer create loose objects that can be copied directly into\na packfile without recompression. And that is sort of Dana's\nproblem here. OK, not entirely, but whatever.\n\n-- \nShawn.\n"},{"id":"43206","messageId":"20070525020642.GS28023@spearce.org","threadId":"8273","inReplyTo":"56b7f5510705241629n192a41adi4c0d63c53cf8472b@mail.gmail.com","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-05-25T02:06:42Z","receivedAt":"2007-05-25T02:06:42Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Dana How wrote:\n> On 5/24/07, Shawn O. Pearce wrote:\n> >Junio C Hamano wrote:\n> >> \"Dana How\" writes:\n> >> > We have three options in this case:\n> >> > (1) Drop the object (do not put it in the new pack(s)).\n> >> > (2) Pass the object into the new pack(s).\n> >> > (3) Write out the object as a new loose object.\n> >> > Option (1) is unacceptable. When you call git-repack -a,\n> >> > it blindly deletes all the non-kept packs at the end. So\n> >> > the megablobs would be lost.\n> >> Ok, I can buy that -- (1) nor (2) are unacceptable and (3) is\n> >> the only sane thing to do for a previously packed objects that\n> >> exceed the size limit.\n> >\n> >I still don't buy the idea that these megablobs shouldn't be packed.\n> >I understand Dana's pain here (at least a little bit, my problems\n> >aren't as bad as his are), but I also hate to see us run away from\n> >packfiles for these really sick cases just because we have some\n> >issues in our current packfile handling.\n> >\n> >Packfiles give us a lot of benefits:\n> >\n> > 1) less inode usage;\n> I agree with Geert that blowing an inode on a 100MB+ object\n> is no big deal.\n\nYou and me both. If the size of the blob is high enough than\noverhead associated with the inode and any tail-block wastage is\nnoise. Filesystems are pretty good at tracking large-ish files.\nMy comment here wasn't so much about blowing an inode on a 100MiB+\nobject, but just in general that packfiles reduce inode usage,\nwhich in the common metadata case (300,000 small objects) is a\nbig difference.\n\nI think you are right; if we get an object in the >100MiB size\nrange we can certainly afford an inode for it.\n\n> > 2) transport can write directly to local disk;\n> > 3) transport can (quickly) copy from local disk;\n> For (2) and (3) see comments on next para plus NFS discussion.\n> > 4) testing for existance is *much* faster;\n> This is true. But I don't care about this cost if it\n> is only incurred on large objects which are leaf nodes\n> in the git \"data relationship tree\" (tags->commits->trees->blobs) anyway.\n\nYes, that's true.\n\n> > 5) deltafication is possible;\n> Again Geert made a good argument that didn't occur to me that\n> you definitely DON'T want to do deltification on such large objects.\n> Junio recently added delta/nodelta attribute; this would be useful\n> to me, but unfortunately I have several continua of files, each with\n> the same suffix, but with largely varying sizes, so attributes won't\n> help me unless the name globs in .gitattributes are expanded to full\n> expressions similar to find(1) [i.e. include testing based on size,\n> perms, type], which I think would be insane.\n\nWhich brings up the comment I think I made (below) about skipping\ndeltas on very large objects. Things over a certain size are not\nlikely to delta well, or in any reasonable time. We probably should\ndefault to not trying to delta those, but let the user force us to\ndo so with a .gitattributes option. Maybe.\n\n> >By pushing the megablob packs to the end of our packed_git search\n> >list we won't tend to scan their indexes, as most of our objects\n> >will be found earlier in the search list. Hence we will generally\n> >avoid any costs associated with their indexes.\n> Good argument and I submitted a patch to do this.\n> Let's see who chokes on the floating arithmetic ;-)\n\nI actually had another thought in this area. I'll try to work up a\npatch to accompany yours. I think we can avoid even touching the\nalternate object databases half of the time, and I'd like to be\nable to do that. Why? Because I started to setup this megablob\napproach on my own Windows based repositories. Unfortunately it\nmakes git-log about 1 second slower, and I suspect its in the\nalternate repository initialization.\n \n> >Huge packfiles probably should be scheduled for keeping with a .keep\n> >automatically. We probably should teach pack-objects to generate a\n> >.keep file if the resulting .pack was over a certain size threshold\n> >(say 1.5 GiB by default) and teach git-repack to rename the .keep\n> >file as it also renames the .idx and .pack.\n> I have experimented with this, and Jakub Narebski made related\n> suggestions. I find this quite hokey, but even if I do it in my central\n> alternate, I still do not want to be packing megablobs in individual user's\n> repos EVER, and need some way to exclude them.\n\nYes, that makes a lot of sense.\n\n> >Better that we degrade gracefully when faced with massive inputs\n> >than we do something stupid by default and make the poor user pay\n> >for their mistake of not throughly reading plumbing documentation\n> >before use.\n> Unnecessary copying of several GB is not degrading gracefully in my view.\n> In fact having repack.maxblobsize = 2000 (K) in the default \"config\"\n> strikes me as degrading much more gracefully than what the code\n> would currently do.\n\nSure. But I think this goes back to #3 (network transport) and\nhow our loose object format now doesn't support it well. And even\nif that's fixed I don't think 2 MiB is a good default; its *far*\ntoo low. I have a number of blobs that are in the 12-16 MiB range\nand they delta very well in a pretty reasonable time.\n\n> This silly patch took my packfile sets from 12GB+ to 13MB,\n> and it's difficult to describe how relieved I now feel.\n\nI think I understand a little bit. Today I took 3 repositories\nthat were about 70 MiB each and dropped them down to 16 MiB, 2\nMiB and 4.5 MiB by creating a single 120 MiB \"megablob\" packfile\nthat spanned all 3 of them. This isn't the same scale as what you\nare dealing with, but now I have a current metadata pack for each\nthat isn't gummed up with large blobs, making repacking faster.\nI also have a smaller working set size. :-\n\n> But I think you have an understandable motivation:\n> you want packfiles to be as good as possible, and any escape\n> mechanism from them decreases the motivation to \"fix\" packing.\n\nYes, that's correct. I'm not against stepping outside of\npackfiles and making usage of loose objects for megablobs easier.\nI just want to make sure its the best way to handle these things.\nGenerally we've made major improvements in things when we've been\npushed by large repositories/datasets.\n\n> Now I agree with this, which is why I just submitted some other patches,\n> but I don't share your goal of the universality of all packfiles --\n> just the ones used for transport. Don't your packv4 plans introduce\n> mods which won't be used for transport as well?\n\nYes, at least initially we'd reencode from pack v4 down to pack v2\nfor transport, because transporting the dictionary with delta reuse\nis an interesting problem. However Nico and I have discussed it\nat length and have plans for how to code a pack v4 based transport,\nand pack v4's file format concepts are partially based upon making\npack v4 transport easier to implement. But from a \"start simple and\nkeep it simple, stupid\" principle we'd like to avoid the complexity\nearly on.\n\n> >Now I would agree that we should punt on deltification of anything\n> >that is just too large, and let the user decide what too large means,\n> >and default it around 500 or 1024 MiB. But I would still stuff it\n> >into a packfile.\n> >\n> >Maybe it still makes sense to have a limit on the maximum size of a\n> >loose object to pack, but I think that's only to avoid the sick case\n> >of a very simple no-argument \"git repack\" taking a long while because\n> >there's 8 loose objects and 6 of them are 900 MiB image files.\n> Perhaps we _will_ make progress if we all agree to describe\n> my situation as \"sick\" ;-) . In this paragraph you seem to agree that\n> there is some argument for keeping megablobs from _entering_ packs?\n\nYes. If your workflow is basically \"git add HUGE; git commit; git push;\ngit prune-alternates\" then you only have to pack the huge object once,\nand can remove the loose object from the user's .git/objects directly\npretty quickly, because its available via your NFS alternate. In such\na configuration yes, it does make some sense to never allow a megablob\nfrom entering a pack.\n\nSo I guess I'm partially in agreement with you...\n\n> >Once in a packfile, I'd keep it there, even if the user decreases\n> >the threshold, as the advantages of it being in the packfile outweigh\n> >the disadvantages of it being in the packfile. And there's like no\n> >advantage to being loose once packed.\n> To (almost) follow this suggestion I would need git-fast-import to respect\n> repack.maxblobsize as well. Is that OK with you?\n\nYes I could implement that (or better accept a patch that does so)\nbut I'd actually wonder why not just categorize the objects into\ntwo different packfiles. Have one for \"small stuff\" and another\nfor \"everything larger than small stuff\". Split the two packfiles\nindependently of each other. Hence fast-import would produce more\npackfiles, but each output packfile would probably have a couple\nof megablobs in it, and you'd have one single packfile with all of\nthe smaller metadata.\n\nAnd actually if you are trying to shove large objects through\nfast-import we really can do a lot better. Like avoiding\ndeltification attempts, adjusting the compression level to something\nbetter suited to your blob (I don't know if its compressable or not)\nand streaming to the output packfile, rather than holding the entire\nthing in memory before writing the first byte.\n\nGiven the advantages discussed above about being in a packfile, and\nthat fast-import was writing specifically for creating packfiles\ninstead of loose objects during large IO transfers into Git, I\nthink it is sort of anti-fast-import to have it create loose objects.\n \n> I previously offered to Junio that the \"write loose object\" thing\n> could be restricted: it would only happen if -f were supplied to\n> git-repack, otherwise the bad blob would pass through to the new pack.\n> Does this \"reduction in strength\" make this feature more palatable to you?\n\nYes. But go back to the .keep discussion above where I suggest\nwe automatically .keep any \"large\" packfile. Once you get a huge\npackfile you probably don't want to redo the disk IO associated with\nrepacking it, unless you really are trying to force a large reorg.\nSo I'd agree with the idea of making a -f needed to eject a megablob,\nbut I think you'd need more than that as you'd also be trying to\nbypass the usual .keep logic.\n \n> If the stats on a repo change significantly, \"write loose object\"\n> becomes more important if you have to make a significant reduction\n> to repack.maxblobsize (or specify it for the first time).\n\nI'm not sure users should be tweaking this... Its fine to make\nthe knob available, but we really should have the knob adjust\nitself somewhat intelligently. Don't ever add a knob that you\ncannot write a rule to control; if you can write a rule than write\nit dammit and don't make the user do your job for you. Knobs are\ngood for when your case is just so far away from the normal that\nthe rule is utterly wrong...\n\nGuess what, your repository is such a repository (its far away from\nour normal rules). But I think its problems are also common enough\nthat we really should attempt to make our rules handle it better.\n \n> I don't agree that once in a packfile, a blob should stay there.\n> Its presence is degrading access to \"normal\" blobs co-habiting with it.\n> So you will want to repack to separate them in different packs\n> (the various .keep-related ideas) or just write them out loose.\n\nIf we're evicting a megablob from one packfile to a loose object\n(because its degrading access to the other objects in its current\npackfile) we're already committed to doing the massive disk IO\nrequired for that eviction. We might as well write it back out\nto a file format that we can more easily work with, than one that\nwe cannot.\n\nBut I have to circle back here and say \"why is a megablob degrading\naccess in a packfile\"? This goes right back to your point above\nabout my wanting to stay in the packfile format just to make the\npackfile format better. What's so wrong with the packfile format\n(and the code that writes/reads it) that makes access for a small\nmetadata more expensive when there are megablobs attached in the\nsame packfile?\n\nOr is it just because we like to repack the smaller metadata\nfrequently, but that's horribly expensive because the megablobs\nare in the same packfile? If its really just about repacking then\n.keep marked megablob packs are the way to go.\n\n-- \nShawn.\n"},{"id":"43218","messageId":"alpine.LFD.0.99.0705250010350.3366@xanadu.home","threadId":"8273","inReplyTo":"20070525020642.GS28023@spearce.org","subject":"Re: [PATCH] Prevent megablobs from gunking up git packs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-25T05:44:58Z","receivedAt":"2007-05-25T05:44:58Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"\nOK..... I ignore git@vger.kernel.org for a day or two and things really \nstart to go wild! ;-)\n\nI'll try to cover only those points that are still debatable. I think \neverybody agrees with huge blobs as loose objects using extra inodes \nbeing the least of our worries.\n\nOn Thu, 24 May 2007, Shawn O. Pearce wrote:\n\n> Dana How wrote:\n> > On 5/24/07, Shawn O. Pearce wrote:\n> > >Junio C Hamano wrote:\n> > >> \"Dana How\" writes:\n> > >> > We have three options in this case:\n> > >> > (1) Drop the object (do not put it in the new pack(s)).\n> > >> > (2) Pass the object into the new pack(s).\n> > >> > (3) Write out the object as a new loose object.\n> > >> > Option (1) is unacceptable. When you call git-repack -a,\n> > >> > it blindly deletes all the non-kept packs at the end. So\n> > >> > the megablobs would be lost.\n> > >> Ok, I can buy that -- (1) nor (2) are unacceptable and (3) is\n> > >> the only sane thing to do for a previously packed objects that\n> > >> exceed the size limit.\n\nOK... I sort of agree, but not entirely.\n\nFirst, let's examine the reasons for wanting to expulse a big blob out \nof a pack.\n\nThe first reason I've seen is that big blobs put surrounding objects way \napart and pack access performance gets bad, especially tree walking. \nThe solution to this problem is trivial: let's simply store big blobs \ntogether at the end of the pack! Problem solved.\n\nThe other reason for keeping huge blobs out is that they bring repack \nperformance down and create unnecessary IO. Well, in that case I think \nthat you should simply avoid (re)packing them in the first place. I \nthink it should be possible to combine both features: the split packs \nand the big-blobs-go-at-the-end solution I mentioned above so that those \nbig blobs could end up in one or more packs of their own.\n\nBut writing loose objects from git-pack-objects... Nah, this is just too \nhacky and ugly. The tool is about packing objects and starting to \ncreate loose objects from there is pushing the packing concept a bit too \nfar for my taste.\n\nI wouldn't mind a _separate_ tool that would load a pack index, \ndetermine object sizes from it, and then extract big objects to write \nthem as loose objects (although I question its usefulness). But not \nwithin pack-objects please.\n\nSo I think the best solution really involves a new parameter to \ngit-pack-objects allowing for objects which size exceed a certain \ntreshold to go at the end of the pack. If they end up in a different \npack because of pack size limit then so be it, at which point you could \nalways explode that huge-blob pack into loose objects, avoiding the need \nfor the extra tool I mention above, but again I don't think that would \nbe that useful.\n\n> > Again Geert made a good argument that didn't occur to me that\n> > you definitely DON'T want to do deltification on such large objects.\n> > Junio recently added delta/nodelta attribute; this would be useful\n> > to me, but unfortunately I have several continua of files, each with\n> > the same suffix, but with largely varying sizes, so attributes won't\n> > help me unless the name globs in .gitattributes are expanded to full\n> > expressions similar to find(1) [i.e. include testing based on size,\n> > perms, type], which I think would be insane.\n\nI think having a parameter to exclude object which size exceed a \nspecified size treshold from deltification attempts would also be a \nvalid option. But...\n\n> Which brings up the comment I think I made (below) about skipping\n> deltas on very large objects. Things over a certain size are not\n> likely to delta well, or in any reasonable time. We probably should\n> default to not trying to delta those, but let the user force us to\n> do so with a .gitattributes option. Maybe.\n\nI don't agree with the presumption that huge objects are unlikely to \ndelta well. It really depends on the data you have. If, for example, \nyou want to store, say, different versions of a filesystem image, then \nthose different images have the potential to be really huge. Yet they \nmight delta extremely well against each other and provide a tremendous \nspace saving.\n\nIt all depends on the kind of data you work with.\nIt is good to have the possibility to skip deltification based on a file \nattribute. It is also good to have the possibility to skip \ndeltification based on object size (through a command line switch or \nconfig entry). But those must remain _options_.\n\n> > >Huge packfiles probably should be scheduled for keeping with a .keep\n> > >automatically. We probably should teach pack-objects to generate a\n> > >.keep file if the resulting .pack was over a certain size threshold\n> > >(say 1.5 GiB by default) and teach git-repack to rename the .keep\n> > >file as it also renames the .idx and .pack.\n\nNah. Those kind of arbitrary defaults are most likely to be fine for \nsome cases and bad for many others. These \"sick\" cases such as Dana's \nare so special that they better be manually tuned for best operations \naccording to the data set, and more importantly to the work flow used, \nbecause different work flows are likely to require different \"defaults\". \nBetter not put any arbitrary default and create a \"Advanced tuning for \nbest performances with insane repositories\" section in the documentation \ninstead.\n\n> > >Better that we degrade gracefully when faced with massive inputs\n> > >than we do something stupid by default and make the poor user pay\n> > >for their mistake of not throughly reading plumbing documentation\n> > >before use.\n\nWell, I think that if someone is seriously considering GIT for a \nmulti-gigabyte repository, that person has better read a little \ndocumentation before starting to play. Of course this advanced tuning \nfor huge repository section I'm suggesting should stand out in the main \nindex. And most \"poor users\" usually don't have such a big repo to \nfool themselves with.\n\n> > >Now I would agree that we should punt on deltification of anything\n> > >that is just too large, and let the user decide what too large means,\n> > >and default it around 500 or 1024 MiB. But I would still stuff it\n> > >into a packfile.\n\nWell, thing is, once deltified, those huge objects won't be subject to \ndeltification attempts anymore, unless -f is used. So the deltification \ncost will happen only once anyway. Then it is only the issue of \nflagging a particular pack with .keep to exclude it from any further \nrepacking which would simply end up wasting disk IO anyway.\n\nThere is certainly a hard default on deltification attempt that we \nshould impose right now though, which is 4GB. The reason is that the \ndelta encoding doesn't do offsets larger than 32 bits at the moment.\n\n> > I previously offered to Junio that the \"write loose object\" thing\n> > could be restricted: it would only happen if -f were supplied to\n> > git-repack, otherwise the bad blob would pass through to the new pack.\n> > Does this \"reduction in strength\" make this feature more palatable to you?\n\nNot really.\n\nLike I said before, I'd much prefer to have a split pack for huge \nobjects, and a separate unpack-object pass on it if you really want them \nloose. If you want to deny entry of loose objects into a pack based on \ntheir size that's understandable, but only if they're already loose.\n\n> > I don't agree that once in a packfile, a blob should stay there.\n> > Its presence is degrading access to \"normal\" blobs co-habiting with it.\n\nAs mentioned at the top I don't think this is a big issue.\n\n> Or is it just because we like to repack the smaller metadata\n> frequently, but that's horribly expensive because the megablobs\n> are in the same packfile? If its really just about repacking then\n> .keep marked megablob packs are the way to go.\n\nI think so as well.\n\n\nNicolas\n"}]}