{"thread":{"id":"24061","subject":"Leaving large binaries out of the packfile","startedAt":"2010-06-10T06:25:26Z","lastAt":"2010-06-24T06:32:47Z","messageCount":5,"participants":["Joshua Jensen","Shawn O. Pearce","Paolo Bonzini"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"143393","messageId":"4C108556.8040102@workspacewhiz.com","threadId":"24061","inReplyTo":null,"subject":"Leaving large binaries out of the packfile","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2010-06-10T06:25:26Z","receivedAt":"2010-06-10T06:25:26Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":" Hi.\n\nI've been dealing with a Subversion repository that contains a lot of \nlarge binaries. Git generally seems to handle them reasonably enough, \nalthough it chokes under the pressure of a 'git gc' with this git-svn \nrepository. The repository packs total 2.7 gigabytes. As it turns out, \nthe 250 individual blob revisions worth of large binaries are about 2.4 \ngigabytes of that.\n\nSometimes, 'git gc' runs out of memory. I have to discover which file \nis causing the problem, so I can add it to .gitattributes with a \n'-delta' flag. Mostly, though, the repacking takes forever, and I dread \nrunning the operation.\n\nAs an experiment, I added a '-pack' flag to .gitattributes. This flag \nwill leave the file type specified by the .gitattributes entry loose in \nthe repository. During a 'git gc', instead of recopying gigabytes of \ndata each time, the loose objects are used. The 'git gc' process runs \nvery quick with this change.\n\nThe only issue I've found is in too_many_loose_objects(). gitk is \nalways telling me the repository needs to be packed, obviously because \nof all the loose objects.\n\nI haven't yet come up with a good idea for handling this. I thought \nabout putting the forced loose objects in a separate directory. (This \nidea goes along with another that I want to build on top of this \nfunctionality, the ability to commit and have -pack binaries go to an \nalternates location.) I have also thought about writing out a file with \nthe count of forced loose objects and using that to drive the \nguesstimate made by too_many_loose_objects() down.\n\nDoes anyone have any thoughts?\n\nThanks!\n\nJosh\n\n---\n builtin/pack-objects.c | 25 +++++++++++++++++++++++++\n 1 files changed, 25 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 214d7ef..f33a7fb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -644,6 +644,28 @@ static int no_try_delta(const char *path)\n return 0;\n }\n\n+static void setup_pack_attr_check(struct git_attr_check *check)\n+{\n+ static struct git_attr *attr_pack;\n+\n+ if (!attr_pack)\n+ attr_pack = git_attr(\"pack\");\n+\n+ check[0].attr = attr_pack;\n+}\n+\n+static int must_pack(const char *path)\n+{\n+ struct git_attr_check check[1];\n+\n+ setup_pack_attr_check(check);\n+ if (git_checkattr(path, ARRAY_SIZE(check), check))\n+ return 1;\n+ if (ATTR_FALSE(check->value))\n+ return 0;\n+ return 1;\n+}\n+\n static int add_object_entry(const unsigned char *sha1, enum \nobject_type type,\n const char *name, int exclude)\n {\n@@ -667,6 +689,9 @@ static int add_object_entry(const unsigned char \n*sha1, enum object_type type,\n if (!exclude && local && has_loose_object_nonlocal(sha1))\n return 0;\n\n+ if (name && !must_pack(name))\n+ return 0;\n+\n for (p = packed_git; p; p = p->next) {\n off_t offset = find_pack_entry_one(sha1, p);\n if (offset) {\n--\n1.7.1.msysgit.3.1.g108b5.dirty\n"},{"id":"143459","messageId":"20100610180457.GO14847@spearce.org","threadId":"24061","inReplyTo":"4C108556.8040102@workspacewhiz.com","subject":"Re: Leaving large binaries out of the packfile","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-06-10T18:04:57Z","receivedAt":"2010-06-10T18:04:57Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Joshua Jensen wrote:\n> Sometimes, 'git gc' runs out of memory. I have to discover which file \n> is causing the problem, so I can add it to .gitattributes with a \n> '-delta' flag. Mostly, though, the repacking takes forever, and I dread \n> running the operation.\n\nIf you have the list of big objects, you can put them into their\nown pack file manually. Feed their SHA-1 names on stdin to git\npack-objects, and save the resulting pack under .git/objects/pack.\n\nAssuming the pack was called pack-DEADC0FFEE.pack, create a file\ncalled pack-DEADC0FFEE.keep in the same directory. This will stop\nGit from trying to repack the contents of that pack file.\n\nNow run `git gc` to remove those huge objects from the pack file\nthat contains all of the other stuff.\n\n-- \nShawn.\n"},{"id":"143516","messageId":"4C12566E.6020607@gnu.org","threadId":"24061","inReplyTo":"20100610180457.GO14847@spearce.org","subject":"Re: Leaving large binaries out of the packfile","fromName":"Paolo Bonzini","fromEmail":"bonzini@gnu.org","sentAt":"2010-06-11T15:29:50Z","receivedAt":"2010-06-11T15:29:50Z","isPatch":false,"sender":{"key":"bonzini@gnu.org","avatar":"https://avatars.githubusercontent.com/u/42082?v=4"},"body":"On 06/10/2010 08:04 PM, Shawn O. Pearce wrote:\n> Joshua Jensen wrote:\n>> Sometimes, 'git gc' runs out of memory. I have to discover which file\n>> is causing the problem, so I can add it to .gitattributes with a\n>> '-delta' flag. Mostly, though, the repacking takes forever, and I dread\n>> running the operation.\n>\n> If you have the list of big objects, you can put them into their\n> own pack file manually. Feed their SHA-1 names on stdin to git\n> pack-objects, and save the resulting pack under .git/objects/pack.\n\nDo you know any simpler way than\n\ngit log --pretty=format:%H | while read x; do\n git ls-tree $x -- ChangeLog | awk '{print $3}'\ndone | sort -u\n\nto do this? I thought it would be nice to add --sha1-only to \ngit-ls-tree, but maybe I'm missing some other trick.\n\n> Assuming the pack was called pack-DEADC0FFEE.pack, create a file\n> called pack-DEADC0FFEE.keep in the same directory. This will stop\n> Git from trying to repack the contents of that pack file.\n>\n> Now run `git gc` to remove those huge objects from the pack file\n> that contains all of the other stuff.\n\nThat obviously wouldn't help if these large binaries are updated often, \nhowever.\n\nPaolo\n"},{"id":"143520","messageId":"20100611161744.GS14847@spearce.org","threadId":"24061","inReplyTo":"4C12566E.6020607@gnu.org","subject":"Re: Leaving large binaries out of the packfile","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-06-11T16:17:44Z","receivedAt":"2010-06-11T16:17:44Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Paolo Bonzini wrote:\n> On 06/10/2010 08:04 PM, Shawn O. Pearce wrote:\n>> Joshua Jensen wrote:\n>>> Sometimes, 'git gc' runs out of memory. I have to discover which file\n>>> is causing the problem, so I can add it to .gitattributes with a\n>>> '-delta' flag. Mostly, though, the repacking takes forever, and I dread\n>>> running the operation.\n>>\n>> If you have the list of big objects, you can put them into their\n>> own pack file manually. Feed their SHA-1 names on stdin to git\n>> pack-objects, and save the resulting pack under .git/objects/pack.\n>\n> Do you know any simpler way than\n>\n> git log --pretty=format:%H | while read x; do\n> git ls-tree $x -- ChangeLog | awk '{print $3}'\n> done | sort -u\n>\n> to do this? I thought it would be nice to add --sha1-only to \n> git-ls-tree, but maybe I'm missing some other trick.\n\nMaybe\n\n git rev-list --objects HEAD | grep ' ChangeLog'\n\npack-objects wants the output of rev-list --objects as input, file\nname and all. So its just a matter of selecting the right lines\nfrom its output.\n\n>> Assuming the pack was called pack-DEADC0FFEE.pack, create a file\n>> called pack-DEADC0FFEE.keep in the same directory. This will stop\n>> Git from trying to repack the contents of that pack file.\n>>\n>> Now run `git gc` to remove those huge objects from the pack file\n>> that contains all of the other stuff.\n>\n> That obviously wouldn't help if these large binaries are updated often, \n> however.\n\nNo, it doesn't. But you still could do this on a periodic basis.\nThat way you only drag around a handful of recently created large\nbinaries during a typical `git gc`, and not the entire project's\nhistory of them.\n\n-- \nShawn.\n"},{"id":"144121","messageId":"4C22FC0F.1060805@workspacewhiz.com","threadId":"24061","inReplyTo":"20100610180457.GO14847@spearce.org","subject":"Re: Leaving large binaries out of the packfile","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2010-06-24T06:32:47Z","receivedAt":"2010-06-24T06:32:47Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":" ----- Original Message -----\nFrom: Shawn O. Pearce\nDate: 6/10/2010 12:04 PM\n> Joshua Jensen wrote:\n>> Sometimes, 'git gc' runs out of memory. I have to discover which file\n>> is causing the problem, so I can add it to .gitattributes with a\n>> '-delta' flag. Mostly, though, the repacking takes forever, and I dread\n>> running the operation.\n> If you have the list of big objects, you can put them into their\n> own pack file manually. Feed their SHA-1 names on stdin to git\n> pack-objects, and save the resulting pack under .git/objects/pack.\n>\n> Assuming the pack was called pack-DEADC0FFEE.pack, create a file\n> called pack-DEADC0FFEE.keep in the same directory. This will stop\n> Git from trying to repack the contents of that pack file.\n>\n> Now run `git gc` to remove those huge objects from the pack file\n> that contains all of the other stuff\nPardon the late response.\n\nThis method can work, but it is a manual process. I am interested in a \nmethod where Git can make the determination for me based on a wildcard \nand flag from .gitattributes.\n\nI am still playing with the feature within a multi-gigabyte repository \nwith lots of large binaries. I'll post more about it when some \nadditional changes have been made.\n\nThanks!\n\nJosh\n"}]}