{"thread":{"id":"6922","subject":"RFC: [PATCH] Support incremental pack files","startedAt":"2007-02-23T07:13:44Z","lastAt":"2007-02-26T22:03:01Z","messageCount":6,"participants":["Martin Koegler","Junio C Hamano","Nicolas Pitre","Shawn O. Pearce","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"35300","messageId":"20070223071344.GA21107@auto.tuwien.ac.at","threadId":"6922","inReplyTo":null,"subject":"RFC: [PATCH] Support incremental pack files","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2007-02-23T07:13:44Z","receivedAt":"2007-02-23T07:13:44Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"With CVS (or RCS), commiting a version, increases the required storage\nonly by the size of the diff.\n\nCommiting a new version in GIT increases the storage by the compressed\nsize of each changed blob. Packing all unpacked objects decreases the\nrequired storage, but does not generate deltas against objects in\npacks. You need to repack all objects to get around this.\n\nFor normal source code, this is not a problem.  But if you want to use\ngit for big files, you waste storage (or CPU time for everything\nrepacking).\n\nThe follwing patch (again git-1.5.0-rc3) is a prototyp for supporting\nincremental pack files in git. The file structures are not changed.\nIt only permits, that the base commit of a delta is located in a\ndifferent pack or as unpacked object.\n\nChanges: \n\n* in sha1_file get_delta_base returns offset 0, if the delta base\nobject is not the same pack. Additionally it stores the sha1 of it in\na parameter. unpack_delta_entry and packed_delta_info are changed, so\nthat they search such a blob in other packs or as sha1\nfile. packed_object_info_detail stops searching, if the delta base\nobject is not in the same pack.\n\n* to builtin-pack-objects.c a loop detector is added (flag\nloop_check).  When adding a excluded object (= preferred base), its\nposition in the pack file is added, if it is in pack.\n\nIf a preferred base object is in a pack, it checks, if it is a delta.\nIf so, it locates the base object and stores it in ->delta. If it was\nnot in the object list, the base object gets added with a exclude\nvalue of 2 and it is recusivly check the base object in the same way\n(check_preferred_object).\n\ntry_delta refuses to produce a delta, if a loop would be created or if\nthe base object was added by check_preferred_object.\n\n* mark_edges_unparsed travers all commit list for not parsed parent\ncommit objects. They are used as heuristic to create a list of\nsuiteable base object for building the delta.\n\nWith the patch, you can eg. pack all unpacked object with a command link like\ngit-pack-objects --non-empty --all --reflog --unpacked --incremental --base-parent\n\nI see the following problems:\n* fetching an incremental pack file over HTTP\n* reusing a incremental pack file via reuse_cached_pack in builtin-pack-objects.c\n\nmfg Martin Kögler\n\n--- builtin-pack-objects.c.orig\t2007-02-22 22:11:11.287809817 +0100\n+++ builtin-pack-objects.c\t2007-02-22 22:11:22.694525976 +0100\n@@ -16,7 +16,7 @@\n static const char pack_usage[] = \"\\\n git-pack-objects [{ -q | --progress | --all-progress }] \\n\\\n \t[--local] [--incremental] [--window=N] [--depth=N] \\n\\\n-\t[--no-reuse-delta] [--delta-base-offset] [--non-empty] \\n\\\n+\t[--no-reuse-delta] [--delta-base-offset] [--non-empty] [--base-parent] \\n\\\n \t[--revs [--unpacked | --all]*] [--reflog] [--stdout | base-name] \\n\\\n \t[<ref-list | <object-list]\";\n \n@@ -65,6 +65,8 @@\n static int local;\n static int incremental;\n static int allow_ofs_delta;\n+static int base_parent;\n+static int loop_check;\n \n static struct object_entry **sorted_by_sha, **sorted_by_type;\n static struct object_entry *objects;\n@@ -692,6 +694,19 @@\n \t\t\t}\n \t\t}\n \t}\n+\n+\tif (loop_check && exclude) {\n+\t\tfor (p = packed_git; p; p = p->next) {\n+\t\t\tunsigned long offset = find_pack_entry_one(sha1, p);\n+\t\t\tif (offset) {\n+\t\t\t\tif (!found_pack) {\n+\t\t\t\t\tfound_offset = offset;\n+\t\t\t\t\tfound_pack = p;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \tif ((entry = locate_object_entry(sha1)) != NULL)\n \t\tgoto already_added;\n \n@@ -722,13 +737,13 @@\n \t\tprogress_update = 0;\n \t}\n \tif (exclude)\n-\t\tentry->preferred_base = 1;\n-\telse {\n-\t\tif (found_pack) {\n-\t\t\tentry->in_pack = found_pack;\n-\t\t\tentry->in_pack_offset = found_offset;\n-\t\t}\n+\t    entry->preferred_base = exclude;\n+\n+\tif (found_pack) {\n+\t    entry->in_pack = found_pack;\n+\t    entry->in_pack_offset = found_offset;\n \t}\n+\n \treturn status;\n }\n \n@@ -976,6 +991,78 @@\n \tit->pcache.tree_size = size;\n }\n \n+static void check_preferred_object(struct object_entry *entry)\n+{\n+    struct packed_git *p = entry->in_pack;\n+    struct pack_window *w_curs = NULL;\n+    unsigned long size, used;\n+    unsigned char *buf;\n+    unsigned long left = p->pack_size - entry->in_pack_offset;\n+    struct object_entry *base_entry = NULL;\n+    unsigned hash;\n+\n+    if (!p || !entry->preferred_base)\n+\treturn;\n+\n+    buf = use_pack(p, &w_curs, entry->in_pack_offset, NULL);\n+    \n+    used = unpack_object_header_gently(buf, left,\n+\t\t\t\t       &entry->in_pack_type, &size);\n+\n+    unsigned char c, *base_name;\n+    unsigned long ofs;\n+    unsigned long used_0;\n+    /* there is at least 20 bytes left in the pack */\n+    switch (entry->in_pack_type) {\n+\tcase OBJ_REF_DELTA:\n+\t    base_name = use_pack(p, &w_curs,\n+\t\t\t\t entry->in_pack_offset + used, NULL);\n+\t    used += 20;\n+\t    break;\n+\tcase OBJ_OFS_DELTA:\n+\t    buf = use_pack(p, &w_curs,\n+\t\t\t   entry->in_pack_offset + used, NULL);\n+\t    used_0 = 0;\n+\t    c = buf[used_0++];\n+\t    ofs = c & 127;\n+\t    while (c & 128) {\n+\t\tofs += 1;\n+\t\tif (!ofs || ofs & ~(~0UL >> 7))\n+\t\t    die(\"delta base offset overflow in pack for %s\",\n+\t\t\tsha1_to_hex(entry->sha1));\n+\t\tc = buf[used_0++];\n+\t\tofs = (ofs << 7) + (c & 127);\n+\t    }\n+\t    if (ofs >= entry->in_pack_offset)\n+\t\tdie(\"delta base offset out of bound for %s\",\n+\t\t    sha1_to_hex(entry->sha1));\n+\t    ofs = entry->in_pack_offset - ofs;\n+\t    base_name = find_packed_object_name(p, ofs);\n+\t    used += used_0;\n+\t    break;\n+\tdefault:\n+\t    base_name = NULL;\n+    }\n+\n+    unuse_pack(&w_curs);\n+\n+    if (!base_name)\n+\treturn;\n+\n+    base_entry = locate_object_entry(base_name);\n+    if (!base_entry) {\n+\thash = name_hash(\"\");\n+\tadd_object_entry(base_name, hash, 2);\n+\n+\tbase_entry = locate_object_entry(base_name);\n+\tcheck_preferred_object(base_entry);\n+    }\n+\n+    entry->delta = base_entry;\n+    entry->delta_sibling = base_entry->delta_child;\n+    base_entry->delta_child = entry;\n+}\n+\n static void check_object(struct object_entry *entry)\n {\n \tchar type[20];\n@@ -1062,6 +1149,9 @@\n \t\t/* Otherwise we would do the usual */\n \t}\n \n+\tif (entry->in_pack && entry->preferred_base) \n+\t    check_preferred_object (entry);\n+\n \tif (sha1_object_info(entry->sha1, type, &entry->size))\n \t\tdie(\"unable to get type of object %s\",\n \t\t    sha1_to_hex(entry->sha1));\n@@ -1218,6 +1308,8 @@\n \t */\n \tif (trg_entry->preferred_base)\n \t\treturn -1;\n+\tif (src_entry->preferred_base == 2)\n+\t\treturn -1;\n \n \t/*\n \t * We do not bother to try a delta that we discarded\n@@ -1242,6 +1334,15 @@\n \tif (src_entry->depth >= max_depth)\n \t\treturn 0;\n \n+\tif (loop_check) {\n+\t    struct object_entry *i = src_entry->delta;\n+\t    while (i) {\n+\t\tif (i == trg_entry)\n+\t\t    return 0;\n+\t\ti = i->delta;\n+\t    }\n+\t}\n+\n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n \tmax_size = trg_size/2 - 20;\n@@ -1540,6 +1641,8 @@\n \n \tprepare_revision_walk(&revs);\n \tmark_edges_uninteresting(revs.commits, &revs, show_edge);\n+\tif (base_parent)\n+\t    mark_edges_unparsed(revs.commits, &revs, show_edge);\n \ttraverse_commit_list(&revs, show_commit, show_object);\n }\n \n@@ -1609,6 +1712,11 @@\n \t\t\tno_reuse_delta = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(\"--base-parent\", arg)) {\n+\t\t\tbase_parent = 1;\n+\t\t\tloop_check = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(\"--delta-base-offset\", arg)) {\n \t\t\tallow_ofs_delta = 1;\n \t\t\tcontinue;\n--- list-objects.c.orig\t2007-02-22 22:05:13.972754239 +0100\n+++ list-objects.c\t2007-02-22 22:11:11.254461051 +0100\n@@ -66,6 +66,27 @@\n \ttree->buffer = NULL;\n }\n \n+void mark_edges_unparsed(struct commit_list *list,\n+\t\t\t struct rev_info *revs,\n+\t\t\t show_edge_fn show_edge)\n+{\n+\tfor ( ; list; list = list->next) {\n+\t\tstruct commit *commit = list->item;\n+\t\tstruct commit_list *parents;\n+\n+\t\tfor (parents = commit->parents; parents; parents = parents->next) {\n+\t\t    struct commit *parent = parents->item;\n+\t\t    if (parent->object.parsed)\n+\t\t\tcontinue;\n+\t\t    if (!(parent->object.flags & SHOWN)) {\n+\t\t\tparent->object.flags |= SHOWN;\n+\t\t\tshow_edge(parent);\n+\t\t    }\n+\t\t}\n+\t}\n+}\n+\n+\n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t    struct rev_info *revs,\n \t\t\t\t\t    show_edge_fn show_edge)\n--- list-objects.h.orig\t2007-02-22 22:05:10.375119739 +0100\n+++ list-objects.h\t2007-02-22 22:05:48.095287586 +0100\n@@ -8,5 +8,6 @@\n void traverse_commit_list(struct rev_info *revs, show_commit_fn, show_object_fn);\n \n void mark_edges_uninteresting(struct commit_list *, struct rev_info *, show_edge_fn);\n+void mark_edges_unparsed(struct commit_list *, struct rev_info *, show_edge_fn);\n \n #endif\n--- sha1_file.c.orig\t2007-02-22 22:39:46.163797786 +0100\n+++ sha1_file.c\t2007-02-22 22:52:36.547048788 +0100\n@@ -1030,7 +1030,8 @@\n \t\t\t\t    unsigned long offset,\n \t\t\t\t    enum object_type kind,\n \t\t\t\t    unsigned long delta_obj_offset,\n-\t\t\t\t    unsigned long *base_obj_offset)\n+\t\t\t\t    unsigned long *base_obj_offset,\n+\t\t\t\t    unsigned char *base_sha1)\n {\n \tunsigned char *base_info = use_pack(p, w_curs, offset, NULL);\n \tunsigned long base_offset;\n@@ -1059,9 +1060,7 @@\n \t} else if (kind == OBJ_REF_DELTA) {\n \t\t/* The base entry _must_ be in the same pack */\n \t\tbase_offset = find_pack_entry_one(base_info, p);\n-\t\tif (!base_offset)\n-\t\t\tdie(\"failed to find delta-pack base object %s\",\n-\t\t\t\tsha1_to_hex(base_info));\n+\t\thashcpy (base_sha1, base_info);\n \t\toffset += 20;\n \t} else\n \t\tdie(\"I am totally screwed\");\n@@ -1081,18 +1080,25 @@\n \t\t\t     char *type,\n \t\t\t     unsigned long *sizep)\n {\n+        unsigned char base_sha1[20];\n \tunsigned long base_offset;\n \n \toffset = get_delta_base(p, w_curs, offset, kind,\n-\t\tobj_offset, &base_offset);\n+\t\tobj_offset, &base_offset, base_sha1);\n \n \t/* We choose to only get the type of the base object and\n \t * ignore potentially corrupt pack file that expects the delta\n \t * based on a base with a wrong size.  This saves tons of\n \t * inflate() calls.\n \t */\n-\tif (packed_object_info(p, base_offset, type, NULL))\n+\tif (base_offset) {\n+\t    if (packed_object_info(p, base_offset, type, NULL))\n \t\tdie(\"cannot get info for delta-pack base\");\n+\t} else {\n+\t    if (sha1_object_info(base_sha1, type, NULL))\n+\t\tdie(\"cannot get info for delta-pack base %s\",\n+\t\t    sha1_to_hex (base_sha1));\n+\t}\n \n \tif (sizep) {\n \t\tconst unsigned char *data;\n@@ -1168,6 +1174,7 @@\n \tstruct pack_window *w_curs = NULL;\n \tunsigned long obj_offset, val;\n \tunsigned char *next_sha1;\n+\tunsigned char sha1[20];\n \tenum object_type kind;\n \n \t*delta_chain_length = 0;\n@@ -1189,7 +1196,7 @@\n \t\t\treturn;\n \t\tcase OBJ_OFS_DELTA:\n \t\t\tget_delta_base(p, &w_curs, offset, kind,\n-\t\t\t\tobj_offset, &offset);\n+\t\t\t\tobj_offset, &offset, sha1);\n \t\t\tif (*delta_chain_length == 0) {\n \t\t\t\t/* TODO: find base_sha1 as pointed by offset */\n \t\t\t}\n@@ -1199,6 +1206,8 @@\n \t\t\tif (*delta_chain_length == 0)\n \t\t\t\thashcpy(base_sha1, next_sha1);\n \t\t\toffset = find_pack_entry_one(next_sha1, p);\n+\t\t\tif (!offset)\n+\t\t\t    return;\n \t\t\tbreak;\n \t\t}\n \t\tobj_offset = offset;\n@@ -1281,11 +1290,15 @@\n \t\t\t\tunsigned long *sizep)\n {\n \tvoid *delta_data, *result, *base;\n+\tunsigned char base_sha1[20];\n \tunsigned long result_size, base_size, base_offset;\n \n \toffset = get_delta_base(p, w_curs, offset, kind,\n-\t\tobj_offset, &base_offset);\n-\tbase = unpack_entry(p, base_offset, type, &base_size);\n+\t\tobj_offset, &base_offset, base_sha1);\n+\tif (base_offset)\n+\t    base = unpack_entry(p, base_offset, type, &base_size);\n+\telse\n+\t    base = read_sha1_file (base_sha1, type, &base_size);\n \tif (!base)\n \t\tdie(\"failed to read delta base object at %lu from %s\",\n \t\t    base_offset, p->pack_name);\n"},{"id":"35302","messageId":"7vfy8x9tvo.fsf@assigned-by-dhcp.cox.net","threadId":"6922","inReplyTo":"20070223071344.GA21107@auto.tuwien.ac.at","subject":"Re: RFC: [PATCH] Support incremental pack files","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-23T08:10:35Z","receivedAt":"2007-02-23T08:10:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"mkoegler@auto.tuwien.ac.at (Martin Koegler) writes:\n\n> Commiting a new version in GIT increases the storage by the compressed\n> size of each changed blob. Packing all unpacked objects decreases the\n> required storage, but does not generate deltas against objects in\n> packs. You need to repack all objects to get around this.\n>\n> For normal source code, this is not a problem.  But if you want to use\n> git for big files, you waste storage (or CPU time for everything\n> repacking).\n\nThree points that might help you without any code change.\n\n - Have you run \"git repack -a -d\" without \"-f\"?  Reusing of\n   existing delta is specifically designed to avoid the \"CPU\n   time for everything repacking\" problem.\n\n - If you are dealing with something other than \"normal source\n   code\", do you know if your objects delta against each other\n   well?  If not, turning core.legacyheaders off might be a\n   win.  It allows the objects that are recorded as non-delta in\n   resulting pack to be copied straight from loose objects.\n\n - Once you accumulated large enough packs with existing\n   objects, marking them with .keep would leave them untouched\n   during subsequent repack.  When \"git repack -a -d\" repacks\n   \"everything\", its definition of \"everything\" becomes \"except\n   things that are in packs marked with .keep files\".\n\nSide note: Is the .keep mechanism sufficiently documented?  I am\ntoo lazy to check that right now, but here is a tip.  After\nreleasing the big one, line v1.5.0, I do:\n\n  $ P=.git/objects/pack\n  $ git rev-list --objects v1.5.0 |\n    git pack-objects --delta-base-offset \\\n          --depth=30 --window=100 --no-reuse-delta pack\n  ...\n  6fba5cb8ed92dfef71ff47def9f95fa1e703ba59\n  $ mv pack-6fba5cb8ed92dfef71ff47def9f95fa1e703ba59.* $P/\n  $ echo 'Post 1.5.0' >$P/pack-6fba5cb8ed92dfef71ff47def9f95fa1e703ba59.keep\n  $ git gc --prune\n\nThis does three things:\n\n - It packs everything reachable from v1.5.0 with delta chain\n   that is deeper than the default.\n\n - The pack is installed in the object store; the presence of\n   .keep file (the contents of it does not matter) tells\n   subsequent repack not to touch it.\n\n - Then the remaining objects are packed into different pack.\n\nWith this, the repository uses two packs, one is what I'll keep\nuntil it's time to do the big repack again, another is what's\nconstantly recreated by repacking but contains only \"recent\"\nobject.\n\n> It only permits, that the base commit of a delta is located in a\n> different pack or as unpacked object.\n\nThis \"only\" change needs to be done _very_ carefully, since\nself-containedness of pack files is one of the important\nelements of the stability of a git repository.\n\nIn effect, you are making the delta and its base object into a\nnew type of \"reachability\" for the purpose of fsck/prune by\nallowing incremental pack to contain a delta against a loose\nobject.  I am not saying it is a bad idea, but making sure you\ncovered every case you could lose necessary objects will be a\nlot of work.\n\nFor example, suppose a delta in your incremental pack is based\non a loose object.  That loose object can become unreachable\nafter rewinding or rebasing your refs.  You have to somehow\narrange that git-prune knows this situation and prevent it from\ngetting pruned -- otherwise your incremental pack becomes\ncorrupt.\n\nAnd that is just one example I could come up with after seeing\nyour message in 3 minutes while watching TV ;-).  I would\nusually say \"I am sure there will be more...\", but in this\nparticular case, I am inclined to say that I do not even want to\nstart thinking about possible fallout from this.  It's scary.\n"},{"id":"35326","messageId":"alpine.LRH.0.82.0702231040510.27932@xanadu.home","threadId":"6922","inReplyTo":"20070223071344.GA21107@auto.tuwien.ac.at","subject":"Re: RFC: [PATCH] Support incremental pack files","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-23T16:04:38Z","receivedAt":"2007-02-23T16:04:38Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 23 Feb 2007, Martin Koegler wrote:\n\n> With CVS (or RCS), commiting a version, increases the required storage\n> only by the size of the diff.\n> \n> Commiting a new version in GIT increases the storage by the compressed\n> size of each changed blob. Packing all unpacked objects decreases the\n> required storage, but does not generate deltas against objects in\n> packs. You need to repack all objects to get around this.\n> \n> For normal source code, this is not a problem.  But if you want to use\n> git for big files, you waste storage (or CPU time for everything\n> repacking).\n\nDid you try repack -a -d (without -f) ?\n\nWhen -f is not used, already deltified objects are simply copied as is \ninto the new pack without further processing.\n\n> The follwing patch (again git-1.5.0-rc3) is a prototyp for supporting\n> incremental pack files in git. The file structures are not changed.\n> It only permits, that the base commit of a delta is located in a\n> different pack or as unpacked object.\n\nWe always refused to have packs in the repository that are not self \ncontained because that would pave the way for all sort of nasty issues.  \nIt is otherwise much harder to prevent circular delta chains, harder to \nensure full reachability when pruning disconnected objects at the \nhierarchical level, etc.  And those are real issues that would bite you \nas soon as you perform a single fetch or push with something else than \nthe native protocol.\n\nIn other words I think this is a bad idea for repository storage.  We do \nit a part of the native GIT protocol because it is obvious that there is \nno possibility for delta loops (ommitted base objects in the transmitted \npack are known to exists in the peer repository) and those packs are \nfixed up with missing objects on the receive side when not exploded into \nloose objects.\n\nAgain a repack without -f should not be that expensive.  If it is then \nsomething is wrong and that should be fixed.\n\nOne thing that is too expensive in GIT is rev-list --objects --all (or \nequivalent) used to list objects to pack.  But Shawn and I have a plan \nto fix that at some point... (if only I can find some spare time to \nwrite more code for it).\n\n\nNicolas\n"},{"id":"35328","messageId":"20070223163255.GA26731@spearce.org","threadId":"6922","inReplyTo":"alpine.LRH.0.82.0702231040510.27932@xanadu.home","subject":"Re: RFC: [PATCH] Support incremental pack files","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-23T16:32:55Z","receivedAt":"2007-02-23T16:32:55Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre <nico@cam.org> wrote:\n> One thing that is too expensive in GIT is rev-list --objects --all (or \n> equivalent) used to list objects to pack.  But Shawn and I have a plan \n> to fix that at some point... (if only I can find some spare time to \n> write more code for it).\n\nDitto.  ;-)\n\nI have been pretty swamped this week, but plan on working on packv4\nprototype code today and this weekend.  (BTW, I did get the string\ncode, its useful, thanks!)\n\n-- \nShawn.\n"},{"id":"35547","messageId":"20070226214552.GA13402@auto.tuwien.ac.at","threadId":"6922","inReplyTo":"7vfy8x9tvo.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: [PATCH] Support incremental pack files","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2007-02-26T21:45:52Z","receivedAt":"2007-02-26T21:45:52Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"On Fri, Feb 23, 2007 at 12:10:35AM -0800, Junio C Hamano wrote:\n> mkoegler@auto.tuwien.ac.at (Martin Koegler) writes:\n> \n> > Commiting a new version in GIT increases the storage by the compressed\n> > size of each changed blob. Packing all unpacked objects decreases the\n> > required storage, but does not generate deltas against objects in\n> > packs. You need to repack all objects to get around this.\n> >\n> > For normal source code, this is not a problem.  But if you want to use\n> > git for big files, you waste storage (or CPU time for everything\n> > repacking).\n> \n> Three points that might help you without any code change.\n> \n>  - Have you run \"git repack -a -d\" without \"-f\"?  Reusing of\n>    existing delta is specifically designed to avoid the \"CPU\n>    time for everything repacking\" problem.\n> \n>  - If you are dealing with something other than \"normal source\n>    code\", do you know if your objects delta against each other\n>    well?  If not, turning core.legacyheaders off might be a\n>    win.  It allows the objects that are recorded as non-delta in\n>    resulting pack to be copied straight from loose objects.\n\nI currently use CVS to save the daily changes in database dumps (files\nmostly containing INSERT INTO xx (...) VALUES (...);). I'm trying to\nswitch this to git.\n\nA commit typically consists of some files with a size of > 100 MB and\nare growing every day. (All unpacked blob objects of) A commit require\ncurrently about 60 MB. A incremental pack file containing one commit\nis smaller than 1 MB, so the delta works well.\n\n>  - Once you accumulated large enough packs with existing\n>    objects, marking them with .keep would leave them untouched\n>    during subsequent repack.  When \"git repack -a -d\" repacks\n>    \"everything\", its definition of \"everything\" becomes \"except\n>    things that are in packs marked with .keep files\".\n> \n> Side note: Is the .keep mechanism sufficiently documented?  I am\n> too lazy to check that right now, but here is a tip.  After\n> releasing the big one, line v1.5.0, I do:\n\nI have not found any notice of this in the git documentation.\n\n>   $ P=.git/objects/pack\n>   $ git rev-list --objects v1.5.0 |\n>     git pack-objects --delta-base-offset \\\n>           --depth=30 --window=100 --no-reuse-delta pack\n>   ...\n>   6fba5cb8ed92dfef71ff47def9f95fa1e703ba59\n>   $ mv pack-6fba5cb8ed92dfef71ff47def9f95fa1e703ba59.* $P/\n>   $ echo 'Post 1.5.0' >$P/pack-6fba5cb8ed92dfef71ff47def9f95fa1e703ba59.keep\n>   $ git gc --prune\n> \n> This does three things:\n> \n>  - It packs everything reachable from v1.5.0 with delta chain\n>    that is deeper than the default.\n> \n>  - The pack is installed in the object store; the presence of\n>    .keep file (the contents of it does not matter) tells\n>    subsequent repack not to touch it.\n> \n>  - Then the remaining objects are packed into different pack.\n> \n> With this, the repository uses two packs, one is what I'll keep\n> until it's time to do the big repack again, another is what's\n> constantly recreated by repacking but contains only \"recent\"\n> object.\n\nThis could be a practical solution for me. The biggest disadvantage\nof this solution is, that each pack file is at least >= 60 MB.\n\nA nice feature of git is, that it normally does not change files,\nwhich keeps incremental backups small. I want to retain this, so I\nwant avoid uncessary repacking.\n\nAs I have no tags, I can base the repacking decision only on file\nsize:\n\n  * Daily: Mark all packs >= eg. 100 MB as keep and repack the\n           repository.\n  * Weekly/Monthly/Yearly: repack repository including packs of the\n           next size class.\n\nMy first idea was to write a script, which delete all keep files,\nrecreates them for packs bigger than a specified size and the starts\ngit-repack.\n\nAs git-repack already calls find, this could be easly added to the\nscript:\n\n--- git-repack  2007-02-17 18:06:09.000000000 +0100\n+++ git-repack1 2007-02-26 22:09:12.000000000 +0100\n@@ -8,11 +8,12 @@\n . git-sh-setup\n\n no_update_info= all_into_one= remove_redundant=\n-local= quiet= no_reuse_delta= extra=\n+local= quiet= no_reuse_delta= extra= sizearg=\n while case \"$#\" in 0) break ;; esac\n do\n        case \"$1\" in\n        -n)     no_update_info=t ;;\n+       -s)     sizearg=\"-size -${2}k\" ; shift; ;;\n        -a)     all_into_one=t ;;\n        -d)     remove_redundant=t ;;\n        -q)     quiet=-q ;;\n@@ -46,7 +47,7 @@\n        ;;\n ,t,)\n        if [ -d \"$PACKDIR\" ]; then\n-               for e in `cd \"$PACKDIR\" && find . -type f -name '*.pack' \\\n+               for e in `cd \"$PACKDIR\" && find . -type f $sizearg -name '*.pack' \\\n                        | sed -e 's/^\\.\\///' -e 's/\\.pack$//'`\n                do\n                        if [ -e \"$PACKDIR/$e.keep\" ]; then\n\n\n> > It only permits, that the base commit of a delta is located in a\n> > different pack or as unpacked object.\n> \n> This \"only\" change needs to be done _very_ carefully, since\n> self-containedness of pack files is one of the important\n> elements of the stability of a git repository.\n\nI understand the problems. GIT would need at least a list of external\nbase objects in the pack to speed up things like eg. git-prune.\n\nmfg Martin Kögler\n"},{"id":"35548","messageId":"Pine.LNX.4.63.0702262258120.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6922","inReplyTo":"20070226214552.GA13402@auto.tuwien.ac.at","subject":"Re: RFC: [PATCH] Support incremental pack files","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-26T22:03:01Z","receivedAt":"2007-02-26T22:03:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 26 Feb 2007, Martin Koegler wrote:\n\n> On Fri, Feb 23, 2007 at 12:10:35AM -0800, Junio C Hamano wrote:\n>\n> >   $ P=.git/objects/pack\n> >   $ git rev-list --objects v1.5.0 |\n> >     git pack-objects --delta-base-offset \\\n> >           --depth=30 --window=100 --no-reuse-delta pack\n> >   ...\n> >   6fba5cb8ed92dfef71ff47def9f95fa1e703ba59\n> >   $ mv pack-6fba5cb8ed92dfef71ff47def9f95fa1e703ba59.* $P/\n> >   $ echo 'Post 1.5.0' >$P/pack-6fba5cb8ed92dfef71ff47def9f95fa1e703ba59.keep\n> >   $ git gc --prune\n> > \n> > This does three things:\n> > \n> >  - It packs everything reachable from v1.5.0 with delta chain\n> >    that is deeper than the default.\n> > \n> >  - The pack is installed in the object store; the presence of\n> >    .keep file (the contents of it does not matter) tells\n> >    subsequent repack not to touch it.\n> > \n> >  - Then the remaining objects are packed into different pack.\n> > \n> > With this, the repository uses two packs, one is what I'll keep\n> > until it's time to do the big repack again, another is what's\n> > constantly recreated by repacking but contains only \"recent\"\n> > object.\n> \n> This could be a practical solution for me. The biggest disadvantage of \n> this solution is, that each pack file is at least >= 60 MB.\n\nJunio has a branch he rewinds sometimes. That's why he does not do the \nobvious, which should work for you:\n\n$ git gc --prune\n$ for p in .git/objects/pack/*.pack; do\n\tkeepfile=`echo $p | sed s/pack$/keep/`\n\techo \"Keep all current packs as-are\" > $keepfile\n  done\n\nYou should run this from time to time. You can run \"git gc --prune\" more \noften, of course...\n\nHth,\nDscho\n"}]}