{"thread":{"id":"5926","subject":"Recent and near future backward incompatibilities","startedAt":"2006-10-15T06:29:17Z","lastAt":"2006-10-17T16:51:12Z","messageCount":22,"participants":["Junio C Hamano","Jakub Narebski","Nicolas Pitre","Linus Torvalds","A Large Angry SCM","Theodore Tso","Stephen Hemminger"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"28790","messageId":"7v4pu62ite.fsf@assigned-by-dhcp.cox.net","threadId":"5926","inReplyTo":null,"subject":"Recent and near future backward incompatibilities","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-15T06:29:17Z","receivedAt":"2006-10-15T06:29:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"It was brought to my attention that the public git.git\nrepository cannot be cloned with older versions of git.  More\nprecisely, packs generated with post 16854571 (NOT contained in\nv1.4.2.3 but in the current \"master\" and more importantly in\nv1.4.3-rc3 which I tagged tonight) can contain deltas that are\nnot compatible with the version of git before d60fc1c8, which\nmeans that v1.1.6 and older (v1.2.0 and later are Ok).\n\nThe older version of git did not know anything about version 3\ndelta (which can express larger copies from source when\ncomputing a delta) before d60fc1c8, and barfs if it is fed a\npack that contains such a delta.  We have generated only version\n2 delta for quite a while until very recently post v1.4.2.3.\n\nThe thing is, I made a mistake to repack my public repository\nwith more recent git (I work with \"next\" usually, and with\n\"master\" after we go into -rcX cycle to prepare for the\nrelease).  Although the version of git that kernel.org runs is\nstill v1.4.2.3 which means its pack-objects does not produce\nversion 3 delta by itself, the problem is that the delta reuse\nlogic happily copies out whatever delta is in existing packs.\n\nI already repacked the public repository with an older git,\nv1.4.2.3, using 'git-repack -a -d -f' to fix this problem.\n\nOne thing we can and should immediately do is to revert 16854571\nfor now until we decide how to resolve this issue cleanly.\n\nThese are what needs to happen but one of them is quite tricky:\n\n - the reusing of delta is what makes pack-objects practical,\n   and it is expensive to look into existing delta to see if it\n   is version 2 or version 3 before deciding to reuse each delta\n   data, so even if we update pack-objects so that we can tell\n   it to generate a pack that contains only version 2 deltas, it\n   would be very expensive to do so and may not be practical.  I\n   am not sure how to resolve this issue efficiently right now;\n   we need a bit of thinking.\n\n - so instead we could just say a public repository that needs\n   to interoperate with older clients should not keep packs with\n   version 3 delta, at least for now.  deltifying from loose\n   objects are done afresh every time pack-objects is run, and\n   it is easier to control.\n\n - git-pack-objects needs to be updated so that we can control\n   whether we generate version 2 or version 3 delta.\n\n - we need to add .git/config item that tells pack-objects to\n   never generate version 3 delta for that particular\n   repository.  This is similar to the way we would need to\n   control the use of delta-base-offset representation currently\n   cooking in \"next\".\n\n - the on-the-wire protocol between fetch-pack and upload-pack\n   needs to be updated so that the server side can tell if the\n   client side is prepared to handle version 3 delta in the\n   pack, and this needs to be passed to pack-objects to control\n   its operation.  This part I can see how to fix.\n\nWe may have a similar issue when enabling generation of loose\nobjects with new style headers.  This is already controlled with\nthe core.legacyheaders configuration item.\n"},{"id":"28791","messageId":"7virim10rb.fsf@assigned-by-dhcp.cox.net","threadId":"5926","inReplyTo":"7v4pu62ite.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-15T07:44:40Z","receivedAt":"2006-10-15T07:44:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This introduces a new configuration item, pack.deltaversion, to\ncontrol whether pack-objects is allowed to use version 3 delta.\nBy default, we keep generating version 2 delta (and version 2\npackfile format) to be compatible with git earlier than v1.2.0.\n\nThis configuration affects the command in the following ways:\n\n - the resulting packfile will have the specified version;\n\n - when generating delta, larger copies are allowed only when\n   deltaversion is 3;\n\n - the logic to reuse delta from existing packs refuses to reuse\n   delta from packs that uses delta version 3 when the\n   configuration is set to 2.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n---\n\n * Nico, I'd really appreciate it if you can eyeball this\n   patch.  For the upcoming v1.4.3, we'd at least want to\n   revert:\n\n     commit 16854571aae6302f457c5fbee41ac64669b09595\n     Author: Nicolas Pitre <nico@cam.org>\n     Date:   Thu Sep 21 00:11:59 2006 -0400\n\n     move pack creation to version 3\n\n     It's been quite a while now that GIT is able to read version 3 packs.\n     Let's create them at last.\n\n     Signed-off-by: Nicolas Pitre <nico@cam.org>\n     Signed-off-by: Junio C Hamano <junkio@cox.net>\n\n   but I'd like to allow people to explicitly tell what version\n   is to be created by default per repository to avoid future\n   problems.\n\n   For now it would be advised to leave this configuration empty\n   in public repositories, which would cause them to have\n   version 2 packs.  People who know their repositories are only\n   used with git v1.2.0 or newer can use the configuration to\n   allow version 3 packs.\n\n   We should later add a command line override to pack-objects\n   to explicitly say which delta version to be used, so that\n   pack protocol can negotiate the allowed delta version between\n   client and the server and pass that command line option when\n   upload-pack runs pack-objects.\n\n builtin-pack-objects.c |   16 ++++++++++++----\n cache.h                |    1 +\n delta.h                |    2 ++\n diff-delta.c           |   15 +++++++++++++--\n environment.c          |    3 +++\n sha1_file.c            |    1 +\n 6 files changed, 32 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 96c069a..4d2147b 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -456,7 +456,7 @@ static void write_pack_file(void)\n \t\tfprintf(stderr, \"Writing %d objects.\\n\", nr_result);\n \n \thdr.hdr_signature = htonl(PACK_SIGNATURE);\n-\thdr.hdr_version = htonl(PACK_VERSION);\n+\thdr.hdr_version = htonl(delta_version);\n \thdr.hdr_entries = htonl(nr_result);\n \tsha1write(f, &hdr, sizeof(hdr));\n \toffset = sizeof(hdr);\n@@ -914,12 +914,15 @@ static void check_object(struct object_e\n \t\t/* Check if it is delta, and the base is also an object\n \t\t * we are going to pack.  If so we will reuse the existing\n \t\t * delta.\n+\t\t *\n+\t\t * Also make sure that we do not reuse delta from an existing\n+\t\t * pack that uses higher delta version than allowed.\n \t\t */\n \t\tif (!no_reuse_delta &&\n \t\t    entry->in_pack_type == OBJ_DELTA &&\n \t\t    (base_entry = locate_object_entry(base)) &&\n-\t\t    (!base_entry->preferred_base)) {\n-\n+\t\t    (!base_entry->preferred_base) &&\n+\t\t    entry->in_pack->pack_version <= delta_version) {\n \t\t\t/* Depth value does not matter - find_deltas()\n \t\t\t * will never consider reused delta as the\n \t\t\t * base object to deltify other objects\n@@ -1326,10 +1329,15 @@ static void setup_progress_signal(void)\n \n static int git_pack_config(const char *k, const char *v)\n {\n-\tif(!strcmp(k, \"pack.window\")) {\n+\tif (!strcmp(k, \"pack.window\")) {\n \t\twindow = git_config_int(k, v);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.deltaversion\")) {\n+\t\tdelta_version = git_config_int(k, v);\n+\t\tif (!pack_version_ok(htonl(delta_version)))\n+\t\t\tdie(\"value %s for '%s' not allowed\", v, k);\n+\t}\n \treturn git_default_config(k, v);\n }\n \ndiff --git a/cache.h b/cache.h\nindex c354701..724c09a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -337,6 +337,7 @@ extern struct packed_git {\n \tunsigned int pack_last_used;\n \tunsigned int pack_use_cnt;\n \tint pack_local;\n+\tint pack_version;\n \tunsigned char sha1[20];\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\n \tchar pack_name[FLEX_ARRAY]; /* more */\ndiff --git a/delta.h b/delta.h\nindex 7b3f86d..55af3d2 100644\n--- a/delta.h\n+++ b/delta.h\n@@ -1,6 +1,8 @@\n #ifndef DELTA_H\n #define DELTA_H\n \n+extern int delta_version;\n+\n /* opaque object for delta index */\n struct delta_index;\n \ndiff --git a/diff-delta.c b/diff-delta.c\nindex fa16d06..2f6dcfb 100644\n--- a/diff-delta.c\n+++ b/diff-delta.c\n@@ -253,10 +253,13 @@ create_delta(const struct delta_index *i\n \tint inscnt;\n \tconst unsigned char *ref_data, *ref_top, *data, *top;\n \tunsigned char *out;\n+\tunsigned int ref_size_limit;\n \n \tif (!trg_buf || !trg_size)\n \t\treturn NULL;\n \n+\tref_size_limit = (delta_version > 2) ? 0xffffff : 0x10000;\n+\n \toutpos = 0;\n \toutsize = 8192;\n \tif (max_size && outsize >= max_size)\n@@ -308,8 +311,8 @@ create_delta(const struct delta_index *i\n \t\t\t\tcontinue;\n \t\t\tif (ref_size > top - src)\n \t\t\t\tref_size = top - src;\n-\t\t\tif (ref_size > 0x10000)\n-\t\t\t\tref_size = 0x10000;\n+\t\t\tif (ref_size > ref_size_limit)\n+\t\t\t\tref_size = ref_size_limit;\n \t\t\tif (ref_size <= msize)\n \t\t\t\tbreak;\n \t\t\twhile (ref_size-- && *src++ == *ref)\n@@ -318,6 +321,8 @@ create_delta(const struct delta_index *i\n \t\t\t\t/* this is our best match so far */\n \t\t\t\tmsize = ref - entry->ptr;\n \t\t\t\tmoff = entry->ptr - ref_data;\n+\t\t\t\tif (delta_version > 2 && msize >= 0x10000)\n+\t\t\t\t\tbreak; /* this is good enough */\n \t\t\t}\n \t\t}\n \n@@ -381,6 +386,12 @@ create_delta(const struct delta_index *i\n \t\t\tif (msize & 0xff) { out[outpos++] = msize; i |= 0x10; }\n \t\t\tmsize >>= 8;\n \t\t\tif (msize & 0xff) { out[outpos++] = msize; i |= 0x20; }\n+\t\t\tif (delta_version > 2) {\n+\t\t\t\tmsize >>= 8;\n+\t\t\t\tif (msize & 0xff) {\n+\t\t\t\t\tout[outpos++] = msize; i |= 0x40;\n+\t\t\t\t}\n+\t\t\t}\n \n \t\t\t*op = i;\n \t\t}\ndiff --git a/environment.c b/environment.c\nindex 63b1d15..e266f83 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -8,6 +8,7 @@\n  * are.\n  */\n #include \"cache.h\"\n+#include \"pack.h\"\n \n char git_default_email[MAX_GITNAME];\n char git_default_name[MAX_GITNAME];\n@@ -25,6 +26,8 @@ const char *apply_default_whitespace;\n int zlib_compression_level = Z_DEFAULT_COMPRESSION;\n int pager_in_use;\n int pager_use_color = 1;\n+/* by default we allow 2 but up to PACK_VERSION is allowed */\n+int delta_version = 2;\n \n static const char *git_dir;\n static char *git_object_dir, *git_index_file, *git_refs_dir, *git_graft_file;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d111be7..6653182 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -527,6 +527,7 @@ int use_packed_git(struct packed_git *p)\n \t\t\t    p->pack_size - 20)) {\n \t\t\tdie(\"packfile %s does not match index.\", p->pack_name);\n \t\t}\n+\t\tp->pack_version = ntohl(hdr->hdr_version);\n \t}\n \tp->pack_last_used = pack_used_ctr++;\n \tp->pack_use_cnt++;\n-- \n1.4.3.rc3.ga5ce\n"},{"id":"28793","messageId":"egsts6$a1o$1@sea.gmane.org","threadId":"5926","inReplyTo":"7virim10rb.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-10-15T09:09:46Z","receivedAt":"2006-10-15T09:09:46Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> This introduces a new configuration item, pack.deltaversion, to\n> control whether pack-objects is allowed to use version 3 delta.\n\nDocumentation, please?\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"28801","messageId":"Pine.LNX.4.64.0610151133450.17085@xanadu.home","threadId":"5926","inReplyTo":"7v4pu62ite.fsf@assigned-by-dhcp.cox.net","subject":"Re: Recent and near future backward incompatibilities","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-15T15:34:20Z","receivedAt":"2006-10-15T15:34:20Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 14 Oct 2006, Junio C Hamano wrote:\n\n> It was brought to my attention that the public git.git\n> repository cannot be cloned with older versions of git.  More\n> precisely, packs generated with post 16854571 (NOT contained in\n> v1.4.2.3 but in the current \"master\" and more importantly in\n> v1.4.3-rc3 which I tagged tonight) can contain deltas that are\n> not compatible with the version of git before d60fc1c8, which\n> means that v1.1.6 and older (v1.2.0 and later are Ok).\n\nAhhhhhh.  DAMN !\n\n> One thing we can and should immediately do is to revert 16854571\n> for now until we decide how to resolve this issue cleanly.\n> \n> These are what needs to happen but one of them is quite tricky:\n> \n>  - the reusing of delta is what makes pack-objects practical,\n>    and it is expensive to look into existing delta to see if it\n>    is version 2 or version 3 before deciding to reuse each delta\n>    data, so even if we update pack-objects so that we can tell\n>    it to generate a pack that contains only version 2 deltas, it\n>    would be very expensive to do so and may not be practical.\n\nWhy not?  After all users of GIT versions that don't understand pack \nversion 3 should be a very small minority by now.\n\n>    am not sure how to resolve this issue efficiently right now;\n>    we need a bit of thinking.\n\nActually it doesn't have to be that expensive to convert deltas v2 to \ndeltas v3 on the fly.  They can be inflated, parsed, the copy ops that \nexceed 0x10000 converted into multiple ops of smaller copy blocks, then \ndeflated.  This is certainly much less costly than rematching deltas \nfrom scratch.\n\nWell I'd say you just revert pack v3 generation patch for now and \nrelease v1.4.3 without it.  Pack v3 generation can wait a bit longer \nuntil we implement the above or users of GIT that can read packs v2 only \nare so few that we shouldn't care anymore and tell them to use an \nintermediate version of GIT in order to clone the latest.  It is not \nlike if that makes such a big difference on pack size anyway (much less \nthan delta with offsets to base actually).\n\n>  - we need to add .git/config item that tells pack-objects to\n>    never generate version 3 delta for that particular\n>    repository.  This is similar to the way we would need to\n>    control the use of delta-base-offset representation currently\n>    cooking in \"next\".\n\nThis is different. The delta-base-offset representation is decided at \nrun time every time a pack is generated and regardless if delta data is \nbeing reused from another pack or regenerated afresh, and so with no \ncost.  So this is no issue for users of old GIT versions since the \nnative GIT protocol already handle it in a backward compatible manner.\n\nThe only issue here concerns users that don't use the native GIT \nprotocol.  But in this case they have two options: either they switch to \nthe native protocol, or they upgrade to the latest GIT version which \ncan always be pulled with the native GIT protocol.\n\n> We may have a similar issue when enabling generation of loose\n> objects with new style headers.  This is already controlled with\n> the core.legacyheaders configuration item.\n\nSure, but those are never passed through the native GIT protocol which \nmakes it a much less critical issue.\n\nNot being able to upgrade to the latest GIT in order to actually cope \nwith the new format because the primary GIT repository started to feed \nold GIT versions with that new format unconditionally really is a \nproblem though.\n\nI think we should not bend backward too much with repository \ncompatibility issues as long as there is no interoperability issues at \nthe protocol level.  But the GIT protocol must always remain \ninteroperable with whatever GIT version still in use.\n\n\nNicolas\n"},{"id":"28803","messageId":"Pine.LNX.4.64.0610151135110.17085@xanadu.home","threadId":"5926","inReplyTo":"7virim10rb.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-15T15:53:35Z","receivedAt":"2006-10-15T15:53:35Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 15 Oct 2006, Junio C Hamano wrote:\n\n> This introduces a new configuration item, pack.deltaversion, to\n> control whether pack-objects is allowed to use version 3 delta.\n> By default, we keep generating version 2 delta (and version 2\n> packfile format) to be compatible with git earlier than v1.2.0.\n> \n> This configuration affects the command in the following ways:\n> \n>  - the resulting packfile will have the specified version;\n> \n>  - when generating delta, larger copies are allowed only when\n>    deltaversion is 3;\n> \n>  - the logic to reuse delta from existing packs refuses to reuse\n>    delta from packs that uses delta version 3 when the\n>    configuration is set to 2.\n> \n> Signed-off-by: Junio C Hamano <junkio@cox.net>\n\nI'd suggest to drop this altogether.  See my previous email for my \nreasoning on this issue.  I think this should be done another way.\n\nIf anything, maybe this patch can be added before v1.4.3 is released:\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7d23a80..1688417 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -165,9 +165,10 @@ static int find_common(int fd[2], unsign\n \t\t\tcontinue;\n \t\t}\n \n-\t\tpacket_write(fd[1], \"want %s%s%s\\n\", sha1_to_hex(remote),\n+\t\tpacket_write(fd[1], \"want %s%s%s%s\\n\", sha1_to_hex(remote),\n \t\t\t     (multi_ack ? \" multi_ack\" : \"\"),\n-\t\t\t     (use_thin_pack ? \" thin-pack\" : \"\"));\n+\t\t\t     (use_thin_pack ? \" thin-pack\" : \"\"),\n+\t\t\t     \" packv3\");\n \t\tfetching++;\n \t}\n \tpacket_flush(fd[1]);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 979e583..8e57316 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -218,7 +218,7 @@ static int receive_needs(void)\n \n static int send_ref(const char *refname, const unsigned char *sha1)\n {\n-\tstatic char *capabilities = \"multi_ack thin-pack\";\n+\tstatic char *capabilities = \"multi_ack thin-pack packv3\";\n \tstruct object *o = parse_object(sha1);\n \n \tif (!o)\n\nThis way pack v3 could be fed to GIT v1.4.3 and above whenever we add \nback pack v3 generation, and a pack converted to v2 from any v3 on the \nfly when that capability is not present.\n\n\nNicolas\n"},{"id":"28805","messageId":"7vac3xzbze.fsf@assigned-by-dhcp.cox.net","threadId":"5926","inReplyTo":"Pine.LNX.4.64.0610151135110.17085@xanadu.home","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-15T18:10:29Z","receivedAt":"2006-10-15T18:10:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre <nico@cam.org> writes:\n\n> On Sun, 15 Oct 2006, Junio C Hamano wrote:\n>\n>> This introduces a new configuration item, pack.deltaversion, to\n>> control whether pack-objects is allowed to use version 3 delta.\n>> By default, we keep generating version 2 delta (and version 2\n>> packfile format) to be compatible with git earlier than v1.2.0.\n>...\n> I'd suggest to drop this altogether.  See my previous email for my \n> reasoning on this issue.  I think this should be done another way.\n\nI'll think about it a bit.\n\n> If anything, maybe this patch can be added before v1.4.3 is released:\n>...\n> This way pack v3 could be fed to GIT v1.4.3 and above whenever we add \n> back pack v3 generation, and a pack converted to v2 from any v3 on the \n> fly when that capability is not present.\n\nI think that is sensible.  I also was thinking that we should\ncall the current one packv3 and the one with delta-base-offset\npackv4.\n"},{"id":"28806","messageId":"7v64elzbtg.fsf@assigned-by-dhcp.cox.net","threadId":"5926","inReplyTo":"Pine.LNX.4.64.0610151133450.17085@xanadu.home","subject":"Re: Recent and near future backward incompatibilities","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-15T18:14:03Z","receivedAt":"2006-10-15T18:14:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre <nico@cam.org> writes:\n\n> Actually it doesn't have to be that expensive to convert deltas v2 to \n> deltas v3 on the fly.  They can be inflated, parsed, the copy ops that \n> exceed 0x10000 converted into multiple ops of smaller copy blocks, then \n> deflated.  This is certainly much less costly than rematching deltas \n> from scratch.\n\nTrue, when I think about it.\n\n>>  - we need to add .git/config item that tells pack-objects to\n>>    never generate version 3 delta for that particular\n>>    repository.  This is similar to the way we would need to\n>>    control the use of delta-base-offset representation currently\n>>    cooking in \"next\".\n>\n> This is different. The delta-base-offset representation is decided at \n> run time every time a pack is generated and regardless if delta data is \n> being reused from another pack or regenerated afresh, and so with no \n> cost.  So this is no issue for users of old GIT versions since the \n> native GIT protocol already handle it in a backward compatible manner.\n>\n> The only issue here concerns users that don't use the native GIT \n> protocol.  But in this case they have two options: either they switch to \n> the native protocol, or they upgrade to the latest GIT version which \n> can always be pulled with the native GIT protocol.\n\nTrue again.  Thanks.\n"},{"id":"28807","messageId":"egtu1r$813$1@sea.gmane.org","threadId":"5926","inReplyTo":"7vac3xzbze.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-10-15T18:18:57Z","receivedAt":"2006-10-15T18:18:57Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> I think that is sensible.  I also was thinking that we should\n> call the current one packv3 and the one with delta-base-offset\n> packv4.\n\nJust curious: what was the difference between packv1 and packv2,\nand packv3 and packv4?\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"28809","messageId":"Pine.LNX.4.64.0610151422510.17085@xanadu.home","threadId":"5926","inReplyTo":"7vac3xzbze.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-15T18:30:46Z","receivedAt":"2006-10-15T18:30:46Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 15 Oct 2006, Junio C Hamano wrote:\n\n> Nicolas Pitre <nico@cam.org> writes:\n> \n> > If anything, maybe this patch can be added before v1.4.3 is released:\n> >...\n> > This way pack v3 could be fed to GIT v1.4.3 and above whenever we add \n> > back pack v3 generation, and a pack converted to v2 from any v3 on the \n> > fly when that capability is not present.\n> \n> I think that is sensible.  I also was thinking that we should\n> call the current one packv3 and the one with delta-base-offset\n> packv4.\n\nI think we should not.  The pack version should be tied to incompatible \npack data to prevent older GIT versions from misinterpreting newer \npacks.  The delta block copy encoding is a perfect example of that where \na bit changed meaning.\n\nThe delta-base-offset case included a new object type that wasn't used \nbefore hence there is no room for confusion, and yet that new delta \nobject could be encoded according to pack version 2 or pack version 3 \nwhich makes it orthogonal to the pack version itself.\n\n\nNicolas\n"},{"id":"28811","messageId":"Pine.LNX.4.64.0610151433310.17085@xanadu.home","threadId":"5926","inReplyTo":"egtu1r$813$1@sea.gmane.org","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-15T18:51:08Z","receivedAt":"2006-10-15T18:51:08Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 15 Oct 2006, Jakub Narebski wrote:\n\n> Junio C Hamano wrote:\n> \n> > I think that is sensible.  I also was thinking that we should\n> > call the current one packv3 and the one with delta-base-offset\n> > packv4.\n> \n> Just curious: what was the difference between packv1 and packv2,\n> and packv3 and packv4?\n\nPack v1 was really short-lived (one day or two).  It used a different \nencoding for object size and delta size than what exists today.  When \nthe current encoding was adopted the pack version was bumped to 2 to \nmake sure anyone, if any, who might have started to rely upon packs in \nthose early days would not end up trying to use incompatible pack data.  \nBackward compatibility was not a concern at all back then of course.  \nSo for all practical purposes just consider that pack version 1 never \nexisted.\n\nPack version 3 simply redefined one bit in the delta encoding that was \nnever used.  The former definition of the bit was implemented in the \ndecode part, but attempts to use it in the encode part turned up to be \nway too costly for really really poor benefits.  for details just have a \nlook at commit d60fc1c8649f80c006b9f493c542461e81608d4b.\n\nAs for pack v4... My opinion is that nothing justifies it so far.  So if \nI can convince Junio there shouldn't be any v4 just yet.\n\n\nNicolas\n"},{"id":"28812","messageId":"Pine.LNX.4.64.0610151150530.3952@g5.osdl.org","threadId":"5926","inReplyTo":"7vac3xzbze.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-10-15T18:57:44Z","receivedAt":"2006-10-15T18:57:44Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sun, 15 Oct 2006, Junio C Hamano wrote:\n> \n> I think that is sensible.  I also was thinking that we should\n> call the current one packv3 and the one with delta-base-offset\n> packv4.\n\nQuite frankly, I wonder if the pure \"copy size extension\" (aka \"v3\") thing \nis really worth it at all. \n\nI mean, seriously, how much does it buy us? A couple of bytes per every \n64kB of delta copied? And the downside is that you can't re-use the deltas \nwith old clients and/or you have to re-create a \"v2\" delta at run-time \nfrom a v3 delta by inflating, fixing and deflating it.\n\nSo I would suggest:\n\n - call the delta-base-offset thing the \"v3\" pack format.\n\n - forget about the current \"v3 delta\" entirely. We might as well continue \n   to support reading it, but there's no point in actually ever generating \n   it. \n\nIn other words, I think the current situation in top-of-master is the \nright situation. There's simply no point in adding code to convert v3 to \nv2 on the fly - even if it's not rocket science, it's just not _worth_ it.\n\n(You could also have the extended copy deltas in v3-only, and only send it \nto clients that you know supports it. However, the \"convert to v2\" format \nissue still rears its ugly head, and as a result I just don't think it's \n_ever_ worth it).\n\n\t\tLinus\n"},{"id":"28814","messageId":"45328C19.8070109@gmail.com","threadId":"5926","inReplyTo":"7vac3xzbze.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-10-15T19:29:29Z","receivedAt":"2006-10-15T19:29:29Z","isPatch":true,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Junio C Hamano wrote:\n[...]\n> \n> I think that is sensible.  I also was thinking that we should\n> call the current one packv3 and the one with delta-base-offset\n> packv4.\n\n+1\n"},{"id":"28816","messageId":"45329359.1030302@gmail.com","threadId":"5926","inReplyTo":"Pine.LNX.4.64.0610151422510.17085@xanadu.home","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-10-15T20:00:25Z","receivedAt":"2006-10-15T20:00:25Z","isPatch":true,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Nicolas Pitre wrote:\n> On Sun, 15 Oct 2006, Junio C Hamano wrote:\n> \n>> Nicolas Pitre <nico@cam.org> writes:\n>>\n>>> If anything, maybe this patch can be added before v1.4.3 is released:\n>>> ...\n>>> This way pack v3 could be fed to GIT v1.4.3 and above whenever we add \n>>> back pack v3 generation, and a pack converted to v2 from any v3 on the \n>>> fly when that capability is not present.\n>> I think that is sensible.  I also was thinking that we should\n>> call the current one packv3 and the one with delta-base-offset\n>> packv4.\n> \n> I think we should not.  The pack version should be tied to incompatible \n> pack data to prevent older GIT versions from misinterpreting newer \n> packs.  The delta block copy encoding is a perfect example of that where \n> a bit changed meaning.\n> \n> The delta-base-offset case included a new object type that wasn't used \n> before hence there is no room for confusion, and yet that new delta \n> object could be encoded according to pack version 2 or pack version 3 \n> which makes it orthogonal to the pack version itself.\n\nIt's not a new object type. It's a new object _encoding_ method.\n"},{"id":"28820","messageId":"20061015224012.GC5092@thunk.org","threadId":"5926","inReplyTo":"7v4pu62ite.fsf@assigned-by-dhcp.cox.net","subject":"Re: Recent and near future backward incompatibilities","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2006-10-15T22:40:12Z","receivedAt":"2006-10-15T22:40:12Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Sat, Oct 14, 2006 at 11:29:17PM -0700, Junio C Hamano wrote:\n> It was brought to my attention that the public git.git\n> repository cannot be cloned with older versions of git.  More\n> precisely, packs generated with post 16854571 (NOT contained in\n> v1.4.2.3 but in the current \"master\" and more importantly in\n> v1.4.3-rc3 which I tagged tonight) can contain deltas that are\n> not compatible with the version of git before d60fc1c8, which\n> means that v1.1.6 and older (v1.2.0 and later are Ok).\n\nBy the way, note that Ubuntu Dapper (the current stable version of\nUbuntu) is shipped with git version 1.1.3, and that incompatibility\nextends not to the git repository, but also the Linux-2.6 repostiory\nat\n\ngit://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git\n\n\t\t\t\t\t\t- Ted\n"},{"id":"28821","messageId":"Pine.LNX.4.64.0610151650570.3962@g5.osdl.org","threadId":"5926","inReplyTo":"20061015224012.GC5092@thunk.org","subject":"Re: Recent and near future backward incompatibilities","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-10-15T23:52:04Z","receivedAt":"2006-10-15T23:52:04Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sun, 15 Oct 2006, Theodore Tso wrote:\n> \n> By the way, note that Ubuntu Dapper (the current stable version of\n> Ubuntu) is shipped with git version 1.1.3, and that incompatibility\n> extends not to the git repository, but also the Linux-2.6 repostiory\n> at\n> \n> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git\n\nThat should have been fixed earlier today, since I forced a repack when \nthis was discovered.\n\nSo if somebody still can't pull it with old git tools, please holler.\n\nThat said, Ubuntu should definitely upgrade.\n\n\t\tLinus\n"},{"id":"28828","messageId":"20061015191305.28f71cc2@dads-laptop","threadId":"5926","inReplyTo":"Pine.LNX.4.64.0610151650570.3962@g5.osdl.org","subject":"Re: Recent and near future backward incompatibilities","fromName":"Stephen Hemminger","fromEmail":"shemminger@osdl.org","sentAt":"2006-10-16T02:13:05Z","receivedAt":"2006-10-16T02:13:05Z","isPatch":false,"sender":{"key":"shemminger@osdl.org","avatar":null},"body":"On Sun, 15 Oct 2006 16:52:04 -0700 (PDT)\nLinus Torvalds <torvalds@osdl.org> wrote:\n\n> \n> \n> On Sun, 15 Oct 2006, Theodore Tso wrote:\n> > \n> > By the way, note that Ubuntu Dapper (the current stable version of\n> > Ubuntu) is shipped with git version 1.1.3, and that incompatibility\n> > extends not to the git repository, but also the Linux-2.6 repostiory\n> > at\n> > \n> > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git\n> \n> That should have been fixed earlier today, since I forced a repack when \n> this was discovered.\n> \n> So if somebody still can't pull it with old git tools, please holler.\n> \n> That said, Ubuntu should definitely upgrade.\n> \n> \t\tLinus\n\nThey made some nice about putting a new version in the backport\nrepository.\n"},{"id":"28830","messageId":"Pine.LNX.4.64.0610152247270.17085@xanadu.home","threadId":"5926","inReplyTo":"45329359.1030302@gmail.com","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-16T02:52:57Z","receivedAt":"2006-10-16T02:52:57Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 15 Oct 2006, A Large Angry SCM wrote:\n\n> Nicolas Pitre wrote:\n> > \n> > The delta-base-offset case included a new object type that wasn't used\n> > before hence there is no room for confusion, and yet that new delta object\n> > could be encoded according to pack version 2 or pack version 3 which makes\n> > it orthogonal to the pack version itself.\n> \n> It's not a new object type. It's a new object _encoding_ method.\n\nNot at all.  If it was only a question of encoding method, then both of \nthem could always be interchangeable and supersede the other, which is \nnot the case here.\n\n\nNicolas\n"},{"id":"28835","messageId":"7v64ekyikn.fsf@assigned-by-dhcp.cox.net","threadId":"5926","inReplyTo":"Pine.LNX.4.64.0610151433310.17085@xanadu.home","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-16T04:45:44Z","receivedAt":"2006-10-16T04:45:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre <nico@cam.org> writes:\n\n> As for pack v4... My opinion is that nothing justifies it so far.  So if \n> I can convince Junio there shouldn't be any v4 just yet.\n\nThe only concern I have is the commit walkers (rsync has the\nsame problem as well but we honestly do not care).  They just\ngrab existing packs and try to use them.  I have been wondering\nif it might be safer to mark the delta-base-offset encoded packs\nv4 to make sure the clients would get \"I know only v2 and v3 but\nyou fed me v4\" message.\n\nPeople who own public repositories and who care about commit\nwalkers need to refrain from using delta-base-offset when\nrepacking (which can be done via configuration mechanism\nalready).\n\nIf download goes over git native protocol, there is no such\nworry -- they can pack using delta-base-offset and older clients\nwill be fed a pack with 20-byte base object name just fine, so my\nworry is really only about commit walkers.\n"},{"id":"28842","messageId":"Pine.LNX.4.64.0610160925580.17085@xanadu.home","threadId":"5926","inReplyTo":"7v64ekyikn.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-16T13:27:54Z","receivedAt":"2006-10-16T13:27:54Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 15 Oct 2006, Junio C Hamano wrote:\n\n> Nicolas Pitre <nico@cam.org> writes:\n> \n> > As for pack v4... My opinion is that nothing justifies it so far.  So if \n> > I can convince Junio there shouldn't be any v4 just yet.\n> \n> The only concern I have is the commit walkers (rsync has the\n> same problem as well but we honestly do not care).  They just\n> grab existing packs and try to use them.  I have been wondering\n> if it might be safer to mark the delta-base-offset encoded packs\n> v4 to make sure the clients would get \"I know only v2 and v3 but\n> you fed me v4\" message.\n\nIt'll get \"this pack contains an unknown object type\" kind of message, \nwhich is almost as good IMHO, with the same end result.\n\n\nNicolas\n"},{"id":"28844","messageId":"Pine.LNX.4.64.0610160929450.17085@xanadu.home","threadId":"5926","inReplyTo":"Pine.LNX.4.64.0610151150530.3952@g5.osdl.org","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-16T13:43:04Z","receivedAt":"2006-10-16T13:43:04Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 15 Oct 2006, Linus Torvalds wrote:\n\n> \n> \n> On Sun, 15 Oct 2006, Junio C Hamano wrote:\n> > \n> > I think that is sensible.  I also was thinking that we should\n> > call the current one packv3 and the one with delta-base-offset\n> > packv4.\n> \n> Quite frankly, I wonder if the pure \"copy size extension\" (aka \"v3\") thing \n> is really worth it at all. \n> \n> I mean, seriously, how much does it buy us? A couple of bytes per every \n> 64kB of delta copied? And the downside is that you can't re-use the deltas \n> with old clients and/or you have to re-create a \"v2\" delta at run-time \n> from a v3 delta by inflating, fixing and deflating it.\n\nRight.  This is why I suggested Junio to just drop it for now.  Let's \njust wait some more until this is just not an issue any longer, say in a \nyear from now when all major distributions have switched to a GIT \nversion that can read V3.\n\nIf until then we find the saving really worth the backward compatibility \nv3-to-v2 conversion then we could reconsider.  But I don't think it is \nworth it just yet.\n\nIn the mean time, if Junio adds the patch I posted yesterday advertising \nthe pack version capability over the native protocol then it'll help us \nmake things forward compatible if ever we decide to go with generating \npacks v3 sooner.\n\n\nNicolas\n"},{"id":"28975","messageId":"7vodsakjkg.fsf@assigned-by-dhcp.cox.net","threadId":"5926","inReplyTo":"Pine.LNX.4.64.0610160929450.17085@xanadu.home","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-17T16:12:31Z","receivedAt":"2006-10-17T16:12:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre <nico@cam.org> writes:\n\n> On Sun, 15 Oct 2006, Linus Torvalds wrote:\n>\n>> Quite frankly, I wonder if the pure \"copy size extension\" (aka \"v3\") thing \n>> is really worth it at all. \n>> \n>> I mean, seriously, how much does it buy us? A couple of bytes per every \n>> 64kB of delta copied? And the downside is that you can't re-use the deltas \n>> with old clients and/or you have to re-create a \"v2\" delta at run-time \n>> from a v3 delta by inflating, fixing and deflating it.\n>\n>...\n> In the mean time, if Junio adds the patch I posted yesterday advertising \n> the pack version capability over the native protocol then it'll help us \n> make things forward compatible if ever we decide to go with generating \n> packs v3 sooner.\n\nI've thought about this, but we hopefully would have ofs-delta\ncapability exchanged soon after 1.4.3, and that would be an\nenough advertisement that the client is recent enough; although\nit is technically incorrect to tie these two independent\nfeatures together, the improvement between v2 and v3 is dubious\nso maybe that is the easiest.\n"},{"id":"28977","messageId":"Pine.LNX.4.64.0610171250530.1971@xanadu.home","threadId":"5926","inReplyTo":"7vodsakjkg.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] pack-objects: use of version 3 delta is now optional.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-17T16:51:12Z","receivedAt":"2006-10-17T16:51:12Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 17 Oct 2006, Junio C Hamano wrote:\n\n> Nicolas Pitre <nico@cam.org> writes:\n> \n> > On Sun, 15 Oct 2006, Linus Torvalds wrote:\n> >\n> >> Quite frankly, I wonder if the pure \"copy size extension\" (aka \"v3\") thing \n> >> is really worth it at all. \n> >> \n> >> I mean, seriously, how much does it buy us? A couple of bytes per every \n> >> 64kB of delta copied? And the downside is that you can't re-use the deltas \n> >> with old clients and/or you have to re-create a \"v2\" delta at run-time \n> >> from a v3 delta by inflating, fixing and deflating it.\n> >\n> >...\n> > In the mean time, if Junio adds the patch I posted yesterday advertising \n> > the pack version capability over the native protocol then it'll help us \n> > make things forward compatible if ever we decide to go with generating \n> > packs v3 sooner.\n> \n> I've thought about this, but we hopefully would have ofs-delta\n> capability exchanged soon after 1.4.3, and that would be an\n> enough advertisement that the client is recent enough; although\n> it is technically incorrect to tie these two independent\n> features together, the improvement between v2 and v3 is dubious\n> so maybe that is the easiest.\n\nFair enough.\n\n\nNicolas\n"}]}