{"thread":{"id":"18886","subject":"Weird growth in packfile during initial push","startedAt":"2009-04-15T18:27:54Z","lastAt":"2009-05-04T22:30:05Z","messageCount":16,"participants":["Robin H. Johnson","Nicolas Pitre","Junio C Hamano","Shawn O. Pearce","A Large Angry SCM","Michael Witten","Ealdwulf Wuffinga"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"111391","messageId":"20090415182754.GF23644@curie-int","threadId":"18886","inReplyTo":null,"subject":"Weird growth in packfile during initial push","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2009-04-15T18:27:54Z","receivedAt":"2009-04-15T18:27:54Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"I was doing a more recent conversion of the Gentoo repo, and ran into\nsome odd behavior in the packfile size.\n\nFor anybody else following the repo, you can now get it on the new hardware at:\nhttp://git-exp.overlays.gentoo.org/gitweb/?p=exp/gentoo-x86.git;a=summary\n\nI did the conversion with cvs2svn, packed, added the remote and pushed, only to\nfind that the pack on the remote side suddenly seemed to be ~60MiB larger.\n\n$ time git repack -adf --window=250 --depth=250\nreal 19m59.339s\nuser 96m48.011s\nsys 0m36.914s\n\n$ ls -la /tmp/convert/gentoo-x86-cvs2git/.git/objects/pack\ntotal 903804\ndrwxr-xr-x 2 robbat2 users 119 Apr 14 08:05 .\ndrwxr-xr-x 4 robbat2 users 28 Apr 14 08:05 ..\n-r--r--r-- 1 robbat2 users 139155472 Apr 14 08:05 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.idx\n-r--r--r-- 1 robbat2 users 786336481 Apr 14 08:05 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.pack\n\n$ git remote add origin git+ssh://git@git-exp.overlays.gentoo.org/exp/gentoo-x86.git\n$ git push origin master:master\nInitialized empty Git repository in /var/gitroot/exp/gentoo-x86.git/\nCounting objects: 4969800, done.\nDelta compression using up to 8 threads.\nCompressing objects: 100% (1217809/1217809), done.\nWriting objects: 100% (4969800/4969800), 810.56 MiB | 21608 KiB/s, done.\nTotal 4969800 (delta 3735812), reused 4969800 (delta 3735812)\nTo git+ssh://git@git-exp.overlays.gentoo.org/exp/gentoo-x86.git\n * [new branch] master -> master\n\n$ ls -la /var/gitroot/exp/gentoo-x86.git/objects/pack\ntotal 966876\ndrwxr-xr-x 2 git git 4096 Apr 14 08:43 .\ndrwxr-xr-x 4 git git 4096 Apr 14 08:35 ..\n-r--r--r-- 1 git git 139155472 Apr 14 08:43 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.idx\n-r--r--r-- 1 git git 849936308 Apr 14 08:43 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.pack\n\nOn the client side after the initial clone, it DOES match (in size) what was\ncloned.\n\n(If you're looking for the 849MB one right now, I'll have to get it back for\nyou, I wanted to save that extra space so just did an rsync of the other pack\nover the too-large one for now).\n\n-- \nRobin Hugh Johnson\nGentoo Linux Developer & Infra Guy\nE-Mail : robbat2@gentoo.org\nGnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85\n"},{"id":"111394","messageId":"alpine.LFD.2.00.0904151443030.6741@xanadu.home","threadId":"18886","inReplyTo":"20090415182754.GF23644@curie-int","subject":"Re: Weird growth in packfile during initial push","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-04-15T19:51:40Z","receivedAt":"2009-04-15T19:51:40Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 15 Apr 2009, Robin H. Johnson wrote:\n\n> I was doing a more recent conversion of the Gentoo repo, and ran into\n> some odd behavior in the packfile size.\n> \n> For anybody else following the repo, you can now get it on the new hardware at:\n> http://git-exp.overlays.gentoo.org/gitweb/?p=exp/gentoo-x86.git;a=summary\n> \n> I did the conversion with cvs2svn, packed, added the remote and pushed, only to\n> find that the pack on the remote side suddenly seemed to be ~60MiB larger.\n\nHmmm.\n\n> $ ls -la /tmp/convert/gentoo-x86-cvs2git/.git/objects/pack\n> total 903804\n> drwxr-xr-x 2 robbat2 users 119 Apr 14 08:05 .\n> drwxr-xr-x 4 robbat2 users 28 Apr 14 08:05 ..\n> -r--r--r-- 1 robbat2 users 139155472 Apr 14 08:05 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.idx\n> -r--r--r-- 1 robbat2 users 786336481 Apr 14 08:05 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.pack\n> \n> $ git remote add origin git+ssh://git@git-exp.overlays.gentoo.org/exp/gentoo-x86.git\n> $ git push origin master:master\n> Initialized empty Git repository in /var/gitroot/exp/gentoo-x86.git/\n> Counting objects: 4969800, done.\n> Delta compression using up to 8 threads.\n> Compressing objects: 100% (1217809/1217809), done.\n> Writing objects: 100% (4969800/4969800), 810.56 MiB | 21608 KiB/s, done.\n> Total 4969800 (delta 3735812), reused 4969800 (delta 3735812)\n\nHere we know for sure that all objects were directly reused, so no \nattempt at recompressing them was done. The only thing that \npack-objects might do in this case in addition to directly streaming the \nexisting pack is to convert delta object headers from OFS_DELTA to \nREF_DELTA.\n\n> $ ls -la /var/gitroot/exp/gentoo-x86.git/objects/pack\n> total 966876\n> drwxr-xr-x 2 git git 4096 Apr 14 08:43 .\n> drwxr-xr-x 4 git git 4096 Apr 14 08:35 ..\n> -r--r--r-- 1 git git 139155472 Apr 14 08:43 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.idx\n> -r--r--r-- 1 git git 849936308 Apr 14 08:43 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.pack\n\nLet's see if my theory stands:\n\n\t849936308 - 786336481 = 63599827\n\t63599827 / 3735812 = 17.02\n\nHence an average difference of 17 bytes per delta. Given that REF_DELTA \nobjects have a 20-byte SHA1 base reference which is replaced with a \nvariable length encoding of a pack offset in the OFS_DELTA case, we're \ntalking about 2.98 bytes for that offset encoding which feels about \nright.\n\n[...]\n\nAnd the code matches this theory as well. Can you try this patch if you \nhave a chance?\n\ndiff --git a/builtin-send-pack.c b/builtin-send-pack.c\nindex 91c3651..e41adbf 100644\n--- a/builtin-send-pack.c\n+++ b/builtin-send-pack.c\n@@ -44,12 +44,16 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \t\t\"--stdout\",\n \t\tNULL,\n \t\tNULL,\n+\t\tNULL,\n \t};\n \tstruct child_process po;\n \tint i;\n \n+\ti = 4;\n \tif (args->use_thin_pack)\n-\t\targv[4] = \"--thin\";\n+\t\targv[i++] = \"--thin\";\n+\tif (args->use_ofs_delta)\n+\t\targv[i++] = \"--delta-base-offset\";\n \tmemset(&po, 0, sizeof(po));\n \tpo.argv = argv;\n \tpo.in = -1;\n@@ -316,6 +320,8 @@ int send_pack(struct send_pack_args *args,\n \t\task_for_status_report = 1;\n \tif (server_supports(\"delete-refs\"))\n \t\tallow_deleting_refs = 1;\n+\tif (server_supports(\"ofs-delta\"))\n+\t\targs->use_ofs_delta = 1;\n \n \tif (!remote_refs) {\n \t\tfprintf(stderr, \"No refs in common and none specified; doing nothing.\\n\"\ndiff --git a/send-pack.h b/send-pack.h\nindex 83d76c7..1d7b1b3 100644\n--- a/send-pack.h\n+++ b/send-pack.h\n@@ -6,6 +6,7 @@ struct send_pack_args {\n \t\tsend_mirror:1,\n \t\tforce_update:1,\n \t\tuse_thin_pack:1,\n+\t\tuse_ofs_delta:1,\n \t\tdry_run:1;\n };\n \n\n\nNicolas\n"},{"id":"112726","messageId":"7vy6tj109a.fsf@gitster.siamese.dyndns.org","threadId":"18886","inReplyTo":"alpine.LFD.2.00.0904151443030.6741@xanadu.home","subject":"Re: Weird growth in packfile during initial push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-04-29T23:57:37Z","receivedAt":"2009-04-29T23:57:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n>> $ git push origin master:master\n>> Initialized empty Git repository in /var/gitroot/exp/gentoo-x86.git/\n>> Counting objects: 4969800, done.\n>> Delta compression using up to 8 threads.\n>> Compressing objects: 100% (1217809/1217809), done.\n>> Writing objects: 100% (4969800/4969800), 810.56 MiB | 21608 KiB/s, done.\n>> Total 4969800 (delta 3735812), reused 4969800 (delta 3735812)\n>\n> Here we know for sure that all objects were directly reused, so no \n> attempt at recompressing them was done. The only thing that \n> pack-objects might do in this case in addition to directly streaming the \n> existing pack is to convert delta object headers from OFS_DELTA to \n> REF_DELTA.\n>\n>> $ ls -la /var/gitroot/exp/gentoo-x86.git/objects/pack\n>> total 966876\n>> drwxr-xr-x 2 git git 4096 Apr 14 08:43 .\n>> drwxr-xr-x 4 git git 4096 Apr 14 08:35 ..\n>> -r--r--r-- 1 git git 139155472 Apr 14 08:43 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.idx\n>> -r--r--r-- 1 git git 849936308 Apr 14 08:43 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.pack\n>\n> Let's see if my theory stands:\n>\n> \t849936308 - 786336481 = 63599827\n> \t63599827 / 3735812 = 17.02\n>\n> Hence an average difference of 17 bytes per delta. Given that REF_DELTA \n> objects have a 20-byte SHA1 base reference which is replaced with a \n> variable length encoding of a pack offset in the OFS_DELTA case, we're \n> talking about 2.98 bytes for that offset encoding which feels about \n> right.\n>\n> [...]\n>\n> And the code matches this theory as well. Can you try this patch if you \n> have a chance?\n\nIs there any progress on this?\n\nI think you did a veryclear analysis. 8% size reduction is not only\nunignorable but use of delta offset should also help runtime efficiency,\nright?\n"},{"id":"112732","messageId":"alpine.LFD.2.00.0904292251250.6741@xanadu.home","threadId":"18886","inReplyTo":"7vy6tj109a.fsf@gitster.siamese.dyndns.org","subject":"Re: Weird growth in packfile during initial push","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-04-30T02:52:29Z","receivedAt":"2009-04-30T02:52:29Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 29 Apr 2009, Junio C Hamano wrote:\n\n> Nicolas Pitre writes:\n> \n> > And the code matches this theory as well. Can you try this patch if you \n> > have a chance?\n> \n> Is there any progress on this?\n\nI'll try to find 5 min tomorrow to test the patch.\n\n\nNicolas\n"},{"id":"112802","messageId":"robbat2.20090501T061700.886743377Z@orbis-terrarum.net","threadId":"18886","inReplyTo":"7vy6tj109a.fsf@gitster.siamese.dyndns.org","subject":"Re: Weird growth in packfile during initial push","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2009-05-01T06:17:57Z","receivedAt":"2009-05-01T06:17:57Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"On Wed, Apr 29, 2009 at 04:57:37PM -0700, Junio C Hamano wrote:\n> > And the code matches this theory as well. Can you try this patch if you \n> > have a chance?\n> Is there any progress on this?\nSorry, I was just away for 2 weeks, only got back late yesterday. I'll\ntry to get to it in the next few days unless Nicolas beats me to it.\n\n-- \nRobin Hugh Johnson\nGentoo Linux Developer & Infra Guy\nE-Mail : robbat2@gentoo.org\nGnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85\n"},{"id":"112839","messageId":"alpine.LFD.2.00.0905011616130.6741@xanadu.home","threadId":"18886","inReplyTo":"7vy6tj109a.fsf@gitster.siamese.dyndns.org","subject":"[PATCH] allow OFS_DELTA objects during a push","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-01T20:56:47Z","receivedAt":"2009-05-01T20:56:47Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"The fetching of OFS_DELTA objects has been negotiated between both peers \nsince git version 1.4.4. However, this was missing from the push side \nwhere every OFS_DELTA objects were always converted to REF_DELTA objects \ncausing an increase in transferred data.\n\nTo fix this, both the client and the server processes have to be \nmodified: the former to invoke pack-objects with --delta-base-offset \nwhen the server provides the ofs-delta capability, and the later to send \nthat capability when OFS_DELTA objects are allowed as already indicated \nby the repack.usedeltabaseoffset config variable which is TRUE by \ndefault since git v1.6.0.\n\nSigned-off-by: Nicolas Pitre \n---\n\nOn Wed, 29 Apr 2009, Junio C Hamano wrote:\n\n> Nicolas Pitre writes:\n> \n> > Hence an average difference of 17 bytes per delta. Given that REF_DELTA \n> > objects have a 20-byte SHA1 base reference which is replaced with a \n> > variable length encoding of a pack offset in the OFS_DELTA case, we're \n> > talking about 2.98 bytes for that offset encoding which feels about \n> > right.\n> >\n> > [...]\n> >\n> > And the code matches this theory as well. Can you try this patch if you \n> > have a chance?\n> \n> Is there any progress on this?\n> \n> I think you did a veryclear analysis. 8% size reduction is not only\n> unignorable but use of delta offset should also help runtime efficiency,\n> right?\n\nIndeed.\n\nHere's the final patch. My initial one didn't work because the server \nside didn't advertise the needed capability. So both sides will have to \nbe updated for pushes with OFS_DELTA to kick in.\n\n builtin-receive-pack.c | 22 +++++++++++++++-------\n builtin-send-pack.c | 8 +++++++-\n send-pack.h | 1 +\n 3 files changed, 23 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin-receive-pack.c b/builtin-receive-pack.c\nindex a970b39..4b9d921 100644\n--- a/builtin-receive-pack.c\n+++ b/builtin-receive-pack.c\n@@ -27,10 +27,9 @@ static int receive_unpack_limit = -1;\n static int transfer_unpack_limit = -1;\n static int unpack_limit = 100;\n static int report_status;\n+static int prefer_ofs_delta = 1;\n static const char *head_name;\n-\n-static char capabilities[] = \" report-status delete-refs \";\n-static int capabilities_sent;\n+static char *capabilities_to_send;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n {\n@@ -84,24 +83,29 @@ static int receive_pack_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (strcmp(var, \"repack.usedeltabaseoffset\") == 0) {\n+\t\tprefer_ofs_delta = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(var, value, cb);\n }\n \n static int show_ref(const char *path, const unsigned char *sha1, int flag, void *cb_data)\n {\n-\tif (capabilities_sent)\n+\tif (!capabilities_to_send)\n \t\tpacket_write(1, \"%s %s\\n\", sha1_to_hex(sha1), path);\n \telse\n \t\tpacket_write(1, \"%s %s%c%s\\n\",\n-\t\t\t sha1_to_hex(sha1), path, 0, capabilities);\n-\tcapabilities_sent = 1;\n+\t\t\t sha1_to_hex(sha1), path, 0, capabilities_to_send);\n+\tcapabilities_to_send = NULL;\n \treturn 0;\n }\n \n static void write_head_info(void)\n {\n \tfor_each_ref(show_ref, NULL);\n-\tif (!capabilities_sent)\n+\tif (capabilities_to_send)\n \t\tshow_ref(\"capabilities^{}\", null_sha1, 0, NULL);\n \n }\n@@ -687,6 +691,10 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \telse if (0 <= receive_unpack_limit)\n \t\tunpack_limit = receive_unpack_limit;\n \n+\tcapabilities_to_send = (prefer_ofs_delta) ?\n+\t\t\" report-status delete-refs ofs-delta \" :\n+\t\t\" report-status delete-refs \";\n+\n \tadd_alternate_refs();\n \twrite_head_info();\n \tclear_extra_refs();\ndiff --git a/builtin-send-pack.c b/builtin-send-pack.c\nindex d5a1c48..473a3de 100644\n--- a/builtin-send-pack.c\n+++ b/builtin-send-pack.c\n@@ -43,12 +43,16 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \t\t\"--stdout\",\n \t\tNULL,\n \t\tNULL,\n+\t\tNULL,\n \t};\n \tstruct child_process po;\n \tint i;\n \n+\ti = 4;\n \tif (args->use_thin_pack)\n-\t\targv[4] = \"--thin\";\n+\t\targv[i++] = \"--thin\";\n+\tif (args->use_ofs_delta)\n+\t\targv[i++] = \"--delta-base-offset\";\n \tmemset(&po, 0, sizeof(po));\n \tpo.argv = argv;\n \tpo.in = -1;\n@@ -315,6 +319,8 @@ int send_pack(struct send_pack_args *args,\n \t\task_for_status_report = 1;\n \tif (server_supports(\"delete-refs\"))\n \t\tallow_deleting_refs = 1;\n+\tif (server_supports(\"ofs-delta\"))\n+\t\targs->use_ofs_delta = 1;\n \n \tif (!remote_refs) {\n \t\tfprintf(stderr, \"No refs in common and none specified; doing nothing.\\n\"\ndiff --git a/send-pack.h b/send-pack.h\nindex 83d76c7..1d7b1b3 100644\n--- a/send-pack.h\n+++ b/send-pack.h\n@@ -6,6 +6,7 @@ struct send_pack_args {\n \t\tsend_mirror:1,\n \t\tforce_update:1,\n \t\tuse_thin_pack:1,\n+\t\tuse_ofs_delta:1,\n \t\tdry_run:1;\n };\n \n"},{"id":"112851","messageId":"7v4ow4v0xl.fsf@gitster.siamese.dyndns.org","threadId":"18886","inReplyTo":"alpine.LFD.2.00.0905011616130.6741@xanadu.home","subject":"Re: [PATCH] allow OFS_DELTA objects during a push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-05-01T23:49:26Z","receivedAt":"2009-05-01T23:49:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks.\n\nThe code looks correct, I am reasonably sure updated server-client\ncombination would work fine, and use of the capability mechanism means\nother combinations like old pusher and new receiver, and/or new pusher and\nold receiver, should be also Ok.\n\nI see Shawn did the same to jgit.\n\nBut I'd like to queue this in 'next', and make it official after 1.6.3\nhappens.\n\nI just do not want to repeat silly mistakes, this close to the final,\nsimilar to the \"github needs to get stuck forever at 1.6.1\" we made with\n40c155f (push: prepare sender to receive extended ref information from the\nreceiver, 2008-09-09); it was done as a good change after a discussion\namong Shawn, Daniel and I. We managed to botch it and had to later fix\nwith 02322e1 (send-pack: do not send unknown object name from \".have\" to\npack-objects, 2009-01-27).\n\n(references)\n\n http://thread.gmane.org/gmane.comp.version-control.git/95351\n http://thread.gmane.org/gmane.comp.version-control.git/95072\n http://thread.gmane.org/gmane.comp.version-control.git/107417/focus=107500\n"},{"id":"112852","messageId":"20090502000123.GF23604@spearce.org","threadId":"18886","inReplyTo":"7v4ow4v0xl.fsf@gitster.siamese.dyndns.org","subject":"Compatibility between git.git and jgit","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-02T00:01:23Z","receivedAt":"2009-05-02T00:01:23Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> The code looks correct, I am reasonably sure updated server-client\n> combination would work fine, and use of the capability mechanism means\n> other combinations like old pusher and new receiver, and/or new pusher and\n> old receiver, should be also Ok.\n> \n> I see Shawn did the same to jgit.\n\nOn an unrelated note, someone asked me recently, how do we ensure\ncompatibility in implementations between git.git and jgit?\n\nThere isn't exactly a great notion of \"a Git implementation can do\nX, Y, Z, and never does Q\". So its not like we have a compability\ntest suite to run between the two systems.\n\nJGit is really starting to gain some traction in the open source\nworld.\n\nA lot of folks at Eclipse are really excited about being able to\nship a BSD licensed VCS. Some of the Maven folks are really excited\nabout being able to link JGit up to Apache MINA SSHD and have a 100%\npure Java server solution for Git, that doesn't require native OS\nauthentication systems. Gerrit Code Review relies entirely on it,\nand some folks within Google are now using Gerrit Code Review and\nits embedded MINA SSHD/JGit server as their only Git daemon.\n\nThus far, our compatibility story with git.git has been, \"it should\nwork, uh, we think, because Shawn understands git reasonably well,\nand wrote some of JGit, so uh, yea....\". :-)\n\nBut I think in another 12 months we'll be seeing people running\nonly JGit in many contexts, making compatibility between the two\nimplementations somewhat more important than it has been in the past.\n\n-- \nShawn.\n"},{"id":"112855","messageId":"alpine.LFD.2.00.0905012021070.6741@xanadu.home","threadId":"18886","inReplyTo":"7v4ow4v0xl.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] allow OFS_DELTA objects during a push","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-02T00:24:46Z","receivedAt":"2009-05-02T00:24:46Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 1 May 2009, Junio C Hamano wrote:\n\n> Thanks.\n> \n> The code looks correct, I am reasonably sure updated server-client\n> combination would work fine, and use of the capability mechanism means\n> other combinations like old pusher and new receiver, and/or new pusher and\n> old receiver, should be also Ok.\n> \n> I see Shawn did the same to jgit.\n> \n> But I'd like to queue this in 'next', and make it official after 1.6.3\n> happens.\n> \n> I just do not want to repeat silly mistakes, this close to the final,\n> similar to the \"github needs to get stuck forever at 1.6.1\" we made with\n> 40c155f (push: prepare sender to receive extended ref information from the\n> receiver, 2008-09-09); it was done as a good change after a discussion\n> among Shawn, Daniel and I.\n\nNo problem. Since this is fixing something that actually never worked \nbefore, it is therefore not what I would call a critical fix.\n\n\nNicolas\n"},{"id":"112857","messageId":"49FB9E5E.30504@gmail.com","threadId":"18886","inReplyTo":"20090502000123.GF23604@spearce.org","subject":"Re: Compatibility between git.git and jgit","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2009-05-02T01:14:06Z","receivedAt":"2009-05-02T01:14:06Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n> Junio C Hamano wrote:\n>> The code looks correct, I am reasonably sure updated server-client\n>> combination would work fine, and use of the capability mechanism means\n>> other combinations like old pusher and new receiver, and/or new pusher and\n>> old receiver, should be also Ok.\n>>\n>> I see Shawn did the same to jgit.\n> \n> On an unrelated note, someone asked me recently, how do we ensure\n> compatibility in implementations between git.git and jgit?\n> \n> There isn't exactly a great notion of \"a Git implementation can do\n> X, Y, Z, and never does Q\". So its not like we have a compability\n> test suite to run between the two systems.\n> \n> JGit is really starting to gain some traction in the open source\n> world.\n> \n> A lot of folks at Eclipse are really excited about being able to\n> ship a BSD licensed VCS. Some of the Maven folks are really excited\n> about being able to link JGit up to Apache MINA SSHD and have a 100%\n> pure Java server solution for Git, that doesn't require native OS\n> authentication systems. Gerrit Code Review relies entirely on it,\n> and some folks within Google are now using Gerrit Code Review and\n> its embedded MINA SSHD/JGit server as their only Git daemon.\n> \n> Thus far, our compatibility story with git.git has been, \"it should\n> work, uh, we think, because Shawn understands git reasonably well,\n> and wrote some of JGit, so uh, yea....\". :-)\n> \n> But I think in another 12 months we'll be seeing people running\n> only JGit in many contexts, making compatibility between the two\n> implementations somewhat more important than it has been in the past.\n> \n\n[A non-answer to this implied question]\n\nUsually one of the implementations is declared \"the reference \nimplementation\".\n\n[And another question]\n\nHasn't this issue come up before in the mailing list?\n"},{"id":"112858","messageId":"alpine.LFD.2.00.0905012032590.6741@xanadu.home","threadId":"18886","inReplyTo":"20090502000123.GF23604@spearce.org","subject":"Re: Compatibility between git.git and jgit","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-02T01:39:48Z","receivedAt":"2009-05-02T01:39:48Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 1 May 2009, Shawn O. Pearce wrote:\n\n> On an unrelated note, someone asked me recently, how do we ensure\n> compatibility in implementations between git.git and jgit?\n> \n> There isn't exactly a great notion of \"a Git implementation can do\n> X, Y, Z, and never does Q\". So its not like we have a compability\n> test suite to run between the two systems.\n> \n> JGit is really starting to gain some traction in the open source\n> world.\n\nWell... this is not exactly easy. As I said in the past \n(http://marc.info/?l=git&m=121035043412788&w=2), I think that the C \nversion must remain the reference with regards to protocols and on-disk \ndata structures. If people go wild with JGit and start making changes \nto data structures then it simply won't be Git compatible anymore and \nthe user base will get fragmented. A good way to prevent this is for \npeople interested in making git compatible tools to monitor and interact \non the git mailing list, however if we look at the results from some \npast GSOC projects we must conclude that not everyone is giving enough \nconsideration to that.\n\nA formal compatibility test suite would imply that every Git \nreimplementation should be compatible with the reference C version. \nYou could add some tests in your test suite which are performed in \nparallel using JGit and the C git, and make sure that the produced \nresults are identical, etc.\n\nBut to which extent should the C version remain backward compatible with \nother implementations? Let's suppose a future protocol extension is \nmade and old unsuspecting C clients work just fine but some other \nimplementation crashes with it? The more you have reimplementations of \nGit, the greater is the possibility for one of them to be flawed and \nbuggier in one way or another but happened to just work with older C git \nversions. And the reference implementation cannot be held back because \nof bugs in all alternative implementations.\n\n> A lot of folks at Eclipse are really excited about being able to\n> ship a BSD licensed VCS. Some of the Maven folks are really excited\n> about being able to link JGit up to Apache MINA SSHD and have a 100%\n> pure Java server solution for Git, that doesn't require native OS\n> authentication systems. Gerrit Code Review relies entirely on it,\n> and some folks within Google are now using Gerrit Code Review and\n> its embedded MINA SSHD/JGit server as their only Git daemon.\n\nAs long as they're futzing^Wdeveloping on top of Jgit then \ninteroperability shouldn't be at risk. If people would start adding new \nobject types and pack formats and the like without obtaining a consensus \nwith people around the C version then I might get extremely worried (and \npissed) though.\n\n> Thus far, our compatibility story with git.git has been, \"it should\n> work, uh, we think, because Shawn understands git reasonably well,\n> and wrote some of JGit, so uh, yea....\". :-)\n\nIf that works, and as I know you I'm sure that works great, then maybe \nthis should just continue that way for as long as it is workable.\n\n> But I think in another 12 months we'll be seeing people running\n> only JGit in many contexts, making compatibility between the two\n> implementations somewhat more important than it has been in the past.\n\nOne defensive approach we could adopt is to use a capability slot to \nidentify the software version of each peer involved in the network \ncommunication. The advantage would be for a later Git version to avoid \ndoing some things that are known to break with client X or Y. Of course \neven such a scheme can be abused and misused, like on some web sites if \nyou don't have the \"right\" browser, leading some of them to allow faking \nthe User-Agent string, etc. But maybe the upsides are more important \nthan the downsides. This doesn't help with on-disk interoperability, \nbut this is probably less important than communication interoperability.\n\n\nNicolas\n"},{"id":"112859","messageId":"b4087cc50905011840j664710f9xbb377f1578dfab5a@mail.gmail.com","threadId":"18886","inReplyTo":"20090502000123.GF23604@spearce.org","subject":"Re: Compatibility between git.git and jgit","fromName":"Michael Witten","fromEmail":"mfwitten@gmail.com","sentAt":"2009-05-02T01:40:09Z","receivedAt":"2009-05-02T01:40:09Z","isPatch":false,"sender":{"key":"mfwitten@gmail.com","avatar":"https://avatars.githubusercontent.com/u/597101?v=4"},"body":"On Fri, May 1, 2009 at 19:01, Shawn O. Pearce wrote:\n>\n> But I think in another 12 months we'll be seeing people running\n> only JGit in many contexts, making compatibility between the two\n> implementations somewhat more important than it has been in the past.\n\n:-D\n\nGetting a little cocky there, eh Shawn?\n\nInterestingly, you don't say \"making compatibility **with git.git**\nmore important\"...\n\nWatch your back Junio!\n\n:-D\n"},{"id":"112860","messageId":"20090502015950.GG23604@spearce.org","threadId":"18886","inReplyTo":"alpine.LFD.2.00.0905012032590.6741@xanadu.home","subject":"Re: Compatibility between git.git and jgit","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-02T01:59:50Z","receivedAt":"2009-05-02T01:59:50Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> On Fri, 1 May 2009, Shawn O. Pearce wrote:\n> \n> > On an unrelated note, someone asked me recently, how do we ensure\n> > compatibility in implementations between git.git and jgit?\n> \n> Well... this is not exactly easy. As I said in the past \n> (http://marc.info/?l=git&m=121035043412788&w=2), I think that the C \n> version must remain the reference with regards to protocols and on-disk \n> data structures.\n\nI agree fully.\n\n> If people go wild with JGit and start making changes \n> to data structures then it simply won't be Git compatible anymore and \n> the user base will get fragmented.\n\nAgree. We may see some prototyping happen in JGit first on some\ntopics, and JGit may even support something earlier than git.git,\ne.g JGit has an amazon-s3:// transport that git.git doesn't have.\nBut it also isn't widely used.\n\n> A formal compatibility test suite would imply that every Git \n> reimplementation should be compatible with the reference C version. \n> You could add some tests in your test suite which are performed in \n> parallel using JGit and the C git, and make sure that the produced \n> results are identical, etc.\n\nYea, and to some extent we try to do that already in JGit, but our\ntests aren't complete enough in that area.\n \n> But to which extent should the C version remain backward compatible with \n> other implementations? Let's suppose a future protocol extension is \n> made and old unsuspecting C clients work just fine but some other \n> implementation crashes with it?\n\nThis is what I think scares both myself and the folks that have\nrecently asked me about compatibility.\n\nIf JGit gets a broader user base, and suddenly it stops working\nagainst a newer C git-daemon because of a protocol change, those\nusers are going to be pissed. Its no worse than the \"github can't\never upgrade past 1.6.1\" issue we had not too long ago.\n\nI think we're doing better these days about embedding file format\nversion numbers into files (e.g. pack idx v2) to help alert older\nclients that the format is different. But we also have a something\nof a history of looking for \"holes\" in older C git parsers in\norder to wedge in new features where we didn't plan for them in\nthe first place. E.g. the protocol capability slots we have now.\n\nI think that as reimplementations become more popular, we need to\nrely less on extending things by exploiting parser quirks in older\nC git.git code, and rely more on at least explicit version markers\nthat everyone can work with.\n\n> And the reference implementation cannot be held back because \n> of bugs in all alternative implementations.\n\nI agree. A bug is a bug. But I'd really like to get away from the\ntrend where we exploit bugs in older C git.git implementations to\nadd new functionality, because maybe JGit doesn't have that same\nbug and will fall flat on its face with that exploit.\n\n> As long as they're futzing^Wdeveloping on top of Jgit then \n> interoperability shouldn't be at risk. If people would start adding new \n> object types and pack formats and the like without obtaining a consensus \n> with people around the C version then I might get extremely worried (and \n> pissed) though.\n\nThat's why JGit is BSD, so everyone can use the one f'king library\nand not risk fragmenting the Java market further.\n\nBut yea, I'd be really pissed too if someone hacked up JGit and made\nit incompatible with anything else. Its a risk that the liberal\nBSD license permits.\n\nI'm really sort of hoping that the development momentum around\ngit.git and JGit trying to keep up will keep them coming back\nto the canonical JGit for updates, forcing them to give back any\nhacks^Wimprovements they have made. If the improvements really are\nworthwhile, they can be easily ported over to C before they become\nwidely used in JGit.\n \n> One defensive approach we could adopt is to use a capability slot to \n> identify the software version of each peer involved in the network \n> communication. The advantage would be for a later Git version to avoid \n> doing some things that are known to break with client X or Y. Of course \n> even such a scheme can be abused and misused, like on some web sites if \n> you don't have the \"right\" browser, leading some of them to allow faking \n> the User-Agent string, etc. But maybe the upsides are more important \n> than the downsides. This doesn't help with on-disk interoperability, \n> but this is probably less important than communication interoperability.\n\nBlargh. I'm with you about the whole User-Agent mess.\n\nAsking clients and servers to identify with implementation and\nversion markers might be useful for analysis of who-is-using-what,\nbut I don't think its a good way to negotiate between the peers of\nwhat functionality to enable or disable, or what bug workarounds\nto use. Reminds me of the Apache hack during output to work around\nan HTTP header parsing bug in Netscape 2 when the \"\\r\\n\" pair was\nexactly at byte 256 in the stream. *shudder*\n\n\nFWIW, an EGit user recently complained that some random Git hosting\nsite they were using couldn't work with EGit, but EGit worked fine\nwith other sites, e.g. GitHub. Apparently this site's SSH forced command\nfilter script didn't like EGit asking for \"git upload-pack 'path.git'\".\n\nIts not strictly a Git protocol issue, how the client launches\nthe remote process over SSH, but this random hosting site was\napparently relying on C git's current calling convention of\n\"git-upload-pack 'path.git'\".\n\nLong story short, I claimed it was the hosting site's bug. :-)\n\n-- \nShawn.\n"},{"id":"112883","messageId":"efe2b6d70905020956p3c99a5fbib85ba00ba842a08e@mail.gmail.com","threadId":"18886","inReplyTo":"alpine.LFD.2.00.0905012032590.6741@xanadu.home","subject":"Re: Compatibility between git.git and jgit","fromName":"Ealdwulf Wuffinga","fromEmail":"ealdwulf@googlemail.com","sentAt":"2009-05-02T16:56:29Z","receivedAt":"2009-05-02T16:56:29Z","isPatch":false,"sender":{"key":"ealdwulf@googlemail.com","avatar":null},"body":"On Sat, May 2, 2009 at 2:39 AM, Nicolas Pitre wrote:\n>\n> A formal compatibility test suite would imply that every Git\n> reimplementation should be compatible with the reference C version.\n> You could add some tests in your test suite which are performed in\n> parallel using JGit and the C git, and make sure that the produced\n> results are identical, etc.\n\n If at all possible, it would be a good idea to make it trivial for\nnew tests in the usual\ngit testsuite to be compatability tests (in a special mode, since it\nwould probably slow them down drastically). Ie, we have special\nseparate copy of all the git.git executables, which\nunderneath run two different versions of git and check that they did\nthe same thing.\nOr alternatively wait for the librarification of git.git to complete,\nand do it at the level of that API.\n\nThis may be hideously slow unless you have some kind of snapshotting filesystem\nunderneath. (Ironically the one that springs to mind is built into\nvesta, another scm: http://www.vestasys.org. Vesta's\nfilesystem-manipulation language would be ideal\nfor this. Maybe you could copy it; it's LGPL).\n\nIt's less obvious how networking related tests could be automatically\nmade into compatability\ntests.\n\nDoing this would be a lot trickier than writing some new conformance\ntests, but once it was working, it would be a lot easier to keep track\nof new features. It may be too tricky\nto get to work, but it strikes me as worth thinking about.\n\nEaldwulf\n"},{"id":"113009","messageId":"20090504221129.GF23604@spearce.org","threadId":"18886","inReplyTo":"alpine.LFD.2.00.0905011616130.6741@xanadu.home","subject":"Re: [PATCH] allow OFS_DELTA objects during a push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-04T22:11:29Z","receivedAt":"2009-05-04T22:11:29Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> The fetching of OFS_DELTA objects has been negotiated between both peers \n> since git version 1.4.4. However, this was missing from the push side \n> where every OFS_DELTA objects were always converted to REF_DELTA objects \n> causing an increase in transferred data.\n\nFolks, this may have broken git push for me.\n\nI'm trying to debug it right now, but something in next between\n46488d2 and 03e1664 has caused \"git push\" to not create a pack\nfile, sending the remote peer 0 objects, when really we should have\ntransmitted objects, e.g. in the case I just looked at, we should\nhave sent 11.\n\nFWIW, I'm currently blaming this change as its the only thing to\ntouch builtin-send-pack.c in that commit range. :-)\n\n/me goes off to debug this further...\n \n> builtin-receive-pack.c | 22 +++++++++++++++-------\n> builtin-send-pack.c | 8 +++++++-\n> send-pack.h | 1 +\n> 3 files changed, 23 insertions(+), 8 deletions(-)\n\n-- \nShawn.\n"},{"id":"113010","messageId":"20090504223005.GG23604@spearce.org","threadId":"18886","inReplyTo":"20090504221129.GF23604@spearce.org","subject":"Re: [PATCH] allow OFS_DELTA objects during a push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-04T22:30:05Z","receivedAt":"2009-05-04T22:30:05Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Shawn O. Pearce\" wrote:\n> Nicolas Pitre wrote:\n> > The fetching of OFS_DELTA objects has been negotiated between both peers \n> > since git version 1.4.4. However, this was missing from the push side \n> > where every OFS_DELTA objects were always converted to REF_DELTA objects \n> > causing an increase in transferred data.\n> \n> Folks, this may have broken git push for me.\n> \n> I'm trying to debug it right now, but something in next between\n> 46488d2 and 03e1664 has caused \"git push\" to not create a pack\n> file, sending the remote peer 0 objects, when really we should have\n> transmitted objects, e.g. in the case I just looked at, we should\n> have sent 11.\n\nUhm, never mind.\n\nSomehow my incremental build failed horribly; it compiled and created\na git-push which worked \"some of the time\". Building again produced\na working git-push.\n\nScary stuff. Now I have to worry about the toolchain on this system.\nBut I don't think there is a problem in git.git.\n \n-- \nShawn.\n"}]}