{"thread":{"id":"21862","subject":"Endianness bug in git-svn or called component?","startedAt":"2009-12-04T17:44:58Z","lastAt":"2009-12-27T19:12:37Z","messageCount":9,"participants":["Nathaniel W Filardo","Andreas Schwab","Erik Faye-Lund","Junio C Hamano","Tomas Carnecky"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"129225","messageId":"20091204174458.GV17192@gradx.cs.jhu.edu","threadId":"21862","inReplyTo":null,"subject":"Endianness bug in git-svn or called component?","fromName":"Nathaniel W Filardo","fromEmail":"nwf@cs.jhu.edu","sentAt":"2009-12-04T17:44:58Z","receivedAt":"2009-12-04T17:44:58Z","isPatch":false,"sender":{"key":"nwf@cs.jhu.edu","avatar":null},"body":"On this machine,\n\nmirrors hydra:/tank0/mirrors/misc% uname -a\nFreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64\n\nattempting to fetch from an svn source yields the following error:\n\nrs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua\nInitialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ \nmirrors hydra:/tank0/mirrors/misc% cd test-joshua \nmirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch\n A scripts/distributedLM/config.template \n[...]\n A build.xml\nr1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)\nerror: index uses extension, which we do not understand\nfatal: index file corrupt\nwrite-tree: command returned error: 128\n\nEven on previously initialized and fetched git-svn repos, \"git svn fetch\"\nproduces the same response.\n\nThis is the same symptom as an older bug, but I have not attempted to track\ndown what's going wrong this time in hopes that somebody more familiar can\nfix it faster; see\nhttp://kerneltrap.org/mailarchive/git/2006/5/28/205750/thread#mid-205750 .\n\nIf there's anything more that would be helpful to know, please just ask.\n\nThanks.\n--nwf;\n"},{"id":"129235","messageId":"m23a3qa40n.fsf@igel.home","threadId":"21862","inReplyTo":"20091204174458.GV17192@gradx.cs.jhu.edu","subject":"Re: Endianness bug in git-svn or called component?","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2009-12-04T20:16:40Z","receivedAt":"2009-12-04T20:16:40Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Nathaniel W Filardo writes:\n\n> On this machine,\n>\n> mirrors hydra:/tank0/mirrors/misc% uname -a\n> FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64\n>\n> attempting to fetch from an svn source yields the following error:\n>\n> rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua\n> Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ \n> mirrors hydra:/tank0/mirrors/misc% cd test-joshua \n> mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch\n> A scripts/distributedLM/config.template \n> [...]\n> A build.xml\n> r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)\n> error: index uses extension, which we do not understand\n> fatal: index file corrupt\n> write-tree: command returned error: 128\n\nI could not reproduce that on powerpc (both 32bit and 64bit).\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"129236","messageId":"20091204202928.GW17192@gradx.cs.jhu.edu","threadId":"21862","inReplyTo":"m23a3qa40n.fsf@igel.home","subject":"Re: Endianness bug in git-svn or called component?","fromName":"Nathaniel W Filardo","fromEmail":"nwf@cs.jhu.edu","sentAt":"2009-12-04T20:29:28Z","receivedAt":"2009-12-04T20:29:28Z","isPatch":false,"sender":{"key":"nwf@cs.jhu.edu","avatar":null},"body":"On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote:\n> Nathaniel W Filardo writes:\n> \n> > On this machine,\n> >\n> > mirrors hydra:/tank0/mirrors/misc% uname -a\n> > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64\n> >\n> > attempting to fetch from an svn source yields the following error:\n> >\n> > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua\n> > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ \n> > mirrors hydra:/tank0/mirrors/misc% cd test-joshua \n> > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch\n> > A scripts/distributedLM/config.template \n> > [...]\n> > A build.xml\n> > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)\n> > error: index uses extension, which we do not understand\n> > fatal: index file corrupt\n> > write-tree: command returned error: 128\n> \n> I could not reproduce that on powerpc (both 32bit and 64bit).\n> \n> Andreas.\n\nHm. I seem to have forgotten to mention that I am running\n\n% git --version\ngit version 1.6.5.3\n\nbuilt from the FreeBSD ports tree, in case that matters, using\n\n% gcc --version\ngcc (GCC) 4.2.1 20070719 [FreeBSD]\n\nKnowing nothing of git's internals, how should I start to investigate what's\ngoing on here?\n\nThanks again.\n--nwf;\n"},{"id":"130065","messageId":"20091218090544.GK14941@gradx.cs.jhu.edu","threadId":"21862","inReplyTo":"20091204202928.GW17192@gradx.cs.jhu.edu","subject":"Re: Endianness bug in git-svn or called component?","fromName":"Nathaniel W Filardo","fromEmail":"nwf@cs.jhu.edu","sentAt":"2009-12-18T09:05:44Z","receivedAt":"2009-12-18T09:05:44Z","isPatch":false,"sender":{"key":"nwf@cs.jhu.edu","avatar":null},"body":"On Fri, Dec 04, 2009 at 03:29:28PM -0500, Nathaniel W Filardo wrote:\n> On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote:\n> > Nathaniel W Filardo writes:\n> > \n> > > On this machine,\n> > >\n> > > mirrors hydra:/tank0/mirrors/misc% uname -a\n> > > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64\n> > >\n> > > attempting to fetch from an svn source yields the following error:\n> > >\n> > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua\n> > > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ \n> > > mirrors hydra:/tank0/mirrors/misc% cd test-joshua \n> > > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch\n> > > A scripts/distributedLM/config.template \n> > > [...]\n> > > A build.xml\n> > > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)\n> > > error: index uses extension, which we do not understand\n> > > fatal: index file corrupt\n> > > write-tree: command returned error: 128\n> > \n> > I could not reproduce that on powerpc (both 32bit and 64bit).\n> > \n> > Andreas.\n\nI got some free time and tracked it down. The following one-line delta\nfixes this issue for me; AIUI on sparc64 \"unsigned long\" is 8 bits and in\nthe wrong endianness for the memcpy trick to work as written? I could be\nsligntly off in my assessment of the problem, tho'.\n\nindex 1bbaf1c..9033dd3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)\n * extension name (4-byte) and section length\n * in 4-byte network byte order.\n */\n- unsigned long extsize;\n+ uint32_t extsize;\n memcpy(&extsize, (char *)mmap + src_offset + 4, 4);\n extsize = ntohl(extsize);\n if (read_index_extension(istate,\n\n--nwf;\n"},{"id":"130360","messageId":"20091227061121.GP14941@gradx.cs.jhu.edu","threadId":"21862","inReplyTo":"20091204202928.GW17192@gradx.cs.jhu.edu","subject":"[RESEND] [PATCH] Endianness bug in index cache logic","fromName":"Nathaniel W Filardo","fromEmail":"nwf@cs.jhu.edu","sentAt":"2009-12-27T06:11:21Z","receivedAt":"2009-12-27T06:11:21Z","isPatch":true,"sender":{"key":"nwf@cs.jhu.edu","avatar":null},"body":"On Fri, Dec 04, 2009 at 03:29:28PM -0500, Nathaniel W Filardo wrote:\n> On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote:\n> > Nathaniel W Filardo writes:\n> > \n> > > On this machine,\n> > >\n> > > mirrors hydra:/tank0/mirrors/misc% uname -a\n> > > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64\n> > >\n> > > attempting to fetch from an svn source yields the following error:\n> > >\n> > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua\n> > > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ \n> > > mirrors hydra:/tank0/mirrors/misc% cd test-joshua \n> > > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch\n> > > A scripts/distributedLM/config.template \n> > > [...]\n> > > A build.xml\n> > > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)\n> > > error: index uses extension, which we do not understand\n> > > fatal: index file corrupt\n> > > write-tree: command returned error: 128\n> > \n> > I could not reproduce that on powerpc (both 32bit and 64bit).\n> > \n> > Andreas.\n\nI got some free time and tracked it down. The following one-line delta\nfixes this issue for me; AIUI on sparc64 \"unsigned long\" is 8 bits and in\nthe wrong endianness for the memcpy trick to work as written? I could be\nsligntly off in my assessment of the problem, tho'.\n\nindex 1bbaf1c..9033dd3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)\n * extension name (4-byte) and section length\n * in 4-byte network byte order.\n */\n- unsigned long extsize;\n+ uint32_t extsize;\n memcpy(&extsize, (char *)mmap + src_offset + 4, 4);\n extsize = ntohl(extsize);\n if (read_index_extension(istate,\n\n--nwf;\n"},{"id":"130361","messageId":"40aa078e0912270439i3948a38fw9784e23e7cdc4407@mail.gmail.com","threadId":"21862","inReplyTo":"20091227061121.GP14941@gradx.cs.jhu.edu","subject":"Re: [RESEND] [PATCH] Endianness bug in index cache logic","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@googlemail.com","sentAt":"2009-12-27T12:39:24Z","receivedAt":"2009-12-27T12:39:24Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Sun, Dec 27, 2009 at 7:11 AM, Nathaniel W Filardo wrote:\n> I got some free time and tracked it down.  The following one-line delta\n> fixes this issue for me; AIUI on sparc64 \"unsigned long\" is 8 bits and in\n\nYou mean 8 bytes, right?\n\n-- \nErik \"kusma\" Faye-Lund\n"},{"id":"130363","messageId":"20091227161246.GQ14941@gradx.cs.jhu.edu","threadId":"21862","inReplyTo":"40aa078e0912270439i3948a38fw9784e23e7cdc4407@mail.gmail.com","subject":"Re: [RESEND] [PATCH] Endianness bug in index cache logic","fromName":"Nathaniel W Filardo","fromEmail":"nwf@cs.jhu.edu","sentAt":"2009-12-27T16:12:46Z","receivedAt":"2009-12-27T16:12:46Z","isPatch":true,"sender":{"key":"nwf@cs.jhu.edu","avatar":null},"body":"On Sun, Dec 27, 2009 at 01:39:24PM +0100, Erik Faye-Lund wrote:\n> On Sun, Dec 27, 2009 at 7:11 AM, Nathaniel W Filardo wrote:\n> > I got some free time and tracked it down.  The following one-line delta\n> > fixes this issue for me; AIUI on sparc64 \"unsigned long\" is 8 bits and in\n> \n> You mean 8 bytes, right?\n\nYes, sorry. How embarrassing. :)\n--nwf;\n"},{"id":"130367","messageId":"7veimgxolb.fsf@alter.siamese.dyndns.org","threadId":"21862","inReplyTo":"20091227061121.GP14941@gradx.cs.jhu.edu","subject":"Re: [RESEND] [PATCH] Endianness bug in index cache logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-12-27T18:24:00Z","receivedAt":"2009-12-27T18:24:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nathaniel W Filardo writes:\n\n> I got some free time and tracked it down. The following one-line delta\n> fixes this issue for me; AIUI on sparc64 \"unsigned long\" is 8 bits and in\n> the wrong endianness for the memcpy trick to work as written? I could be\n> sligntly off in my assessment of the problem, tho'.\n>\n> index 1bbaf1c..9033dd3 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)\n> * extension name (4-byte) and section length\n> * in 4-byte network byte order.\n> */\n> - unsigned long extsize;\n> + uint32_t extsize;\n> memcpy(&extsize, (char *)mmap + src_offset + 4, 4);\n> extsize = ntohl(extsize);\n> if (read_index_extension(istate,\n\nGood catch.\n\nThe original is broken on big endian 64-bit platforms, and your sparc64\nindeed is one. The code expects to see a signature (4-byte) at src_offset\nfollowed by length (4-byte int in network byte order) and it is trying to\nread read the latter in extsize.\n\nThanks for the fix. The bug dates back to late April 2006, and I am kind\nof surprised that nobody reported this since then (perhaps nobody runs git\non big endian 64-bit boxes?).\n"},{"id":"130368","messageId":"4B37B1A5.5030209@dbservice.com","threadId":"21862","inReplyTo":"7veimgxolb.fsf@alter.siamese.dyndns.org","subject":"Re: [RESEND] [PATCH] Endianness bug in index cache logic","fromName":"Tomas Carnecky","fromEmail":"tom@dbservice.com","sentAt":"2009-12-27T19:12:37Z","receivedAt":"2009-12-27T19:12:37Z","isPatch":true,"sender":{"key":"tom@dbservice.com","avatar":"https://gravatar.com/avatar/900a300bdd1a8bbe086008ad78210bbee2ad2803b7d50a5cba04c1e9404bd6d2?d=mp&s=160"},"body":"On 12/27/09 7:24 PM, Junio C Hamano wrote:\n> Nathaniel W Filardo writes:\n>\n>> I got some free time and tracked it down. The following one-line delta\n>> fixes this issue for me; AIUI on sparc64 \"unsigned long\" is 8 bits and in\n>> the wrong endianness for the memcpy trick to work as written? I could be\n>> sligntly off in my assessment of the problem, tho'.\n>>\n>> index 1bbaf1c..9033dd3 100644\n>> --- a/read-cache.c\n>> +++ b/read-cache.c\n>> @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)\n>> * extension name (4-byte) and section length\n>> * in 4-byte network byte order.\n>> */\n>> - unsigned long extsize;\n>> + uint32_t extsize;\n>> memcpy(&extsize, (char *)mmap + src_offset + 4, 4);\n>> extsize = ntohl(extsize);\n>> if (read_index_extension(istate,\n>\n> Good catch.\n>\n> The original is broken on big endian 64-bit platforms, and your sparc64\n> indeed is one. The code expects to see a signature (4-byte) at src_offset\n> followed by length (4-byte int in network byte order) and it is trying to\n> read read the latter in extsize.\n>\n> Thanks for the fix. The bug dates back to late April 2006, and I am kind\n> of surprised that nobody reported this since then (perhaps nobody runs git\n> on big endian 64-bit boxes?).\n\nBoth the native Sun compiler as well as GCC default to 32bit binaries, \neven if the system is capable of running 64bit binaries (unlike for \nexample Linux where x86_64-pc-linux-gnu-gcc produces 64bit binaries by \ndefault). And the git makefile doesn't use -m64, so my guess is that \nnobody bothered changing CFLAGS on sparc64 systems (I have access to a \ncouple such systems and I never changed CFLAGS because git always worked \nout of the box). Though I don't know what the default is on other big \nendian systems.\n\ntom\n"}]}