# Endianness bug in git-svn or called component? 9 messages from 2009-12-04 to 2009-12-27. Participants: Nathaniel W Filardo, Andreas Schwab, Erik Faye-Lund, Junio C Hamano, Tomas Carnecky. Thread: https://gitlist.dev/t/21862 ## Nathaniel W Filardo, 2009-12-04 17:44 Subject: Endianness bug in git-svn or called component? Message-ID: <20091204174458.GV17192@gradx.cs.jhu.edu> URL: https://gitlist.dev/e/20091204174458.GV17192%40gradx.cs.jhu.edu ``` On this machine, mirrors hydra:/tank0/mirrors/misc% uname -a FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64 attempting to fetch from an svn source yields the following error: rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ mirrors hydra:/tank0/mirrors/misc% cd test-joshua mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch A scripts/distributedLM/config.template [...] A build.xml r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk) error: index uses extension, which we do not understand fatal: index file corrupt write-tree: command returned error: 128 Even on previously initialized and fetched git-svn repos, "git svn fetch" produces the same response. This is the same symptom as an older bug, but I have not attempted to track down what's going wrong this time in hopes that somebody more familiar can fix it faster; see http://kerneltrap.org/mailarchive/git/2006/5/28/205750/thread#mid-205750 . If there's anything more that would be helpful to know, please just ask. Thanks. --nwf; ``` ## Andreas Schwab, 2009-12-04 20:16 Subject: Re: Endianness bug in git-svn or called component? Message-ID: URL: https://gitlist.dev/e/m23a3qa40n.fsf%40igel.home In-Reply-To: <20091204174458.GV17192@gradx.cs.jhu.edu> ``` Nathaniel W Filardo writes: > On this machine, > > mirrors hydra:/tank0/mirrors/misc% uname -a > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64 > > attempting to fetch from an svn source yields the following error: > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ > mirrors hydra:/tank0/mirrors/misc% cd test-joshua > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch > A scripts/distributedLM/config.template > [...] > A build.xml > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk) > error: index uses extension, which we do not understand > fatal: index file corrupt > write-tree: command returned error: 128 I could not reproduce that on powerpc (both 32bit and 64bit). Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ``` ## Nathaniel W Filardo, 2009-12-04 20:29 Subject: Re: Endianness bug in git-svn or called component? Message-ID: <20091204202928.GW17192@gradx.cs.jhu.edu> URL: https://gitlist.dev/e/20091204202928.GW17192%40gradx.cs.jhu.edu In-Reply-To: ``` On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote: > Nathaniel W Filardo writes: > > > On this machine, > > > > mirrors hydra:/tank0/mirrors/misc% uname -a > > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64 > > > > attempting to fetch from an svn source yields the following error: > > > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua > > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ > > mirrors hydra:/tank0/mirrors/misc% cd test-joshua > > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch > > A scripts/distributedLM/config.template > > [...] > > A build.xml > > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk) > > error: index uses extension, which we do not understand > > fatal: index file corrupt > > write-tree: command returned error: 128 > > I could not reproduce that on powerpc (both 32bit and 64bit). > > Andreas. Hm. I seem to have forgotten to mention that I am running % git --version git version 1.6.5.3 built from the FreeBSD ports tree, in case that matters, using % gcc --version gcc (GCC) 4.2.1 20070719 [FreeBSD] Knowing nothing of git's internals, how should I start to investigate what's going on here? Thanks again. --nwf; ``` ## Nathaniel W Filardo, 2009-12-18 09:05 Subject: Re: Endianness bug in git-svn or called component? Message-ID: <20091218090544.GK14941@gradx.cs.jhu.edu> URL: https://gitlist.dev/e/20091218090544.GK14941%40gradx.cs.jhu.edu In-Reply-To: <20091204202928.GW17192@gradx.cs.jhu.edu> ``` On Fri, Dec 04, 2009 at 03:29:28PM -0500, Nathaniel W Filardo wrote: > On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote: > > Nathaniel W Filardo writes: > > > > > On this machine, > > > > > > mirrors hydra:/tank0/mirrors/misc% uname -a > > > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64 > > > > > > attempting to fetch from an svn source yields the following error: > > > > > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua > > > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ > > > mirrors hydra:/tank0/mirrors/misc% cd test-joshua > > > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch > > > A scripts/distributedLM/config.template > > > [...] > > > A build.xml > > > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk) > > > error: index uses extension, which we do not understand > > > fatal: index file corrupt > > > write-tree: command returned error: 128 > > > > I could not reproduce that on powerpc (both 32bit and 64bit). > > > > Andreas. I got some free time and tracked it down. The following one-line delta fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in the wrong endianness for the memcpy trick to work as written? I could be sligntly off in my assessment of the problem, tho'. index 1bbaf1c..9033dd3 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path) * extension name (4-byte) and section length * in 4-byte network byte order. */ - unsigned long extsize; + uint32_t extsize; memcpy(&extsize, (char *)mmap + src_offset + 4, 4); extsize = ntohl(extsize); if (read_index_extension(istate, --nwf; ``` ## Nathaniel W Filardo, 2009-12-27 06:11 Subject: [RESEND] [PATCH] Endianness bug in index cache logic Message-ID: <20091227061121.GP14941@gradx.cs.jhu.edu> URL: https://gitlist.dev/e/20091227061121.GP14941%40gradx.cs.jhu.edu In-Reply-To: <20091204202928.GW17192@gradx.cs.jhu.edu> ``` On Fri, Dec 04, 2009 at 03:29:28PM -0500, Nathaniel W Filardo wrote: > On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote: > > Nathaniel W Filardo writes: > > > > > On this machine, > > > > > > mirrors hydra:/tank0/mirrors/misc% uname -a > > > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64 > > > > > > attempting to fetch from an svn source yields the following error: > > > > > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua > > > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/ > > > mirrors hydra:/tank0/mirrors/misc% cd test-joshua > > > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch > > > A scripts/distributedLM/config.template > > > [...] > > > A build.xml > > > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk) > > > error: index uses extension, which we do not understand > > > fatal: index file corrupt > > > write-tree: command returned error: 128 > > > > I could not reproduce that on powerpc (both 32bit and 64bit). > > > > Andreas. I got some free time and tracked it down. The following one-line delta fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in the wrong endianness for the memcpy trick to work as written? I could be sligntly off in my assessment of the problem, tho'. index 1bbaf1c..9033dd3 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path) * extension name (4-byte) and section length * in 4-byte network byte order. */ - unsigned long extsize; + uint32_t extsize; memcpy(&extsize, (char *)mmap + src_offset + 4, 4); extsize = ntohl(extsize); if (read_index_extension(istate, --nwf; ``` ## Erik Faye-Lund, 2009-12-27 12:39 Subject: Re: [RESEND] [PATCH] Endianness bug in index cache logic Message-ID: <40aa078e0912270439i3948a38fw9784e23e7cdc4407@mail.gmail.com> URL: https://gitlist.dev/e/40aa078e0912270439i3948a38fw9784e23e7cdc4407%40mail.gmail.com In-Reply-To: <20091227061121.GP14941@gradx.cs.jhu.edu> ``` On Sun, Dec 27, 2009 at 7:11 AM, Nathaniel W Filardo wrote: > I got some free time and tracked it down.  The following one-line delta > fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in You mean 8 bytes, right? -- Erik "kusma" Faye-Lund ``` ## Nathaniel W Filardo, 2009-12-27 16:12 Subject: Re: [RESEND] [PATCH] Endianness bug in index cache logic Message-ID: <20091227161246.GQ14941@gradx.cs.jhu.edu> URL: https://gitlist.dev/e/20091227161246.GQ14941%40gradx.cs.jhu.edu In-Reply-To: <40aa078e0912270439i3948a38fw9784e23e7cdc4407@mail.gmail.com> ``` On Sun, Dec 27, 2009 at 01:39:24PM +0100, Erik Faye-Lund wrote: > On Sun, Dec 27, 2009 at 7:11 AM, Nathaniel W Filardo wrote: > > I got some free time and tracked it down.  The following one-line delta > > fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in > > You mean 8 bytes, right? Yes, sorry. How embarrassing. :) --nwf; ``` ## Junio C Hamano, 2009-12-27 18:24 Subject: Re: [RESEND] [PATCH] Endianness bug in index cache logic Message-ID: <7veimgxolb.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7veimgxolb.fsf%40alter.siamese.dyndns.org In-Reply-To: <20091227061121.GP14941@gradx.cs.jhu.edu> ``` Nathaniel W Filardo writes: > I got some free time and tracked it down. The following one-line delta > fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in > the wrong endianness for the memcpy trick to work as written? I could be > sligntly off in my assessment of the problem, tho'. > > index 1bbaf1c..9033dd3 100644 > --- a/read-cache.c > +++ b/read-cache.c > @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path) > * extension name (4-byte) and section length > * in 4-byte network byte order. > */ > - unsigned long extsize; > + uint32_t extsize; > memcpy(&extsize, (char *)mmap + src_offset + 4, 4); > extsize = ntohl(extsize); > if (read_index_extension(istate, Good catch. The original is broken on big endian 64-bit platforms, and your sparc64 indeed is one. The code expects to see a signature (4-byte) at src_offset followed by length (4-byte int in network byte order) and it is trying to read read the latter in extsize. Thanks for the fix. The bug dates back to late April 2006, and I am kind of surprised that nobody reported this since then (perhaps nobody runs git on big endian 64-bit boxes?). ``` ## Tomas Carnecky, 2009-12-27 19:12 Subject: Re: [RESEND] [PATCH] Endianness bug in index cache logic Message-ID: <4B37B1A5.5030209@dbservice.com> URL: https://gitlist.dev/e/4B37B1A5.5030209%40dbservice.com In-Reply-To: <7veimgxolb.fsf@alter.siamese.dyndns.org> ``` On 12/27/09 7:24 PM, Junio C Hamano wrote: > Nathaniel W Filardo writes: > >> I got some free time and tracked it down. The following one-line delta >> fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in >> the wrong endianness for the memcpy trick to work as written? I could be >> sligntly off in my assessment of the problem, tho'. >> >> index 1bbaf1c..9033dd3 100644 >> --- a/read-cache.c >> +++ b/read-cache.c >> @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path) >> * extension name (4-byte) and section length >> * in 4-byte network byte order. >> */ >> - unsigned long extsize; >> + uint32_t extsize; >> memcpy(&extsize, (char *)mmap + src_offset + 4, 4); >> extsize = ntohl(extsize); >> if (read_index_extension(istate, > > Good catch. > > The original is broken on big endian 64-bit platforms, and your sparc64 > indeed is one. The code expects to see a signature (4-byte) at src_offset > followed by length (4-byte int in network byte order) and it is trying to > read read the latter in extsize. > > Thanks for the fix. The bug dates back to late April 2006, and I am kind > of surprised that nobody reported this since then (perhaps nobody runs git > on big endian 64-bit boxes?). Both the native Sun compiler as well as GCC default to 32bit binaries, even if the system is capable of running 64bit binaries (unlike for example Linux where x86_64-pc-linux-gnu-gcc produces 64bit binaries by default). And the git makefile doesn't use -m64, so my guess is that nobody bothered changing CFLAGS on sparc64 systems (I have access to a couple such systems and I never changed CFLAGS because git always worked out of the box). Though I don't know what the default is on other big endian systems. tom ```