threads / bug / 21862

Endianness bug in git-svn or called component?

Subject: Endianness bug in git-svn or called component?

## tl;dr

9 messages between Dec 4, 2009 and Dec 27, 2009.

replies: 8people: 5as markdown or json

Nathaniel W Filardo· Dec 4, 2009, 17:44 UTC · lore
On this machine,

mirrors hydra:/tank0/mirrors/misc% uname -a FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc64

attempting to fetch from an svn source yields the following error:
rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
        A       scripts/distributedLM/config.template       
[...]
        A       build.xml
r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
error: index uses  extension, which we do not understand
fatal: index file corrupt
write-tree: command returned error: 128

Even on previously initialized and fetched git-svn repos, "git svn fetch" produces the same response.

This is the same symptom as an older bug, but I have not attempted to track down what's going wrong this time in hopes that somebody more familiar can fix it faster; see http://kerneltrap.org/mailarchive/git/2006/5/28/205750/thread#mid-205750 .

If there's anything more that would be helpful to know, please just ask.

Thanks. --nwf;

Andreas Schwab· Dec 4, 2009, 20:16 UTC · re: Nathaniel W Filardo · lore

Re: Endianness bug in git-svn or called component?

Nathaniel W Filardo <nwf@cs.jhu.edu> writes:
Show 18 quoted lines
> On this machine,
>
> mirrors hydra:/tank0/mirrors/misc% uname -a
> FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN  sparc64
>
> attempting to fetch from an svn source yields the following error:
>
> rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
> Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
> mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
> mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
>         A       scripts/distributedLM/config.template       
> [...]
>         A       build.xml
> r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
> error: index uses  extension, which we do not understand
> fatal: index file corrupt
> write-tree: command returned error: 128
I could not reproduce that on powerpc (both 32bit and 64bit).
Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
Nathaniel W Filardo· Dec 4, 2009, 20:29 UTC · re: Andreas Schwab · lore

Re: Endianness bug in git-svn or called component?

On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote:
Show 24 quoted lines
> Nathaniel W Filardo <nwf@cs.jhu.edu> writes:
> 
> > On this machine,
> >
> > mirrors hydra:/tank0/mirrors/misc% uname -a
> > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN  sparc64
> >
> > attempting to fetch from an svn source yields the following error:
> >
> > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
> > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
> > mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
> > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
> >         A       scripts/distributedLM/config.template       
> > [...]
> >         A       build.xml
> > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
> > error: index uses  extension, which we do not understand
> > fatal: index file corrupt
> > write-tree: command returned error: 128
> 
> I could not reproduce that on powerpc (both 32bit and 64bit).
> 
> Andreas.
Hm.  I seem to have forgotten to mention that I am running

% git --version git version 1.6.5.3

built from the FreeBSD ports tree, in case that matters, using

% gcc --version gcc (GCC) 4.2.1 20070719 [FreeBSD]

Knowing nothing of git's internals, how should I start to investigate what's going on here?

Thanks again. --nwf;

Nathaniel W Filardo· Dec 18, 2009, 09:05 UTC · re: Nathaniel W Filardo · lore

Re: Endianness bug in git-svn or called component?

On Fri, Dec 04, 2009 at 03:29:28PM -0500, Nathaniel W Filardo wrote:
Show 25 quoted lines
> On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote:
> > Nathaniel W Filardo <nwf@cs.jhu.edu> writes:
> > 
> > > On this machine,
> > >
> > > mirrors hydra:/tank0/mirrors/misc% uname -a
> > > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN  sparc64
> > >
> > > attempting to fetch from an svn source yields the following error:
> > >
> > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
> > > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
> > > mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
> > > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
> > >         A       scripts/distributedLM/config.template       
> > > [...]
> > >         A       build.xml
> > > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
> > > error: index uses  extension, which we do not understand
> > > fatal: index file corrupt
> > > write-tree: command returned error: 128
> > 
> > I could not reproduce that on powerpc (both 32bit and 64bit).
> > 
> > Andreas.

I got some free time and tracked it down. The following one-line delta fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in the wrong endianness for the memcpy trick to work as written? I could be sligntly off in my assessment of the problem, tho'.

index 1bbaf1c..9033dd3 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)
                 * extension name (4-byte) and section length
                 * in 4-byte network byte order.
                 */
-               unsigned long extsize;
+               uint32_t extsize;
                memcpy(&extsize, (char *)mmap + src_offset + 4, 4);
                extsize = ntohl(extsize);
                if (read_index_extension(istate,

--nwf;
Nathaniel W Filardo· Dec 27, 2009, 06:11 UTC · re: Nathaniel W Filardo · lore

[RESEND] [PATCH] Endianness bug in index cache logic

On Fri, Dec 04, 2009 at 03:29:28PM -0500, Nathaniel W Filardo wrote:
Show 25 quoted lines
> On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote:
> > Nathaniel W Filardo <nwf@cs.jhu.edu> writes:
> > 
> > > On this machine,
> > >
> > > mirrors hydra:/tank0/mirrors/misc% uname -a
> > > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN  sparc64
> > >
> > > attempting to fetch from an svn source yields the following error:
> > >
> > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
> > > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
> > > mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
> > > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
> > >         A       scripts/distributedLM/config.template       
> > > [...]
> > >         A       build.xml
> > > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
> > > error: index uses  extension, which we do not understand
> > > fatal: index file corrupt
> > > write-tree: command returned error: 128
> > 
> > I could not reproduce that on powerpc (both 32bit and 64bit).
> > 
> > Andreas.

I got some free time and tracked it down. The following one-line delta fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in the wrong endianness for the memcpy trick to work as written? I could be sligntly off in my assessment of the problem, tho'.

index 1bbaf1c..9033dd3 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)
                 * extension name (4-byte) and section length
                 * in 4-byte network byte order.
                 */
-               unsigned long extsize;
+               uint32_t extsize;
                memcpy(&extsize, (char *)mmap + src_offset + 4, 4);
                extsize = ntohl(extsize);
                if (read_index_extension(istate,

--nwf;
Erik Faye-Lund· Dec 27, 2009, 12:39 UTC · re: Nathaniel W Filardo · lore

Re: [RESEND] [PATCH] Endianness bug in index cache logic

On Sun, Dec 27, 2009 at 7:11 AM, Nathaniel W Filardo <nwf@cs.jhu.edu> wrote:
> I got some free time and tracked it down.  The following one-line delta
> fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in
You mean 8 bytes, right?
-- 
Erik "kusma" Faye-Lund
Nathaniel W Filardo· Dec 27, 2009, 16:12 UTC · re: Erik Faye-Lund · lore

Re: [RESEND] [PATCH] Endianness bug in index cache logic

On Sun, Dec 27, 2009 at 01:39:24PM +0100, Erik Faye-Lund wrote:
Show 5 quoted lines
> On Sun, Dec 27, 2009 at 7:11 AM, Nathaniel W Filardo <nwf@cs.jhu.edu> wrote:
> > I got some free time and tracked it down.  The following one-line delta
> > fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in
> 
> You mean 8 bytes, right?

Yes, sorry. How embarrassing. :) --nwf;

Junio C Hamano· Dec 27, 2009, 18:24 UTC · re: Nathaniel W Filardo · lore

Re: [RESEND] [PATCH] Endianness bug in index cache logic

Nathaniel W Filardo <nwf@cs.jhu.edu> writes:
Show 17 quoted lines
> I got some free time and tracked it down.  The following one-line delta
> fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in
> the wrong endianness for the memcpy trick to work as written?  I could be
> sligntly off in my assessment of the problem, tho'.
>
> index 1bbaf1c..9033dd3 100644
> --- a/read-cache.c
> +++ b/read-cache.c
> @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)
>                  * extension name (4-byte) and section length
>                  * in 4-byte network byte order.
>                  */
> -               unsigned long extsize;
> +               uint32_t extsize;
>                 memcpy(&extsize, (char *)mmap + src_offset + 4, 4);
>                 extsize = ntohl(extsize);
>                 if (read_index_extension(istate,
Good catch.

The original is broken on big endian 64-bit platforms, and your sparc64 indeed is one. The code expects to see a signature (4-byte) at src_offset followed by length (4-byte int in network byte order) and it is trying to read read the latter in extsize.

Thanks for the fix. The bug dates back to late April 2006, and I am kind of surprised that nobody reported this since then (perhaps nobody runs git on big endian 64-bit boxes?).

Tomas Carnecky· Dec 27, 2009, 19:12 UTC · re: Junio C Hamano · lore

Re: [RESEND] [PATCH] Endianness bug in index cache logic

On 12/27/09 7:24 PM, Junio C Hamano wrote:
Show 30 quoted lines
> Nathaniel W Filardo<nwf@cs.jhu.edu>  writes:
>
>> I got some free time and tracked it down.  The following one-line delta
>> fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in
>> the wrong endianness for the memcpy trick to work as written?  I could be
>> sligntly off in my assessment of the problem, tho'.
>>
>> index 1bbaf1c..9033dd3 100644
>> --- a/read-cache.c
>> +++ b/read-cache.c
>> @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)
>>                   * extension name (4-byte) and section length
>>                   * in 4-byte network byte order.
>>                   */
>> -               unsigned long extsize;
>> +               uint32_t extsize;
>>                  memcpy(&extsize, (char *)mmap + src_offset + 4, 4);
>>                  extsize = ntohl(extsize);
>>                  if (read_index_extension(istate,
>
> Good catch.
>
> The original is broken on big endian 64-bit platforms, and your sparc64
> indeed is one.  The code expects to see a signature (4-byte) at src_offset
> followed by length (4-byte int in network byte order) and it is trying to
> read read the latter in extsize.
>
> Thanks for the fix.  The bug dates back to late April 2006, and I am kind
> of surprised that nobody reported this since then (perhaps nobody runs git
> on big endian 64-bit boxes?).

Both the native Sun compiler as well as GCC default to 32bit binaries, even if the system is capable of running 64bit binaries (unlike for example Linux where x86_64-pc-linux-gnu-gcc produces 64bit binaries by default). And the git makefile doesn't use -m64, so my guess is that nobody bothered changing CFLAGS on sparc64 systems (I have access to a couple such systems and I never changed CFLAGS because git always worked out of the box). Though I don't know what the default is on other big endian systems.

tom

← back to recent threads