git 1.9.0 segfault

> On Sat, Mar 08, 2014 at 04:23:43PM +0000, Guillaume Gelin wrote:
> > Hi,
> >
> > http://pastebin.com/Np7L54ar
> We're failing to rename because we got an EFAULT, and then we try to
> print the failing filename, and we get a segfault right here:
> 
> 			if (rename(src, dst) < 0 && !ignore_errors)
> 				die_errno (_("renaming '%s' failed"), src);
> 
> I don't know yet if dst is also bad, but clearly src is.  I'm looking
> into it.

> When commit a88c915 (mv: move submodules using a gitfile, 2013-07-30)
> added the submodule_gitfile array, it was not added to the block that
> enlarges the arrays when we are moving a directory so that we do not
> have to worry about it being a directory when we perform the actual
> move.  After this, the loop continues over the enlarged set of sources.
> 
> Since we assume that submodule_gitfile has size argc, if any of the
> items in the source directory are submodules we are guaranteed to write
> beyond the end of submodule_gitfile.
> 
> Fix this by realloc'ing submodule_gitfile at the same time as the other
> arrays.
> 
> Reported-by: Guillaume Gelin <contact@ramnes.eu>
> Signed-off-by: John Keeping <john@keeping.me.uk>
> ---
> On Sat, Mar 08, 2014 at 06:12:18PM +0000, John Keeping wrote:
> > This fixes it for me:
> 
> Here it is as a proper patch.
> 
>  builtin/mv.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/builtin/mv.c b/builtin/mv.c
> index 21c46d1..f99c91e 100644
> --- a/builtin/mv.c
> +++ b/builtin/mv.c
> @@ -179,6 +179,9 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
>  						modes = xrealloc(modes,
>  								(argc + last - first)
>  								* sizeof(enum update_mode));
> +						submodule_gitfile = xrealloc(submodule_gitfile,
> +								(argc + last - first)
> +								* sizeof(char *));
>  					}
>  
>  					dst = add_slash(dst);

> We shrink the source and destination arrays, but not the modes or
> submodule_gitfile arrays, resulting in potentially mismatched data.  Shrink
> all the arrays at the same time to prevent this.
> 
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  builtin/mv.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/builtin/mv.c b/builtin/mv.c
> index f99c91e..b20cd95 100644
> --- a/builtin/mv.c
> +++ b/builtin/mv.c
> @@ -230,6 +230,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
>  					memmove(destination + i,
>  						destination + i + 1,
>  						(argc - i) * sizeof(char *));
> +					memmove(modes + i, modes + i + 1,
> +						(argc - i) * sizeof(char *));
> +					memmove(submodule_gitfile + i,
> +						submodule_gitfile + i + 1,
> +						(argc - i) * sizeof(char *));

> On Sat, Mar 08, 2014 at 07:21:39PM +0000, brian m. carlson wrote:
> 
> > We shrink the source and destination arrays, but not the modes or
> > submodule_gitfile arrays, resulting in potentially mismatched data.  Shrink
> > all the arrays at the same time to prevent this.
> > 
> > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> > ---
> >  builtin/mv.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/builtin/mv.c b/builtin/mv.c
> > index f99c91e..b20cd95 100644
> > --- a/builtin/mv.c
> > +++ b/builtin/mv.c
> > @@ -230,6 +230,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
> >  					memmove(destination + i,
> >  						destination + i + 1,
> >  						(argc - i) * sizeof(char *));
> > +					memmove(modes + i, modes + i + 1,
> > +						(argc - i) * sizeof(char *));
> > +					memmove(submodule_gitfile + i,
> > +						submodule_gitfile + i + 1,
> > +						(argc - i) * sizeof(char *));
> 
> I haven't looked that closely, but would it be crazy to suggest that
> these arrays all be squashed into one array-of-struct? It would be less
> error prone and perhaps more readable.

> We shrink the source and destination arrays, but not the modes or
> submodule_gitfile arrays, resulting in potentially mismatched data.  Shrink
> all the arrays at the same time to prevent this.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  builtin/mv.c | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/builtin/mv.c b/builtin/mv.c
> index f99c91e..b20cd95 100644
> --- a/builtin/mv.c
> +++ b/builtin/mv.c
> @@ -230,6 +230,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
>  					memmove(destination + i,
>  						destination + i + 1,
>  						(argc - i) * sizeof(char *));
> +					memmove(modes + i, modes + i + 1,
> +						(argc - i) * sizeof(char *));
> +					memmove(submodule_gitfile + i,
> +						submodule_gitfile + i + 1,
> +						(argc - i) * sizeof(char *));
>  					i--;
>  				}
>  			} else

> We shrink the source and destination arrays, but not the modes or
> submodule_gitfile arrays, resulting in potentially mismatched data.  Shrink
> all the arrays at the same time to prevent this.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  builtin/mv.c | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/builtin/mv.c b/builtin/mv.c
> index f99c91e..b20cd95 100644
> --- a/builtin/mv.c
> +++ b/builtin/mv.c
> @@ -230,6 +230,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
>  					memmove(destination + i,
>  						destination + i + 1,
>  						(argc - i) * sizeof(char *));
> +					memmove(modes + i, modes + i + 1,
> +						(argc - i) * sizeof(char *));

> > diff --git a/builtin/mv.c b/builtin/mv.c
> > index f99c91e..b20cd95 100644
> > --- a/builtin/mv.c
> > +++ b/builtin/mv.c
> > @@ -230,6 +230,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
> >  					memmove(destination + i,
> >  						destination + i + 1,
> >  						(argc - i) * sizeof(char *));
> > +					memmove(modes + i, modes + i + 1,
> > +						(argc - i) * sizeof(char *));
> 
> This isn't right -- you are computing the size of things to be moved
> based on a type of char*, but 'modes' is an enum.
> 
> (Valgrind spotted this.)

> On Sat, Mar 15, 2014 at 05:05:29PM +0100, Thomas Rast wrote:
>
>> > diff --git a/builtin/mv.c b/builtin/mv.c
>> > index f99c91e..b20cd95 100644
>> > --- a/builtin/mv.c
>> > +++ b/builtin/mv.c
>> > @@ -230,6 +230,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
>> >  					memmove(destination + i,
>> >  						destination + i + 1,
>> >  						(argc - i) * sizeof(char *));
>> > +					memmove(modes + i, modes + i + 1,
>> > +						(argc - i) * sizeof(char *));
>> 
>> This isn't right -- you are computing the size of things to be moved
>> based on a type of char*, but 'modes' is an enum.
>> 
>> (Valgrind spotted this.)
>
> Maybe using sizeof(*destination) and sizeof(*modes) would make this less
> error-prone?
>
> -Peff

> Would it make sense to go one step further to introduce two macros
> to make this kind of screw-up less likely?
> ...
> After letting my eyes coast over hits from "git grep memmove", there
> do seem to be some places that these would help readability, but not
> very many.

> Junio C Hamano <gitster@pobox.com> writes:
> 
>> Would it make sense to go one step further to introduce two macros
>> to make this kind of screw-up less likely?
>> ...
>> After letting my eyes coast over hits from "git grep memmove", there
>> do seem to be some places that these would help readability, but not
>> very many.
> 
> I see quite a many hits that follow this pattern
> 
> 	memmove(array + pos, array + pos + 1, sizeof(*array) * (nr - pos))
> 
> to make a single slot in a middle of array available, which would be
> good candidates to use MOVE_DOWN().  Just to show a few:
> 
> builtin/mv.c:226:	memmove(source + i, source + i + 1,
> builtin/mv.c-227-		(argc - i) * sizeof(char *));
> builtin/mv.c:228:	memmove(destination + i,
> builtin/mv.c-229-		destination + i + 1,
> builtin/mv.c-230-		(argc - i) * sizeof(char *));
> cache-tree.c:92:	memmove(it->down + pos + 1,
> cache-tree.c-93-		it->down + pos,
> cache-tree.c-94-		sizeof(down) * (it->subtree_nr - pos - 1));
> 
> 
> Perhaps something like this patch to start off; I am not sure
> MOVE_DOWN_BOUNDED is needed, though.
> 
>  cache.h | 33 +++++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
> 
> diff --git a/cache.h b/cache.h
> index b66cb49..b2615ab 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -455,6 +455,39 @@ extern int daemonize(void);
>  		} \
>  	} while (0)
>  
> +/*
> + * With an array "array" that currently holds "nr" elements, move
> + * elements at "at" and later down by "count" elements to make room to
> + * add in new elements.  The caller is responsible for making sure
> + * that the array has enough room to hold "nr" + "count" slots.
> + */
> +#define MOVE_DOWN(array, nr, at, count)			\
> +	memmove((array) + (at) + (count),		\
> +		(array) + (at),				\
> +		sizeof((array)[0]) * ((nr) - (at)))
> +
> +/*
> + * With an array "array" that has enough memory to hold "alloc"
> + * elements allocated and currently holds "nr" elements, move elements
> + * at "at" and later down by "count" elements to make room to add in
> + * new elements.
> + */
> +#define MOVE_DOWN_BOUNDED(array, nr, at, count, alloc)		     \
> +	do {							     \
> +		if ((alloc) <= (nr) + (count))			     \
> +			BUG("MOVE_DOWN beyond the end of an array"); \
> +		MOVE_DOWN((array), (nr), (at), (count));	     \
> +	} while (0)
> +
> +/*
> + * With an array "array" that curently holds "nr" elements, move elements
> + * at "at" + "count" and later down by "count" elements, removing the
> + * elements between "at" and "at" + "count".
> + */
> +#define MOVE_UP(array, nr, at, count)				\
> +	memmove((array) + (at), (array) + (at) + (count),	\
> +		sizeof((array)[0]) * ((nr) - ((at) + (count))))
> +
>  /* Initialize and use the cache information */
>  extern int read_index(struct index_state *);
>  extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);

> On 03/17/2014 07:33 AM, Junio C Hamano wrote:
>> Junio C Hamano <gitster@pobox.com> writes:
>>
>>> Would it make sense to go one step further to introduce two macros
>>> to make this kind of screw-up less likely?
> potential callers that I noticed, ALLOC_GROW() was used immediately
> before making space in the array for a new element.  So I suggest
> something more like
>
> +#define MOVE_DOWN(array, nr, at, count)        \
> +       memmove((array) + (at) + (count),               \
> +               (array) + (at),                         \
> +               sizeof((array)[0]) * ((nr) - (at)))

> +#define ALLOC_INSERT_GAP(array, nr, at, count, alloc)               \
> +       do {                                                         \
> +               ALLOC_GROW((array), (nr) + (count), (alloc));        \
> +               MOVE_DOWN((array), (nr), (at), (count));             \
> +       } while (0)
>
> Also, count==1 is so frequent that this special case might deserve its
> own macro pair.
>
> I'm not inspired by these macro names, though.
>
> Michael
>
> --
> Michael Haggerty
> mhagger@alum.mit.edu
> http://softwareswirl.blogspot.com/

> On Mon, Mar 17, 2014 at 11:07 AM, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> > On 03/17/2014 07:33 AM, Junio C Hamano wrote:
> >> Junio C Hamano <gitster@pobox.com> writes:
> >>
> >>> Would it make sense to go one step further to introduce two macros
> >>> to make this kind of screw-up less likely?
> > potential callers that I noticed, ALLOC_GROW() was used immediately
> > before making space in the array for a new element.  So I suggest
> > something more like
> >
> > +#define MOVE_DOWN(array, nr, at, count)        \
> > +       memmove((array) + (at) + (count),               \
> > +               (array) + (at),                         \
> > +               sizeof((array)[0]) * ((nr) - (at)))
> 
> Each time I read these, my brain (for whatever reason) interprets the
> names UP and DOWN opposite of the intended meaning, which makes them
> confusing. Perhaps INSERT_GAP and CLOSE_GAP would avoid such problems,
> and be more consistent with Michael's proposed ALLOC_INSERT_GAP.

> I had recently been thinking along the same lines.  In many of the
> potential callers that I noticed, ALLOC_GROW() was used immediately
> before making space in the array for a new element.  So I suggest
> something more like
>
> +#define MOVE_DOWN(array, nr, at, count)	\
> +	memmove((array) + (at) + (count),		\
> +		(array) + (at),				\
> +		sizeof((array)[0]) * ((nr) - (at)))
> +#define ALLOC_INSERT_GAP(array, nr, at, count, alloc)		     \
> +	do {							     \
> +		ALLOC_GROW((array), (nr) + (count), (alloc));        \
> +		MOVE_DOWN((array), (nr), (at), (count));	     \
> +	} while (0)
>
> Also, count==1 is so frequent that this special case might deserve its
> own macro pair.

> We shrink the source and destination arrays, but not the modes or
> submodule_gitfile arrays, resulting in potentially mismatched data.  Shrink
> all the arrays at the same time to prevent this.  Add tests to ensure the
> problem does not recur.
> 
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> 
> I attempted to come up with a second patch that would refactor out the
> four different arrays into one array of struct, as Jeff suggested, but
> it became very ugly very quickly.  So this patch simply fixes the
> problem and adds tests.