git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] Properly align memory allocations and temporary buffers

From
Jessica Clarke <jrtc27@jrtc27.com>
Date
Jan 12, 2022, 13:58 UTC
Message-ID
<641A33F3-E98A-40B1-BB11-ADC8C1FECA75@jrtc27.com>
In-Reply-To
<20220105132324.6651-1-jrtc27@jrtc27.com>
On 5 Jan 2022, at 13:23, Jessica Clarke <jrtc27@jrtc27.com> wrote:
Show 47 quoted lines
> 
> Currently git_qsort_s allocates a buffer on the stack that has no
> alignment, and mem_pool_alloc assumes uintmax_t's size is adequate
> alignment for any type.
> 
> On CHERI, and thus Arm's Morello prototype, pointers are implemented as
> hardware capabilities which, as well as having a normal integer address,
> have additional bounds, permissions and other metadata in a second word,
> so on a 64-bit architecture they are 128-bit quantities, including their
> alignment requirements. Despite being 128-bit, their integer component
> is still only a 64-bit field, so uintmax_t remains 64-bit, and therefore
> uintmax_t does not sufficiently align an allocation.
> 
> Moreover, these capabilities have an additional "129th" tag bit, which
> tracks the validity of the capability and is cleared on any invalid
> operation that doesn't trap (e.g. partially overwriting a capability
> will invalidate it) which, combined with the architecture's strict
> checks on capability manipulation instructions, ensures it is
> architecturally impossible to construct a capability that gives more
> rights than those you were given in the first place. To store these tag
> bits, each capability sized and aligned word in memory gains a single
> tag bit that is stored in unaddressable (to the processor) memory. This
> means that it is impossible to store a capability at an unaligned
> address: a normal load or store of a capability will always take an
> alignment fault even if the (micro)architecture supports unaligned
> loads/stores for other data types, and a memcpy will, if the destination
> is not appropriately aligned, copy the byte representation but lose the
> tag, meaning that if it is eventually copied back and loaded from an
> aligned location any attempt to dereference it will trap with a tag
> fault. Thus, even char buffers that are memcpy'ed to or from must be
> properly aligned on CHERI architectures if they are to hold pointers.
> 
> Address both of these by introducing a new git_max_align type put in a
> union with the on-stack buffer to force its alignment, as well as a new
> GIT_MAX_ALIGNMENT macro whose value is the alignment of git_max_align
> that gets used for mem_pool_alloc. As well as making the code work on
> CHERI, the former change likely also improves performance on some
> architectures by making memcpy faster (either because it can use larger
> block sizes or because the microarchitecture has inefficient unaligned
> accesses).
> 
> Signed-off-by: Jessica Clarke <jrtc27@jrtc27.com>
> ---
> compat/qsort_s.c  | 11 +++++++----
> git-compat-util.h | 11 +++++++++++
> mem-pool.c        |  6 +++---
> 3 files changed, 21 insertions(+), 7 deletions(-)

I can see the alternative fixes for qsort are now in next, as is the cleanup of register_symlink_changes to use string sets, which just leaves the mem-pool.c change to not assume that uintmax_t is sufficiently aligned for every type that git will use. If I move GIT_MAX_ALIGNMENT and its helper aggregates to mem-pool.c would that be acceptable?

Jess
Show 70 quoted lines
> diff --git a/compat/qsort_s.c b/compat/qsort_s.c
> index 52d1f0a73d..1ccdb87451 100644
> --- a/compat/qsort_s.c
> +++ b/compat/qsort_s.c
> @@ -49,16 +49,19 @@ int git_qsort_s(void *b, size_t n, size_t s,
> 		int (*cmp)(const void *, const void *, void *), void *ctx)
> {
> 	const size_t size = st_mult(n, s);
> -	char buf[1024];
> +	union {
> +		char buf[1024];
> +		git_max_align align;
> +	} u;
> 
> 	if (!n)
> 		return 0;
> 	if (!b || !cmp)
> 		return -1;
> 
> -	if (size < sizeof(buf)) {
> -		/* The temporary array fits on the small on-stack buffer. */
> -		msort_with_tmp(b, n, s, cmp, buf, ctx);
> +	if (size < sizeof(u.buf)) {
> +		/* The temporary array fits in the small on-stack buffer. */
> +		msort_with_tmp(b, n, s, cmp, u.buf, ctx);
> 	} else {
> 		/* It's somewhat large, so malloc it.  */
> 		char *tmp = xmalloc(size);
> diff --git a/git-compat-util.h b/git-compat-util.h
> index 5fa54a7afe..28581a45c5 100644
> --- a/git-compat-util.h
> +++ b/git-compat-util.h
> @@ -274,6 +274,17 @@ typedef unsigned long uintptr_t;
> #define _ALL_SOURCE 1
> #endif
> 
> +typedef union {
> +	uintmax_t max_align_uintmax;
> +	void *max_align_pointer;
> +} git_max_align;
> +
> +typedef struct {
> +	char unalign;
> +	git_max_align aligned;
> +} git_max_alignment;
> +#define GIT_MAX_ALIGNMENT offsetof(git_max_alignment, aligned)
> +
> /* used on Mac OS X */
> #ifdef PRECOMPOSE_UNICODE
> #include "compat/precompose_utf8.h"
> diff --git a/mem-pool.c b/mem-pool.c
> index ccdcad2e3d..748eff925a 100644
> --- a/mem-pool.c
> +++ b/mem-pool.c
> @@ -69,9 +69,9 @@ void *mem_pool_alloc(struct mem_pool *pool, size_t len)
> 	struct mp_block *p = NULL;
> 	void *r;
> 
> -	/* round up to a 'uintmax_t' alignment */
> -	if (len & (sizeof(uintmax_t) - 1))
> -		len += sizeof(uintmax_t) - (len & (sizeof(uintmax_t) - 1));
> +	/* round up to a 'GIT_MAX_ALIGNMENT' alignment */
> +	if (len & (GIT_MAX_ALIGNMENT - 1))
> +		len += GIT_MAX_ALIGNMENT - (len & (GIT_MAX_ALIGNMENT - 1));
> 
> 	if (pool->mp_block &&
> 	    pool->mp_block->end - pool->mp_block->next_free >= len)
> -- 
> 2.33.1
> 
Previous: Jessica ClarkeNext: René Scharfe
Message 24 of 32 in “Properly align memory allocations and temporary buffers”
  1. Properly align memory allocations and temporary buffersJessica Clarke, Jan 5, 2022
  2. Taylor BlauJan 6, 2022
  3. Jessica ClarkeJan 6, 2022
  4. Junio C HamanoJan 6, 2022
  5. Jessica ClarkeJan 6, 2022
  6. Junio C HamanoJan 7, 2022
  7. Jessica ClarkeJan 7, 2022
  8. brian m. carlsonJan 7, 2022
  9. Jessica ClarkeJan 7, 2022
  10. brian m. carlsonJan 7, 2022
  11. Jessica ClarkeJan 7, 2022
  12. Jessica ClarkeJan 7, 2022
  13. Junio C HamanoJan 7, 2022
  14. Jessica ClarkeJan 7, 2022
  15. René ScharfeJan 7, 2022
  16. Junio C HamanoJan 7, 2022
  17. René ScharfeJan 7, 2022
  18. Elijah NewrenJan 8, 2022
  19. brian m. carlsonJan 6, 2022
  20. Jessica ClarkeJan 6, 2022
  21. Philip OakleyJan 7, 2022
  22. René ScharfeJan 7, 2022
  23. Jessica ClarkeJan 7, 2022
  24. Jessica ClarkeJan 12, 2022
  25. René ScharfeJan 12, 2022
  26. Jessica ClarkeJan 12, 2022
  27. mem-pool: Don't assume uintmax_t is aligned enough for all typesJessica Clarke, Jan 23, 2022
  28. Junio C HamanoJan 23, 2022
  29. Jessica ClarkeJan 23, 2022
  30. Junio C HamanoJan 23, 2022
  31. mem-pool: Don't assume uintmax_t is aligned enough for all typesJessica Clarke, Jan 23, 2022
  32. Junio C HamanoJan 24, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.