{"thread":{"id":"58081","subject":"[PATCH 0/3] xdiff: introduce memory allocation macros","startedAt":"2022-06-29T15:25:55Z","lastAt":"2022-07-13T13:24:21Z","messageCount":52,"participants":["Phillip Wood via GitGitGadget","Ævar Arnfjörð Bjarmason","Phillip Wood","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"458099","messageId":"pull.1272.git.1656516334.gitgitgadget@gmail.com","threadId":"58081","inReplyTo":null,"subject":"[PATCH 0/3] xdiff: introduce memory allocation macros","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-29T15:25:31Z","receivedAt":"2022-06-29T15:25:55Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"This patch series introduces macros for allocating and growing arrays in\nxdiff. The macros are similar to ALLOC_ARRAY()/ALLOC_GROW() from the rest of\nthe code base but return an error on failure to allow libgit2 to handle\nmemory allocation failures gracefully rather than dying. The macros\nintroduce overflow checks but these checks are currently redundant as we\nlimit the maximum file size passed to xdiff and these checks alone are\ninsufficient to safely remove the size limit. The aim of this series is to\nmake the xdiff code more readable, there should be no change in behavior (as\nsuch I'm open to the argument that these are just churn and should be\ndropped).\n\nPhillip Wood (3):\n  xdiff: introduce XDL_ALLOC_ARRAY()\n  xdiff: introduce XDL_CALLOC_ARRAY()\n  xdiff: introduce XDL_ALLOC_GROW()\n\n xdiff/xdiffi.c     |  2 +-\n xdiff/xhistogram.c | 19 ++++++-------------\n xdiff/xmacros.h    | 21 +++++++++++++++++++++\n xdiff/xpatience.c  |  9 +++------\n xdiff/xprepare.c   | 41 ++++++++++++-----------------------------\n xdiff/xutils.c     | 17 +++++++++++++++++\n xdiff/xutils.h     |  3 ++-\n 7 files changed, 62 insertions(+), 50 deletions(-)\n\n\nbase-commit: e4a4b31577c7419497ac30cebe30d755b97752c5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1272%2Fphillipwood%2Fwip%2Fxdiff-memory-allocation-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1272/phillipwood/wip/xdiff-memory-allocation-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1272\n-- \ngitgitgadget\n"},{"id":"458100","messageId":"55fd62dc27d2bccfdb8ac300be23fc33d1795366.1656516334.git.gitgitgadget@gmail.com","threadId":"58081","inReplyTo":"pull.1272.git.1656516334.gitgitgadget@gmail.com","subject":"[PATCH 1/3] xdiff: introduce XDL_ALLOC_ARRAY()","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-29T15:25:32Z","receivedAt":"2022-06-29T15:25:57Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood <phillip.wood@dunelm.org.uk>\n\nAdd a helper to allocate an array that automatically calculates the\nallocation size. This is analogous to ALLOC_ARRAY() in the rest of the\ncodebase but returns NULL if the allocation fails to accommodate other\nusers of libxdiff such as libgit2. The helper will also return NULL if\nthe multiplication in the allocation calculation overflows.\n\nSigned-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>\n---\n xdiff/xdiffi.c    | 2 +-\n xdiff/xmacros.h   | 5 +++++\n xdiff/xpatience.c | 4 ++--\n xdiff/xprepare.c  | 8 ++++----\n 4 files changed, 12 insertions(+), 7 deletions(-)\n\ndiff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\nindex 758410c11ac..53e803e6bcb 100644\n--- a/xdiff/xdiffi.c\n+++ b/xdiff/xdiffi.c\n@@ -337,7 +337,7 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n \t * One is to store the forward path and one to store the backward path.\n \t */\n \tndiags = xe->xdf1.nreff + xe->xdf2.nreff + 3;\n-\tif (!(kvd = (long *) xdl_malloc((2 * ndiags + 2) * sizeof(long)))) {\n+\tif (!XDL_ALLOC_ARRAY(kvd, 2 * ndiags + 2)) {\n \n \t\txdl_free_env(xe);\n \t\treturn -1;\ndiff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\nindex ae4636c2477..9fd3c5da91a 100644\n--- a/xdiff/xmacros.h\n+++ b/xdiff/xmacros.h\n@@ -49,5 +49,10 @@ do { \\\n \t\t((unsigned long) __p[2]) << 16 | ((unsigned long) __p[3]) << 24; \\\n } while (0)\n \n+/* Allocate an array of nr elements, returns NULL on failure */\n+#define XDL_ALLOC_ARRAY(p, nr)\t\t\t\t\\\n+\t((p) = SIZE_MAX / sizeof(*(p)) >= (size_t)(nr)\t\\\n+\t\t? xdl_malloc((nr) * sizeof(*(p)))\t\\\n+\t\t: NULL)\n \n #endif /* #if !defined(XMACROS_H) */\ndiff --git a/xdiff/xpatience.c b/xdiff/xpatience.c\nindex 1a21c6a74b3..ce87b9084ca 100644\n--- a/xdiff/xpatience.c\n+++ b/xdiff/xpatience.c\n@@ -200,7 +200,7 @@ static int binary_search(struct entry **sequence, int longest,\n  */\n static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n {\n-\tstruct entry **sequence = xdl_malloc(map->nr * sizeof(struct entry *));\n+\tstruct entry **sequence;\n \tint longest = 0, i;\n \tstruct entry *entry;\n \n@@ -211,7 +211,7 @@ static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n \t */\n \tint anchor_i = -1;\n \n-\tif (!sequence)\n+\tif (!XDL_ALLOC_ARRAY(sequence, map->nr))\n \t\treturn -1;\n \n \tfor (entry = map->first; entry; entry = entry->next) {\ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex 105752758f2..25866a1667a 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -86,7 +86,7 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n \tmemset(cf->rchash, 0, cf->hsize * sizeof(xdlclass_t *));\n \n \tcf->alloc = size;\n-\tif (!(cf->rcrecs = (xdlclass_t **) xdl_malloc(cf->alloc * sizeof(xdlclass_t *)))) {\n+\tif (!XDL_ALLOC_ARRAY(cf->rcrecs, cf->alloc)) {\n \n \t\txdl_free(cf->rchash);\n \t\txdl_cha_free(&cf->ncha);\n@@ -178,7 +178,7 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \n \tif (xdl_cha_init(&xdf->rcha, sizeof(xrecord_t), narec / 4 + 1) < 0)\n \t\tgoto abort;\n-\tif (!(recs = (xrecord_t **) xdl_malloc(narec * sizeof(xrecord_t *))))\n+\tif (!XDL_ALLOC_ARRAY(recs, narec))\n \t\tgoto abort;\n \n \thbits = xdl_hashbits((unsigned int) narec);\n@@ -215,9 +215,9 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \n \tif ((XDF_DIFF_ALG(xpp->flags) != XDF_PATIENCE_DIFF) &&\n \t    (XDF_DIFF_ALG(xpp->flags) != XDF_HISTOGRAM_DIFF)) {\n-\t\tif (!(rindex = xdl_malloc((nrec + 1) * sizeof(*rindex))))\n+\t\tif (!XDL_ALLOC_ARRAY(rindex, nrec + 1))\n \t\t\tgoto abort;\n-\t\tif (!(ha = xdl_malloc((nrec + 1) * sizeof(*ha))))\n+\t\tif (!XDL_ALLOC_ARRAY(ha, nrec + 1))\n \t\t\tgoto abort;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"458101","messageId":"8bead9856be7b39d3d03f688f53d678832d60109.1656516334.git.gitgitgadget@gmail.com","threadId":"58081","inReplyTo":"pull.1272.git.1656516334.gitgitgadget@gmail.com","subject":"[PATCH 2/3] xdiff: introduce XDL_CALLOC_ARRAY()","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-29T15:25:33Z","receivedAt":"2022-06-29T15:25:58Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood <phillip.wood@dunelm.org.uk>\n\nAdd a helper for allocating an array and initialize the elements to\nzero. This is analogous to CALLOC_ARRAY() in the rest of the codebase\nbut it returns NULL on allocation failures rather than dying to\naccommodate other users of libxdiff such as libgit2.\n\nSigned-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>\n---\n xdiff/xhistogram.c | 19 ++++++-------------\n xdiff/xmacros.h    |  6 ++++++\n xdiff/xpatience.c  |  5 +----\n xdiff/xprepare.c   | 14 ++++----------\n 4 files changed, 17 insertions(+), 27 deletions(-)\n\ndiff --git a/xdiff/xhistogram.c b/xdiff/xhistogram.c\nindex 01decffc332..df909004c10 100644\n--- a/xdiff/xhistogram.c\n+++ b/xdiff/xhistogram.c\n@@ -251,7 +251,7 @@ static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n \t\t    int line1, int count1, int line2, int count2)\n {\n \tint b_ptr;\n-\tint sz, ret = -1;\n+\tint ret = -1;\n \tstruct histindex index;\n \n \tmemset(&index, 0, sizeof(index));\n@@ -265,23 +265,16 @@ static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n \tindex.rcha.head = NULL;\n \n \tindex.table_bits = xdl_hashbits(count1);\n-\tsz = index.records_size = 1 << index.table_bits;\n-\tsz *= sizeof(struct record *);\n-\tif (!(index.records = (struct record **) xdl_malloc(sz)))\n+\tindex.records_size = 1 << index.table_bits;\n+\tif (!XDL_CALLOC_ARRAY(index.records, index.records_size))\n \t\tgoto cleanup;\n-\tmemset(index.records, 0, sz);\n \n-\tsz = index.line_map_size = count1;\n-\tsz *= sizeof(struct record *);\n-\tif (!(index.line_map = (struct record **) xdl_malloc(sz)))\n+\tindex.line_map_size = count1;\n+\tif (!XDL_CALLOC_ARRAY(index.line_map, index.line_map_size))\n \t\tgoto cleanup;\n-\tmemset(index.line_map, 0, sz);\n \n-\tsz = index.line_map_size;\n-\tsz *= sizeof(unsigned int);\n-\tif (!(index.next_ptrs = (unsigned int *) xdl_malloc(sz)))\n+\tif (!XDL_CALLOC_ARRAY(index.next_ptrs, index.line_map_size))\n \t\tgoto cleanup;\n-\tmemset(index.next_ptrs, 0, sz);\n \n \t/* lines / 4 + 1 comes from xprepare.c:xdl_prepare_ctx() */\n \tif (xdl_cha_init(&index.rcha, sizeof(struct record), count1 / 4 + 1) < 0)\ndiff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\nindex 9fd3c5da91a..23db8e785d7 100644\n--- a/xdiff/xmacros.h\n+++ b/xdiff/xmacros.h\n@@ -55,4 +55,10 @@ do { \\\n \t\t? xdl_malloc((nr) * sizeof(*(p)))\t\\\n \t\t: NULL)\n \n+/* Allocate an array of nr zeroed out elements, returns NULL on failure */\n+#define XDL_CALLOC_ARRAY(p, nr)\t\t\t\t\\\n+\t(XDL_ALLOC_ARRAY((p), (nr))\t\t\t\\\n+\t\t? memset((p), 0, (nr) * sizeof(*(p)))\t\\\n+\t\t: NULL)\n+\n #endif /* #if !defined(XMACROS_H) */\ndiff --git a/xdiff/xpatience.c b/xdiff/xpatience.c\nindex ce87b9084ca..fe39c2978cb 100644\n--- a/xdiff/xpatience.c\n+++ b/xdiff/xpatience.c\n@@ -151,11 +151,8 @@ static int fill_hashmap(mmfile_t *file1, mmfile_t *file2,\n \n \t/* We know exactly how large we want the hash map */\n \tresult->alloc = count1 * 2;\n-\tresult->entries = (struct entry *)\n-\t\txdl_malloc(result->alloc * sizeof(struct entry));\n-\tif (!result->entries)\n+\tif (!XDL_CALLOC_ARRAY(result->entries, result->alloc))\n \t\treturn -1;\n-\tmemset(result->entries, 0, result->alloc * sizeof(struct entry));\n \n \t/* First, fill with entries from the first file */\n \twhile (count1--)\ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex 25866a1667a..b016570c488 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -78,12 +78,11 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n \n \t\treturn -1;\n \t}\n-\tif (!(cf->rchash = (xdlclass_t **) xdl_malloc(cf->hsize * sizeof(xdlclass_t *)))) {\n+\tif (!XDL_CALLOC_ARRAY(cf->rchash, cf->hsize)) {\n \n \t\txdl_cha_free(&cf->ncha);\n \t\treturn -1;\n \t}\n-\tmemset(cf->rchash, 0, cf->hsize * sizeof(xdlclass_t *));\n \n \tcf->alloc = size;\n \tif (!XDL_ALLOC_ARRAY(cf->rcrecs, cf->alloc)) {\n@@ -183,9 +182,8 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \n \thbits = xdl_hashbits((unsigned int) narec);\n \thsize = 1 << hbits;\n-\tif (!(rhash = (xrecord_t **) xdl_malloc(hsize * sizeof(xrecord_t *))))\n+\tif (!XDL_CALLOC_ARRAY(rhash, hsize))\n \t\tgoto abort;\n-\tmemset(rhash, 0, hsize * sizeof(xrecord_t *));\n \n \tnrec = 0;\n \tif ((cur = blk = xdl_mmfile_first(mf, &bsize))) {\n@@ -209,9 +207,8 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \t\t}\n \t}\n \n-\tif (!(rchg = (char *) xdl_malloc((nrec + 2) * sizeof(char))))\n+\tif (!XDL_CALLOC_ARRAY(rchg, nrec + 2))\n \t\tgoto abort;\n-\tmemset(rchg, 0, (nrec + 2) * sizeof(char));\n \n \tif ((XDF_DIFF_ALG(xpp->flags) != XDF_PATIENCE_DIFF) &&\n \t    (XDF_DIFF_ALG(xpp->flags) != XDF_HISTOGRAM_DIFF)) {\n@@ -383,11 +380,8 @@ static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd\n \txdlclass_t *rcrec;\n \tchar *dis, *dis1, *dis2;\n \n-\tif (!(dis = (char *) xdl_malloc(xdf1->nrec + xdf2->nrec + 2))) {\n-\n+\tif (!XDL_CALLOC_ARRAY(dis, xdf1->nrec + xdf2->nrec + 2))\n \t\treturn -1;\n-\t}\n-\tmemset(dis, 0, xdf1->nrec + xdf2->nrec + 2);\n \tdis1 = dis;\n \tdis2 = dis1 + xdf1->nrec + 1;\n \n-- \ngitgitgadget\n\n"},{"id":"458102","messageId":"da996677f0730ec7a50d399524fb58c44dad468a.1656516334.git.gitgitgadget@gmail.com","threadId":"58081","inReplyTo":"pull.1272.git.1656516334.gitgitgadget@gmail.com","subject":"[PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-29T15:25:34Z","receivedAt":"2022-06-29T15:26:08Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood <phillip.wood@dunelm.org.uk>\n\nAdd a helper to grow an array. This is analogous to ALLOC_GROW() in\nthe rest of the codebase but returns −1 on allocation failure to\naccommodate other users of libxdiff such as libgit2. It will also\nreturn a error if the multiplication overflows while calculating the\nnew allocation size. Note that this keeps doubling on reallocation\nlike the code it is replacing rather than increasing the existing size\nby half like ALLOC_GROW(). It does however copy ALLOC_GROW()'s trick\nof adding a small amount to the new allocation to avoid a lot of\nreallocations at small sizes.\n\nSigned-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>\n---\n xdiff/xmacros.h  | 10 ++++++++++\n xdiff/xprepare.c | 19 ++++---------------\n xdiff/xutils.c   | 17 +++++++++++++++++\n xdiff/xutils.h   |  3 ++-\n 4 files changed, 33 insertions(+), 16 deletions(-)\n\ndiff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\nindex 23db8e785d7..d13a6724629 100644\n--- a/xdiff/xmacros.h\n+++ b/xdiff/xmacros.h\n@@ -61,4 +61,14 @@ do { \\\n \t\t? memset((p), 0, (nr) * sizeof(*(p)))\t\\\n \t\t: NULL)\n \n+/*\n+ * Ensure array p can accommodate at least nr elements, growing the\n+ * array and updating alloc (which is the number of allocated\n+ * elements) as necessary. Frees p and returns -1 on failure, returns\n+ * 0 on success\n+ */\n+#define XDL_ALLOC_GROW(p, nr, alloc)\t\\\n+\t(-!((nr) <= (alloc) ||\t\t\\\n+\t    ((p) = xdl_alloc_grow_helper((p), (nr), &(alloc), sizeof(*(p))))))\n+\n #endif /* #if !defined(XMACROS_H) */\ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex b016570c488..c84549f6c50 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -111,7 +111,6 @@ static int xdl_classify_record(unsigned int pass, xdlclassifier_t *cf, xrecord_t\n \tlong hi;\n \tchar const *line;\n \txdlclass_t *rcrec;\n-\txdlclass_t **rcrecs;\n \n \tline = rec->ptr;\n \thi = (long) XDL_HASHLONG(rec->ha, cf->hbits);\n@@ -127,14 +126,8 @@ static int xdl_classify_record(unsigned int pass, xdlclassifier_t *cf, xrecord_t\n \t\t\treturn -1;\n \t\t}\n \t\trcrec->idx = cf->count++;\n-\t\tif (cf->count > cf->alloc) {\n-\t\t\tcf->alloc *= 2;\n-\t\t\tif (!(rcrecs = (xdlclass_t **) xdl_realloc(cf->rcrecs, cf->alloc * sizeof(xdlclass_t *)))) {\n-\n+\t\tif (XDL_ALLOC_GROW(cf->rcrecs, cf->count, cf->alloc))\n \t\t\t\treturn -1;\n-\t\t\t}\n-\t\t\tcf->rcrecs = rcrecs;\n-\t\t}\n \t\tcf->rcrecs[rcrec->idx] = rcrec;\n \t\trcrec->line = line;\n \t\trcrec->size = rec->size;\n@@ -163,7 +156,7 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \tunsigned long hav;\n \tchar const *blk, *cur, *top, *prev;\n \txrecord_t *crec;\n-\txrecord_t **recs, **rrecs;\n+\txrecord_t **recs;\n \txrecord_t **rhash;\n \tunsigned long *ha;\n \tchar *rchg;\n@@ -190,12 +183,8 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \t\tfor (top = blk + bsize; cur < top; ) {\n \t\t\tprev = cur;\n \t\t\thav = xdl_hash_record(&cur, top, xpp->flags);\n-\t\t\tif (nrec >= narec) {\n-\t\t\t\tnarec *= 2;\n-\t\t\t\tif (!(rrecs = (xrecord_t **) xdl_realloc(recs, narec * sizeof(xrecord_t *))))\n-\t\t\t\t\tgoto abort;\n-\t\t\t\trecs = rrecs;\n-\t\t\t}\n+\t\t\tif (XDL_ALLOC_GROW(recs, nrec + 1, narec))\n+\t\t\t\tgoto abort;\n \t\t\tif (!(crec = xdl_cha_alloc(&xdf->rcha)))\n \t\t\t\tgoto abort;\n \t\t\tcrec->ptr = prev;\ndiff --git a/xdiff/xutils.c b/xdiff/xutils.c\nindex 115b2b1640b..9e36f24875d 100644\n--- a/xdiff/xutils.c\n+++ b/xdiff/xutils.c\n@@ -432,3 +432,20 @@ int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n \n \treturn 0;\n }\n+\n+void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size)\n+{\n+\tvoid *tmp = NULL;\n+\tsize_t n = ((LONG_MAX - 16) / 2 >= *alloc) ? 2 * *alloc + 16 : LONG_MAX;\n+\tif (nr > n)\n+\t\tn = nr;\n+\tif (SIZE_MAX / size >= n)\n+\t\ttmp = xdl_realloc(p, n * size);\n+\tif (tmp) {\n+\t\t*alloc = n;\n+\t} else {\n+\t\txdl_free(p);\n+\t\t*alloc = 0;\n+\t}\n+\treturn tmp;\n+}\ndiff --git a/xdiff/xutils.h b/xdiff/xutils.h\nindex fba7bae03c7..fd0bba94e8b 100644\n--- a/xdiff/xutils.h\n+++ b/xdiff/xutils.h\n@@ -42,6 +42,7 @@ int xdl_emit_hunk_hdr(long s1, long c1, long s2, long c2,\n int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n \t\t       int line1, int count1, int line2, int count2);\n \n-\n+/* Do not call this function, use XDL_ALLOC_GROW instead */\n+void* xdl_alloc_grow_helper(void* p, long nr, long* alloc, size_t size);\n \n #endif /* #if !defined(XUTILS_H) */\n-- \ngitgitgadget\n"},{"id":"458181","messageId":"220630.86h742e81q.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"pull.1272.git.1656516334.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/3] xdiff: introduce memory allocation macros","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-30T10:46:22Z","receivedAt":"2022-06-30T10:54:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 29 2022, Phillip Wood via GitGitGadget wrote:\n\n> This patch series introduces macros for allocating and growing arrays in\n> xdiff. The macros are similar to ALLOC_ARRAY()/ALLOC_GROW() from the rest of\n> the code base but return an error on failure to allow libgit2 to handle\n> memory allocation failures gracefully rather than dying. The macros\n> introduce overflow checks but these checks are currently redundant as we\n> limit the maximum file size passed to xdiff and these checks alone are\n> insufficient to safely remove the size limit. The aim of this series is to\n> make the xdiff code more readable, there should be no change in behavior (as\n> such I'm open to the argument that these are just churn and should be\n> dropped).\n\nI think it's a good direction, but why not make such new macros\nnon-XDL_* specific, add them to git-compat-util.h, and then define our\nexisting macros that call xmalloc() now in terms of these new macros?\n\nI realize that it'll take a bit more careful hacking in wrapper.c and\ngit-compat-util.h, but it would allow us to eventually make some other\nlow-level APIs of ours use such an API.\n\nE.g. we have some hand-rolled replacements for \"struct strbuf\" in at\nleast a couple of places (e.g. vreportf() in usage.c). If you pull on\nthat thread you'll see that it's for no reason other than strbuf.c calls\nALLOC_GROW(), which we'll die() in, and we don't want to die on malloc\nfailure in e.g. BUG().\n\n"},{"id":"458182","messageId":"220630.86czeqe74c.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"da996677f0730ec7a50d399524fb58c44dad468a.1656516334.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-30T10:54:23Z","receivedAt":"2022-06-30T11:14:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 29 2022, Phillip Wood via GitGitGadget wrote:\n\n> From: Phillip Wood <phillip.wood@dunelm.org.uk>\n>\n> Add a helper to grow an array. This is analogous to ALLOC_GROW() in\n> the rest of the codebase but returns −1 on allocation failure to\n> accommodate other users of libxdiff such as libgit2.\n\nUrm, does it? I just skimmed this, so maybe I missed something, but I\ndon't see where you changed the definition of xdl_malloc(),\nxdl_realloc() etc.\n\nAnd those are defined as:\n\n\t#define xdl_malloc(x) xmalloc(x)\n\t#define xdl_free(ptr) free(ptr)\n\t#define xdl_realloc(ptr,x) xrealloc(ptr,x)\n\nAnd for e.g. xmalloc() we do:\n\n        return do_xmalloc(size, 0);\n\nWhere that 0=gently, i.e. we'll die() instead of error(), the xrealloc()\nthen has no \"gently\" option.\n\nIs the (pretty glaring, if that's the case) unstated assumption here\nthat this doesn't in fact return -1 on allocation failure, but you're\nexpected to replace the underlying xmalloc() with an implementation that\ndoes?\n\nIf so I'm doubly confused by this, since you're providing alternatives\nto e.g.:\n\n\t#define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))\n\nSo if that's the plan why would we need an XDL_ALLOC_ARRAY(), can't you\njust check that it returns non-NULL?\n\nThat's not possible with our current xmalloc(), but ditto for this\nseries, and apparently the compiler isn't smart enough to yell at you\nabout that...\n\nI wondered if we were just missing the returns_nonnull attribute, but\nplaying around with it I couldn't get GCC at least to warn about\nchecking xmalloc()'s return value for non-NULL.\n"},{"id":"458184","messageId":"5734b393-57c8-dc63-282e-221ee1937351@gmail.com","threadId":"58081","inReplyTo":"220630.86czeqe74c.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-06-30T12:03:35Z","receivedAt":"2022-06-30T12:03:41Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 30/06/2022 11:54, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Jun 29 2022, Phillip Wood via GitGitGadget wrote:\n> \n>> From: Phillip Wood <phillip.wood@dunelm.org.uk>\n>>\n>> Add a helper to grow an array. This is analogous to ALLOC_GROW() in\n>> the rest of the codebase but returns −1 on allocation failure to\n>> accommodate other users of libxdiff such as libgit2.\n> \n> Urm, does it? I just skimmed this, so maybe I missed something, but I\n> don't see where you changed the definition of xdl_malloc(),\n> xdl_realloc() etc.\n\nXDL_ALLOC_GROW is defined as\n\n+/*\n+ * Ensure array p can accommodate at least nr elements, growing the\n+ * array and updating alloc (which is the number of allocated\n+ * elements) as necessary. Frees p and returns -1 on failure, returns\n+ * 0 on success\n+ */\n+#define XDL_ALLOC_GROW(p, nr, alloc)\t\\\n+\t(-!((nr) <= (alloc) ||\t\t\\\n+\t    ((p) = xdl_alloc_grow_helper((p), (nr), &(alloc), sizeof(*(p))))))\n+\n\nso it returns -1 if xdl_alloc_grow_helper() returns NULL, which it does \nif there is an allocation failure or the multiplication overflows.\n\n> And those are defined as:\n> \n> \t#define xdl_malloc(x) xmalloc(x)\n> \t#define xdl_free(ptr) free(ptr)\n> \t#define xdl_realloc(ptr,x) xrealloc(ptr,x)\n> \n> And for e.g. xmalloc() we do:\n> \n>          return do_xmalloc(size, 0);\n> \n> Where that 0=gently, i.e. we'll die() instead of error(), the xrealloc()\n> then has no \"gently\" option.\n>\n> Is the (pretty glaring, if that's the case) unstated assumption here\n> that this doesn't in fact return -1 on allocation failure, but you're\n> expected to replace the underlying xmalloc() with an implementation that\n> does?\n\nI'm not relying on the return value of xrealloc() in the macro\n\n> If so I'm doubly confused by this, since you're providing alternatives\n> to e.g.:\n> \n> \t#define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))\n> \n> So if that's the plan why would we need an XDL_ALLOC_ARRAY(), can't you\n> just check that it returns non-NULL?\n >\n> That's not possible with our current xmalloc(), but ditto for this\n> series, and apparently the compiler isn't smart enough to yell at you\n> about that...\n> \n> I wondered if we were just missing the returns_nonnull attribute, but\n> playing around with it I couldn't get GCC at least to warn about\n> checking xmalloc()'s return value for non-NULL.\n\nI'm not quite sure what you're saying in these last three paragraphs\n\nBest Wishes\n\nPhillip\n\n"},{"id":"458186","messageId":"0a1650bf-a7f5-0cc5-e6c9-0e02d1f542bf@gmail.com","threadId":"58081","inReplyTo":"5734b393-57c8-dc63-282e-221ee1937351@gmail.com","subject":"Re: [PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-06-30T12:38:45Z","receivedAt":"2022-06-30T12:38:53Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 30/06/2022 13:03, Phillip Wood wrote:\n> On 30/06/2022 11:54, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Wed, Jun 29 2022, Phillip Wood via GitGitGadget wrote:\n>>\n>>> From: Phillip Wood <phillip.wood@dunelm.org.uk>\n>>>\n>>> Add a helper to grow an array. This is analogous to ALLOC_GROW() in\n>>> the rest of the codebase but returns −1 on allocation failure to\n>>> accommodate other users of libxdiff such as libgit2.\n>>\n>> Urm, does it? I just skimmed this, so maybe I missed something, but I\n>> don't see where you changed the definition of xdl_malloc(),\n>> xdl_realloc() etc.\n\nOh I think I might have misunderstood your question. For git.git it will \nstill die() but for other users that arrange for xdl_realloc() to return \nNULL on failure it will return -1. The same applies to the comments in \nthe previous two patches about XDL_[CM]ALLOC_ARRAY() returning NULL on \nallocation failure.\n\nBest Wishes\n\nPhillip\n"},{"id":"458189","messageId":"220630.86zghuclzq.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"0a1650bf-a7f5-0cc5-e6c9-0e02d1f542bf@gmail.com","subject":"Re: [PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-30T13:25:46Z","receivedAt":"2022-06-30T13:35:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jun 30 2022, Phillip Wood wrote:\n\n> On 30/06/2022 13:03, Phillip Wood wrote:\n>> On 30/06/2022 11:54, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> On Wed, Jun 29 2022, Phillip Wood via GitGitGadget wrote:\n>>>\n>>>> From: Phillip Wood <phillip.wood@dunelm.org.uk>\n>>>>\n>>>> Add a helper to grow an array. This is analogous to ALLOC_GROW() in\n>>>> the rest of the codebase but returns −1 on allocation failure to\n>>>> accommodate other users of libxdiff such as libgit2.\n>>>\n>>> Urm, does it? I just skimmed this, so maybe I missed something, but I\n>>> don't see where you changed the definition of xdl_malloc(),\n>>> xdl_realloc() etc.\n>\n> Oh I think I might have misunderstood your question. For git.git it\n> will still die() but for other users that arrange for xdl_realloc() to\n> return NULL on failure it will return -1. The same applies to the\n> comments in the previous two patches about XDL_[CM]ALLOC_ARRAY()\n> returning NULL on allocation failure.\n\nYes, I meant that the \"but returns −1 on allocation failure to\naccommodate other users of libxdiff such as libgit2\" is really more of\na:\n\n\t...but *allows for* dropping in another xmalloc(), xrealloc()\n\tetc. implementation that doesn't die on failure.\n\nSo I think the rest of my upthread question still stands, i.e.:\n\n\t\"So if that's the plan why would we need an XDL_ALLOC_ARRAY(),\n\tcan't you just check that it [I mean ALLOC_ARRAY()] returns\n\tnon-NULL?\"\n\nI.e. if the plan is to replace the underlying x*() functions with\nnon-fatal variants can't you use ALLOC_ARRAY() instead? I haven't tried\nthat, but I don't see a reason we couldn't do that in principle...\n\nAnyway, I'm all for the end goal here, but the way to get there seems to\nbe a bit of an exercise in running with scissors the more I think about\nit.\n\nI.e. the reason we're using these x*alloc() wrappers at all is because\nwe're lazy and want to write stuff like:\n\n\tstruct stuff *foo = xmalloc(...);\n\tfoo->bar = \"baz\";\n\nWhich the compiler is helpfully not yelling at us about, as opposed to\ndoing the same with \"malloc()\", where it would spot the potential null\npointer dereference.\n\n(I'm using \"compiler\" here to be inclusive of extended gcc/clang options\nto detect this sort of thing, & other similar analyzers).\n\nBut re \"scissors\": if we're doing to be maintaining the xdiff code\nin-tree to be defined as our usual x*alloc() functions we're going to be\ncarrying code that can't have test or analysis coverage.\n\nWhich I think brings me back to my suggestion of whether we can't just\nhave non-fatal versions of these helper macros, define our own currently\nfatal versions in terms of those, and use the non-fatal versions in the\nxdiff/ code.\n\n\n"},{"id":"458238","messageId":"xmqqy1xednht.fsf@gitster.g","threadId":"58081","inReplyTo":"8bead9856be7b39d3d03f688f53d678832d60109.1656516334.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/3] xdiff: introduce XDL_CALLOC_ARRAY()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-06-30T18:17:50Z","receivedAt":"2022-06-30T18:17:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Phillip Wood via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> diff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\n> index 9fd3c5da91a..23db8e785d7 100644\n> --- a/xdiff/xmacros.h\n> +++ b/xdiff/xmacros.h\n> @@ -55,4 +55,10 @@ do { \\\n>  \t\t? xdl_malloc((nr) * sizeof(*(p)))\t\\\n>  \t\t: NULL)\n>  \n> +/* Allocate an array of nr zeroed out elements, returns NULL on failure */\n> +#define XDL_CALLOC_ARRAY(p, nr)\t\t\t\t\\\n> +\t(XDL_ALLOC_ARRAY((p), (nr))\t\t\t\\\n> +\t\t? memset((p), 0, (nr) * sizeof(*(p)))\t\\\n> +\t\t: NULL)\n> +\n\nThis implementation is somewhat dissapointing.  Allocating and then\nclearing is conceptually different from allocating an already\ncleared buffer.\n\nWouldn't it make more sense to build on top of xcalloc() or its\ncounterpart in xdl world by introducing xdl_calloc()?  For that,\nthis step would probably need to become two patches.  The first\npatch introduces xdl_calloc(), which uses xcalloc() in our codebase,\nand turn the existing \"alloc and clear\" code to use it, and the\nsecond patch would wrap the use of xdl_calloc() in the size-safe\nmacro XDL_CALLOC_ARRAY().\n\n"},{"id":"458239","messageId":"xmqqtu82dmu2.fsf@gitster.g","threadId":"58081","inReplyTo":"da996677f0730ec7a50d399524fb58c44dad468a.1656516334.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-06-30T18:32:05Z","receivedAt":"2022-06-30T18:32:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Phillip Wood via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> +/*\n> + * Ensure array p can accommodate at least nr elements, growing the\n> + * array and updating alloc (which is the number of allocated\n> + * elements) as necessary. Frees p and returns -1 on failure, returns\n> + * 0 on success\n> + */\n> +#define XDL_ALLOC_GROW(p, nr, alloc)\t\\\n> +\t(-!((nr) <= (alloc) ||\t\t\\\n> +\t    ((p) = xdl_alloc_grow_helper((p), (nr), &(alloc), sizeof(*(p))))))\n> + ...\n> +\n> +void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size)\n> +{\n> +\tvoid *tmp = NULL;\n> +\tsize_t n = ((LONG_MAX - 16) / 2 >= *alloc) ? 2 * *alloc + 16 : LONG_MAX;\n\nNot counting in size_t but in long?\n\nI assume that 16 mimics the ALLOW_GROW(), but ALLOW_GROW() grows by\n1.5, not by 2, so these two are not exactly compatible.\n\nThe computation of 'n' tries to avoid arithmetic in \"long\"\noverflowing, but does it ensure that we actually grow if we truncate\nat LONG_MAX?  After *alloc grew to LONG_MAX by calling this helper\nonce, if we need to grow again and call this helper, 'n' will again\nget LONG_MAX and we end up not growing at all, no?\n\n> +\tif (nr > n)\n> +\t\tn = nr;\n> +\tif (SIZE_MAX / size >= n)\n> +\t\ttmp = xdl_realloc(p, n * size);\n> +\tif (tmp) {\n> +\t\t*alloc = n;\n> +\t} else {\n> +\t\txdl_free(p);\n> +\t\t*alloc = 0;\n> +\t}\n> +\treturn tmp;\n> +}\n\n\n"},{"id":"458519","messageId":"d58292b6-dad6-78d4-2846-18ddc3e30ca5@gmail.com","threadId":"58081","inReplyTo":"xmqqtu82dmu2.fsf@gitster.g","subject":"Re: [PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-06T13:14:25Z","receivedAt":"2022-07-06T13:14:51Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Junio\n\nOn 30/06/2022 19:32, Junio C Hamano wrote:\n> \"Phillip Wood via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> \n>> +/*\n>> + * Ensure array p can accommodate at least nr elements, growing the\n>> + * array and updating alloc (which is the number of allocated\n>> + * elements) as necessary. Frees p and returns -1 on failure, returns\n>> + * 0 on success\n>> + */\n>> +#define XDL_ALLOC_GROW(p, nr, alloc)\t\\\n>> +\t(-!((nr) <= (alloc) ||\t\t\\\n>> +\t    ((p) = xdl_alloc_grow_helper((p), (nr), &(alloc), sizeof(*(p))))))\n>> + ...\n>> +\n>> +void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size)\n>> +{\n>> +\tvoid *tmp = NULL;\n>> +\tsize_t n = ((LONG_MAX - 16) / 2 >= *alloc) ? 2 * *alloc + 16 : LONG_MAX;\n> \n> Not counting in size_t but in long?\n\nYes, that's to match the existing code.\n\n> I assume that 16 mimics the ALLOW_GROW(), but ALLOW_GROW() grows by\n> 1.5, not by 2, so these two are not exactly compatible.\n> \n> The computation of 'n' tries to avoid arithmetic in \"long\"\n> overflowing, but does it ensure that we actually grow if we truncate\n> at LONG_MAX?  After *alloc grew to LONG_MAX by calling this helper\n> once, if we need to grow again and call this helper, 'n' will again\n> get LONG_MAX and we end up not growing at all, no?\n\nRight but the caller can only request up to LONG_MAX items in nr. If we \nwere to grow beyond that alloc would be truncated when we returned it to \nthe caller.\n\nThe code here is written to fit in with xdiff using long where it would \nbe better using size_t (changing that would be a fairly major undertaking).\n\nBest Wishes\n\nPhillip\n\n>> +\tif (nr > n)\n>> +\t\tn = nr;\n>> +\tif (SIZE_MAX / size >= n)\n>> +\t\ttmp = xdl_realloc(p, n * size);\n>> +\tif (tmp) {\n>> +\t\t*alloc = n;\n>> +\t} else {\n>> +\t\txdl_free(p);\n>> +\t\t*alloc = 0;\n>> +\t}\n>> +\treturn tmp;\n>> +}\n> \n> \n"},{"id":"458520","messageId":"114180f5-8c54-b6e1-14d1-4ad1a9666358@gmail.com","threadId":"58081","inReplyTo":"xmqqy1xednht.fsf@gitster.g","subject":"Re: [PATCH 2/3] xdiff: introduce XDL_CALLOC_ARRAY()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-06T13:17:02Z","receivedAt":"2022-07-06T13:17:38Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Junio\n\nOn 30/06/2022 19:17, Junio C Hamano wrote:\n> \"Phillip Wood via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> \n>> diff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\n>> index 9fd3c5da91a..23db8e785d7 100644\n>> --- a/xdiff/xmacros.h\n>> +++ b/xdiff/xmacros.h\n>> @@ -55,4 +55,10 @@ do { \\\n>>   \t\t? xdl_malloc((nr) * sizeof(*(p)))\t\\\n>>   \t\t: NULL)\n>>   \n>> +/* Allocate an array of nr zeroed out elements, returns NULL on failure */\n>> +#define XDL_CALLOC_ARRAY(p, nr)\t\t\t\t\\\n>> +\t(XDL_ALLOC_ARRAY((p), (nr))\t\t\t\\\n>> +\t\t? memset((p), 0, (nr) * sizeof(*(p)))\t\\\n>> +\t\t: NULL)\n>> +\n> \n> This implementation is somewhat dissapointing.  Allocating and then\n> clearing is conceptually different from allocating an already\n> cleared buffer.\n> \n> Wouldn't it make more sense to build on top of xcalloc() or its\n> counterpart in xdl world by introducing xdl_calloc()?  For that,\n> this step would probably need to become two patches.  The first\n> patch introduces xdl_calloc(), which uses xcalloc() in our codebase,\n> and turn the existing \"alloc and clear\" code to use it, and the\n> second patch would wrap the use of xdl_calloc() in the size-safe\n> macro XDL_CALLOC_ARRAY().\n\nI was hoping to avoid that by following the existing pattern of malloc() \n+ memset() but I'll reroll with a preparatory patch that converts the \nexisting code to use xdl_calloc()\n\nBest Wishes\n\nPhillip\n\n"},{"id":"458521","messageId":"fa543f66-6e86-9f95-f164-20a68705cecd@gmail.com","threadId":"58081","inReplyTo":"220630.86zghuclzq.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-06T13:23:32Z","receivedAt":"2022-07-06T13:23:55Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 30/06/2022 14:25, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jun 30 2022, Phillip Wood wrote:\n> \n>> On 30/06/2022 13:03, Phillip Wood wrote:\n>>> On 30/06/2022 11:54, Ævar Arnfjörð Bjarmason wrote:\n>>>>\n>>>> On Wed, Jun 29 2022, Phillip Wood via GitGitGadget wrote:\n>>>>\n>>>>> From: Phillip Wood <phillip.wood@dunelm.org.uk>\n>>>>>\n>>>>> Add a helper to grow an array. This is analogous to ALLOC_GROW() in\n>>>>> the rest of the codebase but returns −1 on allocation failure to\n>>>>> accommodate other users of libxdiff such as libgit2.\n>>>>\n>>>> Urm, does it? I just skimmed this, so maybe I missed something, but I\n>>>> don't see where you changed the definition of xdl_malloc(),\n>>>> xdl_realloc() etc.\n>>\n>> Oh I think I might have misunderstood your question. For git.git it\n>> will still die() but for other users that arrange for xdl_realloc() to\n>> return NULL on failure it will return -1. The same applies to the\n>> comments in the previous two patches about XDL_[CM]ALLOC_ARRAY()\n>> returning NULL on allocation failure.\n> \n> Yes, I meant that the \"but returns −1 on allocation failure to\n> accommodate other users of libxdiff such as libgit2\" is really more of\n> a:\n> \n> \t...but *allows for* dropping in another xmalloc(), xrealloc()\n> \tetc. implementation that doesn't die on failure.\n> \n> So I think the rest of my upthread question still stands, i.e.:\n> \n> \t\"So if that's the plan why would we need an XDL_ALLOC_ARRAY(),\n> \tcan't you just check that it [I mean ALLOC_ARRAY()] returns\n> \tnon-NULL?\"\n> \n> I.e. if the plan is to replace the underlying x*() functions with\n> non-fatal variants can't you use ALLOC_ARRAY() instead? I haven't tried\n> that, but I don't see a reason we couldn't do that in principle...\n\nAs the cover letter says, the aim of this series is not to replace \nxmalloc() etc with non-fatal variants, it is just to make the xdiff code \nmore readable. (One can already use a non-fatal allocation function for \nxdl_malloc()) I don't think that using ALLOC_ARRAY() in xdiff is helpful \nfor other users as they would have to define their own array allocation \nmacros, rather than just providing their own allocation functions. I \nwould like to reduce the friction others have upstreaming xdiff patches \nto us, not increase it.\n\nBest Wishes\n\nPhillip\n\n> Anyway, I'm all for the end goal here, but the way to get there seems to\n> be a bit of an exercise in running with scissors the more I think about\n> it.\n> \n> I.e. the reason we're using these x*alloc() wrappers at all is because\n> we're lazy and want to write stuff like:\n> \n> \tstruct stuff *foo = xmalloc(...);\n> \tfoo->bar = \"baz\";\n> \n> Which the compiler is helpfully not yelling at us about, as opposed to\n> doing the same with \"malloc()\", where it would spot the potential null\n> pointer dereference.\n> \n> (I'm using \"compiler\" here to be inclusive of extended gcc/clang options\n> to detect this sort of thing, & other similar analyzers).\n> \n> But re \"scissors\": if we're doing to be maintaining the xdiff code\n> in-tree to be defined as our usual x*alloc() functions we're going to be\n> carrying code that can't have test or analysis coverage.\n> \n> Which I think brings me back to my suggestion of whether we can't just\n> have non-fatal versions of these helper macros, define our own currently\n> fatal versions in terms of those, and use the non-fatal versions in the\n> xdiff/ code.\n> \n> \n"},{"id":"458560","messageId":"220707.8635fd9meo.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"fa543f66-6e86-9f95-f164-20a68705cecd@gmail.com","subject":"Re: [PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-07T11:17:35Z","receivedAt":"2022-07-07T11:45:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jul 06 2022, Phillip Wood wrote:\n\n> Hi Ævar\n>\n> On 30/06/2022 14:25, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Jun 30 2022, Phillip Wood wrote:\n>> \n>>> On 30/06/2022 13:03, Phillip Wood wrote:\n>>>> On 30/06/2022 11:54, Ævar Arnfjörð Bjarmason wrote:\n>>>>>\n>>>>> On Wed, Jun 29 2022, Phillip Wood via GitGitGadget wrote:\n>>>>>\n>>>>>> From: Phillip Wood <phillip.wood@dunelm.org.uk>\n>>>>>>\n>>>>>> Add a helper to grow an array. This is analogous to ALLOC_GROW() in\n>>>>>> the rest of the codebase but returns −1 on allocation failure to\n>>>>>> accommodate other users of libxdiff such as libgit2.\n>>>>>\n>>>>> Urm, does it? I just skimmed this, so maybe I missed something, but I\n>>>>> don't see where you changed the definition of xdl_malloc(),\n>>>>> xdl_realloc() etc.\n>>>\n>>> Oh I think I might have misunderstood your question. For git.git it\n>>> will still die() but for other users that arrange for xdl_realloc() to\n>>> return NULL on failure it will return -1. The same applies to the\n>>> comments in the previous two patches about XDL_[CM]ALLOC_ARRAY()\n>>> returning NULL on allocation failure.\n>> Yes, I meant that the \"but returns −1 on allocation failure to\n>> accommodate other users of libxdiff such as libgit2\" is really more of\n>> a:\n>> \t...but *allows for* dropping in another xmalloc(), xrealloc()\n>> \tetc. implementation that doesn't die on failure.\n>> So I think the rest of my upthread question still stands, i.e.:\n>> \t\"So if that's the plan why would we need an XDL_ALLOC_ARRAY(),\n>> \tcan't you just check that it [I mean ALLOC_ARRAY()] returns\n>> \tnon-NULL?\"\n>> I.e. if the plan is to replace the underlying x*() functions with\n>> non-fatal variants can't you use ALLOC_ARRAY() instead? I haven't tried\n>> that, but I don't see a reason we couldn't do that in principle...\n>\n> As the cover letter says, the aim of this series is not to replace\n> xmalloc() etc with non-fatal variants, it is just to make the xdiff\n> code more readable.\n\nI don't think it's more readable to carry code in-tree that's\nunreachable except when combined with code out-of-tree. I.e. this series\nleaves us with the equivalent of:\n\n\tptr = xmalloc(...);\n        if (!ptr)\n\t\t/* unreachable in git.git ... */\n\nI don't think it's more readable to have code that rather trivial\nanalysis will show goes against the \"__attribute__((noreturn))\" we're\nplacing on our die() function.\n\nWhich is what I'm pointing out with \"running with scissors\". I.e. I'm\nfully on-board with the end goal, but that can be accomplished in a way\nthat doesn't confuse humans & analyzers alike.\n\n> (One can already use a non-fatal allocation\n> function for xdl_malloc()) [...]\n\nThat just seems inviting a segfault or undefined/untested behavior\n(whether in the sense of \"undefined by C\" or \"untested by git.git's\ncodebase logic\"). Everything around xmalloc() now assumes \"never returns\nNULL\", and you want to:\n\n * Make it return NULL when combined with out-of-tree-code\n * Maintain the code in git.git, where it never returns NULL, but in a\n   way where we won't have bugs when combined with a new macro that\n   behaves differently, in a way we never even test ourselves.\n\nIsn't that correct, or am I missing something?\n\n> I don't think that using ALLOC_ARRAY() in\n> xdiff is helpful for other users as they would have to define their\n> own array allocation macros, rather than just providing their own\n> allocation functions. I would like to reduce the friction others have\n> upstreaming xdiff patches to us, not increase it.\n\nYes, I'm totally on-board with reducing the friction in others using\nxdiff, and would like to see more of that sort of out-of-tree use in\ngeneral (although for things outside of xdiff GPL v.s. LGPL concerns\ncome into play).\n\nI'd even like for us to explicitly make that much easier. I.e. if you\nwant to use xdiff now you search for it, and find the at this point\nunmaintained upstream, and if you find that git has a \"newer\" version\nyou'll have some trouble extracting it already.\n\nAfter this series you'll need to start writing & maintaining your own\nnon-trivial alloc wrapper logic if you're doing that. If you get it\nsubtly wrong you'll have a buggy xdiff, and most likely users will just\ncopy/paste the git.git version from our git-compat-util.h & cache.h,\nwhich is rather silly.\n\n\nWhich is why I'm saying we could/should do this in a much easier way,\ni.e.:\n\n * Factor out the now only-fatal-on-NULL ALLOC_GROW() such that we can\n   have a non-fatal version (ALLOC_GROW) and a non-fatal one.\n\n   I don't know what we'd call it. we usually have X*() meaning \"fatal\",\n   but these are fatal by default, maybe G* for gently? So\n   GALLOC_GROW().  Urgh, anyway, continuing with that ugly name...\n\n * Have xdiff/ use that GALLOC_GROW() & malloc(), not ALLOC_GROW() &\n   xmalloc(), as we really want to have the appropriate code flow\n   analysis etc. spot for us that we should handle NULL returns,\n   otherwise combining this code with libgit2 will be buggy/broken.\n\nThis makes it much easier for libgit2 to use this, as it won't need to\ndo anything special at all. Since our GALLOC_GROW() will eventualy use\nmalloc() instead of xmalloc() you don't need to define anything that\nre-implements the GALLOC_GROW() or whatever other non-fatal versions of\nour only-fatal helpers we have.\n\nThis assumes that we'd move these macros out of git-compat-util.h and a\nnew git-exernal-compat.h, or that instead of *just* copying the xdiff/\ndirectory your import script would need to run some small bit of cc -E\nand or perl/sed to one-off extract the smell bits of\ngit-exernal-compat.h or cache.h that we need.\n"},{"id":"458627","messageId":"b34dcb93-df73-f5de-3c7c-7ab6c3250afe@gmail.com","threadId":"58081","inReplyTo":"220707.8635fd9meo.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 3/3] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-08T09:35:29Z","receivedAt":"2022-07-08T09:35:41Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 07/07/2022 12:17, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Jul 06 2022, Phillip Wood wrote:\n> \n>> Hi Ævar\n>>\n>> On 30/06/2022 14:25, Ævar Arnfjörð Bjarmason wrote:\n>>> On Thu, Jun 30 2022, Phillip Wood wrote:\n>>>\n>>>> On 30/06/2022 13:03, Phillip Wood wrote:\n>>>>> On 30/06/2022 11:54, Ævar Arnfjörð Bjarmason wrote:\n>>>>>>\n>>>>>> On Wed, Jun 29 2022, Phillip Wood via GitGitGadget wrote:\n>>>>>>\n>>>>>>> From: Phillip Wood <phillip.wood@dunelm.org.uk>\n>>>>>>>\n>>>>>>> Add a helper to grow an array. This is analogous to ALLOC_GROW() in\n>>>>>>> the rest of the codebase but returns −1 on allocation failure to\n>>>>>>> accommodate other users of libxdiff such as libgit2.\n>>>>>>\n>>>>>> Urm, does it? I just skimmed this, so maybe I missed something, but I\n>>>>>> don't see where you changed the definition of xdl_malloc(),\n>>>>>> xdl_realloc() etc.\n>>>>\n>>>> Oh I think I might have misunderstood your question. For git.git it\n>>>> will still die() but for other users that arrange for xdl_realloc() to\n>>>> return NULL on failure it will return -1. The same applies to the\n>>>> comments in the previous two patches about XDL_[CM]ALLOC_ARRAY()\n>>>> returning NULL on allocation failure.\n>>> Yes, I meant that the \"but returns −1 on allocation failure to\n>>> accommodate other users of libxdiff such as libgit2\" is really more of\n>>> a:\n>>> \t...but *allows for* dropping in another xmalloc(), xrealloc()\n>>> \tetc. implementation that doesn't die on failure.\n>>> So I think the rest of my upthread question still stands, i.e.:\n>>> \t\"So if that's the plan why would we need an XDL_ALLOC_ARRAY(),\n>>> \tcan't you just check that it [I mean ALLOC_ARRAY()] returns\n>>> \tnon-NULL?\"\n>>> I.e. if the plan is to replace the underlying x*() functions with\n>>> non-fatal variants can't you use ALLOC_ARRAY() instead? I haven't tried\n>>> that, but I don't see a reason we couldn't do that in principle...\n>>\n>> As the cover letter says, the aim of this series is not to replace\n>> xmalloc() etc with non-fatal variants, it is just to make the xdiff\n>> code more readable.\n> \n> I don't think it's more readable to carry code in-tree that's\n> unreachable except when combined with code out-of-tree. I.e. this series\n> leaves us with the equivalent of:\n> \n> \tptr = xmalloc(...);\n>          if (!ptr)\n> \t\t/* unreachable in git.git ... */\n> \n> I don't think it's more readable to have code that rather trivial\n> analysis will show goes against the \"__attribute__((noreturn))\" we're\n> placing on our die() function.\n\nWe're already in this situation. The code in xdiff is written to handle \nallocation failures and we use an allocation function that dies instead. \nThis patch series does nothing to alter that situation.\n\n> Which is what I'm pointing out with \"running with scissors\". I.e. I'm\n> fully on-board with the end goal, but that can be accomplished in a way\n> that doesn't confuse humans & analyzers alike.\n> \n>> (One can already use a non-fatal allocation\n>> function for xdl_malloc()) [...]\n> \n> That just seems inviting a segfault or undefined/untested behavior\n> (whether in the sense of \"undefined by C\" or \"untested by git.git's\n> codebase logic\"). Everything around xmalloc() now assumes \"never returns\n> NULL\", and you want to:\n> \n>   * Make it return NULL when combined with out-of-tree-code\n\nNo I do not want to alter the behavior of xmalloc() at all, that is why \nthis series does not alter the behavior of xmalloc()\n\n>   * Maintain the code in git.git, where it never returns NULL, but in a\n>     way where we won't have bugs when combined with a new macro that\n>     behaves differently, in a way we never even test ourselves.\n\nThat describes the current situation with xdiff, this series does not \nalter that.\n\n> Isn't that correct, or am I missing something?\n\nYou should note that libgit2 uses malloc() as it's default allocator, \nseeming without issue.\n\n>> I don't think that using ALLOC_ARRAY() in\n>> xdiff is helpful for other users as they would have to define their\n>> own array allocation macros, rather than just providing their own\n>> allocation functions. I would like to reduce the friction others have\n>> upstreaming xdiff patches to us, not increase it.\n> \n> Yes, I'm totally on-board with reducing the friction in others using\n> xdiff, and would like to see more of that sort of out-of-tree use in\n> general (although for things outside of xdiff GPL v.s. LGPL concerns\n> come into play).\n> \n> I'd even like for us to explicitly make that much easier. I.e. if you\n> want to use xdiff now you search for it, and find the at this point\n> unmaintained upstream, and if you find that git has a \"newer\" version\n> you'll have some trouble extracting it already.\n> \n> After this series you'll need to start writing & maintaining your own\n> non-trivial alloc wrapper logic if you're doing that. If you get it\n> subtly wrong you'll have a buggy xdiff, and most likely users will just\n> copy/paste the git.git version from our git-compat-util.h & cache.h,\n> which is rather silly.\n\nThis series does not alter what wrappers you need to write whereas your \nsuggestion of using ALLOC_ARRAY() would force more work on potential \nxdiff users (though below I think you're suggesting we provide them in a \nseparate header so they can be reused more easily).\n\n> Which is why I'm saying we could/should do this in a much easier way,\n> i.e.:\n> \n>   * Factor out the now only-fatal-on-NULL ALLOC_GROW() such that we can\n>     have a non-fatal version (ALLOC_GROW) and a non-fatal one.\n> \n>     I don't know what we'd call it. we usually have X*() meaning \"fatal\",\n>     but these are fatal by default, maybe G* for gently? So\n>     GALLOC_GROW().  Urgh, anyway, continuing with that ugly name...\n\nFurther proof that naming is hard...\n\n>   * Have xdiff/ use that GALLOC_GROW() & malloc(), not ALLOC_GROW() &\n>     xmalloc(), as we really want to have the appropriate code flow\n>     analysis etc. spot for us that we should handle NULL returns,\n>     otherwise combining this code with libgit2 will be buggy/broken.\n> \n> This makes it much easier for libgit2 to use this, as it won't need to\n> do anything special at all. Since our GALLOC_GROW() will eventualy use\n> malloc() instead of xmalloc() you don't need to define anything that\n> re-implements the GALLOC_GROW() or whatever other non-fatal versions of\n> our only-fatal helpers we have.\n> \n> This assumes that we'd move these macros out of git-compat-util.h and a\n> new git-exernal-compat.h, or that instead of *just* copying the xdiff/\n> directory your import script would need to run some small bit of cc -E\n> and or perl/sed to one-off extract the smell bits of\n> git-exernal-compat.h or cache.h that we need.\n\nI think there is an argument that we should change our xdiff wrapper to \nuse malloc() rather than xmalloc() so we're able to test the error \nhandling. That then begs the question as to how we actually get the \nallocation functions to fail when they're being tested. I also think \nthat is an orthogonal change that could happen with or without this \npatch series.\n\nBest Wishes\n\nPhillip\n"},{"id":"458632","messageId":"cover-0.7-00000000000-20220708T140354Z-avarab@gmail.com","threadId":"58081","inReplyTo":"b34dcb93-df73-f5de-3c7c-7ab6c3250afe@gmail.com","subject":"[PATCH 0/7] xdiff: use standard alloc macros, share them via git-shared-util.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T14:20:12Z","receivedAt":"2022-07-08T14:20:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series is on top of\nhttps://lore.kernel.org/git/pull.1272.git.1656516334.gitgitgadget@gmail.com/;\nand shows that we can factor our the allocation macros used in cache.h\nand git-compat-util.h instead of defining replacements for them and\nalloc_nr() in xdiff/*.\n\nThe journy towards doing so is slightly longer, but I think worth\ndoing, since...\n\nOn Fri, Jul 08 2022, Phillip Wood wrote:\n\n> On 07/07/2022 12:17, Ævar Arnfjörð Bjarmason wrote:\n\n>> I don't think it's more readable to carry code in-tree that's\n>> unreachable except when combined with code out-of-tree. I.e. this series\n>> leaves us with the equivalent of:\n>>      ptr = xmalloc(...);\n>>          if (!ptr)\n>>              /* unreachable in git.git ... */\n>> I don't think it's more readable to have code that rather trivial\n>> analysis will show goes against the \"__attribute__((noreturn))\" we're\n>> placing on our die() function.\n>\n> We're already in this situation. The code in xdiff is written to\n> handle allocation failures and we use an allocation function that dies\n> instead. This patch series does nothing to alter that situation.\n>[...]\n>> That just seems inviting a segfault or undefined/untested behavior\n>> (whether in the sense of \"undefined by C\" or \"untested by git.git's\n>> codebase logic\"). Everything around xmalloc() now assumes \"never returns\n>> NULL\", and you want to:\n>>   * Make it return NULL when combined with out-of-tree-code\n>\n> No I do not want to alter the behavior of xmalloc() at all, that is\n> why this series does not alter the behavior of xmalloc()\n> [...]\n> I think there is an argument that we should change our xdiff wrapper\n> to use malloc() rather than xmalloc() so we're able to test the error\n> handling. That then begs the question as to how we actually get the\n> allocation functions to fail when they're being tested. I also think\n> that is an orthogonal change that could happen with or without this\n> patch series.\n\nI think part of what I was claiming upthread I was just confused\nabout. I.e. yes we do use xdl_malloc() defined as xmalloc() already,\nwhat *is* true (and I didn't make this clear) was that your proposed\nseries cements that further in place.\n\nBut as 6/7 here notes 36c83197249 (xdiff: use xmalloc/xrealloc,\n2019-04-11) left us with that state of affairs for expediency, but as\nwe're really close to just \"properly lib-ifying\" xdiff I think we\nshould just do that. At the time of 36c83197249 we had ~20 calls to\nxdl_malloc(), after your series we were at ~1/2 of that (including a\ncallers that 36c83197249 explicitly punted on).\n\nThis larger series is something we can do later, but I'm submitting it\nas a non-RFC in case there's consensus to pick it up on top. The sum\nof the two would be smaller if they were squashed together, but I\nhaven't done that here.\n\nThe 1/7 is an amendmend I suggested to 47be28e51e6 (Merge branch\n'pw/xdiff-alloc-fail', 2022-03-09), since it was modifying some of the\nsame lines of code...\n\nÆvar Arnfjörð Bjarmason (7):\n  xdiff: simplify freeing patterns around xdl_free_env()\n  git-shared-util.h: move \"shared\" allocation utilities here\n  git-shared-util.h: add G*() versions of *ALLOC_*()\n  xdiff: use G[C]ALLOC_ARRAY(), not XDL_CALLOC_ARRAY()\n  xdiff: use GALLOC_GROW(), not XDL_ALLOC_GROW()\n  xdiff: remove xdl_malloc() wrapper, use malloc(), not xmalloc()\n  xdiff: remove xdl_free(), use free() instead\n\n cache.h            |  75 -----------------------------\n git-compat-util.h  |  28 ++---------\n git-shared-util.h  | 115 +++++++++++++++++++++++++++++++++++++++++++++\n xdiff/xdiff.h      |   5 --\n xdiff/xdiffi.c     |  47 ++++++++----------\n xdiff/xhistogram.c |  15 +++---\n xdiff/xmacros.h    |  23 ---------\n xdiff/xmerge.c     |  57 ++++++++++++----------\n xdiff/xpatience.c  |  14 +++---\n xdiff/xprepare.c   |  60 +++++++++++++----------\n xdiff/xutils.c     |  33 ++++---------\n xdiff/xutils.h     |   2 -\n 12 files changed, 234 insertions(+), 240 deletions(-)\n create mode 100644 git-shared-util.h\n\n-- \n2.37.0.913.g189dca38629\n\n"},{"id":"458633","messageId":"patch-1.7-ad61f4e20b1-20220708T140354Z-avarab@gmail.com","threadId":"58081","inReplyTo":"cover-0.7-00000000000-20220708T140354Z-avarab@gmail.com","subject":"[PATCH 1/7] xdiff: simplify freeing patterns around xdl_free_env()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T14:20:13Z","receivedAt":"2022-07-08T14:20:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the invocations of xdl_free_env() so that only the function\nthat allocated the \"xe\" variable frees it, rather than passing it to a\nfunction that might use \"&xe\", and on failure having that function\nfree it.\n\nThis simplifies the allocation management, since due to the new \"{ 0\n}\" initialization we can pass \"&xe\" to xdl_free_env() without\naccounting for the \"&xe\" not being initialized yet.\n\nThis change was originally suggested as an amendment of the series\nthat since got merged in 47be28e51e6 (Merge branch\n'pw/xdiff-alloc-fail', 2022-03-09) in [1]. The \"avoid double free of\nxe2\" in that series is one of the pattern we can simplify here.\n\n1. https://lore.kernel.org/git/220216.86tuczt7js.gmgdl@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n xdiff/xdiffi.c | 40 ++++++++++++++++------------------------\n xdiff/xmerge.c | 47 ++++++++++++++++++++++++++++-------------------\n xdiff/xutils.c | 12 ++++++++----\n 3 files changed, 52 insertions(+), 47 deletions(-)\n\ndiff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\nindex 53e803e6bcb..6fded43e87d 100644\n--- a/xdiff/xdiffi.c\n+++ b/xdiff/xdiffi.c\n@@ -320,15 +320,11 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n \tif (xdl_prepare_env(mf1, mf2, xpp, xe) < 0)\n \t\treturn -1;\n \n-\tif (XDF_DIFF_ALG(xpp->flags) == XDF_PATIENCE_DIFF) {\n-\t\tres = xdl_do_patience_diff(mf1, mf2, xpp, xe);\n-\t\tgoto out;\n-\t}\n+\tif (XDF_DIFF_ALG(xpp->flags) == XDF_PATIENCE_DIFF)\n+\t\treturn xdl_do_patience_diff(mf1, mf2, xpp, xe);\n \n-\tif (XDF_DIFF_ALG(xpp->flags) == XDF_HISTOGRAM_DIFF) {\n-\t\tres = xdl_do_histogram_diff(mf1, mf2, xpp, xe);\n-\t\tgoto out;\n-\t}\n+\tif (XDF_DIFF_ALG(xpp->flags) == XDF_HISTOGRAM_DIFF)\n+\t\treturn xdl_do_histogram_diff(mf1, mf2, xpp, xe);\n \n \t/*\n \t * Allocate and setup K vectors to be used by the differential\n@@ -337,11 +333,8 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n \t * One is to store the forward path and one to store the backward path.\n \t */\n \tndiags = xe->xdf1.nreff + xe->xdf2.nreff + 3;\n-\tif (!XDL_ALLOC_ARRAY(kvd, 2 * ndiags + 2)) {\n-\n-\t\txdl_free_env(xe);\n+\tif (!XDL_ALLOC_ARRAY(kvd, 2 * ndiags + 2))\n \t\treturn -1;\n-\t}\n \tkvdf = kvd;\n \tkvdb = kvdf + ndiags;\n \tkvdf += xe->xdf2.nreff + 1;\n@@ -366,9 +359,6 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n \t\t\t   kvdf, kvdb, (xpp->flags & XDF_NEED_MINIMAL) != 0,\n \t\t\t   &xenv);\n \txdl_free(kvd);\n- out:\n-\tif (res < 0)\n-\t\txdl_free_env(xe);\n \n \treturn res;\n }\n@@ -1054,19 +1044,19 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,\n int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n \t     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {\n \txdchange_t *xscr;\n-\txdfenv_t xe;\n+\txdfenv_t xe = { 0 };\n \temit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;\n+\tint status = 0;\n \n \tif (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {\n-\n-\t\treturn -1;\n+\t\tstatus = -1;\n+\t\tgoto cleanup;\n \t}\n \tif (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||\n \t    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||\n \t    xdl_build_script(&xe, &xscr) < 0) {\n-\n-\t\txdl_free_env(&xe);\n-\t\treturn -1;\n+\t\tstatus = -1;\n+\t\tgoto cleanup;\n \t}\n \tif (xscr) {\n \t\tif (xpp->flags & XDF_IGNORE_BLANK_LINES)\n@@ -1078,12 +1068,14 @@ int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n \t\tif (ef(&xe, xscr, ecb, xecfg) < 0) {\n \n \t\t\txdl_free_script(xscr);\n-\t\t\txdl_free_env(&xe);\n-\t\t\treturn -1;\n+\t\t\tstatus = -1;\n+\t\t\tgoto cleanup;\n \t\t}\n \t\txdl_free_script(xscr);\n \t}\n+\n+cleanup:\n \txdl_free_env(&xe);\n \n-\treturn 0;\n+\treturn status;\n }\ndiff --git a/xdiff/xmerge.c b/xdiff/xmerge.c\nindex af40c88a5b3..ac0cf52f3be 100644\n--- a/xdiff/xmerge.c\n+++ b/xdiff/xmerge.c\n@@ -365,7 +365,7 @@ static int xdl_refine_conflicts(xdfenv_t *xe1, xdfenv_t *xe2, xdmerge_t *m,\n {\n \tfor (; m; m = m->next) {\n \t\tmmfile_t t1, t2;\n-\t\txdfenv_t xe;\n+\t\txdfenv_t xe = { 0 };\n \t\txdchange_t *xscr, *x;\n \t\tint i1 = m->i1, i2 = m->i2;\n \n@@ -387,8 +387,10 @@ static int xdl_refine_conflicts(xdfenv_t *xe1, xdfenv_t *xe2, xdmerge_t *m,\n \t\tt2.ptr = (char *)xe2->xdf2.recs[m->i2]->ptr;\n \t\tt2.size = xe2->xdf2.recs[m->i2 + m->chg2 - 1]->ptr\n \t\t\t+ xe2->xdf2.recs[m->i2 + m->chg2 - 1]->size - t2.ptr;\n-\t\tif (xdl_do_diff(&t1, &t2, xpp, &xe) < 0)\n+\t\tif (xdl_do_diff(&t1, &t2, xpp, &xe) < 0) {\n+\t\t\txdl_free_env(&xe);\n \t\t\treturn -1;\n+\t\t}\n \t\tif (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||\n \t\t    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||\n \t\t    xdl_build_script(&xe, &xscr) < 0) {\n@@ -684,30 +686,37 @@ static int xdl_do_merge(xdfenv_t *xe1, xdchange_t *xscr1,\n int xdl_merge(mmfile_t *orig, mmfile_t *mf1, mmfile_t *mf2,\n \t\txmparam_t const *xmp, mmbuffer_t *result)\n {\n-\txdchange_t *xscr1 = NULL, *xscr2 = NULL;\n-\txdfenv_t xe1, xe2;\n-\tint status = -1;\n+\txdchange_t *xscr1, *xscr2;\n+\txdfenv_t xe1 = { 0 };\n+\txdfenv_t xe2 = { 0 };\n+\tint status;\n \txpparam_t const *xpp = &xmp->xpp;\n \n \tresult->ptr = NULL;\n \tresult->size = 0;\n \n-\tif (xdl_do_diff(orig, mf1, xpp, &xe1) < 0)\n-\t\treturn -1;\n-\n-\tif (xdl_do_diff(orig, mf2, xpp, &xe2) < 0)\n-\t\tgoto free_xe1; /* avoid double free of xe2 */\n-\n+\tif (xdl_do_diff(orig, mf1, xpp, &xe1) < 0) {\n+\t\tstatus = -1;\n+\t\tgoto cleanup;\n+\t}\n+\tif (xdl_do_diff(orig, mf2, xpp, &xe2) < 0) {\n+\t\tstatus = -1;\n+\t\tgoto cleanup;\n+\t}\n \tif (xdl_change_compact(&xe1.xdf1, &xe1.xdf2, xpp->flags) < 0 ||\n \t    xdl_change_compact(&xe1.xdf2, &xe1.xdf1, xpp->flags) < 0 ||\n-\t    xdl_build_script(&xe1, &xscr1) < 0)\n-\t\tgoto out;\n-\n+\t    xdl_build_script(&xe1, &xscr1) < 0) {\n+\t\tstatus = -1;\n+\t\tgoto cleanup;\n+\t}\n \tif (xdl_change_compact(&xe2.xdf1, &xe2.xdf2, xpp->flags) < 0 ||\n \t    xdl_change_compact(&xe2.xdf2, &xe2.xdf1, xpp->flags) < 0 ||\n-\t    xdl_build_script(&xe2, &xscr2) < 0)\n-\t\tgoto out;\n-\n+\t    xdl_build_script(&xe2, &xscr2) < 0) {\n+\t\txdl_free_script(xscr1);\n+\t\tstatus = -1;\n+\t\tgoto cleanup;\n+\t}\n+\tstatus = 0;\n \tif (!xscr1) {\n \t\tresult->ptr = xdl_malloc(mf2->size);\n \t\tif (!result->ptr)\n@@ -731,9 +740,9 @@ int xdl_merge(mmfile_t *orig, mmfile_t *mf1, mmfile_t *mf2,\n \txdl_free_script(xscr1);\n \txdl_free_script(xscr2);\n \n-\txdl_free_env(&xe2);\n- free_xe1:\n+cleanup:\n \txdl_free_env(&xe1);\n+\txdl_free_env(&xe2);\n \n \treturn status;\n }\ndiff --git a/xdiff/xutils.c b/xdiff/xutils.c\nindex 9e36f24875d..a6f10353cff 100644\n--- a/xdiff/xutils.c\n+++ b/xdiff/xutils.c\n@@ -414,7 +414,8 @@ int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n \t * ranges of lines instead of the whole files.\n \t */\n \tmmfile_t subfile1, subfile2;\n-\txdfenv_t env;\n+\txdfenv_t env = { 0 };\n+\tint status = 0;\n \n \tsubfile1.ptr = (char *)diff_env->xdf1.recs[line1 - 1]->ptr;\n \tsubfile1.size = diff_env->xdf1.recs[line1 + count1 - 2]->ptr +\n@@ -422,15 +423,18 @@ int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n \tsubfile2.ptr = (char *)diff_env->xdf2.recs[line2 - 1]->ptr;\n \tsubfile2.size = diff_env->xdf2.recs[line2 + count2 - 2]->ptr +\n \t\tdiff_env->xdf2.recs[line2 + count2 - 2]->size - subfile2.ptr;\n-\tif (xdl_do_diff(&subfile1, &subfile2, xpp, &env) < 0)\n-\t\treturn -1;\n+\tif (xdl_do_diff(&subfile1, &subfile2, xpp, &env) < 0) {\n+\t\tstatus = -1;\n+\t\tgoto cleanup;\n+\t}\n \n \tmemcpy(diff_env->xdf1.rchg + line1 - 1, env.xdf1.rchg, count1);\n \tmemcpy(diff_env->xdf2.rchg + line2 - 1, env.xdf2.rchg, count2);\n \n+cleanup:\n \txdl_free_env(&env);\n \n-\treturn 0;\n+\treturn status;\n }\n \n void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size)\n-- \n2.37.0.913.g189dca38629\n\n"},{"id":"458634","messageId":"patch-2.7-87f4760bc05-20220708T140354Z-avarab@gmail.com","threadId":"58081","inReplyTo":"cover-0.7-00000000000-20220708T140354Z-avarab@gmail.com","subject":"[PATCH 2/7] git-shared-util.h: move \"shared\" allocation utilities here","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T14:20:14Z","receivedAt":"2022-07-08T14:20:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the most common allocation utilities from cache.h and\ngit-compat-util.h to a new git-shared-util.h. The idea is that xdiff/\nand other in-tree code can share this, and that external projects\ncould then copy this header and include it.\n\nThey will need to include some things that git-compat-util.h does\nbefore that, e.g. we need a \"size_t\" here, and if they'll end up using\nany of the x*() functions they'll need to declare those. But doing so\nshould be fairly obvious, and we can always extend this to define some\nfallback wrappers here if e.g. GIT_COMPAT_UTIL_H isn't defined.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n cache.h           |  75 ---------------------------------\n git-compat-util.h |  28 ++-----------\n git-shared-util.h | 104 ++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 108 insertions(+), 99 deletions(-)\n create mode 100644 git-shared-util.h\n\ndiff --git a/cache.h b/cache.h\nindex ac5ab4ef9d3..f29dbbadf77 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -677,81 +677,6 @@ void initialize_repository_version(int hash_algo, int reinit);\n void sanitize_stdfds(void);\n int daemonize(void);\n \n-#define alloc_nr(x) (((x)+16)*3/2)\n-\n-/**\n- * Dynamically growing an array using realloc() is error prone and boring.\n- *\n- * Define your array with:\n- *\n- * - a pointer (`item`) that points at the array, initialized to `NULL`\n- *   (although please name the variable based on its contents, not on its\n- *   type);\n- *\n- * - an integer variable (`alloc`) that keeps track of how big the current\n- *   allocation is, initialized to `0`;\n- *\n- * - another integer variable (`nr`) to keep track of how many elements the\n- *   array currently has, initialized to `0`.\n- *\n- * Then before adding `n`th element to the item, call `ALLOC_GROW(item, n,\n- * alloc)`.  This ensures that the array can hold at least `n` elements by\n- * calling `realloc(3)` and adjusting `alloc` variable.\n- *\n- * ------------\n- * sometype *item;\n- * size_t nr;\n- * size_t alloc\n- *\n- * for (i = 0; i < nr; i++)\n- * \tif (we like item[i] already)\n- * \t\treturn;\n- *\n- * // we did not like any existing one, so add one\n- * ALLOC_GROW(item, nr + 1, alloc);\n- * item[nr++] = value you like;\n- * ------------\n- *\n- * You are responsible for updating the `nr` variable.\n- *\n- * If you need to specify the number of elements to allocate explicitly\n- * then use the macro `REALLOC_ARRAY(item, alloc)` instead of `ALLOC_GROW`.\n- *\n- * Consider using ALLOC_GROW_BY instead of ALLOC_GROW as it has some\n- * added niceties.\n- *\n- * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'.\n- */\n-#define ALLOC_GROW(x, nr, alloc) \\\n-\tdo { \\\n-\t\tif ((nr) > alloc) { \\\n-\t\t\tif (alloc_nr(alloc) < (nr)) \\\n-\t\t\t\talloc = (nr); \\\n-\t\t\telse \\\n-\t\t\t\talloc = alloc_nr(alloc); \\\n-\t\t\tREALLOC_ARRAY(x, alloc); \\\n-\t\t} \\\n-\t} while (0)\n-\n-/*\n- * Similar to ALLOC_GROW but handles updating of the nr value and\n- * zeroing the bytes of the newly-grown array elements.\n- *\n- * DO NOT USE any expression with side-effect for any of the\n- * arguments.\n- */\n-#define ALLOC_GROW_BY(x, nr, increase, alloc) \\\n-\tdo { \\\n-\t\tif (increase) { \\\n-\t\t\tsize_t new_nr = nr + (increase); \\\n-\t\t\tif (new_nr < nr) \\\n-\t\t\t\tBUG(\"negative growth in ALLOC_GROW_BY\"); \\\n-\t\t\tALLOC_GROW(x, new_nr, alloc); \\\n-\t\t\tmemset((x) + nr, 0, sizeof(*(x)) * (increase)); \\\n-\t\t\tnr = new_nr; \\\n-\t\t} \\\n-\t} while (0)\n-\n /* Initialize and use the cache information */\n struct lock_file;\n void preload_index(struct index_state *index,\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 58d7708296b..eb4b27f4846 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1034,31 +1034,11 @@ FILE *fopen_or_warn(const char *path, const char *mode);\n  */\n int xstrncmpz(const char *s, const char *t, size_t len);\n \n-/*\n- * FREE_AND_NULL(ptr) is like free(ptr) followed by ptr = NULL. Note\n- * that ptr is used twice, so don't pass e.g. ptr++.\n+/**\n+ * Common allocation utils, including ones xdiff uses, and thus are\n+ * split into this file for sharing with external projects.\n  */\n-#define FREE_AND_NULL(p) do { free(p); (p) = NULL; } while (0)\n-\n-#define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))\n-#define CALLOC_ARRAY(x, alloc) (x) = xcalloc((alloc), sizeof(*(x)))\n-#define REALLOC_ARRAY(x, alloc) (x) = xrealloc((x), st_mult(sizeof(*(x)), (alloc)))\n-\n-#define COPY_ARRAY(dst, src, n) copy_array((dst), (src), (n), sizeof(*(dst)) + \\\n-\tBUILD_ASSERT_OR_ZERO(sizeof(*(dst)) == sizeof(*(src))))\n-static inline void copy_array(void *dst, const void *src, size_t n, size_t size)\n-{\n-\tif (n)\n-\t\tmemcpy(dst, src, st_mult(size, n));\n-}\n-\n-#define MOVE_ARRAY(dst, src, n) move_array((dst), (src), (n), sizeof(*(dst)) + \\\n-\tBUILD_ASSERT_OR_ZERO(sizeof(*(dst)) == sizeof(*(src))))\n-static inline void move_array(void *dst, const void *src, size_t n, size_t size)\n-{\n-\tif (n)\n-\t\tmemmove(dst, src, st_mult(size, n));\n-}\n+#include \"git-shared-util.h\"\n \n /*\n  * These functions help you allocate structs with flex arrays, and copy\ndiff --git a/git-shared-util.h b/git-shared-util.h\nnew file mode 100644\nindex 00000000000..7b4479a0f72\n--- /dev/null\n+++ b/git-shared-util.h\n@@ -0,0 +1,104 @@\n+#ifndef GIT_SHARED_UTIL_H\n+#define GIT_SHARED_UTIL_H\n+\n+/*\n+ * FREE_AND_NULL(ptr) is like free(ptr) followed by ptr = NULL. Note\n+ * that ptr is used twice, so don't pass e.g. ptr++.\n+ */\n+#define FREE_AND_NULL(p) do { free(p); (p) = NULL; } while (0)\n+\n+#define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))\n+#define CALLOC_ARRAY(x, alloc) (x) = xcalloc((alloc), sizeof(*(x)))\n+#define REALLOC_ARRAY(x, alloc) (x) = xrealloc((x), st_mult(sizeof(*(x)), (alloc)))\n+\n+#define COPY_ARRAY(dst, src, n) copy_array((dst), (src), (n), sizeof(*(dst)) + \\\n+\tBUILD_ASSERT_OR_ZERO(sizeof(*(dst)) == sizeof(*(src))))\n+static inline void copy_array(void *dst, const void *src, size_t n, size_t size)\n+{\n+\tif (n)\n+\t\tmemcpy(dst, src, st_mult(size, n));\n+}\n+\n+#define MOVE_ARRAY(dst, src, n) move_array((dst), (src), (n), sizeof(*(dst)) + \\\n+\tBUILD_ASSERT_OR_ZERO(sizeof(*(dst)) == sizeof(*(src))))\n+static inline void move_array(void *dst, const void *src, size_t n, size_t size)\n+{\n+\tif (n)\n+\t\tmemmove(dst, src, st_mult(size, n));\n+}\n+\n+#define alloc_nr(x) (((x)+16)*3/2)\n+\n+/**\n+ * Dynamically growing an array using realloc() is error prone and boring.\n+ *\n+ * Define your array with:\n+ *\n+ * - a pointer (`item`) that points at the array, initialized to `NULL`\n+ *   (although please name the variable based on its contents, not on its\n+ *   type);\n+ *\n+ * - an integer variable (`alloc`) that keeps track of how big the current\n+ *   allocation is, initialized to `0`;\n+ *\n+ * - another integer variable (`nr`) to keep track of how many elements the\n+ *   array currently has, initialized to `0`.\n+ *\n+ * Then before adding `n`th element to the item, call `ALLOC_GROW(item, n,\n+ * alloc)`.  This ensures that the array can hold at least `n` elements by\n+ * calling `realloc(3)` and adjusting `alloc` variable.\n+ *\n+ * ------------\n+ * sometype *item;\n+ * size_t nr;\n+ * size_t alloc\n+ *\n+ * for (i = 0; i < nr; i++)\n+ * \tif (we like item[i] already)\n+ * \t\treturn;\n+ *\n+ * // we did not like any existing one, so add one\n+ * ALLOC_GROW(item, nr + 1, alloc);\n+ * item[nr++] = value you like;\n+ * ------------\n+ *\n+ * You are responsible for updating the `nr` variable.\n+ *\n+ * If you need to specify the number of elements to allocate explicitly\n+ * then use the macro `REALLOC_ARRAY(item, alloc)` instead of `ALLOC_GROW`.\n+ *\n+ * Consider using ALLOC_GROW_BY instead of ALLOC_GROW as it has some\n+ * added niceties.\n+ *\n+ * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'.\n+ */\n+#define ALLOC_GROW(x, nr, alloc) \\\n+\tdo { \\\n+\t\tif ((nr) > alloc) { \\\n+\t\t\tif (alloc_nr(alloc) < (nr)) \\\n+\t\t\t\talloc = (nr); \\\n+\t\t\telse \\\n+\t\t\t\talloc = alloc_nr(alloc); \\\n+\t\t\tREALLOC_ARRAY(x, alloc); \\\n+\t\t} \\\n+\t} while (0)\n+\n+/*\n+ * Similar to ALLOC_GROW but handles updating of the nr value and\n+ * zeroing the bytes of the newly-grown array elements.\n+ *\n+ * DO NOT USE any expression with side-effect for any of the\n+ * arguments.\n+ */\n+#define ALLOC_GROW_BY(x, nr, increase, alloc) \\\n+\tdo { \\\n+\t\tif (increase) { \\\n+\t\t\tsize_t new_nr = nr + (increase); \\\n+\t\t\tif (new_nr < nr) \\\n+\t\t\t\tBUG(\"negative growth in ALLOC_GROW_BY\"); \\\n+\t\t\tALLOC_GROW(x, new_nr, alloc); \\\n+\t\t\tmemset((x) + nr, 0, sizeof(*(x)) * (increase)); \\\n+\t\t\tnr = new_nr; \\\n+\t\t} \\\n+\t} while (0)\n+#endif\n-- \n2.37.0.913.g189dca38629\n\n"},{"id":"458635","messageId":"patch-3.7-d7c14a1cad8-20220708T140354Z-avarab@gmail.com","threadId":"58081","inReplyTo":"cover-0.7-00000000000-20220708T140354Z-avarab@gmail.com","subject":"[PATCH 3/7] git-shared-util.h: add G*() versions of *ALLOC_*()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T14:20:15Z","receivedAt":"2022-07-08T14:20:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add \"gently\" versions of ALLOC_ARRAY(), CALLOC_ARRAY() etc. using the\nnaming convention G*() as a shorthand for \"GENTLY_*()\".\n\nNothing uses these functions yet, but as we'll see in subsequent\ncommit(s) we're able to convert things that need e.g. non-fatal\n\"ALLOC_GROW\" behavior over to this.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n git-shared-util.h | 15 +++++++++++++--\n 1 file changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/git-shared-util.h b/git-shared-util.h\nindex 7b4479a0f72..718a8e00732 100644\n--- a/git-shared-util.h\n+++ b/git-shared-util.h\n@@ -8,8 +8,11 @@\n #define FREE_AND_NULL(p) do { free(p); (p) = NULL; } while (0)\n \n #define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))\n+#define GALLOC_ARRAY(x, alloc) (x) = malloc(st_mult(sizeof(*(x)), (alloc)))\n #define CALLOC_ARRAY(x, alloc) (x) = xcalloc((alloc), sizeof(*(x)))\n+#define GCALLOC_ARRAY(x, alloc) (x) = calloc((alloc), sizeof(*(x)))\n #define REALLOC_ARRAY(x, alloc) (x) = xrealloc((x), st_mult(sizeof(*(x)), (alloc)))\n+#define GREALLOC_ARRAY(x, alloc) (x) = realloc((x), st_mult(sizeof(*(x)), (alloc)))\n \n #define COPY_ARRAY(dst, src, n) copy_array((dst), (src), (n), sizeof(*(dst)) + \\\n \tBUILD_ASSERT_OR_ZERO(sizeof(*(dst)) == sizeof(*(src))))\n@@ -71,17 +74,25 @@ static inline void move_array(void *dst, const void *src, size_t n, size_t size)\n  * added niceties.\n  *\n  * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'.\n+ *\n+ * GALLOC_GROW() behaves like ALLOC_GROW(), except that in malloc()\n+ * failure we'll return NULL rather than dying.\n  */\n-#define ALLOC_GROW(x, nr, alloc) \\\n+#define ALLOC_GROW_1(x, nr, alloc, gently) \\\n \tdo { \\\n \t\tif ((nr) > alloc) { \\\n \t\t\tif (alloc_nr(alloc) < (nr)) \\\n \t\t\t\talloc = (nr); \\\n \t\t\telse \\\n \t\t\t\talloc = alloc_nr(alloc); \\\n-\t\t\tREALLOC_ARRAY(x, alloc); \\\n+\t\t\tif (gently) \\\n+\t\t\t\tGREALLOC_ARRAY(x, alloc); \\\n+\t\t\telse \\\n+\t\t\t\tREALLOC_ARRAY(x, alloc); \\\n \t\t} \\\n \t} while (0)\n+#define ALLOC_GROW(x, nr, alloc) ALLOC_GROW_1(x, nr, alloc, 0)\n+#define GALLOC_GROW(x, nr, alloc) ALLOC_GROW_1(x, nr, alloc, 1)\n \n /*\n  * Similar to ALLOC_GROW but handles updating of the nr value and\n-- \n2.37.0.913.g189dca38629\n\n"},{"id":"458636","messageId":"patch-5.7-3665576576f-20220708T140354Z-avarab@gmail.com","threadId":"58081","inReplyTo":"cover-0.7-00000000000-20220708T140354Z-avarab@gmail.com","subject":"[PATCH 5/7] xdiff: use GALLOC_GROW(), not XDL_ALLOC_GROW()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T14:20:17Z","receivedAt":"2022-07-08T14:20:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Replace the recently introduced XDL_ALLOC_GROW() with invocations of\nthe GALLOC_GROW() from git-shared-util.h.\n\nAs this change shows the macro + function indirection of\nXDL_ALLOC_GROW() is something we needed only because the two callsites\nwe used it in wanted to use it as an expression, and we thus had to\npass the \"sizeof\" down.\n\nLet's just check the value afterwards instead, which allows us to use\nthe shared macro, we can also remove xdl_reallo(), this was its last\nuser.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n xdiff/xdiff.h    |  1 -\n xdiff/xmacros.h  | 11 -----------\n xdiff/xprepare.c |  8 +++++---\n xdiff/xutils.c   | 17 -----------------\n xdiff/xutils.h   |  2 --\n 5 files changed, 5 insertions(+), 34 deletions(-)\n\ndiff --git a/xdiff/xdiff.h b/xdiff/xdiff.h\nindex 72e25a9ffa5..832cf9d977e 100644\n--- a/xdiff/xdiff.h\n+++ b/xdiff/xdiff.h\n@@ -121,7 +121,6 @@ typedef struct s_bdiffparam {\n \n #define xdl_malloc(x) xmalloc(x)\n #define xdl_free(ptr) free(ptr)\n-#define xdl_realloc(ptr,x) xrealloc(ptr,x)\n \n void *xdl_mmfile_first(mmfile_t *mmf, long *size);\n long xdl_mmfile_size(mmfile_t *mmf);\ndiff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\nindex 75506bdf17e..6a6b3057375 100644\n--- a/xdiff/xmacros.h\n+++ b/xdiff/xmacros.h\n@@ -48,15 +48,4 @@ do { \\\n \t(v) = (unsigned long) __p[0] | ((unsigned long) __p[1]) << 8 | \\\n \t\t((unsigned long) __p[2]) << 16 | ((unsigned long) __p[3]) << 24; \\\n } while (0)\n-\n-/*\n- * Ensure array p can accommodate at least nr elements, growing the\n- * array and updating alloc (which is the number of allocated\n- * elements) as necessary. Frees p and returns -1 on failure, returns\n- * 0 on success\n- */\n-#define XDL_ALLOC_GROW(p, nr, alloc)\t\\\n-\t(-!((nr) <= (alloc) ||\t\t\\\n-\t    ((p) = xdl_alloc_grow_helper((p), (nr), &(alloc), sizeof(*(p))))))\n-\n #endif /* #if !defined(XMACROS_H) */\ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex d6cbee32a2a..4182d9e1c0a 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -128,8 +128,9 @@ static int xdl_classify_record(unsigned int pass, xdlclassifier_t *cf, xrecord_t\n \t\t\treturn -1;\n \t\t}\n \t\trcrec->idx = cf->count++;\n-\t\tif (XDL_ALLOC_GROW(cf->rcrecs, cf->count, cf->alloc))\n-\t\t\t\treturn -1;\n+\t\tGALLOC_GROW(cf->rcrecs, cf->count, cf->alloc);\n+\t\tif (!cf->rcrecs)\n+\t\t\treturn -1;\n \t\tcf->rcrecs[rcrec->idx] = rcrec;\n \t\trcrec->line = line;\n \t\trcrec->size = rec->size;\n@@ -187,7 +188,8 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \t\tfor (top = blk + bsize; cur < top; ) {\n \t\t\tprev = cur;\n \t\t\thav = xdl_hash_record(&cur, top, xpp->flags);\n-\t\t\tif (XDL_ALLOC_GROW(recs, nrec + 1, narec))\n+\t\t\tGALLOC_GROW(recs, nrec + 1, narec);\n+\t\t\tif (!recs)\n \t\t\t\tgoto abort;\n \t\t\tif (!(crec = xdl_cha_alloc(&xdf->rcha)))\n \t\t\t\tgoto abort;\ndiff --git a/xdiff/xutils.c b/xdiff/xutils.c\nindex a6f10353cff..c0cd5338c4e 100644\n--- a/xdiff/xutils.c\n+++ b/xdiff/xutils.c\n@@ -436,20 +436,3 @@ int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n \n \treturn status;\n }\n-\n-void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size)\n-{\n-\tvoid *tmp = NULL;\n-\tsize_t n = ((LONG_MAX - 16) / 2 >= *alloc) ? 2 * *alloc + 16 : LONG_MAX;\n-\tif (nr > n)\n-\t\tn = nr;\n-\tif (SIZE_MAX / size >= n)\n-\t\ttmp = xdl_realloc(p, n * size);\n-\tif (tmp) {\n-\t\t*alloc = n;\n-\t} else {\n-\t\txdl_free(p);\n-\t\t*alloc = 0;\n-\t}\n-\treturn tmp;\n-}\ndiff --git a/xdiff/xutils.h b/xdiff/xutils.h\nindex fd0bba94e8b..7ae3f897bef 100644\n--- a/xdiff/xutils.h\n+++ b/xdiff/xutils.h\n@@ -42,7 +42,5 @@ int xdl_emit_hunk_hdr(long s1, long c1, long s2, long c2,\n int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n \t\t       int line1, int count1, int line2, int count2);\n \n-/* Do not call this function, use XDL_ALLOC_GROW instead */\n-void* xdl_alloc_grow_helper(void* p, long nr, long* alloc, size_t size);\n \n #endif /* #if !defined(XUTILS_H) */\n-- \n2.37.0.913.g189dca38629\n\n"},{"id":"458637","messageId":"patch-4.7-19567077b84-20220708T140354Z-avarab@gmail.com","threadId":"58081","inReplyTo":"cover-0.7-00000000000-20220708T140354Z-avarab@gmail.com","subject":"[PATCH 4/7] xdiff: use G[C]ALLOC_ARRAY(), not XDL_CALLOC_ARRAY()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T14:20:16Z","receivedAt":"2022-07-08T14:20:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Use the newly created GCALLOC_ARRAY() helpers rather than the recently\nintroduced XDL_[C]ALLOC_ARRAY().\n\nAs shown in this diff the calling convention differs, we cannot use\nGCALLOC_ARRAY() as an expression, but that's an advantage in not\nhaving to relay the \"sizeof()\" down via a wrapper function.\n\nThis also:\n\n * Fixes long-standing potential overflow issues, as we're using\n   st_mult() in the underlying G_[C]ALLOC().  Note that the\n\n * Slightly optimizes the \"XDL_CALLOC_ARRAY\", as we'll now use\n   calloc() rather than malloc() + memset() (although smart compilers\n   will probably do the same for both...).\n\n * Changes the \"XDL_CALLOC_ARRAY\" behavior where we'd shortcut if the\n   size was already large enough, but this behavior was changed when\n   XDL_ALLOC_ARRAY() was introduced, so this is safe.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n xdiff/xdiffi.c     |  3 ++-\n xdiff/xhistogram.c |  9 ++++++---\n xdiff/xmacros.h    | 12 ------------\n xdiff/xpatience.c  |  6 ++++--\n xdiff/xprepare.c   | 24 ++++++++++++++++--------\n 5 files changed, 28 insertions(+), 26 deletions(-)\n\ndiff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\nindex 6fded43e87d..077cc456087 100644\n--- a/xdiff/xdiffi.c\n+++ b/xdiff/xdiffi.c\n@@ -333,7 +333,8 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n \t * One is to store the forward path and one to store the backward path.\n \t */\n \tndiags = xe->xdf1.nreff + xe->xdf2.nreff + 3;\n-\tif (!XDL_ALLOC_ARRAY(kvd, 2 * ndiags + 2))\n+\tGALLOC_ARRAY(kvd, 2 * ndiags + 2);\n+\tif (!kvd)\n \t\treturn -1;\n \tkvdf = kvd;\n \tkvdb = kvdf + ndiags;\ndiff --git a/xdiff/xhistogram.c b/xdiff/xhistogram.c\nindex df909004c10..f20592bfbdd 100644\n--- a/xdiff/xhistogram.c\n+++ b/xdiff/xhistogram.c\n@@ -266,14 +266,17 @@ static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n \n \tindex.table_bits = xdl_hashbits(count1);\n \tindex.records_size = 1 << index.table_bits;\n-\tif (!XDL_CALLOC_ARRAY(index.records, index.records_size))\n+\tGCALLOC_ARRAY(index.records, index.records_size);\n+\tif (!index.records)\n \t\tgoto cleanup;\n \n \tindex.line_map_size = count1;\n-\tif (!XDL_CALLOC_ARRAY(index.line_map, index.line_map_size))\n+\tGCALLOC_ARRAY(index.line_map, index.line_map_size);\n+\tif (!index.line_map)\n \t\tgoto cleanup;\n \n-\tif (!XDL_CALLOC_ARRAY(index.next_ptrs, index.line_map_size))\n+\tGCALLOC_ARRAY(index.next_ptrs, index.line_map_size);\n+\tif (!index.next_ptrs)\n \t\tgoto cleanup;\n \n \t/* lines / 4 + 1 comes from xprepare.c:xdl_prepare_ctx() */\ndiff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\nindex d13a6724629..75506bdf17e 100644\n--- a/xdiff/xmacros.h\n+++ b/xdiff/xmacros.h\n@@ -49,18 +49,6 @@ do { \\\n \t\t((unsigned long) __p[2]) << 16 | ((unsigned long) __p[3]) << 24; \\\n } while (0)\n \n-/* Allocate an array of nr elements, returns NULL on failure */\n-#define XDL_ALLOC_ARRAY(p, nr)\t\t\t\t\\\n-\t((p) = SIZE_MAX / sizeof(*(p)) >= (size_t)(nr)\t\\\n-\t\t? xdl_malloc((nr) * sizeof(*(p)))\t\\\n-\t\t: NULL)\n-\n-/* Allocate an array of nr zeroed out elements, returns NULL on failure */\n-#define XDL_CALLOC_ARRAY(p, nr)\t\t\t\t\\\n-\t(XDL_ALLOC_ARRAY((p), (nr))\t\t\t\\\n-\t\t? memset((p), 0, (nr) * sizeof(*(p)))\t\\\n-\t\t: NULL)\n-\n /*\n  * Ensure array p can accommodate at least nr elements, growing the\n  * array and updating alloc (which is the number of allocated\ndiff --git a/xdiff/xpatience.c b/xdiff/xpatience.c\nindex fe39c2978cb..bb328d9f852 100644\n--- a/xdiff/xpatience.c\n+++ b/xdiff/xpatience.c\n@@ -151,7 +151,8 @@ static int fill_hashmap(mmfile_t *file1, mmfile_t *file2,\n \n \t/* We know exactly how large we want the hash map */\n \tresult->alloc = count1 * 2;\n-\tif (!XDL_CALLOC_ARRAY(result->entries, result->alloc))\n+\tGCALLOC_ARRAY(result->entries, result->alloc);\n+\tif (!result->entries)\n \t\treturn -1;\n \n \t/* First, fill with entries from the first file */\n@@ -208,7 +209,8 @@ static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n \t */\n \tint anchor_i = -1;\n \n-\tif (!XDL_ALLOC_ARRAY(sequence, map->nr))\n+\tGALLOC_ARRAY(sequence, map->nr);\n+\tif (!sequence)\n \t\treturn -1;\n \n \tfor (entry = map->first; entry; entry = entry->next) {\ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex c84549f6c50..d6cbee32a2a 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -78,15 +78,17 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n \n \t\treturn -1;\n \t}\n-\tif (!XDL_CALLOC_ARRAY(cf->rchash, cf->hsize)) {\n+\tGCALLOC_ARRAY(cf->rchash, cf->hsize);\n+\tif (!cf->rchash) {\n \n \t\txdl_cha_free(&cf->ncha);\n \t\treturn -1;\n \t}\n \n \tcf->alloc = size;\n-\tif (!XDL_ALLOC_ARRAY(cf->rcrecs, cf->alloc)) {\n \n+\tGALLOC_ARRAY(cf->rcrecs, cf->alloc);\n+\tif (!cf->rcrecs) {\n \t\txdl_free(cf->rchash);\n \t\txdl_cha_free(&cf->ncha);\n \t\treturn -1;\n@@ -170,12 +172,14 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \n \tif (xdl_cha_init(&xdf->rcha, sizeof(xrecord_t), narec / 4 + 1) < 0)\n \t\tgoto abort;\n-\tif (!XDL_ALLOC_ARRAY(recs, narec))\n+\tGALLOC_ARRAY(recs, narec);\n+\tif (!recs)\n \t\tgoto abort;\n \n \thbits = xdl_hashbits((unsigned int) narec);\n \thsize = 1 << hbits;\n-\tif (!XDL_CALLOC_ARRAY(rhash, hsize))\n+\tGCALLOC_ARRAY(rhash, hsize);\n+\tif (!rhash)\n \t\tgoto abort;\n \n \tnrec = 0;\n@@ -196,14 +200,17 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \t\t}\n \t}\n \n-\tif (!XDL_CALLOC_ARRAY(rchg, nrec + 2))\n+\tGCALLOC_ARRAY(rchg, nrec + 2);\n+\tif (!rchg)\n \t\tgoto abort;\n \n \tif ((XDF_DIFF_ALG(xpp->flags) != XDF_PATIENCE_DIFF) &&\n \t    (XDF_DIFF_ALG(xpp->flags) != XDF_HISTOGRAM_DIFF)) {\n-\t\tif (!XDL_ALLOC_ARRAY(rindex, nrec + 1))\n+\t\tGALLOC_ARRAY(rindex, nrec + 1);\n+\t\tif (!rindex)\n \t\t\tgoto abort;\n-\t\tif (!XDL_ALLOC_ARRAY(ha, nrec + 1))\n+\t\tGALLOC_ARRAY(ha, nrec + 1);\n+\t\tif (!ha)\n \t\t\tgoto abort;\n \t}\n \n@@ -369,7 +376,8 @@ static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd\n \txdlclass_t *rcrec;\n \tchar *dis, *dis1, *dis2;\n \n-\tif (!XDL_CALLOC_ARRAY(dis, xdf1->nrec + xdf2->nrec + 2))\n+\tGCALLOC_ARRAY(dis, xdf1->nrec + xdf2->nrec + 2);\n+\tif (!dis)\n \t\treturn -1;\n \tdis1 = dis;\n \tdis2 = dis1 + xdf1->nrec + 1;\n-- \n2.37.0.913.g189dca38629\n\n"},{"id":"458638","messageId":"patch-6.7-33d93f121a9-20220708T140354Z-avarab@gmail.com","threadId":"58081","inReplyTo":"cover-0.7-00000000000-20220708T140354Z-avarab@gmail.com","subject":"[PATCH 6/7] xdiff: remove xdl_malloc() wrapper, use malloc(), not xmalloc()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T14:20:18Z","receivedAt":"2022-07-08T14:20:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As noted in 36c83197249 (xdiff: use xmalloc/xrealloc, 2019-04-11) the\nreason we have xdl_malloc() in the first place was to make the xdiff\ncode safer, it was not handling some allocation failures correctly at\nthe time.\n\nBut as noted in that commit doing this was intended as a stopgap, as\nvarious code in xdiff/* did not correctly handle allocation failures,\nand would have segfaulted if malloc() returned NULL.\n\nBut since then and after preceding commits we can be confident that\nmalloc() failures are handled correctly. All of these users of\nxdl_malloc() are checking their return values, and we're returning\n-1 (or similar ) to the top-level in case of failures.\n\nThis also has the big advantage of making the compiler and analysis\ntools less confused, and potentially avoiding undefined (compiler)\nbehavior. I.e. we define our own xmalloc() to call die() on failure,\nand that function uses the non-standard \"noreturn\" attribute on our\nmost common compiler targets.\n\nBut xdl_malloc()'s use conflicted with that, confusing both human\nreaders of this code, and potentially compilers as well.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n xdiff/xdiff.h  |  1 -\n xdiff/xdiffi.c |  2 +-\n xdiff/xmerge.c | 10 +++++-----\n xdiff/xutils.c |  2 +-\n 4 files changed, 7 insertions(+), 8 deletions(-)\n\ndiff --git a/xdiff/xdiff.h b/xdiff/xdiff.h\nindex 832cf9d977e..df048e0099b 100644\n--- a/xdiff/xdiff.h\n+++ b/xdiff/xdiff.h\n@@ -119,7 +119,6 @@ typedef struct s_bdiffparam {\n } bdiffparam_t;\n \n \n-#define xdl_malloc(x) xmalloc(x)\n #define xdl_free(ptr) free(ptr)\n \n void *xdl_mmfile_first(mmfile_t *mmf, long *size);\ndiff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\nindex 077cc456087..6590811634f 100644\n--- a/xdiff/xdiffi.c\n+++ b/xdiff/xdiffi.c\n@@ -368,7 +368,7 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n static xdchange_t *xdl_add_change(xdchange_t *xscr, long i1, long i2, long chg1, long chg2) {\n \txdchange_t *xch;\n \n-\tif (!(xch = (xdchange_t *) xdl_malloc(sizeof(xdchange_t))))\n+\tif (!(xch = (xdchange_t *) malloc(sizeof(xdchange_t))))\n \t\treturn NULL;\n \n \txch->next = xscr;\ndiff --git a/xdiff/xmerge.c b/xdiff/xmerge.c\nindex ac0cf52f3be..676ad39020d 100644\n--- a/xdiff/xmerge.c\n+++ b/xdiff/xmerge.c\n@@ -60,7 +60,7 @@ static int xdl_append_merge(xdmerge_t **merge, int mode,\n \t\tm->chg1 = i1 + chg1 - m->i1;\n \t\tm->chg2 = i2 + chg2 - m->i2;\n \t} else {\n-\t\tm = xdl_malloc(sizeof(xdmerge_t));\n+\t\tm = malloc(sizeof(xdmerge_t));\n \t\tif (!m)\n \t\t\treturn -1;\n \t\tm->next = NULL;\n@@ -409,7 +409,7 @@ static int xdl_refine_conflicts(xdfenv_t *xe1, xdfenv_t *xe2, xdmerge_t *m,\n \t\tm->i2 = xscr->i2 + i2;\n \t\tm->chg2 = xscr->chg2;\n \t\twhile (xscr->next) {\n-\t\t\txdmerge_t *m2 = xdl_malloc(sizeof(xdmerge_t));\n+\t\t\txdmerge_t *m2 = malloc(sizeof(xdmerge_t));\n \t\t\tif (!m2) {\n \t\t\t\txdl_free_env(&xe);\n \t\t\t\txdl_free_script(x);\n@@ -670,7 +670,7 @@ static int xdl_do_merge(xdfenv_t *xe1, xdchange_t *xscr1,\n \t\t\t\t\t\t ancestor_name,\n \t\t\t\t\t\t favor, changes, NULL, style,\n \t\t\t\t\t\t marker_size);\n-\t\tresult->ptr = xdl_malloc(size);\n+\t\tresult->ptr = malloc(size);\n \t\tif (!result->ptr) {\n \t\t\txdl_cleanup_merge(changes);\n \t\t\treturn -1;\n@@ -718,14 +718,14 @@ int xdl_merge(mmfile_t *orig, mmfile_t *mf1, mmfile_t *mf2,\n \t}\n \tstatus = 0;\n \tif (!xscr1) {\n-\t\tresult->ptr = xdl_malloc(mf2->size);\n+\t\tresult->ptr = malloc(mf2->size);\n \t\tif (!result->ptr)\n \t\t\tgoto out;\n \t\tstatus = 0;\n \t\tmemcpy(result->ptr, mf2->ptr, mf2->size);\n \t\tresult->size = mf2->size;\n \t} else if (!xscr2) {\n-\t\tresult->ptr = xdl_malloc(mf1->size);\n+\t\tresult->ptr = malloc(mf1->size);\n \t\tif (!result->ptr)\n \t\t\tgoto out;\n \t\tstatus = 0;\ndiff --git a/xdiff/xutils.c b/xdiff/xutils.c\nindex c0cd5338c4e..865e08f0e93 100644\n--- a/xdiff/xutils.c\n+++ b/xdiff/xutils.c\n@@ -98,7 +98,7 @@ void *xdl_cha_alloc(chastore_t *cha) {\n \tvoid *data;\n \n \tif (!(ancur = cha->ancur) || ancur->icurr == cha->nsize) {\n-\t\tif (!(ancur = (chanode_t *) xdl_malloc(sizeof(chanode_t) + cha->nsize))) {\n+\t\tif (!(ancur = (chanode_t *) malloc(sizeof(chanode_t) + cha->nsize))) {\n \n \t\t\treturn NULL;\n \t\t}\n-- \n2.37.0.913.g189dca38629\n\n"},{"id":"458639","messageId":"patch-7.7-a1bf9a94f0a-20220708T140354Z-avarab@gmail.com","threadId":"58081","inReplyTo":"cover-0.7-00000000000-20220708T140354Z-avarab@gmail.com","subject":"[PATCH 7/7] xdiff: remove xdl_free(), use free() instead","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T14:20:19Z","receivedAt":"2022-07-08T14:21:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the xdl_free() wrapper macro in favor of using free()\ndirectly. The wrapper macro was brought in with the initial import of\nxdiff/ in 3443546f6ef (Use a *real* built-in diff generator,\n2006-03-24).\n\nAs subsequent discussions on the topic[1] have made clear there's no\nreason to use this wrapper. Both git itself as well as any external\nusers such as libgit2 compile the xdiff/* code as part of their own\ncompilation, and can thus find the right malloc() and free() at\nlink-time.\n\nWhen compiling git we already find a custom malloc() and free()\ne.g. if compiled with USE_NED_ALLOCATOR=YesPlease.\n\n1. https://lore.kernel.org/git/220415.867d7qbaad.gmgdl@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n xdiff/xdiff.h      |  3 ---\n xdiff/xdiffi.c     |  4 ++--\n xdiff/xhistogram.c |  6 +++---\n xdiff/xpatience.c  |  8 ++++----\n xdiff/xprepare.c   | 28 ++++++++++++++--------------\n xdiff/xutils.c     |  2 +-\n 6 files changed, 24 insertions(+), 27 deletions(-)\n\ndiff --git a/xdiff/xdiff.h b/xdiff/xdiff.h\nindex df048e0099b..a37d89fcdaf 100644\n--- a/xdiff/xdiff.h\n+++ b/xdiff/xdiff.h\n@@ -118,9 +118,6 @@ typedef struct s_bdiffparam {\n \tlong bsize;\n } bdiffparam_t;\n \n-\n-#define xdl_free(ptr) free(ptr)\n-\n void *xdl_mmfile_first(mmfile_t *mmf, long *size);\n long xdl_mmfile_size(mmfile_t *mmf);\n \ndiff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\nindex 6590811634f..375bb81a8aa 100644\n--- a/xdiff/xdiffi.c\n+++ b/xdiff/xdiffi.c\n@@ -359,7 +359,7 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n \tres = xdl_recs_cmp(&dd1, 0, dd1.nrec, &dd2, 0, dd2.nrec,\n \t\t\t   kvdf, kvdb, (xpp->flags & XDF_NEED_MINIMAL) != 0,\n \t\t\t   &xenv);\n-\txdl_free(kvd);\n+\tfree(kvd);\n \n \treturn res;\n }\n@@ -960,7 +960,7 @@ void xdl_free_script(xdchange_t *xscr) {\n \n \twhile ((xch = xscr) != NULL) {\n \t\txscr = xscr->next;\n-\t\txdl_free(xch);\n+\t\tfree(xch);\n \t}\n }\n \ndiff --git a/xdiff/xhistogram.c b/xdiff/xhistogram.c\nindex f20592bfbdd..be35d9c9697 100644\n--- a/xdiff/xhistogram.c\n+++ b/xdiff/xhistogram.c\n@@ -240,9 +240,9 @@ static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env,\n \n static inline void free_index(struct histindex *index)\n {\n-\txdl_free(index->records);\n-\txdl_free(index->line_map);\n-\txdl_free(index->next_ptrs);\n+\tfree(index->records);\n+\tfree(index->line_map);\n+\tfree(index->next_ptrs);\n \txdl_cha_free(&index->rcha);\n }\n \ndiff --git a/xdiff/xpatience.c b/xdiff/xpatience.c\nindex bb328d9f852..8fffd2b8297 100644\n--- a/xdiff/xpatience.c\n+++ b/xdiff/xpatience.c\n@@ -233,7 +233,7 @@ static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n \t/* No common unique lines were found */\n \tif (!longest) {\n \t\t*res = NULL;\n-\t\txdl_free(sequence);\n+\t\tfree(sequence);\n \t\treturn 0;\n \t}\n \n@@ -245,7 +245,7 @@ static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n \t\tentry = entry->previous;\n \t}\n \t*res = entry;\n-\txdl_free(sequence);\n+\tfree(sequence);\n \treturn 0;\n }\n \n@@ -358,7 +358,7 @@ static int patience_diff(mmfile_t *file1, mmfile_t *file2,\n \t\t\tenv->xdf1.rchg[line1++ - 1] = 1;\n \t\twhile(count2--)\n \t\t\tenv->xdf2.rchg[line2++ - 1] = 1;\n-\t\txdl_free(map.entries);\n+\t\tfree(map.entries);\n \t\treturn 0;\n \t}\n \n@@ -372,7 +372,7 @@ static int patience_diff(mmfile_t *file1, mmfile_t *file2,\n \t\tresult = fall_back_to_classic_diff(&map,\n \t\t\tline1, count1, line2, count2);\n  out:\n-\txdl_free(map.entries);\n+\tfree(map.entries);\n \treturn result;\n }\n \ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex 4182d9e1c0a..169629761c0 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -89,7 +89,7 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n \n \tGALLOC_ARRAY(cf->rcrecs, cf->alloc);\n \tif (!cf->rcrecs) {\n-\t\txdl_free(cf->rchash);\n+\t\tfree(cf->rchash);\n \t\txdl_cha_free(&cf->ncha);\n \t\treturn -1;\n \t}\n@@ -102,8 +102,8 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n \n static void xdl_free_classifier(xdlclassifier_t *cf) {\n \n-\txdl_free(cf->rcrecs);\n-\txdl_free(cf->rchash);\n+\tfree(cf->rcrecs);\n+\tfree(cf->rchash);\n \txdl_cha_free(&cf->ncha);\n }\n \n@@ -230,11 +230,11 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \treturn 0;\n \n abort:\n-\txdl_free(ha);\n-\txdl_free(rindex);\n-\txdl_free(rchg);\n-\txdl_free(rhash);\n-\txdl_free(recs);\n+\tfree(ha);\n+\tfree(rindex);\n+\tfree(rchg);\n+\tfree(rhash);\n+\tfree(recs);\n \txdl_cha_free(&xdf->rcha);\n \treturn -1;\n }\n@@ -242,11 +242,11 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \n static void xdl_free_ctx(xdfile_t *xdf) {\n \n-\txdl_free(xdf->rhash);\n-\txdl_free(xdf->rindex);\n-\txdl_free(xdf->rchg - 1);\n-\txdl_free(xdf->ha);\n-\txdl_free(xdf->recs);\n+\tfree(xdf->rhash);\n+\tfree(xdf->rindex);\n+\tfree(xdf->rchg - 1);\n+\tfree(xdf->ha);\n+\tfree(xdf->recs);\n \txdl_cha_free(&xdf->rcha);\n }\n \n@@ -424,7 +424,7 @@ static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd\n \t}\n \txdf2->nreff = nreff;\n \n-\txdl_free(dis);\n+\tfree(dis);\n \n \treturn 0;\n }\ndiff --git a/xdiff/xutils.c b/xdiff/xutils.c\nindex 865e08f0e93..00eeba452a5 100644\n--- a/xdiff/xutils.c\n+++ b/xdiff/xutils.c\n@@ -88,7 +88,7 @@ void xdl_cha_free(chastore_t *cha) {\n \n \tfor (cur = cha->head; (tmp = cur) != NULL;) {\n \t\tcur = cur->next;\n-\t\txdl_free(tmp);\n+\t\tfree(tmp);\n \t}\n }\n \n-- \n2.37.0.913.g189dca38629\n\n"},{"id":"458644","messageId":"pull.1272.v2.git.1657297519.gitgitgadget@gmail.com","threadId":"58081","inReplyTo":"pull.1272.git.1656516334.gitgitgadget@gmail.com","subject":"[PATCH v2 0/4] xdiff: introduce memory allocation macros","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-08T16:25:15Z","receivedAt":"2022-07-08T16:26:17Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"This patch series introduces macros for allocating and growing arrays in\nxdiff. The macros are similar to ALLOC_ARRAY()/ALLOC_GROW() from the rest of\nthe code base but return an error on failure to allow libgit2 to handle\nmemory allocation failures gracefully rather than dying. The macros\nintroduce overflow checks but these checks are currently redundant as we\nlimit the maximum file size passed to xdiff and these checks alone are\ninsufficient to safely remove the size limit. The aim of this series is to\nmake the xdiff code more readable, there should be no change in behavior (as\nsuch I'm open to the argument that these are just churn and should be\ndropped).\n\nThanks to Junio and Ævar the feedback on V1 Changes since V1:\n\n * Patch 2: new, it replaces xdl_malloc() followed by memset() with\n   xdl_calloc() as suggested by Junio\n * Patch 3: simplified by the new patch 2, the end result is unchanged\n * Patch 4: added a note to the commit message explaining the choice to use\n   long rather than size_t\n\nPhillip Wood (4):\n  xdiff: introduce XDL_ALLOC_ARRAY()\n  xdiff: introduce xdl_calloc\n  xdiff: introduce XDL_CALLOC_ARRAY()\n  xdiff: introduce XDL_ALLOC_GROW()\n\n xdiff/xdiff.h      |  1 +\n xdiff/xdiffi.c     |  2 +-\n xdiff/xhistogram.c | 19 ++++++-------------\n xdiff/xmacros.h    | 18 ++++++++++++++++++\n xdiff/xpatience.c  |  9 +++------\n xdiff/xprepare.c   | 41 ++++++++++++-----------------------------\n xdiff/xutils.c     | 17 +++++++++++++++++\n xdiff/xutils.h     |  3 ++-\n 8 files changed, 60 insertions(+), 50 deletions(-)\n\n\nbase-commit: e4a4b31577c7419497ac30cebe30d755b97752c5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1272%2Fphillipwood%2Fwip%2Fxdiff-memory-allocation-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1272/phillipwood/wip/xdiff-memory-allocation-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1272\n\nRange-diff vs v1:\n\n 1:  55fd62dc27d = 1:  55fd62dc27d xdiff: introduce XDL_ALLOC_ARRAY()\n 2:  8bead9856be ! 2:  ea8993a57ef xdiff: introduce XDL_CALLOC_ARRAY()\n     @@ Metadata\n      Author: Phillip Wood <phillip.wood@dunelm.org.uk>\n      \n       ## Commit message ##\n     -    xdiff: introduce XDL_CALLOC_ARRAY()\n     +    xdiff: introduce xdl_calloc\n      \n     -    Add a helper for allocating an array and initialize the elements to\n     -    zero. This is analogous to CALLOC_ARRAY() in the rest of the codebase\n     -    but it returns NULL on allocation failures rather than dying to\n     -    accommodate other users of libxdiff such as libgit2.\n     +    In preparation for introducing XDL_CALLOC_ARRAY() use calloc() to\n     +    obtain zeroed out memory rather than malloc() followed by memset(). To\n     +    try and keep the lines a reasonable length this commit also stops\n     +    casting the pointer returned by calloc() as this is unnecessary.\n      \n          Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>\n      \n     + ## xdiff/xdiff.h ##\n     +@@ xdiff/xdiff.h: typedef struct s_bdiffparam {\n     + \n     + \n     + #define xdl_malloc(x) xmalloc(x)\n     ++#define xdl_calloc(n, sz) xcalloc(n, sz)\n     + #define xdl_free(ptr) free(ptr)\n     + #define xdl_realloc(ptr,x) xrealloc(ptr,x)\n     + \n     +\n       ## xdiff/xhistogram.c ##\n      @@ xdiff/xhistogram.c: static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n       \t\t    int line1, int count1, int line2, int count2)\n     @@ xdiff/xhistogram.c: static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n      -\tsz *= sizeof(struct record *);\n      -\tif (!(index.records = (struct record **) xdl_malloc(sz)))\n      +\tindex.records_size = 1 << index.table_bits;\n     -+\tif (!XDL_CALLOC_ARRAY(index.records, index.records_size))\n     ++\tif (!(index.records = xdl_calloc(index.records_size,\n     ++\t\t\t\t\t sizeof(*index.records))))\n       \t\tgoto cleanup;\n      -\tmemset(index.records, 0, sz);\n       \n     @@ xdiff/xhistogram.c: static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n      -\tsz *= sizeof(struct record *);\n      -\tif (!(index.line_map = (struct record **) xdl_malloc(sz)))\n      +\tindex.line_map_size = count1;\n     -+\tif (!XDL_CALLOC_ARRAY(index.line_map, index.line_map_size))\n     ++\tif (!(index.line_map = xdl_calloc(index.line_map_size,\n     ++\t\t\t\t\t  sizeof(*index.line_map))))\n       \t\tgoto cleanup;\n      -\tmemset(index.line_map, 0, sz);\n       \n      -\tsz = index.line_map_size;\n      -\tsz *= sizeof(unsigned int);\n      -\tif (!(index.next_ptrs = (unsigned int *) xdl_malloc(sz)))\n     -+\tif (!XDL_CALLOC_ARRAY(index.next_ptrs, index.line_map_size))\n     ++\tif (!(index.next_ptrs = xdl_calloc(index.line_map_size,\n     ++\t\t\t\t\t   sizeof(*index.next_ptrs))))\n       \t\tgoto cleanup;\n      -\tmemset(index.next_ptrs, 0, sz);\n       \n       \t/* lines / 4 + 1 comes from xprepare.c:xdl_prepare_ctx() */\n       \tif (xdl_cha_init(&index.rcha, sizeof(struct record), count1 / 4 + 1) < 0)\n      \n     - ## xdiff/xmacros.h ##\n     -@@ xdiff/xmacros.h: do { \\\n     - \t\t? xdl_malloc((nr) * sizeof(*(p)))\t\\\n     - \t\t: NULL)\n     - \n     -+/* Allocate an array of nr zeroed out elements, returns NULL on failure */\n     -+#define XDL_CALLOC_ARRAY(p, nr)\t\t\t\t\\\n     -+\t(XDL_ALLOC_ARRAY((p), (nr))\t\t\t\\\n     -+\t\t? memset((p), 0, (nr) * sizeof(*(p)))\t\\\n     -+\t\t: NULL)\n     -+\n     - #endif /* #if !defined(XMACROS_H) */\n     -\n       ## xdiff/xpatience.c ##\n      @@ xdiff/xpatience.c: static int fill_hashmap(mmfile_t *file1, mmfile_t *file2,\n       \n     @@ xdiff/xpatience.c: static int fill_hashmap(mmfile_t *file1, mmfile_t *file2,\n       \tresult->alloc = count1 * 2;\n      -\tresult->entries = (struct entry *)\n      -\t\txdl_malloc(result->alloc * sizeof(struct entry));\n     --\tif (!result->entries)\n     -+\tif (!XDL_CALLOC_ARRAY(result->entries, result->alloc))\n     ++\tresult->entries = xdl_calloc(result->alloc, sizeof(*result->entries));\n     + \tif (!result->entries)\n       \t\treturn -1;\n      -\tmemset(result->entries, 0, result->alloc * sizeof(struct entry));\n       \n     @@ xdiff/xprepare.c: static int xdl_init_classifier(xdlclassifier_t *cf, long size,\n       \t\treturn -1;\n       \t}\n      -\tif (!(cf->rchash = (xdlclass_t **) xdl_malloc(cf->hsize * sizeof(xdlclass_t *)))) {\n     -+\tif (!XDL_CALLOC_ARRAY(cf->rchash, cf->hsize)) {\n     ++\tif (!(cf->rchash = xdl_calloc(cf->hsize, sizeof(*cf->rchash)))) {\n       \n       \t\txdl_cha_free(&cf->ncha);\n       \t\treturn -1;\n     @@ xdiff/xprepare.c: static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, lo\n       \thbits = xdl_hashbits((unsigned int) narec);\n       \thsize = 1 << hbits;\n      -\tif (!(rhash = (xrecord_t **) xdl_malloc(hsize * sizeof(xrecord_t *))))\n     -+\tif (!XDL_CALLOC_ARRAY(rhash, hsize))\n     ++\tif (!(rhash = xdl_calloc(hsize, sizeof(*rhash))))\n       \t\tgoto abort;\n      -\tmemset(rhash, 0, hsize * sizeof(xrecord_t *));\n       \n     @@ xdiff/xprepare.c: static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, lo\n       \t}\n       \n      -\tif (!(rchg = (char *) xdl_malloc((nrec + 2) * sizeof(char))))\n     -+\tif (!XDL_CALLOC_ARRAY(rchg, nrec + 2))\n     ++\tif (!(rchg = xdl_calloc((nrec + 2), sizeof(*rchg))))\n       \t\tgoto abort;\n      -\tmemset(rchg, 0, (nrec + 2) * sizeof(char));\n       \n     @@ xdiff/xprepare.c: static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *\n       \tchar *dis, *dis1, *dis2;\n       \n      -\tif (!(dis = (char *) xdl_malloc(xdf1->nrec + xdf2->nrec + 2))) {\n     --\n     -+\tif (!XDL_CALLOC_ARRAY(dis, xdf1->nrec + xdf2->nrec + 2))\n     ++\tif (!(dis = xdl_calloc(xdf1->nrec + xdf2->nrec + 2, sizeof(*dis)))) {\n     + \n       \t\treturn -1;\n     --\t}\n     + \t}\n      -\tmemset(dis, 0, xdf1->nrec + xdf2->nrec + 2);\n       \tdis1 = dis;\n       \tdis2 = dis1 + xdf1->nrec + 1;\n -:  ----------- > 3:  5490fd22579 xdiff: introduce XDL_CALLOC_ARRAY()\n 3:  da996677f07 ! 4:  8c24cd7737b xdiff: introduce XDL_ALLOC_GROW()\n     @@ Commit message\n          of adding a small amount to the new allocation to avoid a lot of\n          reallocations at small sizes.\n      \n     +    Note that xdl_alloc_grow_helper() uses long rather than size_t for\n     +    `nr` and `alloc` to match the existing code.\n     +\n          Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>\n      \n       ## xdiff/xmacros.h ##\n      @@ xdiff/xmacros.h: do { \\\n     - \t\t? memset((p), 0, (nr) * sizeof(*(p)))\t\\\n     - \t\t: NULL)\n     + /* Allocate an array of nr zeroed out elements, returns NULL on failure */\n     + #define XDL_CALLOC_ARRAY(p, nr)\t((p) = xdl_calloc(nr, sizeof(*(p))))\n       \n      +/*\n      + * Ensure array p can accommodate at least nr elements, growing the\n\n-- \ngitgitgadget\n"},{"id":"458645","messageId":"ea8993a57efea4af347e00e25eed6487fec28a53.1657297520.git.gitgitgadget@gmail.com","threadId":"58081","inReplyTo":"pull.1272.v2.git.1657297519.gitgitgadget@gmail.com","subject":"[PATCH v2 2/4] xdiff: introduce xdl_calloc","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-08T16:25:17Z","receivedAt":"2022-07-08T16:26:18Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood <phillip.wood@dunelm.org.uk>\n\nIn preparation for introducing XDL_CALLOC_ARRAY() use calloc() to\nobtain zeroed out memory rather than malloc() followed by memset(). To\ntry and keep the lines a reasonable length this commit also stops\ncasting the pointer returned by calloc() as this is unnecessary.\n\nSigned-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>\n---\n xdiff/xdiff.h      |  1 +\n xdiff/xhistogram.c | 22 +++++++++-------------\n xdiff/xpatience.c  |  4 +---\n xdiff/xprepare.c   | 12 ++++--------\n 4 files changed, 15 insertions(+), 24 deletions(-)\n\ndiff --git a/xdiff/xdiff.h b/xdiff/xdiff.h\nindex 72e25a9ffa5..bb56b23f34c 100644\n--- a/xdiff/xdiff.h\n+++ b/xdiff/xdiff.h\n@@ -120,6 +120,7 @@ typedef struct s_bdiffparam {\n \n \n #define xdl_malloc(x) xmalloc(x)\n+#define xdl_calloc(n, sz) xcalloc(n, sz)\n #define xdl_free(ptr) free(ptr)\n #define xdl_realloc(ptr,x) xrealloc(ptr,x)\n \ndiff --git a/xdiff/xhistogram.c b/xdiff/xhistogram.c\nindex 01decffc332..c97edc1e29e 100644\n--- a/xdiff/xhistogram.c\n+++ b/xdiff/xhistogram.c\n@@ -251,7 +251,7 @@ static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n \t\t    int line1, int count1, int line2, int count2)\n {\n \tint b_ptr;\n-\tint sz, ret = -1;\n+\tint ret = -1;\n \tstruct histindex index;\n \n \tmemset(&index, 0, sizeof(index));\n@@ -265,23 +265,19 @@ static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n \tindex.rcha.head = NULL;\n \n \tindex.table_bits = xdl_hashbits(count1);\n-\tsz = index.records_size = 1 << index.table_bits;\n-\tsz *= sizeof(struct record *);\n-\tif (!(index.records = (struct record **) xdl_malloc(sz)))\n+\tindex.records_size = 1 << index.table_bits;\n+\tif (!(index.records = xdl_calloc(index.records_size,\n+\t\t\t\t\t sizeof(*index.records))))\n \t\tgoto cleanup;\n-\tmemset(index.records, 0, sz);\n \n-\tsz = index.line_map_size = count1;\n-\tsz *= sizeof(struct record *);\n-\tif (!(index.line_map = (struct record **) xdl_malloc(sz)))\n+\tindex.line_map_size = count1;\n+\tif (!(index.line_map = xdl_calloc(index.line_map_size,\n+\t\t\t\t\t  sizeof(*index.line_map))))\n \t\tgoto cleanup;\n-\tmemset(index.line_map, 0, sz);\n \n-\tsz = index.line_map_size;\n-\tsz *= sizeof(unsigned int);\n-\tif (!(index.next_ptrs = (unsigned int *) xdl_malloc(sz)))\n+\tif (!(index.next_ptrs = xdl_calloc(index.line_map_size,\n+\t\t\t\t\t   sizeof(*index.next_ptrs))))\n \t\tgoto cleanup;\n-\tmemset(index.next_ptrs, 0, sz);\n \n \t/* lines / 4 + 1 comes from xprepare.c:xdl_prepare_ctx() */\n \tif (xdl_cha_init(&index.rcha, sizeof(struct record), count1 / 4 + 1) < 0)\ndiff --git a/xdiff/xpatience.c b/xdiff/xpatience.c\nindex ce87b9084ca..94f8886dd14 100644\n--- a/xdiff/xpatience.c\n+++ b/xdiff/xpatience.c\n@@ -151,11 +151,9 @@ static int fill_hashmap(mmfile_t *file1, mmfile_t *file2,\n \n \t/* We know exactly how large we want the hash map */\n \tresult->alloc = count1 * 2;\n-\tresult->entries = (struct entry *)\n-\t\txdl_malloc(result->alloc * sizeof(struct entry));\n+\tresult->entries = xdl_calloc(result->alloc, sizeof(*result->entries));\n \tif (!result->entries)\n \t\treturn -1;\n-\tmemset(result->entries, 0, result->alloc * sizeof(struct entry));\n \n \t/* First, fill with entries from the first file */\n \twhile (count1--)\ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex 25866a1667a..45190e59fc8 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -78,12 +78,11 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n \n \t\treturn -1;\n \t}\n-\tif (!(cf->rchash = (xdlclass_t **) xdl_malloc(cf->hsize * sizeof(xdlclass_t *)))) {\n+\tif (!(cf->rchash = xdl_calloc(cf->hsize, sizeof(*cf->rchash)))) {\n \n \t\txdl_cha_free(&cf->ncha);\n \t\treturn -1;\n \t}\n-\tmemset(cf->rchash, 0, cf->hsize * sizeof(xdlclass_t *));\n \n \tcf->alloc = size;\n \tif (!XDL_ALLOC_ARRAY(cf->rcrecs, cf->alloc)) {\n@@ -183,9 +182,8 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \n \thbits = xdl_hashbits((unsigned int) narec);\n \thsize = 1 << hbits;\n-\tif (!(rhash = (xrecord_t **) xdl_malloc(hsize * sizeof(xrecord_t *))))\n+\tif (!(rhash = xdl_calloc(hsize, sizeof(*rhash))))\n \t\tgoto abort;\n-\tmemset(rhash, 0, hsize * sizeof(xrecord_t *));\n \n \tnrec = 0;\n \tif ((cur = blk = xdl_mmfile_first(mf, &bsize))) {\n@@ -209,9 +207,8 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \t\t}\n \t}\n \n-\tif (!(rchg = (char *) xdl_malloc((nrec + 2) * sizeof(char))))\n+\tif (!(rchg = xdl_calloc((nrec + 2), sizeof(*rchg))))\n \t\tgoto abort;\n-\tmemset(rchg, 0, (nrec + 2) * sizeof(char));\n \n \tif ((XDF_DIFF_ALG(xpp->flags) != XDF_PATIENCE_DIFF) &&\n \t    (XDF_DIFF_ALG(xpp->flags) != XDF_HISTOGRAM_DIFF)) {\n@@ -383,11 +380,10 @@ static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd\n \txdlclass_t *rcrec;\n \tchar *dis, *dis1, *dis2;\n \n-\tif (!(dis = (char *) xdl_malloc(xdf1->nrec + xdf2->nrec + 2))) {\n+\tif (!(dis = xdl_calloc(xdf1->nrec + xdf2->nrec + 2, sizeof(*dis)))) {\n \n \t\treturn -1;\n \t}\n-\tmemset(dis, 0, xdf1->nrec + xdf2->nrec + 2);\n \tdis1 = dis;\n \tdis2 = dis1 + xdf1->nrec + 1;\n \n-- \ngitgitgadget\n\n"},{"id":"458646","messageId":"55fd62dc27d2bccfdb8ac300be23fc33d1795366.1657297520.git.gitgitgadget@gmail.com","threadId":"58081","inReplyTo":"pull.1272.v2.git.1657297519.gitgitgadget@gmail.com","subject":"[PATCH v2 1/4] xdiff: introduce XDL_ALLOC_ARRAY()","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-08T16:25:16Z","receivedAt":"2022-07-08T16:26:20Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood <phillip.wood@dunelm.org.uk>\n\nAdd a helper to allocate an array that automatically calculates the\nallocation size. This is analogous to ALLOC_ARRAY() in the rest of the\ncodebase but returns NULL if the allocation fails to accommodate other\nusers of libxdiff such as libgit2. The helper will also return NULL if\nthe multiplication in the allocation calculation overflows.\n\nSigned-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>\n---\n xdiff/xdiffi.c    | 2 +-\n xdiff/xmacros.h   | 5 +++++\n xdiff/xpatience.c | 4 ++--\n xdiff/xprepare.c  | 8 ++++----\n 4 files changed, 12 insertions(+), 7 deletions(-)\n\ndiff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\nindex 758410c11ac..53e803e6bcb 100644\n--- a/xdiff/xdiffi.c\n+++ b/xdiff/xdiffi.c\n@@ -337,7 +337,7 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n \t * One is to store the forward path and one to store the backward path.\n \t */\n \tndiags = xe->xdf1.nreff + xe->xdf2.nreff + 3;\n-\tif (!(kvd = (long *) xdl_malloc((2 * ndiags + 2) * sizeof(long)))) {\n+\tif (!XDL_ALLOC_ARRAY(kvd, 2 * ndiags + 2)) {\n \n \t\txdl_free_env(xe);\n \t\treturn -1;\ndiff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\nindex ae4636c2477..9fd3c5da91a 100644\n--- a/xdiff/xmacros.h\n+++ b/xdiff/xmacros.h\n@@ -49,5 +49,10 @@ do { \\\n \t\t((unsigned long) __p[2]) << 16 | ((unsigned long) __p[3]) << 24; \\\n } while (0)\n \n+/* Allocate an array of nr elements, returns NULL on failure */\n+#define XDL_ALLOC_ARRAY(p, nr)\t\t\t\t\\\n+\t((p) = SIZE_MAX / sizeof(*(p)) >= (size_t)(nr)\t\\\n+\t\t? xdl_malloc((nr) * sizeof(*(p)))\t\\\n+\t\t: NULL)\n \n #endif /* #if !defined(XMACROS_H) */\ndiff --git a/xdiff/xpatience.c b/xdiff/xpatience.c\nindex 1a21c6a74b3..ce87b9084ca 100644\n--- a/xdiff/xpatience.c\n+++ b/xdiff/xpatience.c\n@@ -200,7 +200,7 @@ static int binary_search(struct entry **sequence, int longest,\n  */\n static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n {\n-\tstruct entry **sequence = xdl_malloc(map->nr * sizeof(struct entry *));\n+\tstruct entry **sequence;\n \tint longest = 0, i;\n \tstruct entry *entry;\n \n@@ -211,7 +211,7 @@ static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n \t */\n \tint anchor_i = -1;\n \n-\tif (!sequence)\n+\tif (!XDL_ALLOC_ARRAY(sequence, map->nr))\n \t\treturn -1;\n \n \tfor (entry = map->first; entry; entry = entry->next) {\ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex 105752758f2..25866a1667a 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -86,7 +86,7 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n \tmemset(cf->rchash, 0, cf->hsize * sizeof(xdlclass_t *));\n \n \tcf->alloc = size;\n-\tif (!(cf->rcrecs = (xdlclass_t **) xdl_malloc(cf->alloc * sizeof(xdlclass_t *)))) {\n+\tif (!XDL_ALLOC_ARRAY(cf->rcrecs, cf->alloc)) {\n \n \t\txdl_free(cf->rchash);\n \t\txdl_cha_free(&cf->ncha);\n@@ -178,7 +178,7 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \n \tif (xdl_cha_init(&xdf->rcha, sizeof(xrecord_t), narec / 4 + 1) < 0)\n \t\tgoto abort;\n-\tif (!(recs = (xrecord_t **) xdl_malloc(narec * sizeof(xrecord_t *))))\n+\tif (!XDL_ALLOC_ARRAY(recs, narec))\n \t\tgoto abort;\n \n \thbits = xdl_hashbits((unsigned int) narec);\n@@ -215,9 +215,9 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \n \tif ((XDF_DIFF_ALG(xpp->flags) != XDF_PATIENCE_DIFF) &&\n \t    (XDF_DIFF_ALG(xpp->flags) != XDF_HISTOGRAM_DIFF)) {\n-\t\tif (!(rindex = xdl_malloc((nrec + 1) * sizeof(*rindex))))\n+\t\tif (!XDL_ALLOC_ARRAY(rindex, nrec + 1))\n \t\t\tgoto abort;\n-\t\tif (!(ha = xdl_malloc((nrec + 1) * sizeof(*ha))))\n+\t\tif (!XDL_ALLOC_ARRAY(ha, nrec + 1))\n \t\t\tgoto abort;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"458647","messageId":"8c24cd7737b29d461788b71f6a94eb74c468ad33.1657297520.git.gitgitgadget@gmail.com","threadId":"58081","inReplyTo":"pull.1272.v2.git.1657297519.gitgitgadget@gmail.com","subject":"[PATCH v2 4/4] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-08T16:25:19Z","receivedAt":"2022-07-08T16:26:22Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood <phillip.wood@dunelm.org.uk>\n\nAdd a helper to grow an array. This is analogous to ALLOC_GROW() in\nthe rest of the codebase but returns −1 on allocation failure to\naccommodate other users of libxdiff such as libgit2. It will also\nreturn a error if the multiplication overflows while calculating the\nnew allocation size. Note that this keeps doubling on reallocation\nlike the code it is replacing rather than increasing the existing size\nby half like ALLOC_GROW(). It does however copy ALLOC_GROW()'s trick\nof adding a small amount to the new allocation to avoid a lot of\nreallocations at small sizes.\n\nNote that xdl_alloc_grow_helper() uses long rather than size_t for\n`nr` and `alloc` to match the existing code.\n\nSigned-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>\n---\n xdiff/xmacros.h  | 10 ++++++++++\n xdiff/xprepare.c | 19 ++++---------------\n xdiff/xutils.c   | 17 +++++++++++++++++\n xdiff/xutils.h   |  3 ++-\n 4 files changed, 33 insertions(+), 16 deletions(-)\n\ndiff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\nindex 0977d1615ac..8487bb396fa 100644\n--- a/xdiff/xmacros.h\n+++ b/xdiff/xmacros.h\n@@ -58,4 +58,14 @@ do { \\\n /* Allocate an array of nr zeroed out elements, returns NULL on failure */\n #define XDL_CALLOC_ARRAY(p, nr)\t((p) = xdl_calloc(nr, sizeof(*(p))))\n \n+/*\n+ * Ensure array p can accommodate at least nr elements, growing the\n+ * array and updating alloc (which is the number of allocated\n+ * elements) as necessary. Frees p and returns -1 on failure, returns\n+ * 0 on success\n+ */\n+#define XDL_ALLOC_GROW(p, nr, alloc)\t\\\n+\t(-!((nr) <= (alloc) ||\t\t\\\n+\t    ((p) = xdl_alloc_grow_helper((p), (nr), &(alloc), sizeof(*(p))))))\n+\n #endif /* #if !defined(XMACROS_H) */\ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex b016570c488..c84549f6c50 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -111,7 +111,6 @@ static int xdl_classify_record(unsigned int pass, xdlclassifier_t *cf, xrecord_t\n \tlong hi;\n \tchar const *line;\n \txdlclass_t *rcrec;\n-\txdlclass_t **rcrecs;\n \n \tline = rec->ptr;\n \thi = (long) XDL_HASHLONG(rec->ha, cf->hbits);\n@@ -127,14 +126,8 @@ static int xdl_classify_record(unsigned int pass, xdlclassifier_t *cf, xrecord_t\n \t\t\treturn -1;\n \t\t}\n \t\trcrec->idx = cf->count++;\n-\t\tif (cf->count > cf->alloc) {\n-\t\t\tcf->alloc *= 2;\n-\t\t\tif (!(rcrecs = (xdlclass_t **) xdl_realloc(cf->rcrecs, cf->alloc * sizeof(xdlclass_t *)))) {\n-\n+\t\tif (XDL_ALLOC_GROW(cf->rcrecs, cf->count, cf->alloc))\n \t\t\t\treturn -1;\n-\t\t\t}\n-\t\t\tcf->rcrecs = rcrecs;\n-\t\t}\n \t\tcf->rcrecs[rcrec->idx] = rcrec;\n \t\trcrec->line = line;\n \t\trcrec->size = rec->size;\n@@ -163,7 +156,7 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \tunsigned long hav;\n \tchar const *blk, *cur, *top, *prev;\n \txrecord_t *crec;\n-\txrecord_t **recs, **rrecs;\n+\txrecord_t **recs;\n \txrecord_t **rhash;\n \tunsigned long *ha;\n \tchar *rchg;\n@@ -190,12 +183,8 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \t\tfor (top = blk + bsize; cur < top; ) {\n \t\t\tprev = cur;\n \t\t\thav = xdl_hash_record(&cur, top, xpp->flags);\n-\t\t\tif (nrec >= narec) {\n-\t\t\t\tnarec *= 2;\n-\t\t\t\tif (!(rrecs = (xrecord_t **) xdl_realloc(recs, narec * sizeof(xrecord_t *))))\n-\t\t\t\t\tgoto abort;\n-\t\t\t\trecs = rrecs;\n-\t\t\t}\n+\t\t\tif (XDL_ALLOC_GROW(recs, nrec + 1, narec))\n+\t\t\t\tgoto abort;\n \t\t\tif (!(crec = xdl_cha_alloc(&xdf->rcha)))\n \t\t\t\tgoto abort;\n \t\t\tcrec->ptr = prev;\ndiff --git a/xdiff/xutils.c b/xdiff/xutils.c\nindex 115b2b1640b..9e36f24875d 100644\n--- a/xdiff/xutils.c\n+++ b/xdiff/xutils.c\n@@ -432,3 +432,20 @@ int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n \n \treturn 0;\n }\n+\n+void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size)\n+{\n+\tvoid *tmp = NULL;\n+\tsize_t n = ((LONG_MAX - 16) / 2 >= *alloc) ? 2 * *alloc + 16 : LONG_MAX;\n+\tif (nr > n)\n+\t\tn = nr;\n+\tif (SIZE_MAX / size >= n)\n+\t\ttmp = xdl_realloc(p, n * size);\n+\tif (tmp) {\n+\t\t*alloc = n;\n+\t} else {\n+\t\txdl_free(p);\n+\t\t*alloc = 0;\n+\t}\n+\treturn tmp;\n+}\ndiff --git a/xdiff/xutils.h b/xdiff/xutils.h\nindex fba7bae03c7..fd0bba94e8b 100644\n--- a/xdiff/xutils.h\n+++ b/xdiff/xutils.h\n@@ -42,6 +42,7 @@ int xdl_emit_hunk_hdr(long s1, long c1, long s2, long c2,\n int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n \t\t       int line1, int count1, int line2, int count2);\n \n-\n+/* Do not call this function, use XDL_ALLOC_GROW instead */\n+void* xdl_alloc_grow_helper(void* p, long nr, long* alloc, size_t size);\n \n #endif /* #if !defined(XUTILS_H) */\n-- \ngitgitgadget\n"},{"id":"458648","messageId":"5490fd22579c8a65e097d79d0da4beea8549a02b.1657297520.git.gitgitgadget@gmail.com","threadId":"58081","inReplyTo":"pull.1272.v2.git.1657297519.gitgitgadget@gmail.com","subject":"[PATCH v2 3/4] xdiff: introduce XDL_CALLOC_ARRAY()","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-08T16:25:18Z","receivedAt":"2022-07-08T16:26:25Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood <phillip.wood@dunelm.org.uk>\n\nAdd a helper for allocating an array and initialize the elements to\nzero. This is analogous to CALLOC_ARRAY() in the rest of the codebase\nbut it returns NULL on allocation failures rather than dying to\naccommodate other users of libxdiff such as libgit2.\n\nSigned-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>\n---\n xdiff/xhistogram.c |  9 +++------\n xdiff/xmacros.h    |  3 +++\n xdiff/xpatience.c  |  3 +--\n xdiff/xprepare.c   | 10 ++++------\n 4 files changed, 11 insertions(+), 14 deletions(-)\n\ndiff --git a/xdiff/xhistogram.c b/xdiff/xhistogram.c\nindex c97edc1e29e..df909004c10 100644\n--- a/xdiff/xhistogram.c\n+++ b/xdiff/xhistogram.c\n@@ -266,17 +266,14 @@ static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n \n \tindex.table_bits = xdl_hashbits(count1);\n \tindex.records_size = 1 << index.table_bits;\n-\tif (!(index.records = xdl_calloc(index.records_size,\n-\t\t\t\t\t sizeof(*index.records))))\n+\tif (!XDL_CALLOC_ARRAY(index.records, index.records_size))\n \t\tgoto cleanup;\n \n \tindex.line_map_size = count1;\n-\tif (!(index.line_map = xdl_calloc(index.line_map_size,\n-\t\t\t\t\t  sizeof(*index.line_map))))\n+\tif (!XDL_CALLOC_ARRAY(index.line_map, index.line_map_size))\n \t\tgoto cleanup;\n \n-\tif (!(index.next_ptrs = xdl_calloc(index.line_map_size,\n-\t\t\t\t\t   sizeof(*index.next_ptrs))))\n+\tif (!XDL_CALLOC_ARRAY(index.next_ptrs, index.line_map_size))\n \t\tgoto cleanup;\n \n \t/* lines / 4 + 1 comes from xprepare.c:xdl_prepare_ctx() */\ndiff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\nindex 9fd3c5da91a..0977d1615ac 100644\n--- a/xdiff/xmacros.h\n+++ b/xdiff/xmacros.h\n@@ -55,4 +55,7 @@ do { \\\n \t\t? xdl_malloc((nr) * sizeof(*(p)))\t\\\n \t\t: NULL)\n \n+/* Allocate an array of nr zeroed out elements, returns NULL on failure */\n+#define XDL_CALLOC_ARRAY(p, nr)\t((p) = xdl_calloc(nr, sizeof(*(p))))\n+\n #endif /* #if !defined(XMACROS_H) */\ndiff --git a/xdiff/xpatience.c b/xdiff/xpatience.c\nindex 94f8886dd14..fe39c2978cb 100644\n--- a/xdiff/xpatience.c\n+++ b/xdiff/xpatience.c\n@@ -151,8 +151,7 @@ static int fill_hashmap(mmfile_t *file1, mmfile_t *file2,\n \n \t/* We know exactly how large we want the hash map */\n \tresult->alloc = count1 * 2;\n-\tresult->entries = xdl_calloc(result->alloc, sizeof(*result->entries));\n-\tif (!result->entries)\n+\tif (!XDL_CALLOC_ARRAY(result->entries, result->alloc))\n \t\treturn -1;\n \n \t/* First, fill with entries from the first file */\ndiff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\nindex 45190e59fc8..b016570c488 100644\n--- a/xdiff/xprepare.c\n+++ b/xdiff/xprepare.c\n@@ -78,7 +78,7 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n \n \t\treturn -1;\n \t}\n-\tif (!(cf->rchash = xdl_calloc(cf->hsize, sizeof(*cf->rchash)))) {\n+\tif (!XDL_CALLOC_ARRAY(cf->rchash, cf->hsize)) {\n \n \t\txdl_cha_free(&cf->ncha);\n \t\treturn -1;\n@@ -182,7 +182,7 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \n \thbits = xdl_hashbits((unsigned int) narec);\n \thsize = 1 << hbits;\n-\tif (!(rhash = xdl_calloc(hsize, sizeof(*rhash))))\n+\tif (!XDL_CALLOC_ARRAY(rhash, hsize))\n \t\tgoto abort;\n \n \tnrec = 0;\n@@ -207,7 +207,7 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n \t\t}\n \t}\n \n-\tif (!(rchg = xdl_calloc((nrec + 2), sizeof(*rchg))))\n+\tif (!XDL_CALLOC_ARRAY(rchg, nrec + 2))\n \t\tgoto abort;\n \n \tif ((XDF_DIFF_ALG(xpp->flags) != XDF_PATIENCE_DIFF) &&\n@@ -380,10 +380,8 @@ static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd\n \txdlclass_t *rcrec;\n \tchar *dis, *dis1, *dis2;\n \n-\tif (!(dis = xdl_calloc(xdf1->nrec + xdf2->nrec + 2, sizeof(*dis)))) {\n-\n+\tif (!XDL_CALLOC_ARRAY(dis, xdf1->nrec + xdf2->nrec + 2))\n \t\treturn -1;\n-\t}\n \tdis1 = dis;\n \tdis2 = dis1 + xdf1->nrec + 1;\n \n-- \ngitgitgadget\n\n"},{"id":"458653","messageId":"1bf5d574-4202-31e3-4dc3-d7890465d28f@gmail.com","threadId":"58081","inReplyTo":"patch-6.7-33d93f121a9-20220708T140354Z-avarab@gmail.com","subject":"Re: [PATCH 6/7] xdiff: remove xdl_malloc() wrapper, use malloc(), not xmalloc()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-08T17:42:17Z","receivedAt":"2022-07-08T17:42:37Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n> As noted in 36c83197249 (xdiff: use xmalloc/xrealloc, 2019-04-11) the\n> reason we have xdl_malloc() in the first place was to make the xdiff\n> code safer, it was not handling some allocation failures correctly at\n> the time.\n\nThis is untrue, we do not have xdl_malloc to make the xdiff code safer, \nthat macro was introduced with the initial import of xdiff. 36c83197249 \n  just changed its definition, the entire commit consists of\n\n-#define xdl_malloc(x) malloc(x)\n+#define xdl_malloc(x) xmalloc(x)\n  #define xdl_free(ptr) free(ptr)\n-#define xdl_realloc(ptr,x) realloc(ptr,x)\n+#define xdl_realloc(ptr,x) xrealloc(ptr,x)\n\nI can see the argument for reverting that change now that we have fixed \nthe error checking but that is not a good reason to remove xdl_malloc().\n\nBest Wishes\n\nPhillip\n\n> But as noted in that commit doing this was intended as a stopgap, as\n> various code in xdiff/* did not correctly handle allocation failures,\n> and would have segfaulted if malloc() returned NULL.\n> \n> But since then and after preceding commits we can be confident that\n> malloc() failures are handled correctly. All of these users of\n> xdl_malloc() are checking their return values, and we're returning\n> -1 (or similar ) to the top-level in case of failures.\n> \n> This also has the big advantage of making the compiler and analysis\n> tools less confused, and potentially avoiding undefined (compiler)\n> behavior. I.e. we define our own xmalloc() to call die() on failure,\n> and that function uses the non-standard \"noreturn\" attribute on our\n> most common compiler targets.\n> \n> But xdl_malloc()'s use conflicted with that, confusing both human\n> readers of this code, and potentially compilers as well.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>   xdiff/xdiff.h  |  1 -\n>   xdiff/xdiffi.c |  2 +-\n>   xdiff/xmerge.c | 10 +++++-----\n>   xdiff/xutils.c |  2 +-\n>   4 files changed, 7 insertions(+), 8 deletions(-)\n> \n> diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h\n> index 832cf9d977e..df048e0099b 100644\n> --- a/xdiff/xdiff.h\n> +++ b/xdiff/xdiff.h\n> @@ -119,7 +119,6 @@ typedef struct s_bdiffparam {\n>   } bdiffparam_t;\n>   \n>   \n> -#define xdl_malloc(x) xmalloc(x)\n>   #define xdl_free(ptr) free(ptr)\n>   \n>   void *xdl_mmfile_first(mmfile_t *mmf, long *size);\n> diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\n> index 077cc456087..6590811634f 100644\n> --- a/xdiff/xdiffi.c\n> +++ b/xdiff/xdiffi.c\n> @@ -368,7 +368,7 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n>   static xdchange_t *xdl_add_change(xdchange_t *xscr, long i1, long i2, long chg1, long chg2) {\n>   \txdchange_t *xch;\n>   \n> -\tif (!(xch = (xdchange_t *) xdl_malloc(sizeof(xdchange_t))))\n> +\tif (!(xch = (xdchange_t *) malloc(sizeof(xdchange_t))))\n>   \t\treturn NULL;\n>   \n>   \txch->next = xscr;\n> diff --git a/xdiff/xmerge.c b/xdiff/xmerge.c\n> index ac0cf52f3be..676ad39020d 100644\n> --- a/xdiff/xmerge.c\n> +++ b/xdiff/xmerge.c\n> @@ -60,7 +60,7 @@ static int xdl_append_merge(xdmerge_t **merge, int mode,\n>   \t\tm->chg1 = i1 + chg1 - m->i1;\n>   \t\tm->chg2 = i2 + chg2 - m->i2;\n>   \t} else {\n> -\t\tm = xdl_malloc(sizeof(xdmerge_t));\n> +\t\tm = malloc(sizeof(xdmerge_t));\n>   \t\tif (!m)\n>   \t\t\treturn -1;\n>   \t\tm->next = NULL;\n> @@ -409,7 +409,7 @@ static int xdl_refine_conflicts(xdfenv_t *xe1, xdfenv_t *xe2, xdmerge_t *m,\n>   \t\tm->i2 = xscr->i2 + i2;\n>   \t\tm->chg2 = xscr->chg2;\n>   \t\twhile (xscr->next) {\n> -\t\t\txdmerge_t *m2 = xdl_malloc(sizeof(xdmerge_t));\n> +\t\t\txdmerge_t *m2 = malloc(sizeof(xdmerge_t));\n>   \t\t\tif (!m2) {\n>   \t\t\t\txdl_free_env(&xe);\n>   \t\t\t\txdl_free_script(x);\n> @@ -670,7 +670,7 @@ static int xdl_do_merge(xdfenv_t *xe1, xdchange_t *xscr1,\n>   \t\t\t\t\t\t ancestor_name,\n>   \t\t\t\t\t\t favor, changes, NULL, style,\n>   \t\t\t\t\t\t marker_size);\n> -\t\tresult->ptr = xdl_malloc(size);\n> +\t\tresult->ptr = malloc(size);\n>   \t\tif (!result->ptr) {\n>   \t\t\txdl_cleanup_merge(changes);\n>   \t\t\treturn -1;\n> @@ -718,14 +718,14 @@ int xdl_merge(mmfile_t *orig, mmfile_t *mf1, mmfile_t *mf2,\n>   \t}\n>   \tstatus = 0;\n>   \tif (!xscr1) {\n> -\t\tresult->ptr = xdl_malloc(mf2->size);\n> +\t\tresult->ptr = malloc(mf2->size);\n>   \t\tif (!result->ptr)\n>   \t\t\tgoto out;\n>   \t\tstatus = 0;\n>   \t\tmemcpy(result->ptr, mf2->ptr, mf2->size);\n>   \t\tresult->size = mf2->size;\n>   \t} else if (!xscr2) {\n> -\t\tresult->ptr = xdl_malloc(mf1->size);\n> +\t\tresult->ptr = malloc(mf1->size);\n>   \t\tif (!result->ptr)\n>   \t\t\tgoto out;\n>   \t\tstatus = 0;\n> diff --git a/xdiff/xutils.c b/xdiff/xutils.c\n> index c0cd5338c4e..865e08f0e93 100644\n> --- a/xdiff/xutils.c\n> +++ b/xdiff/xutils.c\n> @@ -98,7 +98,7 @@ void *xdl_cha_alloc(chastore_t *cha) {\n>   \tvoid *data;\n>   \n>   \tif (!(ancur = cha->ancur) || ancur->icurr == cha->nsize) {\n> -\t\tif (!(ancur = (chanode_t *) xdl_malloc(sizeof(chanode_t) + cha->nsize))) {\n> +\t\tif (!(ancur = (chanode_t *) malloc(sizeof(chanode_t) + cha->nsize))) {\n>   \n>   \t\t\treturn NULL;\n>   \t\t}\n\n"},{"id":"458654","messageId":"6b8fc0e8-c446-1275-12f3-e6520de9584d@gmail.com","threadId":"58081","inReplyTo":"patch-7.7-a1bf9a94f0a-20220708T140354Z-avarab@gmail.com","subject":"Re: [PATCH 7/7] xdiff: remove xdl_free(), use free() instead","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-08T17:51:43Z","receivedAt":"2022-07-08T17:51:48Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n> Remove the xdl_free() wrapper macro in favor of using free()\n> directly. The wrapper macro was brought in with the initial import of\n> xdiff/ in 3443546f6ef (Use a *real* built-in diff generator,\n> 2006-03-24).\n> \n> As subsequent discussions on the topic[1] have made clear there's no\n> reason to use this wrapper.\n\nThat link is to a message where you assert that there is no reason for \nthe wrapper, you seem to be the only person in that thread making that \nargument. The libgit2 maintainer and others are arguing that it is worth \nkeeping to make it easy to swap the allocator.\n\n> Both git itself as well as any external\n> users such as libgit2 compile the xdiff/* code as part of their own\n> compilation, and can thus find the right malloc() and free() at\n> link-time.\n\nI'm struggling to see how that is simpler than the current situation \nwith xdl_malloc(). Perhaps you could show how you think libgit2 could \neasily make xdiff use git__malloc() instead of malloc() without changing \nthe malloc() calls (the message you linked to essentially suggests they \ndo a search and replace). If people cannot swap in another allocator \nwithout changing the code then you are imposing a barrier to them \ncontributing patches back to git's xdiff.\n\nBest Wishes\n\nPhillip\n\n> When compiling git we already find a custom malloc() and free()\n> e.g. if compiled with USE_NED_ALLOCATOR=YesPlease.\n> \n> 1. https://lore.kernel.org/git/220415.867d7qbaad.gmgdl@evledraar.gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>   xdiff/xdiff.h      |  3 ---\n>   xdiff/xdiffi.c     |  4 ++--\n>   xdiff/xhistogram.c |  6 +++---\n>   xdiff/xpatience.c  |  8 ++++----\n>   xdiff/xprepare.c   | 28 ++++++++++++++--------------\n>   xdiff/xutils.c     |  2 +-\n>   6 files changed, 24 insertions(+), 27 deletions(-)\n> \n> diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h\n> index df048e0099b..a37d89fcdaf 100644\n> --- a/xdiff/xdiff.h\n> +++ b/xdiff/xdiff.h\n> @@ -118,9 +118,6 @@ typedef struct s_bdiffparam {\n>   \tlong bsize;\n>   } bdiffparam_t;\n>   \n> -\n> -#define xdl_free(ptr) free(ptr)\n> -\n>   void *xdl_mmfile_first(mmfile_t *mmf, long *size);\n>   long xdl_mmfile_size(mmfile_t *mmf);\n>   \n> diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\n> index 6590811634f..375bb81a8aa 100644\n> --- a/xdiff/xdiffi.c\n> +++ b/xdiff/xdiffi.c\n> @@ -359,7 +359,7 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n>   \tres = xdl_recs_cmp(&dd1, 0, dd1.nrec, &dd2, 0, dd2.nrec,\n>   \t\t\t   kvdf, kvdb, (xpp->flags & XDF_NEED_MINIMAL) != 0,\n>   \t\t\t   &xenv);\n> -\txdl_free(kvd);\n> +\tfree(kvd);\n>   \n>   \treturn res;\n>   }\n> @@ -960,7 +960,7 @@ void xdl_free_script(xdchange_t *xscr) {\n>   \n>   \twhile ((xch = xscr) != NULL) {\n>   \t\txscr = xscr->next;\n> -\t\txdl_free(xch);\n> +\t\tfree(xch);\n>   \t}\n>   }\n>   \n> diff --git a/xdiff/xhistogram.c b/xdiff/xhistogram.c\n> index f20592bfbdd..be35d9c9697 100644\n> --- a/xdiff/xhistogram.c\n> +++ b/xdiff/xhistogram.c\n> @@ -240,9 +240,9 @@ static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env,\n>   \n>   static inline void free_index(struct histindex *index)\n>   {\n> -\txdl_free(index->records);\n> -\txdl_free(index->line_map);\n> -\txdl_free(index->next_ptrs);\n> +\tfree(index->records);\n> +\tfree(index->line_map);\n> +\tfree(index->next_ptrs);\n>   \txdl_cha_free(&index->rcha);\n>   }\n>   \n> diff --git a/xdiff/xpatience.c b/xdiff/xpatience.c\n> index bb328d9f852..8fffd2b8297 100644\n> --- a/xdiff/xpatience.c\n> +++ b/xdiff/xpatience.c\n> @@ -233,7 +233,7 @@ static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n>   \t/* No common unique lines were found */\n>   \tif (!longest) {\n>   \t\t*res = NULL;\n> -\t\txdl_free(sequence);\n> +\t\tfree(sequence);\n>   \t\treturn 0;\n>   \t}\n>   \n> @@ -245,7 +245,7 @@ static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n>   \t\tentry = entry->previous;\n>   \t}\n>   \t*res = entry;\n> -\txdl_free(sequence);\n> +\tfree(sequence);\n>   \treturn 0;\n>   }\n>   \n> @@ -358,7 +358,7 @@ static int patience_diff(mmfile_t *file1, mmfile_t *file2,\n>   \t\t\tenv->xdf1.rchg[line1++ - 1] = 1;\n>   \t\twhile(count2--)\n>   \t\t\tenv->xdf2.rchg[line2++ - 1] = 1;\n> -\t\txdl_free(map.entries);\n> +\t\tfree(map.entries);\n>   \t\treturn 0;\n>   \t}\n>   \n> @@ -372,7 +372,7 @@ static int patience_diff(mmfile_t *file1, mmfile_t *file2,\n>   \t\tresult = fall_back_to_classic_diff(&map,\n>   \t\t\tline1, count1, line2, count2);\n>    out:\n> -\txdl_free(map.entries);\n> +\tfree(map.entries);\n>   \treturn result;\n>   }\n>   \n> diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\n> index 4182d9e1c0a..169629761c0 100644\n> --- a/xdiff/xprepare.c\n> +++ b/xdiff/xprepare.c\n> @@ -89,7 +89,7 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n>   \n>   \tGALLOC_ARRAY(cf->rcrecs, cf->alloc);\n>   \tif (!cf->rcrecs) {\n> -\t\txdl_free(cf->rchash);\n> +\t\tfree(cf->rchash);\n>   \t\txdl_cha_free(&cf->ncha);\n>   \t\treturn -1;\n>   \t}\n> @@ -102,8 +102,8 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n>   \n>   static void xdl_free_classifier(xdlclassifier_t *cf) {\n>   \n> -\txdl_free(cf->rcrecs);\n> -\txdl_free(cf->rchash);\n> +\tfree(cf->rcrecs);\n> +\tfree(cf->rchash);\n>   \txdl_cha_free(&cf->ncha);\n>   }\n>   \n> @@ -230,11 +230,11 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n>   \treturn 0;\n>   \n>   abort:\n> -\txdl_free(ha);\n> -\txdl_free(rindex);\n> -\txdl_free(rchg);\n> -\txdl_free(rhash);\n> -\txdl_free(recs);\n> +\tfree(ha);\n> +\tfree(rindex);\n> +\tfree(rchg);\n> +\tfree(rhash);\n> +\tfree(recs);\n>   \txdl_cha_free(&xdf->rcha);\n>   \treturn -1;\n>   }\n> @@ -242,11 +242,11 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n>   \n>   static void xdl_free_ctx(xdfile_t *xdf) {\n>   \n> -\txdl_free(xdf->rhash);\n> -\txdl_free(xdf->rindex);\n> -\txdl_free(xdf->rchg - 1);\n> -\txdl_free(xdf->ha);\n> -\txdl_free(xdf->recs);\n> +\tfree(xdf->rhash);\n> +\tfree(xdf->rindex);\n> +\tfree(xdf->rchg - 1);\n> +\tfree(xdf->ha);\n> +\tfree(xdf->recs);\n>   \txdl_cha_free(&xdf->rcha);\n>   }\n>   \n> @@ -424,7 +424,7 @@ static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd\n>   \t}\n>   \txdf2->nreff = nreff;\n>   \n> -\txdl_free(dis);\n> +\tfree(dis);\n>   \n>   \treturn 0;\n>   }\n> diff --git a/xdiff/xutils.c b/xdiff/xutils.c\n> index 865e08f0e93..00eeba452a5 100644\n> --- a/xdiff/xutils.c\n> +++ b/xdiff/xutils.c\n> @@ -88,7 +88,7 @@ void xdl_cha_free(chastore_t *cha) {\n>   \n>   \tfor (cur = cha->head; (tmp = cur) != NULL;) {\n>   \t\tcur = cur->next;\n> -\t\txdl_free(tmp);\n> +\t\tfree(tmp);\n>   \t}\n>   }\n>   \n\n"},{"id":"458659","messageId":"YsiHGliw2dYmGcDY@coredump.intra.peff.net","threadId":"58081","inReplyTo":"patch-6.7-33d93f121a9-20220708T140354Z-avarab@gmail.com","subject":"Re: [PATCH 6/7] xdiff: remove xdl_malloc() wrapper, use malloc(), not xmalloc()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-07-08T19:35:54Z","receivedAt":"2022-07-08T19:35:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 08, 2022 at 04:20:18PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> As noted in 36c83197249 (xdiff: use xmalloc/xrealloc, 2019-04-11) the\n> reason we have xdl_malloc() in the first place was to make the xdiff\n> code safer, it was not handling some allocation failures correctly at\n> the time.\n> \n> But as noted in that commit doing this was intended as a stopgap, as\n> various code in xdiff/* did not correctly handle allocation failures,\n> and would have segfaulted if malloc() returned NULL.\n> \n> But since then and after preceding commits we can be confident that\n> malloc() failures are handled correctly. All of these users of\n> xdl_malloc() are checking their return values, and we're returning\n> -1 (or similar ) to the top-level in case of failures.\n\nThis is also losing the other parts mentioned in 36c83197249:\nrespecting GIT_ALLOC_LIMIT and any memory reclamation strategies.\n\nI think you'd want an xmalloc_gently() instead of a raw malloc().\n\nFor the same reason, I suspect it's better to leave this with a layer of\npreprocessor indirection. Even if we chose to use malloc() here, libgit2\nmight not, and having the macro makes it easier to share the code.\n\n-Peff\n"},{"id":"458666","messageId":"220708.86czef9t6y.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"6b8fc0e8-c446-1275-12f3-e6520de9584d@gmail.com","subject":"Re: [PATCH 7/7] xdiff: remove xdl_free(), use free() instead","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T21:26:59Z","receivedAt":"2022-07-08T21:43:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 08 2022, Phillip Wood wrote:\n\n> Hi Ævar\n>\n> On 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n>> Remove the xdl_free() wrapper macro in favor of using free()\n>> directly. The wrapper macro was brought in with the initial import of\n>> xdiff/ in 3443546f6ef (Use a *real* built-in diff generator,\n>> 2006-03-24).\n>> As subsequent discussions on the topic[1] have made clear there's no\n>> reason to use this wrapper.\n>\n> That link is to a message where you assert that there is no reason for\n> the wrapper, you seem to be the only person in that thread making that \n> argument. The libgit2 maintainer and others are arguing that it is\n> worth keeping to make it easy to swap the allocator.\n\nArguing that it's not needed for a technical reason, but an \"aesthetic\nand ergonomic one\", per:\nhttps://lore.kernel.org/git/20220217225804.GC7@edef91d97c94/;\n\nPerhaps I misread it, but I took this as Junio concurring with what I\nwas pointing out there:\nhttps://lore.kernel.org/git/xmqqfsohbdre.fsf@gitster.g/\n\n>> Both git itself as well as any external\n>> users such as libgit2 compile the xdiff/* code as part of their own\n>> compilation, and can thus find the right malloc() and free() at\n>> link-time.\n>\n> I'm struggling to see how that is simpler than the current situation\n> with xdl_malloc().\n\nIt's simpler because when maintaining that code in git.git it's less of\na special case, e.g. we have coccinelle rules that match free(), now\nevery such rule needs to account for the xdiff special-case.\n\nAnd it's less buggy because if you're relying on us always using a\nwrapper bugs will creep in, current master has this:\n\t\n\t$ git -P grep '\\bfree\\(' -- xdiff \n\txdiff/xdiff.h:#define xdl_free(ptr) free(ptr)\n\txdiff/xmerge.c:         free(c);\n\txdiff/xmerge.c: free(next_m);\n\n> Perhaps you could show how you think libgit2 could \n> easily make xdiff use git__malloc() instead of malloc() without\n> changing the malloc() calls (the message you linked to essentially\n> suggests they do a search and replace). If people cannot swap in\n> another allocator without changing the code then you are imposing a\n> barrier to them contributing patches back to git's xdiff.\n\nI was suggesting that if libgit2 wanted to maintain a copy of xdiff that\ncatered to the asthetic desires of the maintainer adjusting whatever\nimport script you use to apply a trivial coccinelle transformation would\ngive you what you wanted. Except without a maintenance burden on\ngit.git, or the bugs you'd get now since you're not catching those two\nfree() cases (or any future such cases).\n\nBut just having the code use malloc() and free() directly and getting\nyou what you get now is easy, and doesn't require any such\nsearch-replacement.\n\nHere's how:\n\nI imported the xdiff/*.[ch] files at the tip of my branch into current\nlibgit2.git's src/libgit2/xdiff/, and then applied this on top, which is\npartially re-applying libgit2's own monkeypatches, and partially\nadjusting for upstream changes (including this one):\n\t\n\tdiff --git a/src/libgit2/xdiff/git-xdiff.h b/src/libgit2/xdiff/git-xdiff.h\n\tindex b75dba819..2e00764d4 100644\n\t--- a/src/libgit2/xdiff/git-xdiff.h\n\t+++ b/src/libgit2/xdiff/git-xdiff.h\n\t@@ -14,6 +14,7 @@\n\t #ifndef INCLUDE_git_xdiff_h__\n\t #define INCLUDE_git_xdiff_h__\n\t \n\t+#include <regex.h>\n\t #include \"regexp.h\"\n\t \n\t /* Work around C90-conformance issues */\n\t@@ -27,11 +28,10 @@\n\t # endif\n\t #endif\n\t \n\t-#define xdl_malloc(x) git__malloc(x)\n\t-#define xdl_free(ptr) git__free(ptr)\n\t-#define xdl_realloc(ptr, x) git__realloc(ptr, x)\n\t+#define malloc(x) git__malloc(x)\n\t+#define free(ptr) git__free(ptr)\n\t \n\t-#define XDL_BUG(msg) GIT_ASSERT(msg)\n\t+#define BUG(msg) GIT_ASSERT(msg)\n\t \n\t #define xdl_regex_t git_regexp\n\t #define xdl_regmatch_t git_regmatch\n\t@@ -50,4 +50,17 @@ GIT_INLINE(int) xdl_regexec_buf(\n\t \treturn -1;\n\t }\n\t \n\t+static inline size_t st_mult(size_t a, size_t b)\n\t+{\n\t+\treturn a * b;\n\t+}\n\t+\n\t+static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,\n\t+\t\t\t      size_t nmatch, regmatch_t pmatch[], int eflags)\n\t+{\n\t+\tassert(nmatch > 0 && pmatch);\n\t+\tpmatch[0].rm_so = 0;\n\t+\tpmatch[0].rm_eo = size;\n\t+\treturn regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);\n\t+}\n\t #endif\n\tdiff --git a/src/libgit2/xdiff/xdiff.h b/src/libgit2/xdiff/xdiff.h\n\tindex a37d89fcd..5e5b525c2 100644\n\t--- a/src/libgit2/xdiff/xdiff.h\n\t+++ b/src/libgit2/xdiff/xdiff.h\n\t@@ -23,6 +23,8 @@\n\t #if !defined(XDIFF_H)\n\t #define XDIFF_H\n\t \n\t+#include \"git-xdiff.h\"\n\t+\n\t #ifdef __cplusplus\n\t extern \"C\" {\n\t #endif /* #ifdef __cplusplus */\n\tdiff --git a/src/libgit2/xdiff/xinclude.h b/src/libgit2/xdiff/xinclude.h\n\tindex a4285ac0e..79812ad8a 100644\n\t--- a/src/libgit2/xdiff/xinclude.h\n\t+++ b/src/libgit2/xdiff/xinclude.h\n\t@@ -23,7 +23,8 @@\n\t #if !defined(XINCLUDE_H)\n\t #define XINCLUDE_H\n\t \n\t-#include \"git-compat-util.h\"\n\t+#include \"git-xdiff.h\"\n\t+#include \"git-shared-util.h\"\n\t #include \"xmacros.h\"\n\t #include \"xdiff.h\"\n\t #include \"xtypes.h\"\n\nIf you then build it and run e.g.:\n\n\tgdb -args ./libgit2_tests -smerge::files\n\nYou'll get stack traces like this if you break on stdalloc__malloc\n(which it uses for me in its default configuration):\n\t\n\t(gdb) bt\n\t#0  stdalloc__malloc (len=144, file=0x87478d \"/home/avar/g/libgit2/src/libgit2/xdiff/xutils.c\", line=101) at /home/avar/g/libgit2/src/util/allocators/stdalloc.c:14\n\t#1  0x00000000006ec15c in xdl_cha_alloc (cha=0x7fffffffcaa8) at /home/avar/g/libgit2/src/libgit2/xdiff/xutils.c:101\n\t#2  0x00000000006eaee9 in xdl_prepare_ctx (pass=1, mf=0x7fffffffcc98, narec=13, xpp=0x7fffffffcca8, cf=0x7fffffffc7d0, xdf=0x7fffffffcaa8)\n\t    at /home/avar/g/libgit2/src/libgit2/xdiff/xprepare.c:194\n\nIOW this was all seamlessly routed to your git__malloc() without us\nhaving to maintain an xdl_{malloc,free}().\n\nNow, I think that's obviously worth adjusting, e.g. the series I've got\nhere could make this easier for libgit2 by including st_mult() at least,\nI'm not sure what you want to do about regexec_buf().\n\nFor the monkeypatching you do now of creating a \"git-xdiff.h\" I think it\nwould be very good to get a patch like what I managed to get\nsha1collisiondetection.git to accept for our own use-case, which allows\nus to use their upstream code as-is from a submodule:\n\n\thttps://github.com/cr-marcstevens/sha1collisiondetection/commit/b45fcef\n"},{"id":"458667","messageId":"220708.868rp39t28.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"1bf5d574-4202-31e3-4dc3-d7890465d28f@gmail.com","subject":"Re: [PATCH 6/7] xdiff: remove xdl_malloc() wrapper, use malloc(), not xmalloc()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T21:44:27Z","receivedAt":"2022-07-08T21:46:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 08 2022, Phillip Wood wrote:\n\n> On 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n>> As noted in 36c83197249 (xdiff: use xmalloc/xrealloc, 2019-04-11) the\n>> reason we have xdl_malloc() in the first place was to make the xdiff\n>> code safer, it was not handling some allocation failures correctly at\n>> the time.\n>\n> This is untrue, we do not have xdl_malloc to make the xdiff code\n> safer, that macro was introduced with the initial import of\n> xdiff. 36c83197249   just changed its definition, the entire commit\n> consists of\n>\n> -#define xdl_malloc(x) malloc(x)\n> +#define xdl_malloc(x) xmalloc(x)\n>  #define xdl_free(ptr) free(ptr)\n> -#define xdl_realloc(ptr,x) realloc(ptr,x)\n> +#define xdl_realloc(ptr,x) xrealloc(ptr,x)\n\nYes sorry about that, I'm missing a few words there, and meant \"the\nreason we have this incarnation of xdl_malloc()[...]\", or something to\nthat effect.\n\n> I can see the argument for reverting that change now that we have\n> fixed the error checking but that is not a good reason to remove\n> xdl_malloc().\n\nIndeed, but hopefully\nhttps://lore.kernel.org/git/220708.86czef9t6y.gmgdl@evledraar.gmail.com/\nis that argument.\n\n>> But as noted in that commit doing this was intended as a stopgap, as\n>> various code in xdiff/* did not correctly handle allocation failures,\n>> and would have segfaulted if malloc() returned NULL.\n>> But since then and after preceding commits we can be confident that\n>> malloc() failures are handled correctly. All of these users of\n>> xdl_malloc() are checking their return values, and we're returning\n>> -1 (or similar ) to the top-level in case of failures.\n>> This also has the big advantage of making the compiler and analysis\n>> tools less confused, and potentially avoiding undefined (compiler)\n>> behavior. I.e. we define our own xmalloc() to call die() on failure,\n>> and that function uses the non-standard \"noreturn\" attribute on our\n>> most common compiler targets.\n>> But xdl_malloc()'s use conflicted with that, confusing both human\n>> readers of this code, and potentially compilers as well.\n>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n>> ---\n>>   xdiff/xdiff.h  |  1 -\n>>   xdiff/xdiffi.c |  2 +-\n>>   xdiff/xmerge.c | 10 +++++-----\n>>   xdiff/xutils.c |  2 +-\n>>   4 files changed, 7 insertions(+), 8 deletions(-)\n>> diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h\n>> index 832cf9d977e..df048e0099b 100644\n>> --- a/xdiff/xdiff.h\n>> +++ b/xdiff/xdiff.h\n>> @@ -119,7 +119,6 @@ typedef struct s_bdiffparam {\n>>   } bdiffparam_t;\n>>     \n>> -#define xdl_malloc(x) xmalloc(x)\n>>   #define xdl_free(ptr) free(ptr)\n>>     void *xdl_mmfile_first(mmfile_t *mmf, long *size);\n>> diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\n>> index 077cc456087..6590811634f 100644\n>> --- a/xdiff/xdiffi.c\n>> +++ b/xdiff/xdiffi.c\n>> @@ -368,7 +368,7 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n>>   static xdchange_t *xdl_add_change(xdchange_t *xscr, long i1, long i2, long chg1, long chg2) {\n>>   \txdchange_t *xch;\n>>   -\tif (!(xch = (xdchange_t *) xdl_malloc(sizeof(xdchange_t))))\n>> +\tif (!(xch = (xdchange_t *) malloc(sizeof(xdchange_t))))\n>>   \t\treturn NULL;\n>>     \txch->next = xscr;\n>> diff --git a/xdiff/xmerge.c b/xdiff/xmerge.c\n>> index ac0cf52f3be..676ad39020d 100644\n>> --- a/xdiff/xmerge.c\n>> +++ b/xdiff/xmerge.c\n>> @@ -60,7 +60,7 @@ static int xdl_append_merge(xdmerge_t **merge, int mode,\n>>   \t\tm->chg1 = i1 + chg1 - m->i1;\n>>   \t\tm->chg2 = i2 + chg2 - m->i2;\n>>   \t} else {\n>> -\t\tm = xdl_malloc(sizeof(xdmerge_t));\n>> +\t\tm = malloc(sizeof(xdmerge_t));\n>>   \t\tif (!m)\n>>   \t\t\treturn -1;\n>>   \t\tm->next = NULL;\n>> @@ -409,7 +409,7 @@ static int xdl_refine_conflicts(xdfenv_t *xe1, xdfenv_t *xe2, xdmerge_t *m,\n>>   \t\tm->i2 = xscr->i2 + i2;\n>>   \t\tm->chg2 = xscr->chg2;\n>>   \t\twhile (xscr->next) {\n>> -\t\t\txdmerge_t *m2 = xdl_malloc(sizeof(xdmerge_t));\n>> +\t\t\txdmerge_t *m2 = malloc(sizeof(xdmerge_t));\n>>   \t\t\tif (!m2) {\n>>   \t\t\t\txdl_free_env(&xe);\n>>   \t\t\t\txdl_free_script(x);\n>> @@ -670,7 +670,7 @@ static int xdl_do_merge(xdfenv_t *xe1, xdchange_t *xscr1,\n>>   \t\t\t\t\t\t ancestor_name,\n>>   \t\t\t\t\t\t favor, changes, NULL, style,\n>>   \t\t\t\t\t\t marker_size);\n>> -\t\tresult->ptr = xdl_malloc(size);\n>> +\t\tresult->ptr = malloc(size);\n>>   \t\tif (!result->ptr) {\n>>   \t\t\txdl_cleanup_merge(changes);\n>>   \t\t\treturn -1;\n>> @@ -718,14 +718,14 @@ int xdl_merge(mmfile_t *orig, mmfile_t *mf1, mmfile_t *mf2,\n>>   \t}\n>>   \tstatus = 0;\n>>   \tif (!xscr1) {\n>> -\t\tresult->ptr = xdl_malloc(mf2->size);\n>> +\t\tresult->ptr = malloc(mf2->size);\n>>   \t\tif (!result->ptr)\n>>   \t\t\tgoto out;\n>>   \t\tstatus = 0;\n>>   \t\tmemcpy(result->ptr, mf2->ptr, mf2->size);\n>>   \t\tresult->size = mf2->size;\n>>   \t} else if (!xscr2) {\n>> -\t\tresult->ptr = xdl_malloc(mf1->size);\n>> +\t\tresult->ptr = malloc(mf1->size);\n>>   \t\tif (!result->ptr)\n>>   \t\t\tgoto out;\n>>   \t\tstatus = 0;\n>> diff --git a/xdiff/xutils.c b/xdiff/xutils.c\n>> index c0cd5338c4e..865e08f0e93 100644\n>> --- a/xdiff/xutils.c\n>> +++ b/xdiff/xutils.c\n>> @@ -98,7 +98,7 @@ void *xdl_cha_alloc(chastore_t *cha) {\n>>   \tvoid *data;\n>>     \tif (!(ancur = cha->ancur) || ancur->icurr == cha->nsize) {\n>> -\t\tif (!(ancur = (chanode_t *) xdl_malloc(sizeof(chanode_t) + cha->nsize))) {\n>> +\t\tif (!(ancur = (chanode_t *) malloc(sizeof(chanode_t) + cha->nsize))) {\n>>     \t\t\treturn NULL;\n>>   \t\t}\n\n"},{"id":"458668","messageId":"220708.864jzr9sq4.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"YsiHGliw2dYmGcDY@coredump.intra.peff.net","subject":"Re: [PATCH 6/7] xdiff: remove xdl_malloc() wrapper, use malloc(), not xmalloc()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T21:47:58Z","receivedAt":"2022-07-08T21:53:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 08 2022, Jeff King wrote:\n\n> On Fri, Jul 08, 2022 at 04:20:18PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> As noted in 36c83197249 (xdiff: use xmalloc/xrealloc, 2019-04-11) the\n>> reason we have xdl_malloc() in the first place was to make the xdiff\n>> code safer, it was not handling some allocation failures correctly at\n>> the time.\n>> \n>> But as noted in that commit doing this was intended as a stopgap, as\n>> various code in xdiff/* did not correctly handle allocation failures,\n>> and would have segfaulted if malloc() returned NULL.\n>> \n>> But since then and after preceding commits we can be confident that\n>> malloc() failures are handled correctly. All of these users of\n>> xdl_malloc() are checking their return values, and we're returning\n>> -1 (or similar ) to the top-level in case of failures.\n>\n> This is also losing the other parts mentioned in 36c83197249:\n> respecting GIT_ALLOC_LIMIT and any memory reclamation strategies.\n>\n> I think you'd want an xmalloc_gently() instead of a raw malloc().\n\n...\n\n> For the same reason, I suspect it's better to leave this with a layer of\n> preprocessor indirection. Even if we chose to use malloc() here, libgit2\n> might not, and having the macro makes it easier to share the code.\n\nI think\nhttps://lore.kernel.org/git/220708.86czef9t6y.gmgdl@evledraar.gmail.com/\nin the related side-thread shows a workable way to do that.\n\nBut I didn't think about the GIT_ALLOC_LIMIT, so if we want that still\nwe'd need to have this code routed to not just malloc() but an\n\"xmalloc_gently()\" as you suggest.\n\nBut we also have a few other uses of malloc() in the codebase. I wonder\nif the right thing here isn't to just use malloc(), but to have\ngit-compat-util.h override malloc() (similar to how we we override\ne.g. exit() now...), which would also catch those.\n\nOr we could just say that it's not worth the complexity, and xdiff won't\nrespect GIT_ALLOC_LIMIT .. :(\n"},{"id":"458670","messageId":"220709.86v8s78chj.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"8c24cd7737b29d461788b71f6a94eb74c468ad33.1657297520.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/4] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-08T22:17:42Z","receivedAt":"2022-07-08T22:29:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 08 2022, Phillip Wood via GitGitGadget wrote:\n\n> From: Phillip Wood <phillip.wood@dunelm.org.uk>\n>\n> Add a helper to grow an array. This is analogous to ALLOC_GROW() in\n> the rest of the codebase but returns −1 on allocation failure to\n> accommodate other users of libxdiff such as libgit2. \n> [...]\n> +\t\tif (XDL_ALLOC_GROW(cf->rcrecs, cf->count, cf->alloc))\n> +void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size)\n> +{\n> +\tvoid *tmp = NULL;\n> +\tsize_t n = ((LONG_MAX - 16) / 2 >= *alloc) ? 2 * *alloc + 16 : LONG_MAX;\n> +\tif (nr > n)\n> +\t\tn = nr;\n> +\tif (SIZE_MAX / size >= n)\n> +\t\ttmp = xdl_realloc(p, n * size);\n> +\tif (tmp) {\n> +\t\t*alloc = n;\n> +\t} else {\n> +\t\txdl_free(p);\n> +\t\t*alloc = 0;\n> +\t}\n> +\treturn tmp;\n> +}\n\nWhether you agree with the entire approach in my series-on-top[1] or not\n(and it looks like not), this way of doing it seems needlessly complex.\n\nI.e. the whole \"pass the size\" business is here because you're wanting\nto use this as an expression, which ALLOC_GROW() isn't able to do.\n\nBut you've also changed every single callsite anyway.\n\nSo why not just change:\n\n    if (XDL_ALLOC_GROW(recs, ...))\n\nTo:\n\n    XDL_ALLOC_GROW(recs, ...);\n    if (!recs)\n\nAnd do away with needing to pass this through a function where we get\nthe size, and thus losing the type information before we can call\nsizeof().\n\nThen, as that series shows the implementation here could be pretty much\nan exact line-for-line copy of what we have in cache.h, including the\nsame alloc_nr(), all without casts etc.\n\nAll of which seems much simpler than trying to maintain the constraint\nthat this must be usable in an expression.\n\nYour commit message sort-of addresses this by mentioning that this\n\"returns −1 on allocation failure to accommodate other users of libxdiff\nsuch as libgit2\".\n\nBut since libgit2 won't use this, but only a copy of this xdiff code in\nlibgit2 I don't see how that makes sense. We're only talking about using\nthis in the xdiff code we maintain, and even if libgit2 wanted to use it\nit could deal with not being able to use it in an expression, no?\n    \t\n1. https://lore.kernel.org/git/patch-5.7-3665576576f-20220708T140354Z-avarab@gmail.com/\n"},{"id":"458759","messageId":"dcde61a3-4d96-6cd5-f05e-45781599f8bd@gmail.com","threadId":"58081","inReplyTo":"220708.86czef9t6y.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 7/7] xdiff: remove xdl_free(), use free() instead","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-11T09:26:20Z","receivedAt":"2022-07-11T09:56:17Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 08/07/2022 22:26, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Jul 08 2022, Phillip Wood wrote:\n> \n>> Hi Ævar\n>>\n>> On 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n>>> Remove the xdl_free() wrapper macro in favor of using free()\n>>> directly. The wrapper macro was brought in with the initial import of\n>>> xdiff/ in 3443546f6ef (Use a *real* built-in diff generator,\n>>> 2006-03-24).\n>>> As subsequent discussions on the topic[1] have made clear there's no\n>>> reason to use this wrapper.\n>>\n>> That link is to a message where you assert that there is no reason for\n>> the wrapper, you seem to be the only person in that thread making that\n>> argument. The libgit2 maintainer and others are arguing that it is\n>> worth keeping to make it easy to swap the allocator.\n> \n> Arguing that it's not needed for a technical reason, but an \"aesthetic\n> and ergonomic one\", per:\n> https://lore.kernel.org/git/20220217225804.GC7@edef91d97c94/;\n> \n> Perhaps I misread it, but I took this as Junio concurring with what I\n> was pointing out there:\n> https://lore.kernel.org/git/xmqqfsohbdre.fsf@gitster.g/\n> \n>>> Both git itself as well as any external\n>>> users such as libgit2 compile the xdiff/* code as part of their own\n>>> compilation, and can thus find the right malloc() and free() at\n>>> link-time.\n>>\n>> I'm struggling to see how that is simpler than the current situation\n>> with xdl_malloc().\n> \n> It's simpler because when maintaining that code in git.git it's less of\n> a special case, e.g. we have coccinelle rules that match free(), now\n> every such rule needs to account for the xdiff special-case.\n> \n> And it's less buggy because if you're relying on us always using a\n> wrapper bugs will creep in, current master has this:\n> \t\n> \t$ git -P grep '\\bfree\\(' -- xdiff\n> \txdiff/xdiff.h:#define xdl_free(ptr) free(ptr)\n> \txdiff/xmerge.c:         free(c);\n> \txdiff/xmerge.c: free(next_m);\n> \n>> Perhaps you could show how you think libgit2 could\n>> easily make xdiff use git__malloc() instead of malloc() without\n>> changing the malloc() calls (the message you linked to essentially\n>> suggests they do a search and replace). If people cannot swap in\n>> another allocator without changing the code then you are imposing a\n>> barrier to them contributing patches back to git's xdiff.\n> \n> I was suggesting that if libgit2 wanted to maintain a copy of xdiff that\n> catered to the asthetic desires of the maintainer adjusting whatever\n> import script you use to apply a trivial coccinelle transformation would\n> give you what you wanted. Except without a maintenance burden on\n> git.git, or the bugs you'd get now since you're not catching those two\n> free() cases (or any future such cases).\n> \n> But just having the code use malloc() and free() directly and getting\n> you what you get now is easy, and doesn't require any such\n> search-replacement.\n> \n> Here's how:\n> \n> I imported the xdiff/*.[ch] files at the tip of my branch into current\n> libgit2.git's src/libgit2/xdiff/, and then applied this on top, which is\n> partially re-applying libgit2's own monkeypatches, and partially\n> adjusting for upstream changes (including this one):\n> \t\n> \tdiff --git a/src/libgit2/xdiff/git-xdiff.h b/src/libgit2/xdiff/git-xdiff.h\n> \tindex b75dba819..2e00764d4 100644\n> \t--- a/src/libgit2/xdiff/git-xdiff.h\n> \t+++ b/src/libgit2/xdiff/git-xdiff.h\n> \t@@ -14,6 +14,7 @@\n> \t #ifndef INCLUDE_git_xdiff_h__\n> \t #define INCLUDE_git_xdiff_h__\n> \t\n> \t+#include <regex.h>\n> \t #include \"regexp.h\"\n> \t\n> \t /* Work around C90-conformance issues */\n> \t@@ -27,11 +28,10 @@\n> \t # endif\n> \t #endif\n> \t\n> \t-#define xdl_malloc(x) git__malloc(x)\n> \t-#define xdl_free(ptr) git__free(ptr)\n> \t-#define xdl_realloc(ptr, x) git__realloc(ptr, x)\n> \t+#define malloc(x) git__malloc(x)\n> \t+#define free(ptr) git__free(ptr)\n> \t\n> \t-#define XDL_BUG(msg) GIT_ASSERT(msg)\n> \t+#define BUG(msg) GIT_ASSERT(msg)\n> \t\n> \t #define xdl_regex_t git_regexp\n> \t #define xdl_regmatch_t git_regmatch\n> \t@@ -50,4 +50,17 @@ GIT_INLINE(int) xdl_regexec_buf(\n> \t \treturn -1;\n> \t }\n> \t\n> \t+static inline size_t st_mult(size_t a, size_t b)\n> \t+{\n> \t+\treturn a * b;\n> \t+}\n> \t+\n> \t+static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,\n> \t+\t\t\t      size_t nmatch, regmatch_t pmatch[], int eflags)\n> \t+{\n> \t+\tassert(nmatch > 0 && pmatch);\n> \t+\tpmatch[0].rm_so = 0;\n> \t+\tpmatch[0].rm_eo = size;\n> \t+\treturn regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);\n> \t+}\n> \t #endif\n> \tdiff --git a/src/libgit2/xdiff/xdiff.h b/src/libgit2/xdiff/xdiff.h\n> \tindex a37d89fcd..5e5b525c2 100644\n> \t--- a/src/libgit2/xdiff/xdiff.h\n> \t+++ b/src/libgit2/xdiff/xdiff.h\n> \t@@ -23,6 +23,8 @@\n> \t #if !defined(XDIFF_H)\n> \t #define XDIFF_H\n> \t\n> \t+#include \"git-xdiff.h\"\n> \t+\n> \t #ifdef __cplusplus\n> \t extern \"C\" {\n> \t #endif /* #ifdef __cplusplus */\n> \tdiff --git a/src/libgit2/xdiff/xinclude.h b/src/libgit2/xdiff/xinclude.h\n> \tindex a4285ac0e..79812ad8a 100644\n> \t--- a/src/libgit2/xdiff/xinclude.h\n> \t+++ b/src/libgit2/xdiff/xinclude.h\n> \t@@ -23,7 +23,8 @@\n> \t #if !defined(XINCLUDE_H)\n> \t #define XINCLUDE_H\n> \t\n> \t-#include \"git-compat-util.h\"\n> \t+#include \"git-xdiff.h\"\n> \t+#include \"git-shared-util.h\"\n> \t #include \"xmacros.h\"\n> \t #include \"xdiff.h\"\n> \t #include \"xtypes.h\"\n> \n> If you then build it and run e.g.:\n> \n> \tgdb -args ./libgit2_tests -smerge::files\n> \n> You'll get stack traces like this if you break on stdalloc__malloc\n> (which it uses for me in its default configuration):\n> \t\n> \t(gdb) bt\n> \t#0  stdalloc__malloc (len=144, file=0x87478d \"/home/avar/g/libgit2/src/libgit2/xdiff/xutils.c\", line=101) at /home/avar/g/libgit2/src/util/allocators/stdalloc.c:14\n> \t#1  0x00000000006ec15c in xdl_cha_alloc (cha=0x7fffffffcaa8) at /home/avar/g/libgit2/src/libgit2/xdiff/xutils.c:101\n> \t#2  0x00000000006eaee9 in xdl_prepare_ctx (pass=1, mf=0x7fffffffcc98, narec=13, xpp=0x7fffffffcca8, cf=0x7fffffffc7d0, xdf=0x7fffffffcaa8)\n> \t    at /home/avar/g/libgit2/src/libgit2/xdiff/xprepare.c:194\n> \n> IOW this was all seamlessly routed to your git__malloc() without us\n> having to maintain an xdl_{malloc,free}().\n\nThanks for showing this, it is really helpful to see a concrete example. \nI was especially interested to see how you went about the conversion \nwithout redefining standard library functions or introducing \nnon-namespaced identifiers in files that included xdiff.h. Unfortunately \nyou have not done that and so I don't think the approach above is viable \n  for a well behaved library.\n\n> Now, I think that's obviously worth adjusting, e.g. the series I've got\n> here could make this easier for libgit2 by including st_mult() at least,\n> I'm not sure what you want to do about regexec_buf().\n> \n> For the monkeypatching you do now of creating a \"git-xdiff.h\" I think it\n> would be very good to get a patch like what I managed to get\n> sha1collisiondetection.git to accept for our own use-case, which allows\n> us to use their upstream code as-is from a submodule:\n> \n> \thttps://github.com/cr-marcstevens/sha1collisiondetection/commit/b45fcef\n\nThanks for the link, That's a really good example where all the \nidentifiers are namespaced and the public include file does not pollute \nthe code that is including it. If you come up with something like that \nwhere the client code can set up same #defines for malloc, calloc, \nrealloc and free that would be a good way forward. I do not think we \nshould require projects to be defining there own versions of \n[C]ALLOC_*() and BUG() just to use xdiff.\n\nBest Wishes\n\nPhillip\n\n"},{"id":"458760","messageId":"YsvuYX7xekLGIdVt@coredump.intra.peff.net","threadId":"58081","inReplyTo":"220708.864jzr9sq4.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 6/7] xdiff: remove xdl_malloc() wrapper, use malloc(), not xmalloc()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-07-11T09:33:21Z","receivedAt":"2022-07-11T10:11:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 08, 2022 at 11:47:58PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> But we also have a few other uses of malloc() in the codebase. I wonder\n> if the right thing here isn't to just use malloc(), but to have\n> git-compat-util.h override malloc() (similar to how we we override\n> e.g. exit() now...), which would also catch those.\n\nI suspect that introduces new complexities, as some calls really may\nwant an actual no-frills malloc. I'm thinking stuff in compat/, for\nexample, where extra actions taken by xmalloc() could cause weird\nlooping or races. This was probably a lot more likely when we closed\npacks via xmalloc (e.g., via git_mmap()'s malloc() call) but I think the\ngeneral principle still holds. E.g., gitsetenv() calling getenv() is a\npotential questionable area.\n\nIt does look like the call in submodule--helper.c is just wrong, though,\nand should be changed. You could probably detect these with the\npreprocessor, but again, you run into complexities with the cases that\n_should_ be vanilla malloc. Given how little of a problem this has been\nhistorically, I'm mostly content to notice these in review and\noccasionally grep for fixes.\n\nI suppose a coccinelle rule could help, because it makes it easy to\nsuppress false positives (like all of compat/), which the preprocessor\ndoesn't.\n\n-Peff\n"},{"id":"458762","messageId":"81b1f792-f8e9-efa5-fb3f-856264b85287@gmail.com","threadId":"58081","inReplyTo":"dcde61a3-4d96-6cd5-f05e-45781599f8bd@gmail.com","subject":"Re: [PATCH 7/7] xdiff: remove xdl_free(), use free() instead","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-11T09:54:56Z","receivedAt":"2022-07-11T10:49:04Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 11/07/2022 10:26, Phillip Wood wrote:\n>> For the monkeypatching you do now of creating a \"git-xdiff.h\" I think it\n>> would be very good to get a patch like what I managed to get\n>> sha1collisiondetection.git to accept for our own use-case, which allows\n>> us to use their upstream code as-is from a submodule:\n>>\n>>     https://github.com/cr-marcstevens/sha1collisiondetection/commit/b45fcef \n>>\n> \n> Thanks for the link, That's a really good example where all the \n> identifiers are namespaced and the public include file does not pollute \n> the code that is including it. If you come up with something like that \n> where the client code can set up same #defines for malloc, calloc, \n> realloc and free that would be a good way forward.\n\nTo be clear those should be namespaced, so we'd have \n-Dxdl_malloc=xmalloc (or xmalloc_gently) and libgit2 would have\n-D xdl_malloc=git__malloc\n\nBest Wishes\n\nPhillip\n\n> I do not think we \n> should require projects to be defining there own versions of \n> [C]ALLOC_*() and BUG() just to use xdiff.\n> \n> Best Wishes\n> \n> Phillip\n> \n\n"},{"id":"458763","messageId":"bb1c85e4-50be-04a5-3c9f-f4c4dab6a50e@gmail.com","threadId":"58081","inReplyTo":"220709.86v8s78chj.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 4/4] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-11T10:00:06Z","receivedAt":"2022-07-11T10:54:46Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 08/07/2022 23:17, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Jul 08 2022, Phillip Wood via GitGitGadget wrote:\n> \n>> From: Phillip Wood <phillip.wood@dunelm.org.uk>\n>>\n>> Add a helper to grow an array. This is analogous to ALLOC_GROW() in\n>> the rest of the codebase but returns −1 on allocation failure to\n>> accommodate other users of libxdiff such as libgit2.\n>> [...]\n>> +\t\tif (XDL_ALLOC_GROW(cf->rcrecs, cf->count, cf->alloc))\n>> +void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size)\n>> +{\n>> +\tvoid *tmp = NULL;\n>> +\tsize_t n = ((LONG_MAX - 16) / 2 >= *alloc) ? 2 * *alloc + 16 : LONG_MAX;\n>> +\tif (nr > n)\n>> +\t\tn = nr;\n>> +\tif (SIZE_MAX / size >= n)\n>> +\t\ttmp = xdl_realloc(p, n * size);\n>> +\tif (tmp) {\n>> +\t\t*alloc = n;\n>> +\t} else {\n>> +\t\txdl_free(p);\n>> +\t\t*alloc = 0;\n>> +\t}\n>> +\treturn tmp;\n>> +}\n> \n> Whether you agree with the entire approach in my series-on-top[1] or not\n> (and it looks like not), this way of doing it seems needlessly complex.\n> \n> I.e. the whole \"pass the size\" business is here because you're wanting\n> to use this as an expression, which ALLOC_GROW() isn't able to do.\n> \n> But you've also changed every single callsite anyway.\n> \n> So why not just change:\n> \n>      if (XDL_ALLOC_GROW(recs, ...))\n> \n> To:\n> \n>      XDL_ALLOC_GROW(recs, ...);\n>      if (!recs)\n\nBecause I think it's ugly, we'd never define a function as void(int* \nresult, args...) and I don't think we should do that for a macro where \nwe can avoid it. The existing ALLOC_* macros make sense as statements \nbecause they die on failure.\n\nBest Wishes\n\nPhillip\n\n> And do away with needing to pass this through a function where we get\n> the size, and thus losing the type information before we can call\n> sizeof().\n> \n> Then, as that series shows the implementation here could be pretty much\n> an exact line-for-line copy of what we have in cache.h, including the\n> same alloc_nr(), all without casts etc.\n >\n> All of which seems much simpler than trying to maintain the constraint\n> that this must be usable in an expression.\n> \n> Your commit message sort-of addresses this by mentioning that this\n> \"returns −1 on allocation failure to accommodate other users of libxdiff\n> such as libgit2\".\n> \n> But since libgit2 won't use this, but only a copy of this xdiff code in\n> libgit2 I don't see how that makes sense. We're only talking about using\n> this in the xdiff code we maintain, and even if libgit2 wanted to use it\n> it could deal with not being able to use it in an expression, no?\n>      \t\n> 1. https://lore.kernel.org/git/patch-5.7-3665576576f-20220708T140354Z-avarab@gmail.com/\n\n"},{"id":"458764","messageId":"622ae936-b45d-885a-d779-64156cb8e3ba@gmail.com","threadId":"58081","inReplyTo":"patch-3.7-d7c14a1cad8-20220708T140354Z-avarab@gmail.com","subject":"Re: [PATCH 3/7] git-shared-util.h: add G*() versions of *ALLOC_*()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-11T10:06:33Z","receivedAt":"2022-07-11T11:02:19Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n> Add \"gently\" versions of ALLOC_ARRAY(), CALLOC_ARRAY() etc. using the\n> naming convention G*() as a shorthand for \"GENTLY_*()\".\n\nIt might be nicer just to call them ALLOC_ARRAY_GENTLY() etc. As the \nreturn value needs to be checked it would make sense to implement them \nas expressions as I have done for XDL_ALLOC_ARRAY() etc.\n\n> Nothing uses these functions yet, but as we'll see in subsequent\n> commit(s) we're able to convert things that need e.g. non-fatal\n> \"ALLOC_GROW\" behavior over to this.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>   git-shared-util.h | 15 +++++++++++++--\n>   1 file changed, 13 insertions(+), 2 deletions(-)\n> \n> diff --git a/git-shared-util.h b/git-shared-util.h\n> index 7b4479a0f72..718a8e00732 100644\n> --- a/git-shared-util.h\n> +++ b/git-shared-util.h\n> @@ -8,8 +8,11 @@\n>   #define FREE_AND_NULL(p) do { free(p); (p) = NULL; } while (0)\n>   \n>   #define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))\n> +#define GALLOC_ARRAY(x, alloc) (x) = malloc(st_mult(sizeof(*(x)), (alloc)))\n>   #define CALLOC_ARRAY(x, alloc) (x) = xcalloc((alloc), sizeof(*(x)))\n> +#define GCALLOC_ARRAY(x, alloc) (x) = calloc((alloc), sizeof(*(x)))\n>   #define REALLOC_ARRAY(x, alloc) (x) = xrealloc((x), st_mult(sizeof(*(x)), (alloc)))\n> +#define GREALLOC_ARRAY(x, alloc) (x) = realloc((x), st_mult(sizeof(*(x)), (alloc)))\n>   \n>   #define COPY_ARRAY(dst, src, n) copy_array((dst), (src), (n), sizeof(*(dst)) + \\\n>   \tBUILD_ASSERT_OR_ZERO(sizeof(*(dst)) == sizeof(*(src))))\n> @@ -71,17 +74,25 @@ static inline void move_array(void *dst, const void *src, size_t n, size_t size)\n>    * added niceties.\n>    *\n>    * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'.\n> + *\n> + * GALLOC_GROW() behaves like ALLOC_GROW(), except that in malloc()\n> + * failure we'll return NULL rather than dying.\n>    */\n> -#define ALLOC_GROW(x, nr, alloc) \\\n> +#define ALLOC_GROW_1(x, nr, alloc, gently) \\\n>   \tdo { \\\n>   \t\tif ((nr) > alloc) { \\\n>   \t\t\tif (alloc_nr(alloc) < (nr)) \\\n>   \t\t\t\talloc = (nr); \\\n>   \t\t\telse \\\n>   \t\t\t\talloc = alloc_nr(alloc); \\\n> -\t\t\tREALLOC_ARRAY(x, alloc); \\\n\nThis leaks the old allocation if realloc() fails because it overwrites \nthe original pointer with NULL, that is particularly bad as if realloc() \nfails we're already short of memory. XDL_ALLOC_GROW() shows how a \npossible way to do this.\n\nBest Wishes\n\nPhillip\n\n> +\t\t\tif (gently) \\\n> +\t\t\t\tGREALLOC_ARRAY(x, alloc); \\\n> +\t\t\telse \\\n> +\t\t\t\tREALLOC_ARRAY(x, alloc); \\\n>   \t\t} \\\n>   \t} while (0)\n> +#define ALLOC_GROW(x, nr, alloc) ALLOC_GROW_1(x, nr, alloc, 0)\n> +#define GALLOC_GROW(x, nr, alloc) ALLOC_GROW_1(x, nr, alloc, 1)\n>   \n>   /*\n>    * Similar to ALLOC_GROW but handles updating of the nr value and\n\n"},{"id":"458765","messageId":"37667911-3049-4e78-e67d-9fcb7389dfc0@gmail.com","threadId":"58081","inReplyTo":"patch-4.7-19567077b84-20220708T140354Z-avarab@gmail.com","subject":"Re: [PATCH 4/7] xdiff: use G[C]ALLOC_ARRAY(), not XDL_CALLOC_ARRAY()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-11T10:10:20Z","receivedAt":"2022-07-11T11:03:39Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n> Use the newly created GCALLOC_ARRAY() helpers rather than the recently\n> introduced XDL_[C]ALLOC_ARRAY().\n> \n> As shown in this diff the calling convention differs, we cannot use\n> GCALLOC_ARRAY() as an expression, but that's an advantage in not\n> having to relay the \"sizeof()\" down via a wrapper function.\n> \n> This also:\n> \n>   * Fixes long-standing potential overflow issues, as we're using\n>     st_mult() in the underlying G_[C]ALLOC().  Note that the\n\nWhat issues is this fixing? XDL_ALLOC_ARRAY() already checks for overflow.\n\n>   * Slightly optimizes the \"XDL_CALLOC_ARRAY\", as we'll now use\n>     calloc() rather than malloc() + memset() (although smart compilers\n>     will probably do the same for both...).\n\nThat's addressed in V2 of my series, unfortunately I sent it just after \nyou'd sent this series.\n\n>   * Changes the \"XDL_CALLOC_ARRAY\" behavior where we'd shortcut if the\n>     size was already large enough, but this behavior was changed when\n>     XDL_ALLOC_ARRAY() was introduced, so this is safe.\n\nI'm not sure what you mean here - how did we shortcut before?\n\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>   xdiff/xdiffi.c     |  3 ++-\n>   xdiff/xhistogram.c |  9 ++++++---\n>   xdiff/xmacros.h    | 12 ------------\n>   xdiff/xpatience.c  |  6 ++++--\n>   xdiff/xprepare.c   | 24 ++++++++++++++++--------\n>   5 files changed, 28 insertions(+), 26 deletions(-)\n> \n> diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c\n> index 6fded43e87d..077cc456087 100644\n> --- a/xdiff/xdiffi.c\n> +++ b/xdiff/xdiffi.c\n> @@ -333,7 +333,8 @@ int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,\n>   \t * One is to store the forward path and one to store the backward path.\n>   \t */\n>   \tndiags = xe->xdf1.nreff + xe->xdf2.nreff + 3;\n> -\tif (!XDL_ALLOC_ARRAY(kvd, 2 * ndiags + 2))\n> +\tGALLOC_ARRAY(kvd, 2 * ndiags + 2);\n> +\tif (!kvd)\n>   \t\treturn -1;\n>   \tkvdf = kvd;\n>   \tkvdb = kvdf + ndiags;\n> diff --git a/xdiff/xhistogram.c b/xdiff/xhistogram.c\n> index df909004c10..f20592bfbdd 100644\n> --- a/xdiff/xhistogram.c\n> +++ b/xdiff/xhistogram.c\n> @@ -266,14 +266,17 @@ static int find_lcs(xpparam_t const *xpp, xdfenv_t *env,\n>   \n>   \tindex.table_bits = xdl_hashbits(count1);\n>   \tindex.records_size = 1 << index.table_bits;\n> -\tif (!XDL_CALLOC_ARRAY(index.records, index.records_size))\n> +\tGCALLOC_ARRAY(index.records, index.records_size);\n> +\tif (!index.records)\n\nI don't think that having GALLOC_ARRAY() as a statement is an \nimprovement here.\n\nBest Wishes\n\nPhillip\n\n>   \t\tgoto cleanup;\n>   \n>   \tindex.line_map_size = count1;\n> -\tif (!XDL_CALLOC_ARRAY(index.line_map, index.line_map_size))\n> +\tGCALLOC_ARRAY(index.line_map, index.line_map_size);\n> +\tif (!index.line_map)\n>   \t\tgoto cleanup;\n>   \n> -\tif (!XDL_CALLOC_ARRAY(index.next_ptrs, index.line_map_size))\n> +\tGCALLOC_ARRAY(index.next_ptrs, index.line_map_size);\n> +\tif (!index.next_ptrs)\n>   \t\tgoto cleanup;\n>   \n>   \t/* lines / 4 + 1 comes from xprepare.c:xdl_prepare_ctx() */\n> diff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\n> index d13a6724629..75506bdf17e 100644\n> --- a/xdiff/xmacros.h\n> +++ b/xdiff/xmacros.h\n> @@ -49,18 +49,6 @@ do { \\\n>   \t\t((unsigned long) __p[2]) << 16 | ((unsigned long) __p[3]) << 24; \\\n>   } while (0)\n>   \n> -/* Allocate an array of nr elements, returns NULL on failure */\n> -#define XDL_ALLOC_ARRAY(p, nr)\t\t\t\t\\\n> -\t((p) = SIZE_MAX / sizeof(*(p)) >= (size_t)(nr)\t\\\n> -\t\t? xdl_malloc((nr) * sizeof(*(p)))\t\\\n> -\t\t: NULL)\n> -\n> -/* Allocate an array of nr zeroed out elements, returns NULL on failure */\n> -#define XDL_CALLOC_ARRAY(p, nr)\t\t\t\t\\\n> -\t(XDL_ALLOC_ARRAY((p), (nr))\t\t\t\\\n> -\t\t? memset((p), 0, (nr) * sizeof(*(p)))\t\\\n> -\t\t: NULL)\n> -\n>   /*\n>    * Ensure array p can accommodate at least nr elements, growing the\n>    * array and updating alloc (which is the number of allocated\n> diff --git a/xdiff/xpatience.c b/xdiff/xpatience.c\n> index fe39c2978cb..bb328d9f852 100644\n> --- a/xdiff/xpatience.c\n> +++ b/xdiff/xpatience.c\n> @@ -151,7 +151,8 @@ static int fill_hashmap(mmfile_t *file1, mmfile_t *file2,\n>   \n>   \t/* We know exactly how large we want the hash map */\n>   \tresult->alloc = count1 * 2;\n> -\tif (!XDL_CALLOC_ARRAY(result->entries, result->alloc))\n> +\tGCALLOC_ARRAY(result->entries, result->alloc);\n> +\tif (!result->entries)\n>   \t\treturn -1;\n>   \n>   \t/* First, fill with entries from the first file */\n> @@ -208,7 +209,8 @@ static int find_longest_common_sequence(struct hashmap *map, struct entry **res)\n>   \t */\n>   \tint anchor_i = -1;\n>   \n> -\tif (!XDL_ALLOC_ARRAY(sequence, map->nr))\n> +\tGALLOC_ARRAY(sequence, map->nr);\n> +\tif (!sequence)\n>   \t\treturn -1;\n>   \n>   \tfor (entry = map->first; entry; entry = entry->next) {\n> diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\n> index c84549f6c50..d6cbee32a2a 100644\n> --- a/xdiff/xprepare.c\n> +++ b/xdiff/xprepare.c\n> @@ -78,15 +78,17 @@ static int xdl_init_classifier(xdlclassifier_t *cf, long size, long flags) {\n>   \n>   \t\treturn -1;\n>   \t}\n> -\tif (!XDL_CALLOC_ARRAY(cf->rchash, cf->hsize)) {\n> +\tGCALLOC_ARRAY(cf->rchash, cf->hsize);\n> +\tif (!cf->rchash) {\n>   \n>   \t\txdl_cha_free(&cf->ncha);\n>   \t\treturn -1;\n>   \t}\n>   \n>   \tcf->alloc = size;\n> -\tif (!XDL_ALLOC_ARRAY(cf->rcrecs, cf->alloc)) {\n>   \n> +\tGALLOC_ARRAY(cf->rcrecs, cf->alloc);\n> +\tif (!cf->rcrecs) {\n>   \t\txdl_free(cf->rchash);\n>   \t\txdl_cha_free(&cf->ncha);\n>   \t\treturn -1;\n> @@ -170,12 +172,14 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n>   \n>   \tif (xdl_cha_init(&xdf->rcha, sizeof(xrecord_t), narec / 4 + 1) < 0)\n>   \t\tgoto abort;\n> -\tif (!XDL_ALLOC_ARRAY(recs, narec))\n> +\tGALLOC_ARRAY(recs, narec);\n> +\tif (!recs)\n>   \t\tgoto abort;\n>   \n>   \thbits = xdl_hashbits((unsigned int) narec);\n>   \thsize = 1 << hbits;\n> -\tif (!XDL_CALLOC_ARRAY(rhash, hsize))\n> +\tGCALLOC_ARRAY(rhash, hsize);\n> +\tif (!rhash)\n>   \t\tgoto abort;\n>   \n>   \tnrec = 0;\n> @@ -196,14 +200,17 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n>   \t\t}\n>   \t}\n>   \n> -\tif (!XDL_CALLOC_ARRAY(rchg, nrec + 2))\n> +\tGCALLOC_ARRAY(rchg, nrec + 2);\n> +\tif (!rchg)\n>   \t\tgoto abort;\n>   \n>   \tif ((XDF_DIFF_ALG(xpp->flags) != XDF_PATIENCE_DIFF) &&\n>   \t    (XDF_DIFF_ALG(xpp->flags) != XDF_HISTOGRAM_DIFF)) {\n> -\t\tif (!XDL_ALLOC_ARRAY(rindex, nrec + 1))\n> +\t\tGALLOC_ARRAY(rindex, nrec + 1);\n> +\t\tif (!rindex)\n>   \t\t\tgoto abort;\n> -\t\tif (!XDL_ALLOC_ARRAY(ha, nrec + 1))\n> +\t\tGALLOC_ARRAY(ha, nrec + 1);\n> +\t\tif (!ha)\n>   \t\t\tgoto abort;\n>   \t}\n>   \n> @@ -369,7 +376,8 @@ static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd\n>   \txdlclass_t *rcrec;\n>   \tchar *dis, *dis1, *dis2;\n>   \n> -\tif (!XDL_CALLOC_ARRAY(dis, xdf1->nrec + xdf2->nrec + 2))\n> +\tGCALLOC_ARRAY(dis, xdf1->nrec + xdf2->nrec + 2);\n> +\tif (!dis)\n>   \t\treturn -1;\n>   \tdis1 = dis;\n>   \tdis2 = dis1 + xdf1->nrec + 1;\n\n"},{"id":"458766","messageId":"c2bc5c79-e71b-f9dc-ba97-261d3454e150@gmail.com","threadId":"58081","inReplyTo":"patch-5.7-3665576576f-20220708T140354Z-avarab@gmail.com","subject":"Re: [PATCH 5/7] xdiff: use GALLOC_GROW(), not XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-11T10:13:06Z","receivedAt":"2022-07-11T11:07:13Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n> Replace the recently introduced XDL_ALLOC_GROW() with invocations of\n> the GALLOC_GROW() from git-shared-util.h.\n> \n> As this change shows the macro + function indirection of\n> XDL_ALLOC_GROW() is something we needed only because the two callsites\n> we used it in wanted to use it as an expression, and we thus had to\n> pass the \"sizeof\" down.\n> \n> Let's just check the value afterwards instead, which allows us to use\n> the shared macro, we can also remove xdl_reallo(), this was its last\n> user.\n\nI don't think this expression->statement change is an improvement. This \nchange also removes the overflow checks that are present in \nXDL_ALLOC_GROW() and fails to free the old allocation when realloc() \nfails. It is not a like for like replacement.\n\nBest Wishes\n\nPhillip\n\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>   xdiff/xdiff.h    |  1 -\n>   xdiff/xmacros.h  | 11 -----------\n>   xdiff/xprepare.c |  8 +++++---\n>   xdiff/xutils.c   | 17 -----------------\n>   xdiff/xutils.h   |  2 --\n>   5 files changed, 5 insertions(+), 34 deletions(-)\n> \n> diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h\n> index 72e25a9ffa5..832cf9d977e 100644\n> --- a/xdiff/xdiff.h\n> +++ b/xdiff/xdiff.h\n> @@ -121,7 +121,6 @@ typedef struct s_bdiffparam {\n>   \n>   #define xdl_malloc(x) xmalloc(x)\n>   #define xdl_free(ptr) free(ptr)\n> -#define xdl_realloc(ptr,x) xrealloc(ptr,x)\n>   \n>   void *xdl_mmfile_first(mmfile_t *mmf, long *size);\n>   long xdl_mmfile_size(mmfile_t *mmf);\n> diff --git a/xdiff/xmacros.h b/xdiff/xmacros.h\n> index 75506bdf17e..6a6b3057375 100644\n> --- a/xdiff/xmacros.h\n> +++ b/xdiff/xmacros.h\n> @@ -48,15 +48,4 @@ do { \\\n>   \t(v) = (unsigned long) __p[0] | ((unsigned long) __p[1]) << 8 | \\\n>   \t\t((unsigned long) __p[2]) << 16 | ((unsigned long) __p[3]) << 24; \\\n>   } while (0)\n> -\n> -/*\n> - * Ensure array p can accommodate at least nr elements, growing the\n> - * array and updating alloc (which is the number of allocated\n> - * elements) as necessary. Frees p and returns -1 on failure, returns\n> - * 0 on success\n> - */\n> -#define XDL_ALLOC_GROW(p, nr, alloc)\t\\\n> -\t(-!((nr) <= (alloc) ||\t\t\\\n> -\t    ((p) = xdl_alloc_grow_helper((p), (nr), &(alloc), sizeof(*(p))))))\n> -\n>   #endif /* #if !defined(XMACROS_H) */\n> diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c\n> index d6cbee32a2a..4182d9e1c0a 100644\n> --- a/xdiff/xprepare.c\n> +++ b/xdiff/xprepare.c\n> @@ -128,8 +128,9 @@ static int xdl_classify_record(unsigned int pass, xdlclassifier_t *cf, xrecord_t\n>   \t\t\treturn -1;\n>   \t\t}\n>   \t\trcrec->idx = cf->count++;\n> -\t\tif (XDL_ALLOC_GROW(cf->rcrecs, cf->count, cf->alloc))\n> -\t\t\t\treturn -1;\n> +\t\tGALLOC_GROW(cf->rcrecs, cf->count, cf->alloc);\n> +\t\tif (!cf->rcrecs)\n> +\t\t\treturn -1;\n>   \t\tcf->rcrecs[rcrec->idx] = rcrec;\n>   \t\trcrec->line = line;\n>   \t\trcrec->size = rec->size;\n> @@ -187,7 +188,8 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_\n>   \t\tfor (top = blk + bsize; cur < top; ) {\n>   \t\t\tprev = cur;\n>   \t\t\thav = xdl_hash_record(&cur, top, xpp->flags);\n> -\t\t\tif (XDL_ALLOC_GROW(recs, nrec + 1, narec))\n> +\t\t\tGALLOC_GROW(recs, nrec + 1, narec);\n> +\t\t\tif (!recs)\n>   \t\t\t\tgoto abort;\n>   \t\t\tif (!(crec = xdl_cha_alloc(&xdf->rcha)))\n>   \t\t\t\tgoto abort;\n> diff --git a/xdiff/xutils.c b/xdiff/xutils.c\n> index a6f10353cff..c0cd5338c4e 100644\n> --- a/xdiff/xutils.c\n> +++ b/xdiff/xutils.c\n> @@ -436,20 +436,3 @@ int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n>   \n>   \treturn status;\n>   }\n> -\n> -void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size)\n> -{\n> -\tvoid *tmp = NULL;\n> -\tsize_t n = ((LONG_MAX - 16) / 2 >= *alloc) ? 2 * *alloc + 16 : LONG_MAX;\n> -\tif (nr > n)\n> -\t\tn = nr;\n> -\tif (SIZE_MAX / size >= n)\n> -\t\ttmp = xdl_realloc(p, n * size);\n> -\tif (tmp) {\n> -\t\t*alloc = n;\n> -\t} else {\n> -\t\txdl_free(p);\n> -\t\t*alloc = 0;\n> -\t}\n> -\treturn tmp;\n> -}\n> diff --git a/xdiff/xutils.h b/xdiff/xutils.h\n> index fd0bba94e8b..7ae3f897bef 100644\n> --- a/xdiff/xutils.h\n> +++ b/xdiff/xutils.h\n> @@ -42,7 +42,5 @@ int xdl_emit_hunk_hdr(long s1, long c1, long s2, long c2,\n>   int xdl_fall_back_diff(xdfenv_t *diff_env, xpparam_t const *xpp,\n>   \t\t       int line1, int count1, int line2, int count2);\n>   \n> -/* Do not call this function, use XDL_ALLOC_GROW instead */\n> -void* xdl_alloc_grow_helper(void* p, long nr, long* alloc, size_t size);\n>   \n>   #endif /* #if !defined(XUTILS_H) */\n\n"},{"id":"458767","messageId":"220711.865yk47x54.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"dcde61a3-4d96-6cd5-f05e-45781599f8bd@gmail.com","subject":"Re: [PATCH 7/7] xdiff: remove xdl_free(), use free() instead","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-11T10:02:39Z","receivedAt":"2022-07-11T11:16:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 11 2022, Phillip Wood wrote:\n\n> Hi Ævar\n>\n> On 08/07/2022 22:26, Ævar Arnfjörð Bjarmason wrote:\n>> On Fri, Jul 08 2022, Phillip Wood wrote:\n>> \n>>> Hi Ævar\n>>>\n>>> On 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n>>>> Remove the xdl_free() wrapper macro in favor of using free()\n>>>> directly. The wrapper macro was brought in with the initial import of\n>>>> xdiff/ in 3443546f6ef (Use a *real* built-in diff generator,\n>>>> 2006-03-24).\n>>>> As subsequent discussions on the topic[1] have made clear there's no\n>>>> reason to use this wrapper.\n>>>\n>>> That link is to a message where you assert that there is no reason for\n>>> the wrapper, you seem to be the only person in that thread making that\n>>> argument. The libgit2 maintainer and others are arguing that it is\n>>> worth keeping to make it easy to swap the allocator.\n>> Arguing that it's not needed for a technical reason, but an\n>> \"aesthetic\n>> and ergonomic one\", per:\n>> https://lore.kernel.org/git/20220217225804.GC7@edef91d97c94/;\n>> Perhaps I misread it, but I took this as Junio concurring with what\n>> I\n>> was pointing out there:\n>> https://lore.kernel.org/git/xmqqfsohbdre.fsf@gitster.g/\n>> \n>>>> Both git itself as well as any external\n>>>> users such as libgit2 compile the xdiff/* code as part of their own\n>>>> compilation, and can thus find the right malloc() and free() at\n>>>> link-time.\n>>>\n>>> I'm struggling to see how that is simpler than the current situation\n>>> with xdl_malloc().\n>> It's simpler because when maintaining that code in git.git it's less\n>> of\n>> a special case, e.g. we have coccinelle rules that match free(), now\n>> every such rule needs to account for the xdiff special-case.\n>> And it's less buggy because if you're relying on us always using a\n>> wrapper bugs will creep in, current master has this:\n>> \t\n>> \t$ git -P grep '\\bfree\\(' -- xdiff\n>> \txdiff/xdiff.h:#define xdl_free(ptr) free(ptr)\n>> \txdiff/xmerge.c:         free(c);\n>> \txdiff/xmerge.c: free(next_m);\n>> \n>>> Perhaps you could show how you think libgit2 could\n>>> easily make xdiff use git__malloc() instead of malloc() without\n>>> changing the malloc() calls (the message you linked to essentially\n>>> suggests they do a search and replace). If people cannot swap in\n>>> another allocator without changing the code then you are imposing a\n>>> barrier to them contributing patches back to git's xdiff.\n>> I was suggesting that if libgit2 wanted to maintain a copy of xdiff\n>> that\n>> catered to the asthetic desires of the maintainer adjusting whatever\n>> import script you use to apply a trivial coccinelle transformation would\n>> give you what you wanted. Except without a maintenance burden on\n>> git.git, or the bugs you'd get now since you're not catching those two\n>> free() cases (or any future such cases).\n>> But just having the code use malloc() and free() directly and\n>> getting\n>> you what you get now is easy, and doesn't require any such\n>> search-replacement.\n>> Here's how:\n>> I imported the xdiff/*.[ch] files at the tip of my branch into\n>> current\n>> libgit2.git's src/libgit2/xdiff/, and then applied this on top, which is\n>> partially re-applying libgit2's own monkeypatches, and partially\n>> adjusting for upstream changes (including this one):\n>> \t\n>> \tdiff --git a/src/libgit2/xdiff/git-xdiff.h b/src/libgit2/xdiff/git-xdiff.h\n>> \tindex b75dba819..2e00764d4 100644\n>> \t--- a/src/libgit2/xdiff/git-xdiff.h\n>> \t+++ b/src/libgit2/xdiff/git-xdiff.h\n>> \t@@ -14,6 +14,7 @@\n>> \t #ifndef INCLUDE_git_xdiff_h__\n>> \t #define INCLUDE_git_xdiff_h__\n>> \t\n>> \t+#include <regex.h>\n>> \t #include \"regexp.h\"\n>> \t\n>> \t /* Work around C90-conformance issues */\n>> \t@@ -27,11 +28,10 @@\n>> \t # endif\n>> \t #endif\n>> \t\n>> \t-#define xdl_malloc(x) git__malloc(x)\n>> \t-#define xdl_free(ptr) git__free(ptr)\n>> \t-#define xdl_realloc(ptr, x) git__realloc(ptr, x)\n>> \t+#define malloc(x) git__malloc(x)\n>> \t+#define free(ptr) git__free(ptr)\n>> \t\n>> \t-#define XDL_BUG(msg) GIT_ASSERT(msg)\n>> \t+#define BUG(msg) GIT_ASSERT(msg)\n>> \t\n>> \t #define xdl_regex_t git_regexp\n>> \t #define xdl_regmatch_t git_regmatch\n>> \t@@ -50,4 +50,17 @@ GIT_INLINE(int) xdl_regexec_buf(\n>> \t \treturn -1;\n>> \t }\n>> \t\n>> \t+static inline size_t st_mult(size_t a, size_t b)\n>> \t+{\n>> \t+\treturn a * b;\n>> \t+}\n>> \t+\n>> \t+static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,\n>> \t+\t\t\t      size_t nmatch, regmatch_t pmatch[], int eflags)\n>> \t+{\n>> \t+\tassert(nmatch > 0 && pmatch);\n>> \t+\tpmatch[0].rm_so = 0;\n>> \t+\tpmatch[0].rm_eo = size;\n>> \t+\treturn regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);\n>> \t+}\n>> \t #endif\n>> \tdiff --git a/src/libgit2/xdiff/xdiff.h b/src/libgit2/xdiff/xdiff.h\n>> \tindex a37d89fcd..5e5b525c2 100644\n>> \t--- a/src/libgit2/xdiff/xdiff.h\n>> \t+++ b/src/libgit2/xdiff/xdiff.h\n>> \t@@ -23,6 +23,8 @@\n>> \t #if !defined(XDIFF_H)\n>> \t #define XDIFF_H\n>> \t\n>> \t+#include \"git-xdiff.h\"\n>> \t+\n>> \t #ifdef __cplusplus\n>> \t extern \"C\" {\n>> \t #endif /* #ifdef __cplusplus */\n>> \tdiff --git a/src/libgit2/xdiff/xinclude.h b/src/libgit2/xdiff/xinclude.h\n>> \tindex a4285ac0e..79812ad8a 100644\n>> \t--- a/src/libgit2/xdiff/xinclude.h\n>> \t+++ b/src/libgit2/xdiff/xinclude.h\n>> \t@@ -23,7 +23,8 @@\n>> \t #if !defined(XINCLUDE_H)\n>> \t #define XINCLUDE_H\n>> \t\n>> \t-#include \"git-compat-util.h\"\n>> \t+#include \"git-xdiff.h\"\n>> \t+#include \"git-shared-util.h\"\n>> \t #include \"xmacros.h\"\n>> \t #include \"xdiff.h\"\n>> \t #include \"xtypes.h\"\n>> If you then build it and run e.g.:\n>> \tgdb -args ./libgit2_tests -smerge::files\n>> You'll get stack traces like this if you break on stdalloc__malloc\n>> (which it uses for me in its default configuration):\n>> \t\n>> \t(gdb) bt\n>> \t#0  stdalloc__malloc (len=144, file=0x87478d \"/home/avar/g/libgit2/src/libgit2/xdiff/xutils.c\", line=101) at /home/avar/g/libgit2/src/util/allocators/stdalloc.c:14\n>> \t#1  0x00000000006ec15c in xdl_cha_alloc (cha=0x7fffffffcaa8) at /home/avar/g/libgit2/src/libgit2/xdiff/xutils.c:101\n>> \t#2  0x00000000006eaee9 in xdl_prepare_ctx (pass=1, mf=0x7fffffffcc98, narec=13, xpp=0x7fffffffcca8, cf=0x7fffffffc7d0, xdf=0x7fffffffcaa8)\n>> \t    at /home/avar/g/libgit2/src/libgit2/xdiff/xprepare.c:194\n>> IOW this was all seamlessly routed to your git__malloc() without us\n>> having to maintain an xdl_{malloc,free}().\n>\n> Thanks for showing this, it is really helpful to see a concrete\n> example. I was especially interested to see how you went about the\n> conversion without redefining standard library functions or\n> introducing non-namespaced identifiers in files that included\n> xdiff.h. Unfortunately you have not done that and so I don't think the\n> approach above is viable   for a well behaved library.\n\nWhy? Because there's some header where doing e.g.:\n\n\t#include \"git2/something.h\"\n\nWill directly include git-xdiff.h by proxy into the user's program?\n\nI admit I didn't check that, and assumed that these were only included\nby other *.c files in libgit2 itself. I.e. it would compile xdiff for\nits own use, but then expose its own API.\n\nAnyway, if it is directly included in some user-exposed *.h file then\nyes, you don't want to overwrite their \"malloc\", but that's a small\nmatter of doing an \"#undef malloc\" at the end (which as the below\nlinked-to patch shows we'd support by having macros like\nSHA1DC_CUSTOM_TRAILING_INCLUDE_UBC_CHECK_C).\n\nAside from what I'm proposing here doing such \"undef at the end\" seems\nlike a good idea in any case, because if there is any macro leakage here\nyou're e.g. re-defining \"XDL_BUG\" and other things that aren't clearly\nin the libgit2 namespace now, no?\n\n>> Now, I think that's obviously worth adjusting, e.g. the series I've got\n>> here could make this easier for libgit2 by including st_mult() at least,\n>> I'm not sure what you want to do about regexec_buf().\n>> For the monkeypatching you do now of creating a \"git-xdiff.h\" I\n>> think it\n>> would be very good to get a patch like what I managed to get\n>> sha1collisiondetection.git to accept for our own use-case, which allows\n>> us to use their upstream code as-is from a submodule:\n>> \thttps://github.com/cr-marcstevens/sha1collisiondetection/commit/b45fcef\n>\n> Thanks for the link, That's a really good example where all the\n> identifiers are namespaced and the public include file does not\n> pollute the code that is including it. If you come up with something\n> like that where the client code can set up same #defines for malloc,\n> calloc, realloc and free that would be a good way forward.\n\nI don't need to come up with that, you've got the linker to do that.\n\nI.e. for almost any \"normal\" use of xdiff you'd simply compile it with\nits reference to malloc(), and then you either link that library that\nalready links to libc into your program.\n\nSo if you use a custom malloc the xdiff code might still use libc\nmalloc, or you'd link the two together and link the resulting program\nwith your own custom malloc.\n\nAs explained in the previous & linked-to threads this is how almost\neveryone would include & use such a library, and indeed that's how git\nitself can use malloc() and free() in its sources, but have options to\nlink to libc malloc, nedmalloc etc.\n\nBut instead of using the linker to resolve \"malloc\" to git__malloc you'd\nlike to effectively include the upstream *.[ch] files directly, and\npretend as though the upstream source used git__malloc() or whatever to\nbegin with.\n\nI don't really understand why you can't just do that at the linker\nlevel, especially as doing so guards you better against namespace\npollution. But whatever, the suggestion(s) above assume you've got a\ngood reason, but show that we don't need to prefix every standard\nfunction just to accommodate that.\n\nInstead we can just provide a knob to include a header of yours after\nall others have been included (which the libgit2 monkeypatches to xdiff\nalready include), and have that header define \"malloc\" as \"git__malloc\",\n\"BUG\" as \"GIT_ASSERT\" etc. \n\nAnd yes, if you're in turn providing an interface where others will then\ninclude your header that's doing that you'll need to apply some\nnamespacing paranoia, namely to \"#undef malloc\" etc.\n\nBut none of that requires git to carry prefixed versions of standard\nfunctions you'd wish to replace, which the patches here show.\n\n> I do not think we should require projects to be defining there own\n> versions of [C]ALLOC_*() and BUG() just to use xdiff.\n\nNo, I don't think so either. I.e. the idea with these patches is that we\ncould bundle up xdiff/* and git-shared-util.h into one distributed\nlibgit, which down the road we could even roll independent releases for\n(as upstream seems to have forever gone away).\n\nWhereas the proposals coming out of libgit2[1] for generalizing xdiff/\nfor general use seem to stop just short of the specific hacks we need to\nget it running for libgit2, but e.g. leave \"xdl_malloc\" defined as\n\"xmalloc\".\n\nThat isn't a standard library function, and therefore the first thing\nyou'd need to do when using it as a library is to find out how git.git\nuses that, and copy/paste it or define your own replacement.\n\nWhereas I think it should be the other way around, we should e.g. ship a\nshimmy BUG() that calls fprintf(stderr, ...) and abort(), but for\nadvanced users such as libgit2 guard those with \"ifndef\" or whatever, so\nyou can have it call GIT_ASSERT() and the like instead.\n\n1. https://lore.kernel.org/git/20220209013354.GB7@abe733c6e288/\n"},{"id":"458768","messageId":"220711.861qur9ays.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"c2bc5c79-e71b-f9dc-ba97-261d3454e150@gmail.com","subject":"Re: [PATCH 5/7] xdiff: use GALLOC_GROW(), not XDL_ALLOC_GROW()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-11T10:48:47Z","receivedAt":"2022-07-11T11:24:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 11 2022, Phillip Wood wrote:\n\n> Hi Ævar\n>\n> On 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n>> Replace the recently introduced XDL_ALLOC_GROW() with invocations of\n>> the GALLOC_GROW() from git-shared-util.h.\n>> As this change shows the macro + function indirection of\n>> XDL_ALLOC_GROW() is something we needed only because the two callsites\n>> we used it in wanted to use it as an expression, and we thus had to\n>> pass the \"sizeof\" down.\n>> Let's just check the value afterwards instead, which allows us to\n>> use\n>> the shared macro, we can also remove xdl_reallo(), this was its last\n>> user.\n>\n> I don't think this expression->statement change is an\n> improvement.\n\nI think the use-as-statement is prettier too, but I think the uglyness\nof having to pass down the sizeof() & re-implementing the macro version\nof the alloc-or-die variant outweights that.\n\n> This change also removes the overflow checks that are\n> present in XDL_ALLOC_GROW()[...]\n\nWe end up calling st_mult(), which does that overflow check. Do you mean\nthat the POC shimmy layer I showed in another reply for libgit2 doesn't\nhave an st_mult() that detects overflows?\n\nThat's true, but as noted downthread of that we can & could ship that as\npart of the shimmy layer, but that's unrelated to this change.\n\nIn your pre-image you use LONG_MAX instead of UINTMAX_MAX & I don't see\n(but maybe I haven't looked at it carefully enough) how it does the same\ndying on overflows. Doesn't it just fall back to LONG_MAX?\n\nPart of this is that it's not clear to me from your commit(s) why you\nneed to rewrite alloc_nr() and rewrite (or drop?) st_mult().\n\n> and fails to free the old allocation when\n> realloc() fails. It is not a like for like replacement.\n\nYes, we should have a free() there. Wel spotted. But again, doing that\nas part of the \"gently\" branch seems preferrable to have duplicate\nversions for expression (non-fatal) v.s. statement (fatal) variants.\n"},{"id":"458860","messageId":"Ys0gerokoiLC9llA@coredump.intra.peff.net","threadId":"58081","inReplyTo":"bb1c85e4-50be-04a5-3c9f-f4c4dab6a50e@gmail.com","subject":"Re: [PATCH v2 4/4] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-07-12T07:19:22Z","receivedAt":"2022-07-12T07:19:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 11, 2022 at 11:00:06AM +0100, Phillip Wood wrote:\n\n> > But you've also changed every single callsite anyway.\n> > \n> > So why not just change:\n> > \n> >      if (XDL_ALLOC_GROW(recs, ...))\n> > \n> > To:\n> > \n> >      XDL_ALLOC_GROW(recs, ...);\n> >      if (!recs)\n> \n> Because I think it's ugly, we'd never define a function as void(int* result,\n> args...) and I don't think we should do that for a macro where we can avoid\n> it. The existing ALLOC_* macros make sense as statements because they die on\n> failure.\n\nThose macros are already unlike functions, though. They take a variable\n_name_, not a pointer to a variable, and assign to it. A version which\nlooked like a function would have to be:\n\n  if (XDL_ALLOC_GROW(&recs, ...))\n\nSo in my head I read them as assignment statements already, making the\nexpression-like form a bit jarring.\n\nJust my two cents. I don't care _too_ much either way, but I'd probably\nbe swayed if one implementation is much simpler than the other.\n\n-Peff\n"},{"id":"458943","messageId":"e86cac3b-4e05-be20-41d8-ed5006463556@gmail.com","threadId":"58081","inReplyTo":"220711.861qur9ays.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 5/7] xdiff: use GALLOC_GROW(), not XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-13T09:09:33Z","receivedAt":"2022-07-13T09:09:46Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 11/07/2022 11:48, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Jul 11 2022, Phillip Wood wrote:\n> \n>> Hi Ævar\n>>\n>> On 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n>>> Replace the recently introduced XDL_ALLOC_GROW() with invocations of\n>>> the GALLOC_GROW() from git-shared-util.h.\n>>> As this change shows the macro + function indirection of\n>>> XDL_ALLOC_GROW() is something we needed only because the two callsites\n>>> we used it in wanted to use it as an expression, and we thus had to\n>>> pass the \"sizeof\" down.\n>>> Let's just check the value afterwards instead, which allows us to\n>>> use\n>>> the shared macro, we can also remove xdl_reallo(), this was its last\n>>> user.\n>>\n>> I don't think this expression->statement change is an\n>> improvement.\n> \n> I think the use-as-statement is prettier too, but I think the uglyness\n> of having to pass down the sizeof() & re-implementing the macro version\n> of the alloc-or-die variant outweights that.\n\nI think this is partly a choice between prioritizing ease of \nimplementation or ease of use for callers.\n\n>> This change also removes the overflow checks that are\n>> present in XDL_ALLOC_GROW()[...]\n> \n> We end up calling st_mult(), which does that overflow check. Do you mean\n> that the POC shimmy layer I showed in another reply for libgit2 doesn't\n> have an st_mult() that detects overflows?\n\nI was referring to\n\n#define alloc_nr(x) (((x)+16)*3/2)\n\nin cache.h. XDL_ALLOC_GROW() detects overflows when growing the number \nof items as well as when calculating the number of bytes to allocate.\n\n> That's true, but as noted downthread of that we can & could ship that as\n> part of the shimmy layer, but that's unrelated to this change.\n> \n> In your pre-image you use LONG_MAX instead of UINTMAX_MAX & I don't see\n> (but maybe I haven't looked at it carefully enough) how it does the same\n> dying on overflows. Doesn't it just fall back to LONG_MAX?\n\nIt does not die on overflow as we want to return errors rather than die \nin the xdiff code. It uses long to match the existing code.\n\n> Part of this is that it's not clear to me from your commit(s) why you\n> need to rewrite alloc_nr() and rewrite (or drop?) st_mult().\n\nSo that we don't die on overflow and so that the xdiff code is self \ncontained.\n\nI'm a bit disappointed that this patch seems to have been written \nwithout really taking the time to understand exactly what the code it is \nreplacing is doing.\n\nBest Wishes\n\nPhillip\n\n>> and fails to free the old allocation when\n>> realloc() fails. It is not a like for like replacement.\n> \n> Yes, we should have a free() there. Wel spotted. But again, doing that\n> as part of the \"gently\" branch seems preferrable to have duplicate\n> versions for expression (non-fatal) v.s. statement (fatal) variants.\n"},{"id":"458944","messageId":"46f70aff-59f1-01ca-8f1f-b07b230011bc@gmail.com","threadId":"58081","inReplyTo":"Ys0gerokoiLC9llA@coredump.intra.peff.net","subject":"Re: [PATCH v2 4/4] xdiff: introduce XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-13T09:38:13Z","receivedAt":"2022-07-13T09:38:20Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Peff\n\nIt's great to see you back on the list\n\nOn 12/07/2022 08:19, Jeff King wrote:\n> On Mon, Jul 11, 2022 at 11:00:06AM +0100, Phillip Wood wrote:\n> \n>>> But you've also changed every single callsite anyway.\n>>>\n>>> So why not just change:\n>>>\n>>>       if (XDL_ALLOC_GROW(recs, ...))\n>>>\n>>> To:\n>>>\n>>>       XDL_ALLOC_GROW(recs, ...);\n>>>       if (!recs)\n>>\n>> Because I think it's ugly, we'd never define a function as void(int* result,\n>> args...) and I don't think we should do that for a macro where we can avoid\n>> it. The existing ALLOC_* macros make sense as statements because they die on\n>> failure.\n> \n> Those macros are already unlike functions, though. They take a variable\n> _name_, not a pointer to a variable, and assign to it. A version which\n> looked like a function would have to be:\n> \n>    if (XDL_ALLOC_GROW(&recs, ...))\n> \n> So in my head I read them as assignment statements already, making the\n> expression-like form a bit jarring.\n\nI see what you're saying, I agree it is not exactly analogous. I tend to \nthink of assignment as an expression because it returns the value that's \nbeing assigned, though here it's a bit muddy because the assignment is \nhidden inside the macro and there is no tell-tale '&' as there would be \nif it were calling function.\n\n> Just my two cents. I don't care _too_ much either way, but I'd probably\n> be swayed if one implementation is much simpler than the other.\n\nI don't think there is much difference in the complexity in this case. \nIn general I think that using a function with a macro wrapper can make \nthe implementation more readable as the function it is not littered with \nthe extra parentheses and backslashes that the same code in a macro \nrequires.\n\nBest Wishes\n\nPhillip\n"},{"id":"458945","messageId":"220713.86edyp46ld.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"e86cac3b-4e05-be20-41d8-ed5006463556@gmail.com","subject":"Re: [PATCH 5/7] xdiff: use GALLOC_GROW(), not XDL_ALLOC_GROW()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-13T10:48:22Z","receivedAt":"2022-07-13T11:04:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jul 13 2022, Phillip Wood wrote:\n\n> Hi Ævar\n>\n> On 11/07/2022 11:48, Ævar Arnfjörð Bjarmason wrote:\n>> On Mon, Jul 11 2022, Phillip Wood wrote:\n>> \n>>> Hi Ævar\n>>>\n>>> On 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n>>>> Replace the recently introduced XDL_ALLOC_GROW() with invocations of\n>>>> the GALLOC_GROW() from git-shared-util.h.\n>>>> As this change shows the macro + function indirection of\n>>>> XDL_ALLOC_GROW() is something we needed only because the two callsites\n>>>> we used it in wanted to use it as an expression, and we thus had to\n>>>> pass the \"sizeof\" down.\n>>>> Let's just check the value afterwards instead, which allows us to\n>>>> use\n>>>> the shared macro, we can also remove xdl_reallo(), this was its last\n>>>> user.\n>>>\n>>> I don't think this expression->statement change is an\n>>> improvement.\n>> I think the use-as-statement is prettier too, but I think the\n>> uglyness\n>> of having to pass down the sizeof() & re-implementing the macro version\n>> of the alloc-or-die variant outweights that.\n>\n> I think this is partly a choice between prioritizing ease of\n> implementation or ease of use for callers.\n>\n>>> This change also removes the overflow checks that are\n>>> present in XDL_ALLOC_GROW()[...]\n>> We end up calling st_mult(), which does that overflow check. Do you\n>> mean\n>> that the POC shimmy layer I showed in another reply for libgit2 doesn't\n>> have an st_mult() that detects overflows?\n>\n> I was referring to\n>\n> #define alloc_nr(x) (((x)+16)*3/2)\n>\n> in cache.h. XDL_ALLOC_GROW() detects overflows when growing the number\n> of items as well as when calculating the number of bytes to allocate.\n>\n>> That's true, but as noted downthread of that we can & could ship that as\n>> part of the shimmy layer, but that's unrelated to this change.\n>> In your pre-image you use LONG_MAX instead of UINTMAX_MAX & I don't\n>> see\n>> (but maybe I haven't looked at it carefully enough) how it does the same\n>> dying on overflows. Doesn't it just fall back to LONG_MAX?\n>\n> It does not die on overflow as we want to return errors rather than\n> die in the xdiff code. It uses long to match the existing code.\n>\n>> Part of this is that it's not clear to me from your commit(s) why you\n>> need to rewrite alloc_nr() and rewrite (or drop?) st_mult().\n>\n> So that we don't die on overflow and so that the xdiff code is self\n> contained.\n>\n> I'm a bit disappointed that this patch seems to have been written\n> without really taking the time to understand exactly what the code it\n> is replacing is doing.\n\nWell, there's a lot to understand :) So it's also an implicit comment on\nthe complexity of your series.\n\nIn case it wasn't clear the main thrust of what I've been commenting on\nhere is asking why what you have here needs to *structurally* look the\nway it does, i.e. why you think the malloc() & free() naming can't\nresolve to libgit2's wrappers (per the thread ending at [1]).\n\nAnd, if we can't end up with an xdiff/* in our tree that doesn't have\nreturn value checking flying in the face of xmalloc's NORETURN()\nbehavior on allocation failures.\n\nBut yes, the suggested replacement isn't behaving exactly as yours does,\nbut I think that's just an implementation detail as far as the stuctural\nquestions above go. I.e.:\n\n * You're trying to fix a long-standing overflow issue in alloc_nr(),\n   but in such a way that only leaves xdiff/* with the fix.\n\n   Can't we address that seperately (e.g. turning alloc_nr into a static\n   inline function using the st_* helper), and then make xdiff and\n   cache.h use that new shared helper?\n\n * You seem to be set on the idea that you absolutely must be rewriting\n   large parts of the existing allocation macro, because you'd really\n   like to use it as an expression v.s. a statement.\n\n   I really disagree with that trade-off, i.e. the whole endeavor of\n   duplicating the implementation in ways that are mostly the same but\n   not quite (e.g. that alloc_grow case) doesn't seem worth it\n   v.s. sharing the behavior.\n\n   But likewise it seems to be an implementation detail of your series\n   that we can't have both, i.e. if you're set on using these as an\n   expression factoring the shared behavior in cache.h out into some\n   static inline functions, then calling those from both variants.\n\n1. https://lore.kernel.org/git/220711.865yk47x54.gmgdl@evledraar.gmail.com/\n"},{"id":"458952","messageId":"bbfa1340-9cab-35d7-2245-f6f8369d5cd4@gmail.com","threadId":"58081","inReplyTo":"220711.865yk47x54.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 7/7] xdiff: remove xdl_free(), use free() instead","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-13T13:00:07Z","receivedAt":"2022-07-13T13:00:14Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 11/07/2022 11:02, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Jul 11 2022, Phillip Wood wrote:\n> \n>> Hi Ævar\n>>\n>> On 08/07/2022 22:26, Ævar Arnfjörð Bjarmason wrote:\n>>> On Fri, Jul 08 2022, Phillip Wood wrote:\n>>>\n>>>> Hi Ævar\n>>>>\n>>>> On 08/07/2022 15:20, Ævar Arnfjörð Bjarmason wrote:\n>>>>> Remove the xdl_free() wrapper macro in favor of using free()\n>>>>> directly. The wrapper macro was brought in with the initial import of\n>>>>> xdiff/ in 3443546f6ef (Use a *real* built-in diff generator,\n>>>>> 2006-03-24).\n>>>>> As subsequent discussions on the topic[1] have made clear there's no\n>>>>> reason to use this wrapper.\n>>>>\n>>>> That link is to a message where you assert that there is no reason for\n>>>> the wrapper, you seem to be the only person in that thread making that\n>>>> argument. The libgit2 maintainer and others are arguing that it is\n>>>> worth keeping to make it easy to swap the allocator.\n>>> Arguing that it's not needed for a technical reason, but an\n>>> \"aesthetic\n>>> and ergonomic one\", per:\n>>> https://lore.kernel.org/git/20220217225804.GC7@edef91d97c94/;\n>>> Perhaps I misread it, but I took this as Junio concurring with what\n>>> I\n>>> was pointing out there:\n>>> https://lore.kernel.org/git/xmqqfsohbdre.fsf@gitster.g/\n>>>\n>>>>> Both git itself as well as any external\n>>>>> users such as libgit2 compile the xdiff/* code as part of their own\n>>>>> compilation, and can thus find the right malloc() and free() at\n>>>>> link-time.\n>>>>\n>>>> I'm struggling to see how that is simpler than the current situation\n>>>> with xdl_malloc().\n>>> It's simpler because when maintaining that code in git.git it's less\n>>> of\n>>> a special case, e.g. we have coccinelle rules that match free(), now\n>>> every such rule needs to account for the xdiff special-case.\n>>> And it's less buggy because if you're relying on us always using a\n>>> wrapper bugs will creep in, current master has this:\n>>> \t\n>>> \t$ git -P grep '\\bfree\\(' -- xdiff\n>>> \txdiff/xdiff.h:#define xdl_free(ptr) free(ptr)\n>>> \txdiff/xmerge.c:         free(c);\n>>> \txdiff/xmerge.c: free(next_m);\n>>>\n>>>> Perhaps you could show how you think libgit2 could\n>>>> easily make xdiff use git__malloc() instead of malloc() without\n>>>> changing the malloc() calls (the message you linked to essentially\n>>>> suggests they do a search and replace). If people cannot swap in\n>>>> another allocator without changing the code then you are imposing a\n>>>> barrier to them contributing patches back to git's xdiff.\n>>> I was suggesting that if libgit2 wanted to maintain a copy of xdiff\n>>> that\n>>> catered to the asthetic desires of the maintainer adjusting whatever\n>>> import script you use to apply a trivial coccinelle transformation would\n>>> give you what you wanted. Except without a maintenance burden on\n>>> git.git, or the bugs you'd get now since you're not catching those two\n>>> free() cases (or any future such cases).\n>>> But just having the code use malloc() and free() directly and\n>>> getting\n>>> you what you get now is easy, and doesn't require any such\n>>> search-replacement.\n>>> Here's how:\n>>> I imported the xdiff/*.[ch] files at the tip of my branch into\n>>> current\n>>> libgit2.git's src/libgit2/xdiff/, and then applied this on top, which is\n>>> partially re-applying libgit2's own monkeypatches, and partially\n>>> adjusting for upstream changes (including this one):\n>>> \t\n>>> \tdiff --git a/src/libgit2/xdiff/git-xdiff.h b/src/libgit2/xdiff/git-xdiff.h\n>>> \tindex b75dba819..2e00764d4 100644\n>>> \t--- a/src/libgit2/xdiff/git-xdiff.h\n>>> \t+++ b/src/libgit2/xdiff/git-xdiff.h\n>>> \t@@ -14,6 +14,7 @@\n>>> \t #ifndef INCLUDE_git_xdiff_h__\n>>> \t #define INCLUDE_git_xdiff_h__\n>>> \t\n>>> \t+#include <regex.h>\n>>> \t #include \"regexp.h\"\n>>> \t\n>>> \t /* Work around C90-conformance issues */\n>>> \t@@ -27,11 +28,10 @@\n>>> \t # endif\n>>> \t #endif\n>>> \t\n>>> \t-#define xdl_malloc(x) git__malloc(x)\n>>> \t-#define xdl_free(ptr) git__free(ptr)\n>>> \t-#define xdl_realloc(ptr, x) git__realloc(ptr, x)\n>>> \t+#define malloc(x) git__malloc(x)\n>>> \t+#define free(ptr) git__free(ptr)\n>>> \t\n>>> \t-#define XDL_BUG(msg) GIT_ASSERT(msg)\n>>> \t+#define BUG(msg) GIT_ASSERT(msg)\n>>> \t\n>>> \t #define xdl_regex_t git_regexp\n>>> \t #define xdl_regmatch_t git_regmatch\n>>> \t@@ -50,4 +50,17 @@ GIT_INLINE(int) xdl_regexec_buf(\n>>> \t \treturn -1;\n>>> \t }\n>>> \t\n>>> \t+static inline size_t st_mult(size_t a, size_t b)\n>>> \t+{\n>>> \t+\treturn a * b;\n>>> \t+}\n>>> \t+\n>>> \t+static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,\n>>> \t+\t\t\t      size_t nmatch, regmatch_t pmatch[], int eflags)\n>>> \t+{\n>>> \t+\tassert(nmatch > 0 && pmatch);\n>>> \t+\tpmatch[0].rm_so = 0;\n>>> \t+\tpmatch[0].rm_eo = size;\n>>> \t+\treturn regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);\n>>> \t+}\n>>> \t #endif\n>>> \tdiff --git a/src/libgit2/xdiff/xdiff.h b/src/libgit2/xdiff/xdiff.h\n>>> \tindex a37d89fcd..5e5b525c2 100644\n>>> \t--- a/src/libgit2/xdiff/xdiff.h\n>>> \t+++ b/src/libgit2/xdiff/xdiff.h\n>>> \t@@ -23,6 +23,8 @@\n>>> \t #if !defined(XDIFF_H)\n>>> \t #define XDIFF_H\n>>> \t\n>>> \t+#include \"git-xdiff.h\"\n>>> \t+\n>>> \t #ifdef __cplusplus\n>>> \t extern \"C\" {\n>>> \t #endif /* #ifdef __cplusplus */\n>>> \tdiff --git a/src/libgit2/xdiff/xinclude.h b/src/libgit2/xdiff/xinclude.h\n>>> \tindex a4285ac0e..79812ad8a 100644\n>>> \t--- a/src/libgit2/xdiff/xinclude.h\n>>> \t+++ b/src/libgit2/xdiff/xinclude.h\n>>> \t@@ -23,7 +23,8 @@\n>>> \t #if !defined(XINCLUDE_H)\n>>> \t #define XINCLUDE_H\n>>> \t\n>>> \t-#include \"git-compat-util.h\"\n>>> \t+#include \"git-xdiff.h\"\n>>> \t+#include \"git-shared-util.h\"\n>>> \t #include \"xmacros.h\"\n>>> \t #include \"xdiff.h\"\n>>> \t #include \"xtypes.h\"\n>>> If you then build it and run e.g.:\n>>> \tgdb -args ./libgit2_tests -smerge::files\n>>> You'll get stack traces like this if you break on stdalloc__malloc\n>>> (which it uses for me in its default configuration):\n>>> \t\n>>> \t(gdb) bt\n>>> \t#0  stdalloc__malloc (len=144, file=0x87478d \"/home/avar/g/libgit2/src/libgit2/xdiff/xutils.c\", line=101) at /home/avar/g/libgit2/src/util/allocators/stdalloc.c:14\n>>> \t#1  0x00000000006ec15c in xdl_cha_alloc (cha=0x7fffffffcaa8) at /home/avar/g/libgit2/src/libgit2/xdiff/xutils.c:101\n>>> \t#2  0x00000000006eaee9 in xdl_prepare_ctx (pass=1, mf=0x7fffffffcc98, narec=13, xpp=0x7fffffffcca8, cf=0x7fffffffc7d0, xdf=0x7fffffffcaa8)\n>>> \t    at /home/avar/g/libgit2/src/libgit2/xdiff/xprepare.c:194\n>>> IOW this was all seamlessly routed to your git__malloc() without us\n>>> having to maintain an xdl_{malloc,free}().\n>>\n>> Thanks for showing this, it is really helpful to see a concrete\n>> example. I was especially interested to see how you went about the\n>> conversion without redefining standard library functions or\n>> introducing non-namespaced identifiers in files that included\n>> xdiff.h. Unfortunately you have not done that and so I don't think the\n>> approach above is viable   for a well behaved library.\n> \n> Why? Because there's some header where doing e.g.:\n> \n> \t#include \"git2/something.h\"\n> \n> Will directly include git-xdiff.h by proxy into the user's program?\n\nNo because you're redefining malloc() and introducing ALLOC_GROW() etc \nin \nsrc/libgit2/{Blame_git.c,Diff_xdiff.c,Merge_file.c,Patch_generate.c,Checkout.c} \nand\nTest/libgit2/merge/files.c. It is not expected that including xdiff.h \nwill change a bunch of symbols in the file it is included in.\n\nBest Wishes\n\nPhillip\n\n> \n> I admit I didn't check that, and assumed that these were only included\n> by other *.c files in libgit2 itself. I.e. it would compile xdiff for\n> its own use, but then expose its own API.\n> \n> Anyway, if it is directly included in some user-exposed *.h file then\n> yes, you don't want to overwrite their \"malloc\", but that's a small\n> matter of doing an \"#undef malloc\" at the end (which as the below\n> linked-to patch shows we'd support by having macros like\n> SHA1DC_CUSTOM_TRAILING_INCLUDE_UBC_CHECK_C).\n> \n> Aside from what I'm proposing here doing such \"undef at the end\" seems\n> like a good idea in any case, because if there is any macro leakage here\n> you're e.g. re-defining \"XDL_BUG\" and other things that aren't clearly\n> in the libgit2 namespace now, no?\n> \n>>> Now, I think that's obviously worth adjusting, e.g. the series I've got\n>>> here could make this easier for libgit2 by including st_mult() at least,\n>>> I'm not sure what you want to do about regexec_buf().\n>>> For the monkeypatching you do now of creating a \"git-xdiff.h\" I\n>>> think it\n>>> would be very good to get a patch like what I managed to get\n>>> sha1collisiondetection.git to accept for our own use-case, which allows\n>>> us to use their upstream code as-is from a submodule:\n>>> \thttps://github.com/cr-marcstevens/sha1collisiondetection/commit/b45fcef\n>>\n>> Thanks for the link, That's a really good example where all the\n>> identifiers are namespaced and the public include file does not\n>> pollute the code that is including it. If you come up with something\n>> like that where the client code can set up same #defines for malloc,\n>> calloc, realloc and free that would be a good way forward.\n> \n> I don't need to come up with that, you've got the linker to do that.\n> \n> I.e. for almost any \"normal\" use of xdiff you'd simply compile it with\n> its reference to malloc(), and then you either link that library that\n> already links to libc into your program.\n> \n> So if you use a custom malloc the xdiff code might still use libc\n> malloc, or you'd link the two together and link the resulting program\n> with your own custom malloc.\n> \n> As explained in the previous & linked-to threads this is how almost\n> everyone would include & use such a library, and indeed that's how git\n> itself can use malloc() and free() in its sources, but have options to\n> link to libc malloc, nedmalloc etc.\n> \n> But instead of using the linker to resolve \"malloc\" to git__malloc you'd\n> like to effectively include the upstream *.[ch] files directly, and\n> pretend as though the upstream source used git__malloc() or whatever to\n> begin with.\n> \n> I don't really understand why you can't just do that at the linker\n> level, especially as doing so guards you better against namespace\n> pollution. But whatever, the suggestion(s) above assume you've got a\n> good reason, but show that we don't need to prefix every standard\n> function just to accommodate that.\n> \n> Instead we can just provide a knob to include a header of yours after\n> all others have been included (which the libgit2 monkeypatches to xdiff\n> already include), and have that header define \"malloc\" as \"git__malloc\",\n> \"BUG\" as \"GIT_ASSERT\" etc.\n> \n> And yes, if you're in turn providing an interface where others will then\n> include your header that's doing that you'll need to apply some\n> namespacing paranoia, namely to \"#undef malloc\" etc.\n> \n> But none of that requires git to carry prefixed versions of standard\n> functions you'd wish to replace, which the patches here show.\n> \n>> I do not think we should require projects to be defining there own\n>> versions of [C]ALLOC_*() and BUG() just to use xdiff.\n> \n> No, I don't think so either. I.e. the idea with these patches is that we\n> could bundle up xdiff/* and git-shared-util.h into one distributed\n> libgit, which down the road we could even roll independent releases for\n> (as upstream seems to have forever gone away).\n> \n> Whereas the proposals coming out of libgit2[1] for generalizing xdiff/\n> for general use seem to stop just short of the specific hacks we need to\n> get it running for libgit2, but e.g. leave \"xdl_malloc\" defined as\n> \"xmalloc\".\n> \n> That isn't a standard library function, and therefore the first thing\n> you'd need to do when using it as a library is to find out how git.git\n> uses that, and copy/paste it or define your own replacement.\n> \n> Whereas I think it should be the other way around, we should e.g. ship a\n> shimmy BUG() that calls fprintf(stderr, ...) and abort(), but for\n> advanced users such as libgit2 guard those with \"ifndef\" or whatever, so\n> you can have it call GIT_ASSERT() and the like instead.\n> \n> 1. https://lore.kernel.org/git/20220209013354.GB7@abe733c6e288/\n\n"},{"id":"458977","messageId":"24148494-72ab-23f8-58b5-c8371c44d167@gmail.com","threadId":"58081","inReplyTo":"220713.86edyp46ld.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 5/7] xdiff: use GALLOC_GROW(), not XDL_ALLOC_GROW()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-13T13:21:34Z","receivedAt":"2022-07-13T13:21:39Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 13/07/2022 11:48, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Jul 13 2022, Phillip Wood wrote:\n>>\n>> I'm a bit disappointed that this patch seems to have been written\n>> without really taking the time to understand exactly what the code it\n>> is replacing is doing.\n> \n> Well, there's a lot to understand :) So it's also an implicit comment on\n> the complexity of your series.\n\nI'm surprised you think it is complex, the implementation of \nXDL_ALLOC_GROW() and its helper is pretty short.\n\n> In case it wasn't clear the main thrust of what I've been commenting on\n> here is asking why what you have here needs to *structurally* look the\n> way it does, i.e. why you think the malloc() & free() naming can't\n> resolve to libgit2's wrappers (per the thread ending at [1]).\n\nI think the different structural approach stems in part from the subtle \ndifferences between ALLOC_GROW() and XDL_ALLOC_GROW(). Once one \nappreciates that the latter needs to free the original allocation on \noverflow and allocation failure and work with code that uses long rather \nthan size_t then there is not much code left to be shared between them.\n\n> And, if we can't end up with an xdiff/* in our tree that doesn't have\n> return value checking flying in the face of xmalloc's NORETURN()\n> behavior on allocation failures.\n\nI don't think xmalloc() is marked as NORETURN in wrapper.h so the \ncompiler would need somehow determine that from looking at the \nimplementation in wrapper.c but even if it is using LTO I'm not sure \nit'll have that information when creating xdiff/lib.a\n\n> But yes, the suggested replacement isn't behaving exactly as yours does,\n> but I think that's just an implementation detail as far as the stuctural\n> questions above go. I.e.:\n> \n>   * You're trying to fix a long-standing overflow issue in alloc_nr(),\n>     but in such a way that only leaves xdiff/* with the fix.\n\nI didn't set out to fix that issue per-se, I only realized it existed \nwhen I reviewed this series.\n\n>     Can't we address that seperately (e.g. turning alloc_nr into a static\n>     inline function using the st_* helper), and then make xdiff and\n>     cache.h use that new shared helper?\n\nAs I've said before xdiff does not want to die on overflow so it cannot \nuse st_mult(). Also you cannot assume that you're dealing with size_t \nwhen you do the overflow check in alloc_nr() so I think it is a tricky \nproblem to solve.\n\n>   * You seem to be set on the idea that you absolutely must be rewriting\n>     large parts of the existing allocation macro, because you'd really\n>     like to use it as an expression v.s. a statement.\n\nIt is not just that - there are plenty of differences that stem from \nreturning an error rather that dying that reduce the amount of common \ncode. Having a separate definition was also driven by a desire to keep \nthe xdiff code self contained.\n\nThis will likely be the last message from me in this thread for a while \n- I'm going offline later next week and want to make sure I have time to \nreview Stolee's rebase patches before then.\n\nBest Wishes\n\nPhillip\n"},{"id":"458978","messageId":"220713.861qup403n.gmgdl@evledraar.gmail.com","threadId":"58081","inReplyTo":"bbfa1340-9cab-35d7-2245-f6f8369d5cd4@gmail.com","subject":"Re: [PATCH 7/7] xdiff: remove xdl_free(), use free() instead","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-13T13:18:29Z","receivedAt":"2022-07-13T13:24:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jul 13 2022, Phillip Wood wrote:\n\n> On 11/07/2022 11:02, Ævar Arnfjörð Bjarmason wrote:\n>> On Mon, Jul 11 2022, Phillip Wood wrote:\n> [...]\n>>> Thanks for showing this, it is really helpful to see a concrete\n>>> example. I was especially interested to see how you went about the\n>>> conversion without redefining standard library functions or\n>>> introducing non-namespaced identifiers in files that included\n>>> xdiff.h. Unfortunately you have not done that and so I don't think the\n>>> approach above is viable   for a well behaved library.\n>> Why? Because there's some header where doing e.g.:\n>> \t#include \"git2/something.h\"\n>> Will directly include git-xdiff.h by proxy into the user's program?\n>\n> No because you're redefining malloc() and introducing ALLOC_GROW() etc\n> in \n> src/libgit2/{Blame_git.c,Diff_xdiff.c,Merge_file.c,Patch_generate.c,Checkout.c}\n> and\n> Test/libgit2/merge/files.c. It is not expected that including xdiff.h\n> will change a bunch of symbols in the file it is included in.\n\n...which is why if you read to the sha1collisiondetection commit below\nyou'd follow that up with including a header at the end of xdiff.h which\nwould do:\n\n\t#undef malloc\n\netc., so you wouldn't leak that macro beyond the code that needs it, and\nyou wouldn't leak the xdl_* macros either, which are purely needed\ninternally in that code. So even aside from my suggestion of doing it\nthis way it seems to me the structure has macro hygene issues worth\nfixing, see e.g. how we have refs-internal.h v.s. refs.h in git.git for\nsimilar reasons.\n\n>> I admit I didn't check that, and assumed that these were only\n>> included\n>> by other *.c files in libgit2 itself. I.e. it would compile xdiff for\n>> its own use, but then expose its own API.\n>> Anyway, if it is directly included in some user-exposed *.h file\n>> then\n>> yes, you don't want to overwrite their \"malloc\", but that's a small\n>> matter of doing an \"#undef malloc\" at the end (which as the below\n>> linked-to patch shows we'd support by having macros like\n>> SHA1DC_CUSTOM_TRAILING_INCLUDE_UBC_CHECK_C).\n>> Aside from what I'm proposing here doing such \"undef at the end\"\n>> seems\n>> like a good idea in any case, because if there is any macro leakage here\n>> you're e.g. re-defining \"XDL_BUG\" and other things that aren't clearly\n>> in the libgit2 namespace now, no?\n>> \n>>>> Now, I think that's obviously worth adjusting, e.g. the series I've got\n>>>> here could make this easier for libgit2 by including st_mult() at least,\n>>>> I'm not sure what you want to do about regexec_buf().\n>>>> For the monkeypatching you do now of creating a \"git-xdiff.h\" I\n>>>> think it\n>>>> would be very good to get a patch like what I managed to get\n>>>> sha1collisiondetection.git to accept for our own use-case, which allows\n>>>> us to use their upstream code as-is from a submodule:\n>>>> \thttps://github.com/cr-marcstevens/sha1collisiondetection/commit/b45fcef\n>>>\n>>> Thanks for the link, That's a really good example where all the\n>>> identifiers are namespaced and the public include file does not\n>>> pollute the code that is including it. If you come up with something\n>>> like that where the client code can set up same #defines for malloc,\n>>> calloc, realloc and free that would be a good way forward.\n>> I don't need to come up with that, you've got the linker to do that.\n>> I.e. for almost any \"normal\" use of xdiff you'd simply compile it\n>> with\n>> its reference to malloc(), and then you either link that library that\n>> already links to libc into your program.\n>> So if you use a custom malloc the xdiff code might still use libc\n>> malloc, or you'd link the two together and link the resulting program\n>> with your own custom malloc.\n>> As explained in the previous & linked-to threads this is how almost\n>> everyone would include & use such a library, and indeed that's how git\n>> itself can use malloc() and free() in its sources, but have options to\n>> link to libc malloc, nedmalloc etc.\n>> But instead of using the linker to resolve \"malloc\" to git__malloc\n>> you'd\n>> like to effectively include the upstream *.[ch] files directly, and\n>> pretend as though the upstream source used git__malloc() or whatever to\n>> begin with.\n>> I don't really understand why you can't just do that at the linker\n>> level, especially as doing so guards you better against namespace\n>> pollution. But whatever, the suggestion(s) above assume you've got a\n>> good reason, but show that we don't need to prefix every standard\n>> function just to accommodate that.\n>> Instead we can just provide a knob to include a header of yours\n>> after\n>> all others have been included (which the libgit2 monkeypatches to xdiff\n>> already include), and have that header define \"malloc\" as \"git__malloc\",\n>> \"BUG\" as \"GIT_ASSERT\" etc.\n>> And yes, if you're in turn providing an interface where others will\n>> then\n>> include your header that's doing that you'll need to apply some\n>> namespacing paranoia, namely to \"#undef malloc\" etc.\n>> But none of that requires git to carry prefixed versions of standard\n>> functions you'd wish to replace, which the patches here show.\n>> \n>>> I do not think we should require projects to be defining there own\n>>> versions of [C]ALLOC_*() and BUG() just to use xdiff.\n>> No, I don't think so either. I.e. the idea with these patches is\n>> that we\n>> could bundle up xdiff/* and git-shared-util.h into one distributed\n>> libgit, which down the road we could even roll independent releases for\n>> (as upstream seems to have forever gone away).\n>> Whereas the proposals coming out of libgit2[1] for generalizing\n>> xdiff/\n>> for general use seem to stop just short of the specific hacks we need to\n>> get it running for libgit2, but e.g. leave \"xdl_malloc\" defined as\n>> \"xmalloc\".\n>> That isn't a standard library function, and therefore the first\n>> thing\n>> you'd need to do when using it as a library is to find out how git.git\n>> uses that, and copy/paste it or define your own replacement.\n>> Whereas I think it should be the other way around, we should\n>> e.g. ship a\n>> shimmy BUG() that calls fprintf(stderr, ...) and abort(), but for\n>> advanced users such as libgit2 guard those with \"ifndef\" or whatever, so\n>> you can have it call GIT_ASSERT() and the like instead.\n>> 1. https://lore.kernel.org/git/20220209013354.GB7@abe733c6e288/\n\n^ I.e. this.\n"}]}