In the file cache-tree.c, the function "static struct cache_tree *read_one(const char **buffer, unsigned long *size_p)" has the following code (link:https://github.com/git/git/blob/9d530dc0024503ab4218fe6c4395b8a0aa245478/cache-tree.c#L592-#L593):
int subtree_nr; subtree_nr = strtol(cp, &ep, 10);
it->subtree_alloc = subtree_nr + 2; CALLOC_ARRAY(it->down, it->subtree_alloc);
The variable subtree_nr is converted from the string buffer. Suppose it equals INT_MAX, the computation of it->subtree_alloc can trigger a signed integer overflow, which is later used to calculate allocation size.
This can cause security problems if the function do_read_index is given a crafted index file.