git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/6] object-file: extract logic to approximate object count

From
Patrick Steinhardt <ps@pks.im>
Date
Mar 11, 2026, 13:58 UTC
Message-ID
<abF084pB38G5Nyv6@pks.im>
In-Reply-To
<87v7f2lei6.fsf@iotcl.com>
On Wed, Mar 11, 2026 at 01:47:13PM +0100, Toon Claes wrote:
Show 9 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > In "builtin/gc.c" we have some logic that checks whether we need to
> > repack objects. This is done by counting the number of objects that we
> > have and checking whether it exceeds a certain threshold. We don't
> > really need an accurate object count though, which is why we only
> > open a single object diretcroy shard and then extrapolate from there.
> 
> s/diretcroy/directory/
Thanks, fixed locally.
Show 45 quoted lines
> > diff --git a/object-file.c b/object-file.c
> > index a3ff7f586c..da67e3c9ff 100644
> > --- a/object-file.c
> > +++ b/object-file.c
> > @@ -1868,6 +1868,47 @@ int odb_source_loose_for_each_object(struct odb_source *source,
> >  					     NULL, NULL, &data);
> >  }
> >  
> > +int odb_source_loose_approximate_object_count(struct odb_source *source,
> > +					      unsigned long *out)
> > +{
> > +	const unsigned hexsz = source->odb->repo->hash_algo->hexsz - 2;
> > +	unsigned long count = 0;
> > +	struct dirent *ent;
> > +	char *path = NULL;
> > +	DIR *dir = NULL;
> > +	int ret;
> > +
> > +	path = xstrfmt("%s/17", source->path);
> > +
> > +	dir = opendir(path);
> > +	if (!dir) {
> > +		if (errno == ENOENT) {
> > +			*out = 0;
> > +			ret = 0;
> > +			goto out;
> > +		}
> > +
> > +		ret = error_errno("cannot open object shard '%s'", path);
> > +		goto out;
> > +	}
> > +
> > +	while ((ent = readdir(dir)) != NULL) {
> > +		if (strspn(ent->d_name, "0123456789abcdef") != hexsz ||
> > +		    ent->d_name[hexsz] != '\0')
> > +			continue;
> > +		count++;
> > +	}
> > +
> > +	*out = count * 256;
> 
> This makes the number way larger, but I don't think we need to worry
> getting anywhere near ULONG_MAX, because I would expect to have Git
> coming to a grind way before that happens (not to mention filesystems
> would get unhappy about it too).

Yup. Even if `unsigned long` was 32 bits that would be >128 million loose objects in a single directory. I agree that this is probably going to make some things in Git unhappy. So we could have overflow checks here, but I'm not sure it's worth it.

Thanks!
Patrick
Previous: Toon ClaesNext: Patrick Steinhardt
Message 9 of 27 in “odb: introduce generic object counting”
  1. 0/6 odb: introduce generic object countingPatrick Steinhardt, Mar 10, 2026
  2. 1/6 odb: stop including "odb/source.h"Patrick Steinhardt, Mar 10, 2026
  3. 2/6 packfile: extract logic to count number of objectsPatrick Steinhardt, Mar 10, 2026
  4. Toon ClaesMar 11, 2026
  5. Patrick SteinhardtMar 11, 2026
  6. 3/6 object-file: extract logic to approximate object countPatrick Steinhardt, Mar 10, 2026
  7. Junio C HamanoMar 10, 2026
  8. Toon ClaesMar 11, 2026
  9. Patrick SteinhardtMar 11, 2026
  10. 4/6 object-file: generalize counting objectsPatrick Steinhardt, Mar 10, 2026
  11. Toon ClaesMar 11, 2026
  12. Patrick SteinhardtMar 11, 2026
  13. 5/6 odb/source: introduce generic object countingPatrick Steinhardt, Mar 10, 2026
  14. Junio C HamanoMar 10, 2026
  15. Patrick SteinhardtMar 11, 2026
  16. Toon ClaesMar 11, 2026
  17. 6/6 odb: introduce generic object countingPatrick Steinhardt, Mar 10, 2026
  18. Toon ClaesMar 11, 2026
  19. Patrick SteinhardtMar 12, 2026
  20. 0/6 odb: introduce generic object countingPatrick Steinhardt, Mar 12, 2026
  21. 1/6 odb: stop including "odb/source.h"Patrick Steinhardt, Mar 12, 2026
  22. 2/6 packfile: extract logic to count number of objectsPatrick Steinhardt, Mar 12, 2026
  23. 3/6 object-file: extract logic to approximate object countPatrick Steinhardt, Mar 12, 2026
  24. 4/6 object-file: generalize counting objectsPatrick Steinhardt, Mar 12, 2026
  25. 5/6 odb/source: introduce generic object countingPatrick Steinhardt, Mar 12, 2026
  26. 6/6 odb: introduce generic object countingPatrick Steinhardt, Mar 12, 2026
  27. Toon ClaesMar 13, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.