Re: Feature Request: Option to make "git rev-list --objects" output duplicate objects

From: Jeff King <peff@peff.net>
Date: Mar 28, 2023, 18:32 UTC
Message-ID: <20230328183253.GD18558@coredump.intra.peff.net>
In-Reply-To: <AS1PR02MB8185DF947EBC583318481E1994889@AS1PR02MB8185.eurprd02.prod.outlook.com>

On Tue, Mar 28, 2023 at 08:08:02AM +0000, Baumann, Moritz wrote:

Show 21 quoted lines

> > But if you want
> > to know all of the names touched in a set of commits, I have used
> > something like this before:
> >
> >   git rev-list $new --not --all |
> >   git diff-tree --stdin --format= -r -c --name-only
> 
> Thanks, that looks promising and solves at least one of my use cases. The only
> minor problem is that there seems to be no way to pipe the diff-tree output to
> cat-file without massaging it with awk first.
> 
> I have three uses cases in my pre-receive hooks:
> 
> 1. Filters solely based on the file name
>    ? your suggestions works perfectly here
> 2. Filters based only on file contents
>    ? git rev-list --objects + git cat-file provide everything I need
> 3. One filter based on file size and name (forbid large files, with exceptions)
>    ? I'm guessing "git rev-list | git diff-tree --stdin | awk |
>      git cat-file --batch-check" is the best solution to extract the necessary
>      information from git in this case?

Yes, that's how I would do all of those. Having to massage the output between diff-tree and cat-file is a little annoying, but at least can still be done in O(1) processes. And you really need some language capable of parsing cat-file output anyway, so it's not too big of a lift.

One thing we did at GitHub is teach index-pack to collect a list of too-big paths (since naturally it knows the size of every blob as it indexes it), and write them out to a special path. Then our pre-receive hook could quickly check the list and act on it (warning above a certain size, rejecting above another), without having to traverse again.

I haven't sent those patches upstream, though, for a few reasons:

  - we only hooked index-pack, not unpack-objects (we only use the
    former, not the latter, but in stock Git you might see either)

  - getting just the object names is kind of awkward. You have to then
    invoke rev-list to find the names of hits (though at least you only
    do so when there is a problem object; the happy case remains fast).

  - it's actually not that helpful to avoid the traversal if you have
    other stuff you want to check anyway (like file contents, or names).
    It was one of those things we optimized a long time ago, and I kind
    of doubt is doing much good these days.

-Peff