threads / discuss / 55032

RE: git archive setting user and group

Subject: RE: git archive setting user and group

## tl;dr

12 messages between Jan 22, 2021 and Jan 23, 2021.

replies: 11people: 7as markdown or json

Jason Pyeron· Jan 22, 2021, 20:40 UTC · lore
Show 6 quoted lines
> From: Jason Pyeron <jpyeron@pdinc.us>
> Sent: Friday, January 22, 2021 3:09 PM
> 
> I am about to make a release for logwatch tonight. Historically the files are owned by logwatch in the
> tgz file. When I use git archive it is owned by uid 0, is there an option to set the uid/uname,
> gid/gname owner of the files?
Answer: not at this time, as it is hard coded in the source.
archive-tar.c:
static void prepare_header(struct archiver_args *args,
                           struct ustar_header *header,
                           unsigned int mode, unsigned long size)
{
        xsnprintf(header->mode, sizeof(header->mode), "%07o", mode & 07777);
        xsnprintf(header->size, sizeof(header->size), "%011"PRIoMAX , S_ISREG(mode) ? (uintmax_t)size : (uintmax_t)0);
        xsnprintf(header->mtime, sizeof(header->mtime), "%011lo", (unsigned long) args->time);
        xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
        xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
        strlcpy(header->uname, "root", sizeof(header->uname));
        strlcpy(header->gname, "root", sizeof(header->gname));
meh.
René Scharfe· Jan 22, 2021, 21:00 UTC · re: Jason Pyeron · lore

Re: git archive setting user and group

Am 22.01.21 um 21:40 schrieb Jason Pyeron:
Show 26 quoted lines
>> From: Jason Pyeron <jpyeron@pdinc.us>
>> Sent: Friday, January 22, 2021 3:09 PM
>>
>> I am about to make a release for logwatch tonight. Historically the files are owned by logwatch in the
>> tgz file. When I use git archive it is owned by uid 0, is there an option to set the uid/uname,
>> gid/gname owner of the files?
>
> Answer: not at this time, as it is hard coded in the source.
>
> archive-tar.c:
> static void prepare_header(struct archiver_args *args,
>                            struct ustar_header *header,
>                            unsigned int mode, unsigned long size)
> {
>         xsnprintf(header->mode, sizeof(header->mode), "%07o", mode & 07777);
>         xsnprintf(header->size, sizeof(header->size), "%011"PRIoMAX , S_ISREG(mode) ? (uintmax_t)size : (uintmax_t)0);
>         xsnprintf(header->mtime, sizeof(header->mtime), "%011lo", (unsigned long) args->time);
>
>         xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
>         xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
>         strlcpy(header->uname, "root", sizeof(header->uname));
>         strlcpy(header->gname, "root", sizeof(header->gname));
>
>
> meh.
>

Adding support for using a custom user and group should be easy. Is this just a cosmetic thing? Regular users would ignore the user info in the archive, and root should not be used for extracting, and on systems that don't have a logwatch user this wouldn't make a difference anyway, right?

René
Jason Pyeron· Jan 22, 2021, 21:13 UTC · re: René Scharfe · lore
Show 35 quoted lines
> From: René Scharfe 
> Sent: Friday, January 22, 2021 4:00 PM
> 
> Am 22.01.21 um 21:40 schrieb Jason Pyeron:
> >> From: Jason Pyeron
> >> Sent: Friday, January 22, 2021 3:09 PM
> >>
> >> I am about to make a release for logwatch tonight. Historically the files are owned by logwatch in
> the
> >> tgz file. When I use git archive it is owned by uid 0, is there an option to set the uid/uname,
> >> gid/gname owner of the files?
> >
> > Answer: not at this time, as it is hard coded in the source.
> >
> > archive-tar.c:
> > static void prepare_header(struct archiver_args *args,
> >                            struct ustar_header *header,
> >                            unsigned int mode, unsigned long size)
> > {
> >         xsnprintf(header->mode, sizeof(header->mode), "%07o", mode & 07777);
> >         xsnprintf(header->size, sizeof(header->size), "%011"PRIoMAX , S_ISREG(mode) ?
> (uintmax_t)size : (uintmax_t)0);
> >         xsnprintf(header->mtime, sizeof(header->mtime), "%011lo", (unsigned long) args->time);
> >
> >         xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
> >         xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
> >         strlcpy(header->uname, "root", sizeof(header->uname));
> >         strlcpy(header->gname, "root", sizeof(header->gname));
> >
> >
> > meh.
> >
> 
> Adding support for using a custom user and group should be easy.  Is
> this just a cosmetic thing?  Regular users would ignore the user info in
In this case, likely a cosmetic thing. 
> the archive, and root should not be used for extracting, and on systems
But I can think of situations, where it would matter - and I do not agree that "root should not be used for extraction".
> that don't have a logwatch user this wouldn't make a difference anyway,
> right?
I updated the hard coded values to match what was needed, did a make and presto all was happy.
The backend should take:

--group=NAME --group-id=GID --owner=NAME --owner-id=UID

Patch wanted?
Konstantin Ryabitsev· Jan 22, 2021, 21:39 UTC · re: René Scharfe · lore

Re: git archive setting user and group

On Fri, Jan 22, 2021 at 10:00:04PM +0100, René Scharfe wrote:
Show 5 quoted lines
> Adding support for using a custom user and group should be easy.  Is
> this just a cosmetic thing?  Regular users would ignore the user info in
> the archive, and root should not be used for extracting, and on systems
> that don't have a logwatch user this wouldn't make a difference anyway,
> right?

Right now, "git archive" operations are bit-for-bit identical across all versions going back at least 8+ years. In fact, we've been relying on this to support bundling tarball signatures with git tags themselves (via git notes). E.g. you can see this in action here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9

If you click on "(sig)", you will download a signature that can be used to verify the tarball generated using "git archive".

I would argue that adding user/group support to "git archive" operation is not really solving any problems other than "it's different from when I run it as a regular user" -- and can introduce potential compatibility problems if implemented.

So, I would selfishly vote not to implement this.
-K
Jason Pyeron· Jan 22, 2021, 22:02 UTC · re: Konstantin Ryabitsev · lore
Show 23 quoted lines
> From: Konstantin Ryabitsev
> Sent: Friday, January 22, 2021 4:40 PM
> Subject: Re: git archive setting user and group
> 
> On Fri, Jan 22, 2021 at 10:00:04PM +0100, René Scharfe wrote:
> > Adding support for using a custom user and group should be easy.  Is
> > this just a cosmetic thing?  Regular users would ignore the user info in
> > the archive, and root should not be used for extracting, and on systems
> > that don't have a logwatch user this wouldn't make a difference anyway,
> > right?
> 
> Right now, "git archive" operations are bit-for-bit identical across all
> versions going back at least 8+ years. In fact, we've been relying on this to
> support bundling tarball signatures with git tags themselves (via git notes).
> E.g. you can see this in action here:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
> 
> If you click on "(sig)", you will download a signature that can be used to
> verify the tarball generated using "git archive".
> 
> I would argue that adding user/group support to "git archive" operation is
> not really solving any problems other than "it's different from when I run it
> as a regular user" -- and can introduce potential compatibility problems if
Being pedantic here, it is different than when I run it as any user - including root.
Don’t confuse tar x with tar c.
tar c captures the current owner of the files, or allows override with a single user / map file.
Show 5 quoted lines
> implemented.
> 
> So, I would selfishly vote not to implement this.
> 
> -K
Ævar Arnfjörð Bjarmason· Jan 22, 2021, 22:28 UTC · re: Konstantin Ryabitsev · lore

Re: git archive setting user and group

On Fri, Jan 22 2021, Konstantin Ryabitsev wrote:
Show 22 quoted lines
> On Fri, Jan 22, 2021 at 10:00:04PM +0100, René Scharfe wrote:
>> Adding support for using a custom user and group should be easy.  Is
>> this just a cosmetic thing?  Regular users would ignore the user info in
>> the archive, and root should not be used for extracting, and on systems
>> that don't have a logwatch user this wouldn't make a difference anyway,
>> right?
>
> Right now, "git archive" operations are bit-for-bit identical across all
> versions going back at least 8+ years. In fact, we've been relying on this to
> support bundling tarball signatures with git tags themselves (via git notes).
> E.g. you can see this in action here:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
>
> If you click on "(sig)", you will download a signature that can be used to
> verify the tarball generated using "git archive".
>
> I would argue that adding user/group support to "git archive" operation is
> not really solving any problems other than "it's different from when I run it
> as a regular user" -- and can introduce potential compatibility problems if
> implemented.
>
> So, I would selfishly vote not to implement this.

It seems "logwatch" has the same situation, except their backwards compatibility is with non-"git archive" tool that used user != root.

If it solves a problem for some users and someone comes up with a patch I don't see why it shouldn't be implemented, but I think it's important that "root" is the default unless configured or the relevant option is invoked.

Or do you mean that the kernel.org use-case is that users are expected to run "git archive" on their own machines and upload the result to kernel.org, and that kernel.org relies on two such files bit-for-bit identical?

brian m. carlson· Jan 23, 2021, 01:05 UTC · re: Konstantin Ryabitsev · lore

Re: git archive setting user and group

On 2021-01-22 at 21:39:54, Konstantin Ryabitsev wrote:
Show 15 quoted lines
> On Fri, Jan 22, 2021 at 10:00:04PM +0100, René Scharfe wrote:
> > Adding support for using a custom user and group should be easy.  Is
> > this just a cosmetic thing?  Regular users would ignore the user info in
> > the archive, and root should not be used for extracting, and on systems
> > that don't have a logwatch user this wouldn't make a difference anyway,
> > right?
> 
> Right now, "git archive" operations are bit-for-bit identical across all
> versions going back at least 8+ years. In fact, we've been relying on this to
> support bundling tarball signatures with git tags themselves (via git notes).
> E.g. you can see this in action here:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
> 
> If you click on "(sig)", you will download a signature that can be used to
> verify the tarball generated using "git archive".

Please do not rely on this behavior. I want to state in the strongest possible terms that this is not guaranteed behavior and it may change at any time. We have explicitly said so on the list multiple times. If you need reproducible archives, you need to add a tool to canonicalize them in a suitable format and not rely on Git to never change things.

If you are relying on this behavior right now, I urge you to change that at your earliest possible convenience. I don't want to break kernel.org's infrastructure again, but I'm also not going to tiptoe around sending patches in fear of that, nor feel bad if it happens again for this reason.

> I would argue that adding user/group support to "git archive" operation is
> not really solving any problems other than "it's different from when I run it
> as a regular user" -- and can introduce potential compatibility problems if
> implemented.

I agree that this feature isn't really something we want. Git produces tar archives for software interchange, in which case producing an intentionally anonymous tarball is the desired behavior.

-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US
Jeff King· Jan 23, 2021, 04:58 UTC · re: brian m. carlson · lore

Re: git archive setting user and group

On Sat, Jan 23, 2021 at 01:05:00AM +0000, brian m. carlson wrote:
Show 14 quoted lines
> > Right now, "git archive" operations are bit-for-bit identical across all
> > versions going back at least 8+ years. In fact, we've been relying on this to
> > support bundling tarball signatures with git tags themselves (via git notes).
> > E.g. you can see this in action here:
> > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
> > 
> > If you click on "(sig)", you will download a signature that can be used to
> > verify the tarball generated using "git archive".
> 
> Please do not rely on this behavior.  I want to state in the strongest
> possible terms that this is not guaranteed behavior and it may change at
> any time.  We have explicitly said so on the list multiple times.  If
> you need reproducible archives, you need to add a tool to canonicalize
> them in a suitable format and not rely on Git to never change things.
I strongly second this. :)

It's also not quite true that things have remained bit-for-bit identical for all that time. We have fixed bugs in that time, although they do not always cause a change in every output tarball (they often depend on corner cases like having long pathnames). Two off the top of my head (that have indeed caused people to complain about changing checksums):

  - 22f0dcd963 (archive-tar: split long paths more carefully,
    2013-01-05)
  - 82a46af13e (archive-tar: fix pax extended header length calculation,
    2019-08-17)

We also rely on system gzip. That's pretty stable, but I have heard tell that even `gzip -n` may differ on platforms.

Another fun one I saw recently: using export-subst with $Format:%h$ will produce different results depending on how many objects are present in the repository running git-archive.

-Peff
Konstantin Ryabitsev· Jan 23, 2021, 05:16 UTC · re: Jeff King · lore

Re: git archive setting user and group

On Fri, Jan 22, 2021 at 11:58:19PM -0500, Jeff King wrote:
> We also rely on system gzip. That's pretty stable, but I have heard tell
> that even `gzip -n` may differ on platforms.

The signatures are made against uncompressed .tar output, so this is not a consideration.

> Another fun one I saw recently: using export-subst with $Format:%h$ will
> produce different results depending on how many objects are present in
> the repository running git-archive.

As long as the output is the same with thhe flags we specified in the comment, we're still okay. E.g.:

    -----BEGIN PGP SIGNATURE-----
    Comment: This signature is for the .tar version of the archive
    Comment: git archive --format tar --prefix=linux-5.10.9/ v5.10.9
    Comment: git version 2.30.0

If running "git archive --format tar --prefix=linux-5.10.9/ v5.10.9" becomes non-deterministic within the same version of git, *then* I'm in trouble. It's been remarkably stable within the past 8 years, so I don't expect there's a dramatic reason why "--format tar" output would need to change -- it's not like the tar spec is much of a moving target.

-K
Konstantin Ryabitsev· Jan 23, 2021, 05:11 UTC · re: brian m. carlson · lore

Re: git archive setting user and group

On Sat, Jan 23, 2021 at 01:05:00AM +0000, brian m. carlson wrote:
Show 14 quoted lines
> > Right now, "git archive" operations are bit-for-bit identical across all
> > versions going back at least 8+ years. In fact, we've been relying on this to
> > support bundling tarball signatures with git tags themselves (via git notes).
> > E.g. you can see this in action here:
> > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
> > 
> > If you click on "(sig)", you will download a signature that can be used to
> > verify the tarball generated using "git archive".
> 
> Please do not rely on this behavior.  I want to state in the strongest
> possible terms that this is not guaranteed behavior and it may change at
> any time.  We have explicitly said so on the list multiple times.  If
> you need reproducible archives, you need to add a tool to canonicalize
> them in a suitable format and not rely on Git to never change things.

It doesn't need to be perpetually the same. This is the reason the comments mention git version -- "to generate an archive that matches this signature, run this command (with this git version)." I know there is no guarantee that this won't ever change across versions, but I don't see why we can't expect git-archive to return the same bit-for-bit output when using the same version of git. Adding --owner and --group would potentially change that, especially if that's something settable via git-config.

Show 5 quoted lines
> If you are relying on this behavior right now, I urge you to change that
> at your earliest possible convenience.  I don't want to break
> kernel.org's infrastructure again, but I'm also not going to tiptoe
> around sending patches in fear of that, nor feel bad if it happens again
> for this reason.

It won't break, it will just inconvenience folks who are using this perk -- Greg KH will just have to use the same git version that we have on the server when he generates the signatures. This has happened before.

-K
Junio C Hamano· Jan 22, 2021, 22:29 UTC · re: René Scharfe · lore

Re: git archive setting user and group

René Scharfe <l.s.r@web.de> writes:
Show 12 quoted lines
> Am 22.01.21 um 21:40 schrieb Jason Pyeron:
> ...
>>         xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
>>         xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
>>         strlcpy(header->uname, "root", sizeof(header->uname));
>>         strlcpy(header->gname, "root", sizeof(header->gname));
>
> Adding support for using a custom user and group should be easy.  Is
> this just a cosmetic thing?  Regular users would ignore the user info in
> the archive, and root should not be used for extracting, and on systems
> that don't have a logwatch user this wouldn't make a difference anyway,
> right?

I am not particularly interested in cosmetics, but it probably is OK to make uname/gname overridable. I do not see any point in uid/gid numeric values overridable, though. Just like user names and group names do not name the same user and group on every machine, uid/gid are even less so.

Jason Pyeron· Jan 22, 2021, 22:51 UTC · re: Junio C Hamano · lore
Show 21 quoted lines
> From: Junio C Hamano
> Sent: Friday, January 22, 2021 5:29 PM
> 
> René Scharfe <l.s.r@web.de> writes:
> 
> > Am 22.01.21 um 21:40 schrieb Jason Pyeron:
> > ...
> >>         xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
> >>         xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
> >>         strlcpy(header->uname, "root", sizeof(header->uname));
> >>         strlcpy(header->gname, "root", sizeof(header->gname));
> >
> > Adding support for using a custom user and group should be easy.  Is
> > this just a cosmetic thing?  Regular users would ignore the user info in
> > the archive, and root should not be used for extracting, and on systems
> > that don't have a logwatch user this wouldn't make a difference anyway,
> > right?
> 
> I am not particularly interested in cosmetics, but it probably is OK
> to make uname/gname overridable.  I do not see any point in uid/gid
> numeric values overridable, though.  Just like user names and group
Interesting. So logwatch(0) vs logwatch(1100) ? I guess it is as good as any value, but tar never uses 0.
On a system without the logwatch user/group tar behaves as follows:
$ tar czf /tmp/logwatch.tgz --owner=logwatch --group=logwatch .

$ tar tvzf /tmp/logwatch.tgz | head -n 1 drwxrwxr-x logwatch/logwatch 0 2020-03-14 15:01 ./

$ tar tvzf /tmp/logwatch.tgz --numeric-owner | head -n 1 drwxrwxr-x 1049681/1049088 0 2020-03-14 15:01 ./

$ id uid=1049681(myuser) gid=1049088(mygroup) groups=no one cares...

but when the user / group is known...
$ tar czf /tmp/logwatch.tgz --owner=Guest --group=Users .

$ tar tvzf /tmp/logwatch.tgz | head -n 1 drwxrwxr-x Guest/Users 0 2020-03-14 15:01 ./

$ tar tvzf /tmp/logwatch.tgz --numeric-owner | head -n 1 drwxrwxr-x 1049077/545 0 2020-03-14 15:01 ./

$ getent passwd Guest | cut -f 3 -d : 1049077

$ getent group Users | cut -f 3 -d : 545

> names do not name the same user and group on every machine, uid/gid
> are even less so.
> 
>

← back to recent threads