threads / patch / 27048

patchsupply '-n' to gzip to produce identical tarballs

Subject: [PATCH] supply '-n' to gzip to produce identical tarballs

## tl;dr

6 messages between Apr 10, 2011 and Apr 11, 2011. Diffs are folded; open one to read it.

replies: 5people: 3as markdown or json

Fraser Tweedale· Apr 10, 2011, 06:12 UTC · lore

Without the '-n' ('--no-name') argument, gzip includes timestamp in output which results in different files. Important systems like FreeBSD ports and perhaps many others hash/checksum downloaded files to ensure integrity. For projects that do not release official archives, gitweb's snapshot feature would be an excellent stand-in but for the fact that the files it produces are not identical.

Supply '-n' to gzip to exclude timestamp from output and produce idential output every time.

Signed-off-by: Fraser Tweedale <frase@frase.id.au>
---
 gitweb/gitweb.perl |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

Show changes to gitweb/gitweb.perl +1 −1

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 46186ab..2ab08da 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -186,7 +186,7 @@ our %known_snapshot_formats = (
 		'type' => 'application/x-gzip',
 		'suffix' => '.tar.gz',
 		'format' => 'tar',
-		'compressor' => ['gzip']},
+		'compressor' => ['gzip', '-n']},
 
 	'tbz2' => {
 		'display' => 'tar.bz2',

-- 
1.7.4.3

Jakub Narebski· Apr 10, 2011, 07:38 UTC · re: Fraser Tweedale · lore

Re: [PATCH] supply '-n' to gzip to produce identical tarballs

Fraser Tweedale <frase@frase.id.au> writes:

Show 13 quoted lines

> Subject: [PATCH] supply '-n' to gzip to produce identical tarballs
>
> Without the '-n' ('--no-name') argument, gzip includes timestamp in
> output which results in different files.  Important systems like FreeBSD
> ports and perhaps many others hash/checksum downloaded files to ensure
> integrity.  For projects that do not release official archives, gitweb's
> snapshot feature would be an excellent stand-in but for the fact that the
> files it produces are not identical.
> 
> Supply '-n' to gzip to exclude timestamp from output and produce idential
> output every time.
> 
> Signed-off-by: Fraser Tweedale <frase@frase.id.au>

Very good description, except subject line should denote which subsystem this commit affects, i.e.:

  gitweb: supply '-n' to gzip to produce identical tarballs

Hmmm... gzip in gitweb's 'snapshot' action gets data compressed from standard input, not from filesystem. Isn't -n / --no-name no-op then? Just asking...

Show 14 quoted lines

> ---
>  gitweb/gitweb.perl |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 46186ab..2ab08da 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -186,7 +186,7 @@ our %known_snapshot_formats = (
>  		'type' => 'application/x-gzip',
>  		'suffix' => '.tar.gz',
>  		'format' => 'tar',
> -		'compressor' => ['gzip']},
> +		'compressor' => ['gzip', '-n']},

Perhaps it would be more clear to use

  +		'compressor' => ['gzip', '--no-name']},

Show 6 quoted lines

>  
>  	'tbz2' => {
>  		'display' => 'tar.bz2',
> -- 
> 1.7.4.3
>

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Fraser Tweedale· Apr 10, 2011, 10:13 UTC · re: Jakub Narebski · lore

Re: [PATCH] supply '-n' to gzip to produce identical tarballs

On Sun, Apr 10, 2011 at 12:38:32AM -0700, Jakub Narebski wrote:

Show 21 quoted lines

> Fraser Tweedale <frase@frase.id.au> writes:
> 
> > Subject: [PATCH] supply '-n' to gzip to produce identical tarballs
> >
> > Without the '-n' ('--no-name') argument, gzip includes timestamp in
> > output which results in different files.  Important systems like FreeBSD
> > ports and perhaps many others hash/checksum downloaded files to ensure
> > integrity.  For projects that do not release official archives, gitweb's
> > snapshot feature would be an excellent stand-in but for the fact that the
> > files it produces are not identical.
> > 
> > Supply '-n' to gzip to exclude timestamp from output and produce idential
> > output every time.
> > 
> > Signed-off-by: Fraser Tweedale <frase@frase.id.au>
> 
> Very good description, except subject line should denote which
> subsystem this commit affects, i.e.:
> 
>   gitweb: supply '-n' to gzip to produce identical tarballs
>

Thank you. Do I need to amend the message and resubmit the patch? (first time submitting a patch to git; I used git send-email).

> Hmmm... gzip in gitweb's 'snapshot' action gets data compressed from
> standard input, not from filesystem.  Isn't -n / --no-name no-op then?
> Just asking...
>

It is not no-op; I have tested to confirm this. I'm not sure whether a file name is recorded in the stdin case, or if so what it is, but the timestamp is recorded and that makes the difference.

Show 26 quoted lines

> > ---
> >  gitweb/gitweb.perl |    2 +-
> >  1 files changed, 1 insertions(+), 1 deletions(-)
> > 
> > diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> > index 46186ab..2ab08da 100755
> > --- a/gitweb/gitweb.perl
> > +++ b/gitweb/gitweb.perl
> > @@ -186,7 +186,7 @@ our %known_snapshot_formats = (
> >  		'type' => 'application/x-gzip',
> >  		'suffix' => '.tar.gz',
> >  		'format' => 'tar',
> > -		'compressor' => ['gzip']},
> > +		'compressor' => ['gzip', '-n']},
> 
> Perhaps it would be more clear to use
> 
>   +		'compressor' => ['gzip', '--no-name']},
> 
> >  
> >  	'tbz2' => {
> >  		'display' => 'tar.bz2',
> > -- 
> > 1.7.4.3
> > 
>

Definitely, if the argument is the same (or even present) on all systems. On FreeBSD and GNU both '-n' and '--no-name' are do the job, but an audit of other systems should be done to ensure they don't break. I chose '-n' as it seemed the more conservative choice.

> -- 
> Jakub Narebski
> Poland
> ShadeHawk on #git

Jakub Narebski· Apr 10, 2011, 13:55 UTC · re: Fraser Tweedale · lore

Re: [PATCH] supply '-n' to gzip to produce identical tarballs

On Sun, 10 Apr 2011, Fraser Tweedale wrote:

> On Sun, Apr 10, 2011 at 12:38:32AM -0700, Jakub Narebski wrote:
> > Fraser Tweedale <frase@frase.id.au> writes:
> > 
> > > Subject: [PATCH] supply '-n' to gzip to produce identical tarballs

Show 7 quoted lines

> > Very good description, except subject line should denote which
> > subsystem this commit affects, i.e.:
> > 
> >   gitweb: supply '-n' to gzip to produce identical tarballs
>  
> Thank you.  Do I need to amend the message and resubmit the patch?  (first
> time submitting a patch to git; I used git send-email).

I don't think so. I guess that Junio can do such trivial amend when applying, at the time he is adding his signoff.

Show 7 quoted lines

> > Hmmm... gzip in gitweb's 'snapshot' action gets data compressed from
> > standard input, not from filesystem.  Isn't -n / --no-name no-op then?
> > Just asking...
> 
> It is not no-op; I have tested to confirm this.  I'm not sure whether
> a file name is recorded in the stdin case, or if so what it is, but the
> timestamp is recorded and that makes the difference.

Thanks for the clarification.

For what it is worth:

Acked-by: Jakub Narebski <jnareb@gmail.com>

Show 6 quoted lines

> > > -		'compressor' => ['gzip']},
> > > +		'compressor' => ['gzip', '-n']},
> > 
> > Perhaps it would be more clear to use
> > 
> >   +		'compressor' => ['gzip', '--no-name']},

> Definitely, if the argument is the same (or even present) on all systems.
> On FreeBSD and GNU both '-n' and '--no-name' are do the job, but an audit
> of other systems should be done to ensure they don't break.  I chose '-n'
> as it seemed the more conservative choice.

So you choose '-n' because it has more chance of being widely supported, isn't it? Good enough for me.

-- 
Jakub Narebski
Poland

Junio C Hamano· Apr 11, 2011, 19:24 UTC · re: Jakub Narebski · lore

Re: [PATCH] supply '-n' to gzip to produce identical tarballs

Jakub Narebski <jnareb@gmail.com> writes:

Show 11 quoted lines

>> > Perhaps it would be more clear to use
>> > 
>> >   +		'compressor' => ['gzip', '--no-name']},
>
>> Definitely, if the argument is the same (or even present) on all systems.
>> On FreeBSD and GNU both '-n' and '--no-name' are do the job, but an audit
>> of other systems should be done to ensure they don't break.  I chose '-n'
>> as it seemed the more conservative choice.
>
> So you choose '-n' because it has more chance of being widely supported,
> isn't it?  Good enough for me.

Interesting. "gzip <COPYING" does get a consistent result because it can fstat to get the timestamp, but "cat COPYING | gzip" does change its output every time it is run. Good catch and a solution. Thanks, both.

So I should expect a pull request sometime after 1.7.5 final from you, with "an audit of other systems" done by others on the list noted in the final commit message?

Fraser Tweedale· Apr 11, 2011, 20:59 UTC · re: Junio C Hamano · lore

Re: [PATCH] supply '-n' to gzip to produce identical tarballs

On Mon, Apr 11, 2011 at 12:24:05PM -0700, Junio C Hamano wrote:

Show 23 quoted lines

> Jakub Narebski <jnareb@gmail.com> writes:
> 
> >> > Perhaps it would be more clear to use
> >> > 
> >> >   +		'compressor' => ['gzip', '--no-name']},
> >
> >> Definitely, if the argument is the same (or even present) on all systems.
> >> On FreeBSD and GNU both '-n' and '--no-name' are do the job, but an audit
> >> of other systems should be done to ensure they don't break.  I chose '-n'
> >> as it seemed the more conservative choice.
> >
> > So you choose '-n' because it has more chance of being widely supported,
> > isn't it?  Good enough for me.
> 
> Interesting.  "gzip <COPYING" does get a consistent result because it can
> fstat to get the timestamp, but "cat COPYING | gzip" does change its
> output every time it is run.  Good catch and a solution.  Thanks, both.
> 
> So I should expect a pull request sometime after 1.7.5 final from you,
> with "an audit of other systems" done by others on the list noted in the
> final commit message?
> 
>

Sure, no problem.

Thanks,

Fraser