threads / discuss / 32795

"git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

Subject: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

## tl;dr

11 messages between Jan 31, 2013 and Feb 4, 2013.

replies: 10people: 4as markdown or json

Greg KH· Jan 31, 2013, 17:28 UTC · lore

Hi,

The way we upload the Linux kernel to kernel.org involves creating a tar archive, signing the archive, and then just uploading the signature. The server then checks out the repo based on the tag, generates the tar archive and checks the signature to make sure they match.

A few days ago I released the 3.0.61 kernel, and it turned out that I couldn't upload the kernel release because 'git archive' now creates a binary file that differs from an older version of git.

I tracked this down to commit 22f0dcd9634a818a0c83f23ea1a48f2d620c0546 (archive-tar: split long paths more carefully). The diff of a hex dump of the tar archives shows the following difference:

--- old_git_archive 2013-01-31 17:31:24.466343388 +0100 +++ new_git_archive 2013-01-31 17:32:21.509674417 +0100

@@ -19239998,8 +19239998,8 @@
 125943d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125943e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125943f0:0000 0000 0000 0000 0000 0000 0000 0000  ................
-12594400:0000 0000 0000 0000 0000 0000 0000 0000  ................
-12594410:0000 0000 0000 0000 0000 0000 0000 0000  ................
+12594400:7765 7374 6272 6964 6765 2d6f 6d61 7033  westbridge-omap3
+12594410:2d70 6e61 6e64 2d68 616c 2f00 0000 0000  -pnand-hal/.....
 12594420:0000 0000 0000 0000 0000 0000 0000 0000  ................
 12594430:0000 0000 0000 0000 0000 0000 0000 0000  ................
 12594440:0000 0000 0000 0000 0000 0000 0000 0000  ................
@@ -19240025,8 +19240025,8 @@
 12594580:2f61 7374 6f72 6961 2f61 7263 682f 6172  /astoria/arch/ar
 12594590:6d2f 706c 6174 2d6f 6d61 702f 696e 636c  m/plat-omap/incl
 125945a0:7564 652f 6d61 6368 2f77 6573 7462 7269  ude/mach/westbri
-125945b0:6467 652f 7765 7374 6272 6964 6765 2d6f  dge/westbridge-o
-125945c0:6d61 7033 2d70 6e61 6e64 2d68 616c 0000  map3-pnand-hal..
+125945b0:6467 6500 0000 0000 0000 0000 0000 0000  dge.............
+125945c0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125945d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125945e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125945f0:0000 0000 0000 0000 0000 0000 0000 0000  ................

Interestingly, the output of uncompressing the tar archives is
identical, so the data is correct, but the binary isn't.

Now keeping binary compatibility of tar archive files isn't really a big
deal, but, the commit to git that causes this seems a bit odd, is it
really needed?  Or can we just fix the version of tar with NetBSD
instead?  :)

Any ideas?

thanks,

greg k-h

Junio C Hamano· Jan 31, 2013, 17:32 UTC · re: Greg KH · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

Greg KH <gregkh@linuxfoundation.org> writes:

Show 15 quoted lines

> The way we upload the Linux kernel to kernel.org involves creating a tar
> archive, signing the archive, and then just uploading the signature.
> The server then checks out the repo based on the tag, generates the tar
> archive and checks the signature to make sure they match.
> 
> A few days ago I released the 3.0.61 kernel, and it turned out that I
> couldn't upload the kernel release because 'git archive' now creates a
> binary file that differs from an older version of git.
> ...
> Now keeping binary compatibility of tar archive files isn't really a big
> deal, but, the commit to git that causes this seems a bit odd, is it
> really needed?  Or can we just fix the version of tar with NetBSD
> instead?  :)
>
> Any ideas?

How about fixing kup to teach the "let's cheat and let the other end run 'git archive', if the resulting archive and GPG signature locally created does match, we do not have to transfer the tarball itself" trick a fall-back mode that says "but if the signature does not match, then transfer the bulk used to create the signature to the remote anyway". This fallback can and should of course be useful for the compressed patch transfer.

Greg KH· Jan 31, 2013, 17:41 UTC · re: Junio C Hamano · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

On Thu, Jan 31, 2013 at 09:32:12AM -0800, Junio C Hamano wrote:

Show 25 quoted lines

> Greg KH <gregkh@linuxfoundation.org> writes:
> 
> > The way we upload the Linux kernel to kernel.org involves creating a tar
> > archive, signing the archive, and then just uploading the signature.
> > The server then checks out the repo based on the tag, generates the tar
> > archive and checks the signature to make sure they match.
> > 
> > A few days ago I released the 3.0.61 kernel, and it turned out that I
> > couldn't upload the kernel release because 'git archive' now creates a
> > binary file that differs from an older version of git.
> > ...
> > Now keeping binary compatibility of tar archive files isn't really a big
> > deal, but, the commit to git that causes this seems a bit odd, is it
> > really needed?  Or can we just fix the version of tar with NetBSD
> > instead?  :)
> >
> > Any ideas?
> 
> How about fixing kup to teach the "let's cheat and let the other end
> run 'git archive', if the resulting archive and GPG signature
> locally created does match, we do not have to transfer the tarball
> itself" trick a fall-back mode that says "but if the signature does
> not match, then transfer the bulk used to create the signature to
> the remote anyway".  This fallback can and should of course be
> useful for the compressed patch transfer.

Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up doing lots of kernel releases while traveling), would be a horrible change. I'd rather just keep using the same older version of git that kernel.org is running instead.

thanks,

greg k-h

Konstantin Ryabitsev· Jan 31, 2013, 17:52 UTC · re: Greg KH · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

On 31/01/13 12:41 PM, Greg KH wrote:

> Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up
> doing lots of kernel releases while traveling), would be a horrible
> change.  I'd rather just keep using the same older version of git that
> kernel.org is running instead.

Well, we do accept compressed archives, so you would be uploading about 80MB instead of 431MB, but that would still be a problem for anyone releasing large tarballs over unreliable connections. I know you routinely do 2-3 releases at once, so that would still mean uploading 120-180MB.

I don't have immediate statistics on how many people release using "kup --tar", but I know that at least you and Linus rely on that exclusively.

Regards,

-- 
Konstantin Ryabitsev
Systems Administrator
Linux Foundation, kernel.org
Montréal, Québec

Greg KH· Feb 4, 2013, 00:48 UTC · re: Konstantin Ryabitsev · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

On Thu, Jan 31, 2013 at 12:52:15PM -0500, Konstantin Ryabitsev wrote:

Show 11 quoted lines

> On 31/01/13 12:41 PM, Greg KH wrote:
> > Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up
> > doing lots of kernel releases while traveling), would be a horrible
> > change.  I'd rather just keep using the same older version of git that
> > kernel.org is running instead.
> 
> Well, we do accept compressed archives, so you would be uploading about
> 80MB instead of 431MB, but that would still be a problem for anyone
> releasing large tarballs over unreliable connections. I know you
> routinely do 2-3 releases at once, so that would still mean uploading
> 120-180MB.

That would mean I can't do kernel releases while on ferry rides, which is probably a good thing in the end :)

> I don't have immediate statistics on how many people release using "kup
> --tar", but I know that at least you and Linus rely on that exclusively.

What causes you to upgrade the version of git on the server? Are you relying on packages for a distro, or is this "hand installed" by yourself? As long as I stay in lock-step with your updates, all should be fine.

Oh, maybe we can report back to the user, the version of git that is being used on the server, if the checksums don't match, so that I know to at least see if my version is different from yours?

thanks,

greg k-h

Junio C Hamano· Jan 31, 2013, 18:16 UTC · re: Greg KH · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

Greg KH <gregkh@linuxfoundation.org> writes:

Show 14 quoted lines

> On Thu, Jan 31, 2013 at 09:32:12AM -0800, Junio C Hamano wrote:
>
>> How about fixing kup to teach the "let's cheat and let the other end
>> run 'git archive', if the resulting archive and GPG signature
>> locally created does match, we do not have to transfer the tarball
>> itself" trick a fall-back mode that says "but if the signature does
>> not match, then transfer the bulk used to create the signature to
>> the remote anyway".  This fallback can and should of course be
>> useful for the compressed patch transfer.
>
> Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up
> doing lots of kernel releases while traveling), would be a horrible
> change.  I'd rather just keep using the same older version of git that
> kernel.org is running instead.

Then how about fixing kup to try both versions of Git? There will be people who run different versions of Git anyway, and kup should not be preventing Git from helping people on other platforms, or improving its output in general.

Greg KH· Feb 4, 2013, 00:45 UTC · re: Junio C Hamano · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

On Thu, Jan 31, 2013 at 10:16:27AM -0800, Junio C Hamano wrote:

Show 21 quoted lines

> Greg KH <gregkh@linuxfoundation.org> writes:
> 
> > On Thu, Jan 31, 2013 at 09:32:12AM -0800, Junio C Hamano wrote:
> >
> >> How about fixing kup to teach the "let's cheat and let the other end
> >> run 'git archive', if the resulting archive and GPG signature
> >> locally created does match, we do not have to transfer the tarball
> >> itself" trick a fall-back mode that says "but if the signature does
> >> not match, then transfer the bulk used to create the signature to
> >> the remote anyway".  This fallback can and should of course be
> >> useful for the compressed patch transfer.
> >
> > Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up
> > doing lots of kernel releases while traveling), would be a horrible
> > change.  I'd rather just keep using the same older version of git that
> > kernel.org is running instead.
> 
> Then how about fixing kup to try both versions of Git?  There will
> be people who run different versions of Git anyway, and kup should
> not be preventing Git from helping people on other platforms, or
> improving its output in general.

I think the combinations of different versions of git that would have to be installed on kernel.org to handle stuff like this as things change over time, wouldn't be worth it.

The number of people this affects right now is only one (me), given that the offending file is not in Linus's tree right now, so he doesn't have issues with uploading new releases.

So I'll just work to ensure I have the same version of git in place if I ever run into this problem again. Or just break down and do full-compressed tarballs instead, if I'm in a place where I have a good network connection.

thanks,

greg k-h

Junio C Hamano· Feb 4, 2013, 05:05 UTC · re: Greg KH · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

Greg KH <gregkh@linuxfoundation.org> writes:

Show 8 quoted lines

>> Then how about fixing kup to try both versions of Git?  There will
>> be people who run different versions of Git anyway, and kup should
>> not be preventing Git from helping people on other platforms, or
>> improving its output in general.
>
> I think the combinations of different versions of git that would have to
> be installed on kernel.org to handle stuff like this as things change
> over time, wouldn't be worth it.

You make it sound as if you have to pick the right one among 47 different versions, but I think over the lifetime of Git there was only one time that output from "diff" would have affected the kup's trick to avoid the transmission of patch text, and another that output from "tar-tree" (aka "archive --format=tar") would have afffected the transmission of tarballs. I do think it is feasible if "kup" wanted to.

> The number of people this affects right now is only one (me), given that
> the offending file is not in Linus's tree right now, so he doesn't have
> issues with uploading new releases.

As a tree grows larger over time, it may be just a matter of time for somebody else to be hit by another deep path, though.

Greg KH· Feb 4, 2013, 18:26 UTC · re: Junio C Hamano · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

On Sun, Feb 03, 2013 at 09:05:55PM -0800, Junio C Hamano wrote:

Show 7 quoted lines

> Greg KH <gregkh@linuxfoundation.org> writes:
> > The number of people this affects right now is only one (me), given that
> > the offending file is not in Linus's tree right now, so he doesn't have
> > issues with uploading new releases.
> 
> As a tree grows larger over time, it may be just a matter of time
> for somebody else to be hit by another deep path, though.

I agree, and over time, everyone will have updated to a version of git newer than 1.8.2.1 so we all will be fine again :)

thanks,

greg k-h

René Scharfe· Jan 31, 2013, 18:33 UTC · re: Greg KH · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

Am 31.01.2013 18:28, schrieb Greg KH:

Show 28 quoted lines

> I tracked this down to commit 22f0dcd9634a818a0c83f23ea1a48f2d620c0546
> (archive-tar: split long paths more carefully).  The diff of a hex dump
> of the tar archives shows the following difference:
> 
> --- old_git_archive	2013-01-31 17:31:24.466343388 +0100
> +++ new_git_archive	2013-01-31 17:32:21.509674417 +0100
> @@ -19239998,8 +19239998,8 @@
>  125943d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125943e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125943f0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> -12594400:0000 0000 0000 0000 0000 0000 0000 0000  ................
> -12594410:0000 0000 0000 0000 0000 0000 0000 0000  ................
> +12594400:7765 7374 6272 6964 6765 2d6f 6d61 7033  westbridge-omap3
> +12594410:2d70 6e61 6e64 2d68 616c 2f00 0000 0000  -pnand-hal/.....
>  12594420:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  12594430:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  12594440:0000 0000 0000 0000 0000 0000 0000 0000  ................
> @@ -19240025,8 +19240025,8 @@
>  12594580:2f61 7374 6f72 6961 2f61 7263 682f 6172  /astoria/arch/ar
>  12594590:6d2f 706c 6174 2d6f 6d61 702f 696e 636c  m/plat-omap/incl
>  125945a0:7564 652f 6d61 6368 2f77 6573 7462 7269  ude/mach/westbri
> -125945b0:6467 652f 7765 7374 6272 6964 6765 2d6f  dge/westbridge-o
> -125945c0:6d61 7033 2d70 6e61 6e64 2d68 616c 0000  map3-pnand-hal..
> +125945b0:6467 6500 0000 0000 0000 0000 0000 0000  dge.............
> +125945c0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125945d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125945e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125945f0:0000 0000 0000 0000 0000 0000 0000 0000  ................

This is the only directory in the repository whose path is long enough to make a difference with the patch, 105 characters in total:

drivers/staging/westbridge/astoria/arch/arm/plat-omap/include/mach/westbridge/westbridge-omap3-pnand-hal/

Five characters less and you wouldn't notice a thing. It contains "westbridge" thrice, so I think it's cheating just to reach that length, though. ;-)

> Interestingly, the output of uncompressing the tar archives is
> identical, so the data is correct, but the binary isn't.

The path is split differently between two header fields, that's all.

> Now keeping binary compatibility of tar archive files isn't really a big
> deal, but, the commit to git that causes this seems a bit odd, is it
> really needed?  Or can we just fix the version of tar with NetBSD
> instead?  :)

Apart from Junio's suggestion, I can't think of a practical solution.

You could downgrade your git to a version before the fix. A downside is that you won't be able to extract the archive on NetBSD without getting an error message (but the contents would be intact, except perhaps for permission bits of the directory above).

You could upgrade the kernel.org version of git, but that might cause the same problem for other maintainers with long directory paths who in their repositories who still use git without the fix.

You could make the path shorter. Won't help at all with the release you just did, of course.

I don't know if other tar implementations freak out when they see an empty name field. NetBSD's tar might seem a bit too strict here, but overall I think it's right in complaining.

What makes the commit odd, by the way?

Thanks, René

Greg KH· Feb 4, 2013, 00:42 UTC · re: René Scharfe · lore

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

On Thu, Jan 31, 2013 at 07:33:52PM +0100, René Scharfe wrote:

Show 38 quoted lines

> Am 31.01.2013 18:28, schrieb Greg KH:
> > I tracked this down to commit 22f0dcd9634a818a0c83f23ea1a48f2d620c0546
> > (archive-tar: split long paths more carefully).  The diff of a hex dump
> > of the tar archives shows the following difference:
> > 
> > --- old_git_archive	2013-01-31 17:31:24.466343388 +0100
> > +++ new_git_archive	2013-01-31 17:32:21.509674417 +0100
> > @@ -19239998,8 +19239998,8 @@
> >  125943d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125943e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125943f0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> > -12594400:0000 0000 0000 0000 0000 0000 0000 0000  ................
> > -12594410:0000 0000 0000 0000 0000 0000 0000 0000  ................
> > +12594400:7765 7374 6272 6964 6765 2d6f 6d61 7033  westbridge-omap3
> > +12594410:2d70 6e61 6e64 2d68 616c 2f00 0000 0000  -pnand-hal/.....
> >  12594420:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  12594430:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  12594440:0000 0000 0000 0000 0000 0000 0000 0000  ................
> > @@ -19240025,8 +19240025,8 @@
> >  12594580:2f61 7374 6f72 6961 2f61 7263 682f 6172  /astoria/arch/ar
> >  12594590:6d2f 706c 6174 2d6f 6d61 702f 696e 636c  m/plat-omap/incl
> >  125945a0:7564 652f 6d61 6368 2f77 6573 7462 7269  ude/mach/westbri
> > -125945b0:6467 652f 7765 7374 6272 6964 6765 2d6f  dge/westbridge-o
> > -125945c0:6d61 7033 2d70 6e61 6e64 2d68 616c 0000  map3-pnand-hal..
> > +125945b0:6467 6500 0000 0000 0000 0000 0000 0000  dge.............
> > +125945c0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125945d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125945e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125945f0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> 
> This is the only directory in the repository whose path is long enough to
> make a difference with the patch, 105 characters in total:
> 
> drivers/staging/westbridge/astoria/arch/arm/plat-omap/include/mach/westbridge/westbridge-omap3-pnand-hal/
> 
> Five characters less and you wouldn't notice a thing.  It contains
> "westbridge" thrice, so I think it's cheating just to reach that
> length, though. ;-)

Yeah, that file path was bad, which is just one reason why it was deleted from the kernel in newer versions :)

> > Interestingly, the output of uncompressing the tar archives is
> > identical, so the data is correct, but the binary isn't.
> 
> The path is split differently between two header fields, that's all.

Ok, thanks for the explaination, I didn't realize that.

Show 18 quoted lines

> > Now keeping binary compatibility of tar archive files isn't really a big
> > deal, but, the commit to git that causes this seems a bit odd, is it
> > really needed?  Or can we just fix the version of tar with NetBSD
> > instead?  :)
> 
> Apart from Junio's suggestion, I can't think of a practical solution.
> 
> You could downgrade your git to a version before the fix.  A downside is
> that you won't be able to extract the archive on NetBSD without getting
> an error message (but the contents would be intact, except perhaps for
> permission bits of the directory above).
> 
> You could upgrade the kernel.org version of git, but that might cause the
> same problem for other maintainers with long directory paths who in their
> repositories who still use git without the fix.
> 
> You could make the path shorter.  Won't help at all with the release you
> just did, of course.

What I ended up doing was just to revert your patch, generating a tar archive that matches what the version on kernel.org.

And originally I now recall that this was something we were worried about, but we put off dealing with it until it caused problems :)

> I don't know if other tar implementations freak out when they see an
> empty name field.  NetBSD's tar might seem a bit too strict here, but
> overall I think it's right in complaining.

Ok, thanks, I now agree.

> What makes the commit odd, by the way?

Sorry, I was originally thinking that you were working around a bug in the NetBSD version of tar, not making it "more correct", which is arguably the right thing to do here.

So I'll work with Konstantine to ensure we both are using the same version of git in the future, it's our kernel.org infrastructure issue here, not a git one, sorry for the noise.

greg k-h