git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/2] archive-tar: write extended headers for far-future mtime

From
René Scharfe <l.s.r@web.de>
Date
Jun 20, 2016, 22:54 UTC
Message-ID
<57687417.4020009@web.de>
In-Reply-To
<20160616043758.GB18323@sigill.intra.peff.net>
Am 16.06.2016 um 06:37 schrieb Jeff King:
Show 43 quoted lines
> The ustar format represents timestamps as seconds since the
> epoch, but only has room to store 11 octal digits.  To
> express anything larger, we need to use an extended header.
> This is exactly the same case we fixed for the size field in
> the previous commit, and the solution here follows the same
> pattern.
>
> This is even mentioned as an issue in f2f0267 (archive-tar:
> use xsnprintf for trivial formatting, 2015-09-24), but since
> it only affected things far in the future, it wasn't worth
> dealing with. But note that my calculations claiming
> thousands of years were off there; because our xsnprintf
> produces a NUL byte, we only have until the year 2242 to
> fix this.
>
> Given that this is just around the corner (geologically
> speaking), and because the fix for "size" makes doing this
> on top trivial, let's just make it work.
>
> Note that we don't (and never did) handle negative
> timestamps (i.e., before 1970). This would probably not be
> too hard to support in the same way, but since git does not
> support negative timestamps at all, I didn't bother here.
>
> Unlike the last patch for "size", this one is easy to test
> efficiently (the magic date is octal 01000000000001):
>
>    $ echo content >file
>    $ git add file
>    $ GIT_COMMITTER_DATE='@68719476737 +0000' \
>      git commit -q -m 'tempori parendum'
>    $ git archive HEAD | tar tvf -
>    -rw-rw-r-- root/root         8 4147-08-20 03:32 file
>
> With the current tip of master, this dies in xsnprintf (and
> prior to f2f0267, it overflowed into the checksum field, and
> the resulting tarfile claimed to be from the year 2242).
>
> However, I decided not to include this in the test suite,
> because I suspect it will create portability headaches,
> including:
>
>    1. The exact format of the system tar's "t" output.
Probably not worth it.
>    2. System tars that cannot handle pax headers.

In t5000 there is a simple interpreter for path headers for systems whose tar doesn't handle them. Adding one for mtime headers may be feasible.

It's just a bit complicated to link a pax header file to the file it applies to when it doesn't also contain a path header. But we know that the mtime of all entries in tar files created by git archive are is the same, so we could simply read one header and then adjust the mtime of all files accordingly.

Show 5 quoted lines
>    3. System tars tha cannot handle dates that far in the
>       future.
>
>    4. If we replace "tar t" with "tar x", then filesystems
>       that cannot handle dates that far in the future.

We can test that by supplying a test archive and set a prerequisite if tar (possibly with our header interpreter) and the filesystem can cope with that.

Show 20 quoted lines
> In other words, we do not really have a reliable tar reader
> for these corner cases, so the best we could do is a
> byte-for-byte comparison of the output.
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
>   archive-tar.c | 13 ++++++++++++-
>   1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/archive-tar.c b/archive-tar.c
> index 7340b64..749722f 100644
> --- a/archive-tar.c
> +++ b/archive-tar.c
> @@ -185,6 +185,14 @@ static inline unsigned long ustar_size(uintmax_t size)
>   		return 0;
>   }
>
> +static inline unsigned long ustar_mtime(time_t mtime)
> +{
> +	if (mtime < 077777777777)
That should be less-or-equal, right?
Show 28 quoted lines
> +		return mtime;
> +	else
> +		return 0;
> +}
> +
>   static void prepare_header(struct archiver_args *args,
>   			   struct ustar_header *header,
>   			   unsigned int mode, unsigned long size)
> @@ -192,7 +200,8 @@ static void prepare_header(struct archiver_args *args,
>   	xsnprintf(header->mode, sizeof(header->mode), "%07o", mode & 07777);
>   	xsnprintf(header->size, sizeof(header->size), "%011lo",
>   		  S_ISREG(mode) ? ustar_size(size) : 0);
> -	xsnprintf(header->mtime, sizeof(header->mtime), "%011lo", (unsigned long) args->time);
> +	xsnprintf(header->mtime, sizeof(header->mtime), "%011lo",
> +		  ustar_mtime(args->time));
>
>   	xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
>   	xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
> @@ -292,6 +301,8 @@ static int write_tar_entry(struct archiver_args *args,
>
>   	if (ustar_size(size) != size)
>   		strbuf_append_ext_header_uint(&ext_header, "size", size);
> +	if (ustar_mtime(args->time) != args->time)
> +		strbuf_append_ext_header_uint(&ext_header, "mtime", args->time);
>
>   	prepare_header(args, &header, mode, size);
>
>
Previous: Jeff KingNext: René Scharfe
Message 17 of 61 in “friendlier handling of overflows in archive-tar”
  1. 0/2 friendlier handling of overflows in archive-tarJeff King, Jun 16, 2016
  2. 1/2 archive-tar: write extended headers for file sizes >= 8GBJeff King, Jun 16, 2016
  3. René ScharfeJun 20, 2016
  4. Jeff KingJun 21, 2016
  5. Jeff KingJun 21, 2016
  6. René ScharfeJun 21, 2016
  7. René ScharfeJun 21, 2016
  8. Jeff KingJun 21, 2016
  9. René ScharfeJun 22, 2016
  10. Jeff KingJun 21, 2016
  11. René ScharfeJun 22, 2016
  12. Jeff KingJun 23, 2016
  13. René ScharfeJun 21, 2016
  14. Robin H. JohnsonJun 21, 2016
  15. Jeff KingJun 21, 2016
  16. 2/2 archive-tar: write extended headers for far-future mtimeJeff King, Jun 16, 2016
  17. René ScharfeJun 20, 2016
  18. René ScharfeJun 22, 2016
  19. Jeff KingJun 23, 2016
  20. René ScharfeJun 23, 2016
  21. Jeff KingJun 23, 2016
  22. Junio C HamanoJun 16, 2016
  23. Jeff KingJun 21, 2016
  24. 1/2 archive-tar: write extended headers for file sizes >= 8GBJeff King, Jun 21, 2016
  25. 2/2 archive-tar: write extended headers for far-future mtimeJeff King, Jun 21, 2016
  26. Junio C HamanoJun 21, 2016
  27. friendlier handling of overflows in archive-tarJeff King, Jun 23, 2016
  28. 1/4 t5000: test tar files that overflow ustar headersJeff King, Jun 23, 2016
  29. Jeff KingJun 23, 2016
  30. Johannes SixtJun 24, 2016
  31. Jeff KingJun 24, 2016
  32. Johannes SixtJun 24, 2016
  33. 0/4 portable signal-checking in testsJeff King, Jun 24, 2016
  34. 1/4 tests: factor portable signal check out of t0005Jeff King, Jun 24, 2016
  35. Johannes SixtJun 24, 2016
  36. Jeff KingJun 24, 2016
  37. Johannes SixtJun 24, 2016
  38. 2/4 t0005: use test_match_signal as appropriateJeff King, Jun 24, 2016
  39. 3/4 test_must_fail: use test_match_signalJeff King, Jun 24, 2016
  40. 4/4 t/lib-git-daemon: use test_match_signalJeff King, Jun 24, 2016
  41. Jeff KingJun 24, 2016
  42. Junio C HamanoJun 24, 2016
  43. Jeff KingJun 24, 2016
  44. Junio C HamanoJun 24, 2016
  45. Eric SunshineJun 24, 2016
  46. Jeff KingJun 24, 2016
  47. Jeff KingJun 24, 2016
  48. Junio C HamanoJun 24, 2016
  49. Jeff KingJun 24, 2016
  50. 2/4 archive-tar: write extended headers for file sizes >= 8GBJeff King, Jun 23, 2016
  51. Junio C HamanoJun 24, 2016
  52. Jeff KingJun 24, 2016
  53. Junio C HamanoJun 24, 2016
  54. Jeff KingJun 24, 2016
  55. 3/4 archive-tar: write extended headers for far-future mtimeJeff King, Jun 23, 2016
  56. Junio C HamanoJun 24, 2016
  57. Jeff KingJun 24, 2016
  58. 4/4 archive-tar: drop return valueJeff King, Jun 23, 2016
  59. Remi Galan AlfonsoJun 24, 2016
  60. Jeff KingJun 24, 2016
  61. Junio C HamanoJun 24, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.