{"thread":{"id":"51825","subject":"Patching Git to handle dates before the Unix epoch","startedAt":"2019-09-10T14:21:57Z","lastAt":"2019-09-13T05:12:06Z","messageCount":6,"participants":["Diomidis Spinellis","Randall S. Becker","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"382132","messageId":"c12e3505-7a02-3eaf-7d71-57a3b4df28d1@aueb.gr","threadId":"51825","inReplyTo":null,"subject":"Patching Git to handle dates before the Unix epoch","fromName":"Diomidis Spinellis","fromEmail":"dds@aueb.gr","sentAt":"2019-09-10T14:14:53Z","receivedAt":"2019-09-10T14:21:57Z","isPatch":false,"sender":{"key":"dds@aueb.gr","avatar":"https://avatars.githubusercontent.com/u/1212716?v=4"},"body":"As people use Git to create synthetic commits of code written in the \npast [1,2] it becomes important to handle dates before the Unix epoch \n(1/1/1970).  I see that modern C libraries, Unix kernels, and tools can \nhandle such dates.  However Git seems to mishandle such dates in several \nplaces, such as in date.c [3,4].  I'm planning to work on a fix, but \nbefore I embark on this I have a few questions.\n\n- Do you see any reasons that may prevent the acceptance of such a patch?\n- Can you think of any non-obvious issues (e.g. backward compatibility, \nswitch to the Gregorian calendar) I should be aware of?\n- Should I submit changes as a bug fix on the maint branch or as a new \nfeature on master?\n\n[1] https://github.com/dspinellis/unix-history-repo (original dates)\n[2] https://github.com/chrislgarry/Apollo-11 (wrong dates)\n[3] https://github.com/git/git/blob/master/date.c#L21\n[4] https://github.com/git/git/blob/master/date.c#L776\n\nDiomidis Spinellis - https://www.spinellis.gr\n"},{"id":"382134","messageId":"003301d567f2$03550df0$09ff29d0$@nexbridge.com","threadId":"51825","inReplyTo":"c12e3505-7a02-3eaf-7d71-57a3b4df28d1@aueb.gr","subject":"RE: Patching Git to handle dates before the Unix epoch","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2019-09-10T16:08:34Z","receivedAt":"2019-09-10T16:08:49Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 10, 2019 10:15 AM, Diomidis Spinellis wrote:\n> As people use Git to create synthetic commits of code written in the past\n> [1,2] it becomes important to handle dates before the Unix epoch\n> (1/1/1970).  I see that modern C libraries, Unix kernels, and tools can handle\n> such dates.  However Git seems to mishandle such dates in several places,\n> such as in date.c [3,4].  I'm planning to work on a fix, but before I embark on\n> this I have a few questions.\n> \n> - Do you see any reasons that may prevent the acceptance of such a patch?\n> - Can you think of any non-obvious issues (e.g. backward compatibility,\n> switch to the Gregorian calendar) I should be aware of?\n> - Should I submit changes as a bug fix on the maint branch or as a new\n> feature on master?\n\nMy suggestion is a new feature as a patch. See other contributions. While you're at this, especially given how extensive this may be given the time_t references, it might be useful to examine the end of epoch concerns as well. 2036 is not that far off and some of the repositories I am managing will likely last beyond the Unix date rollover. There are no time64_t to get us to 2106, but even that is probably not sufficient (thinking beyond my own expiry date). The concept of unlimited date ranges is intriguing if we are going to store broader range artifacts in git, like signatures of physical core samples and use the carbon dating in git, or some Sci-Fi conceptual commit on old 1701D. So if I had a preference, it would be to store an extensible date range going from the Big Bang to the Big Crunch (or beyond), but that is excessive.\n\nSadly my platform's own date ranges are 64-bit microsecond that range from year 1 (AD) - 10000 (AD) and I can't map easily to Unix dates.\n\nJust my $0.02,\nRandall\n\n"},{"id":"382135","messageId":"20190910162629.GA12758@sigill.intra.peff.net","threadId":"51825","inReplyTo":"c12e3505-7a02-3eaf-7d71-57a3b4df28d1@aueb.gr","subject":"Re: Patching Git to handle dates before the Unix epoch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-09-10T16:26:29Z","receivedAt":"2019-09-10T16:26:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 10, 2019 at 05:14:53PM +0300, Diomidis Spinellis wrote:\n\n> As people use Git to create synthetic commits of code written in the past\n> [1,2] it becomes important to handle dates before the Unix epoch (1/1/1970).\n> I see that modern C libraries, Unix kernels, and tools can handle such\n> dates.  However Git seems to mishandle such dates in several places, such as\n> in date.c [3,4].  I'm planning to work on a fix, but before I embark on this\n> I have a few questions.\n\nCoincidentally I was looking into this yesterday. See below.\n\n> - Do you see any reasons that may prevent the acceptance of such a patch?\n\nNo, I think this is where we want to go. One of the long-standing\nblockers was that timestamps were treated as \"unsigned long\" all over\nthe code base. That was cleaned up into a single date type around\ndddbad728c (timestamp_t: a new data type for timestamps, 2017-04-26).\n\nSo there's definitely still work to be done before switching that to a\nsigned type, but most of the far-reaching changes have already been\ndone.\n\n> - Can you think of any non-obvious issues (e.g. backward compatibility,\n> switch to the Gregorian calendar) I should be aware of?\n\nThe big question is: what will existing implementations do with a commit\nobject that contains a negative timestamp?\n\nThere are a few answers I found (at least for existing versions of Git;\nI didn't check other tools):\n\n  - git-log and friends will realize they can't parse it and quietly\n    rewrite it to \"0 +0000\" (e.g., when formatting human-readable author\n    dates). That's pretty reasonable.\n\n  - fsck will complain, which may cause servers to reject objects if\n    they use transfer.fsckObjects. Once there are patches, presumably\n    most hosting sites would apply them relatively quickly, but it's\n    likely to cause some friction. OTOH, you only see that friction if\n    you're working with a history that has such negative timestamps.\n\n  - I'm not sure what we'd do for commit timestamps, which are still\n    stored as 32-bit unsigned integers for the purposes of traversal\n    (and ditto in commit graphs). IMHO the correct thing would be having\n    modern commit timestamps and \"real\" author timestamps. I'm not sure\n    if fsck should enforce that or not.\n\n> - Should I submit changes as a bug fix on the maint branch or as a new\n> feature on master?\n\nWhile it is a bug fix, I think it's a big enough change that it should\nbe part of a major release. So base it on master.\n\nHere's how far I got working on this yesterday. It's quite messy, but\nseems to work at least for basic tests like:\n\n  $ git commit --date='@-6105998402 -0500' --allow-empty -m foo\n  [master f83702e] foo\n   Date: Thu Jul 4 12:19:58 1776 -0500\n\n  $ git show\n  commit f83702e850366c0a9a6dc5a5fb70d21072cf40e5 (HEAD -> master)\n  Author: Jeff King <peff@peff.net>\n  Date:   Thu Jul 4 12:19:58 1776 -0500\n\n      foo\n\nIt obviously needs split up into individual patches with each bit\nexplained. The most glaring problem is that it switches our custom\ntm_to_time_t() to use timegm(), which isn't portable. We can probably\npull a fallback implementation from glibc or elsewhere (I've seen a few\nfloating around the net, and they're less horrible than you might think\nbecause they don't have to deal with timezones).\n\nThere are probably more corner cases to be poked at and found. Notably\nthis _doesn't_ work:\n\n  $ git commit --date='1776-07-04 12:19:58' ...\n\nbecause we're too picky in our parsing. There might be some improvement\npossible there, at least for unambiguous iso8601-looking cases\n(approxidate will probably always have use heuristics that assume a\nrecent-ish date).\n\n---\n Makefile          |  5 ++---\n date.c            | 42 ++++++++++++++++++------------------------\n git-compat-util.h |  9 +++++----\n ident.c           |  4 +++-\n 4 files changed, 28 insertions(+), 32 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex ad71ae1219..41cdb4bd8c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2525,8 +2525,7 @@ endif\n ## Git repository (not a tarball extract), (2) any local modifications\n ## will be lost.\n ## Gettext tools cannot work with our own custom PRItime type, so\n-## we replace PRItime with PRIuMAX.  We need to update this to\n-## PRIdMAX if we switch to a signed type later.\n+## we replace PRItime with PRIdMAX.\n \n po/git.pot: $(GENERATED_H) FORCE\n \t# All modifications will be reverted at the end, so we do not\n@@ -2535,7 +2534,7 @@ po/git.pot: $(GENERATED_H) FORCE\n \n \t@for s in $(LOCALIZED_C) $(LOCALIZED_SH) $(LOCALIZED_PERL); \\\n \tdo \\\n-\t\tsed -e 's|PRItime|PRIuMAX|g' <\"$$s\" >\"$$s+\" && \\\n+\t\tsed -e 's|PRItime|PRIdMAX|g' <\"$$s\" >\"$$s+\" && \\\n \t\tcat \"$$s+\" >\"$$s\" && rm \"$$s+\"; \\\n \tdone\n \ndiff --git a/date.c b/date.c\nindex 8126146c50..52e54e3e2a 100644\n--- a/date.c\n+++ b/date.c\n@@ -7,27 +7,14 @@\n #include \"cache.h\"\n \n /*\n- * This is like mktime, but without normalization of tm_wday and tm_yday.\n+ * This is like mktime, but without normalization of tm_wday and tm_yday,\n+ * and always operating in UTC, not the local timezone.\n  */\n-static time_t tm_to_time_t(const struct tm *tm)\n+static time_t tm_to_time_t(const struct tm *tm_const)\n {\n-\tstatic const int mdays[] = {\n-\t    0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334\n-\t};\n-\tint year = tm->tm_year - 70;\n-\tint month = tm->tm_mon;\n-\tint day = tm->tm_mday;\n-\n-\tif (year < 0 || year > 129) /* algo only works for 1970-2099 */\n-\t\treturn -1;\n-\tif (month < 0 || month > 11) /* array bounds */\n-\t\treturn -1;\n-\tif (month < 2 || (year + 2) % 4)\n-\t\tday--;\n-\tif (tm->tm_hour < 0 || tm->tm_min < 0 || tm->tm_sec < 0)\n-\t\treturn -1;\n-\treturn (year * 365 + (year + 1) / 4 + mdays[month] + day) * 24*60*60UL +\n-\t\ttm->tm_hour * 60*60 + tm->tm_min * 60 + tm->tm_sec;\n+\tstruct tm tm = *tm_const;\n+\t/* this is a GNUism; we need to provide a fallback */\n+\treturn timegm(&tm);\n }\n \n static const char *month_names[] = {\n@@ -51,8 +38,7 @@ static time_t gm_time_t(timestamp_t time, int tz)\n \t\tif (unsigned_add_overflows(time, minutes * 60))\n \t\t\tdie(\"Timestamp+tz too large: %\"PRItime\" +%04d\",\n \t\t\t    time, tz);\n-\t} else if (time < -minutes * 60)\n-\t\tdie(\"Timestamp before Unix epoch: %\"PRItime\" %04d\", time, tz);\n+\t}\n \ttime += minutes * 60;\n \tif (date_overflows(time))\n \t\tdie(\"Timestamp too large for this system: %\"PRItime, time);\n@@ -773,10 +759,11 @@ static int match_object_header_date(const char *date, timestamp_t *timestamp, in\n \ttimestamp_t stamp;\n \tint ofs;\n \n-\tif (*date < '0' || '9' < *date)\n+\tif (*date != '-' && (*date < '0' || '9' < *date))\n \t\treturn -1;\n+\terrno = 0;\n \tstamp = parse_timestamp(date, &end, 10);\n-\tif (*end != ' ' || stamp == TIME_MAX || (end[1] != '+' && end[1] != '-'))\n+\tif (*end != ' ' || errno == ERANGE || (end[1] != '+' && end[1] != '-'))\n \t\treturn -1;\n \tdate = end + 2;\n \tofs = strtol(date, &end, 10);\n@@ -841,6 +828,13 @@ int parse_date_basic(const char *date, timestamp_t *timestamp, int *offset)\n \t\tdate += match;\n \t}\n \n+\tif (tm.tm_year < 0 ||\n+\t    tm.tm_mon < 0 || tm.tm_mon > 11 ||\n+\t    tm.tm_hour < 0 || tm.tm_hour > 23 ||\n+\t    tm.tm_min < 0 || tm.tm_min > 59 ||\n+\t    tm.tm_sec < 0 || tm.tm_sec > 59)\n+\t\treturn -1;\n+\n \t/* do not use mktime(), which uses local timezone, here */\n \t*timestamp = tm_to_time_t(&tm);\n \tif (*timestamp == -1)\n@@ -1322,7 +1316,7 @@ int date_overflows(timestamp_t t)\n \ttime_t sys;\n \n \t/* If we overflowed our timestamp data type, that's bad... */\n-\tif ((uintmax_t)t >= TIME_MAX)\n+\tif (t >= TIME_MAX || t <= TIME_MIN)\n \t\treturn 1;\n \n \t/*\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 83be89de0a..9a93d43429 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -340,10 +340,11 @@ char *gitdirname(char *);\n #define PRIo32 \"o\"\n #endif\n \n-typedef uintmax_t timestamp_t;\n-#define PRItime PRIuMAX\n-#define parse_timestamp strtoumax\n-#define TIME_MAX UINTMAX_MAX\n+typedef intmax_t timestamp_t;\n+#define PRItime PRIdMAX\n+#define parse_timestamp strtoimax\n+#define TIME_MAX INTMAX_MAX\n+#define TIME_MIN INTMAX_MIN\n \n #ifndef PATH_SEP\n #define PATH_SEP ':'\ndiff --git a/ident.c b/ident.c\nindex e666ee4e59..2734bf890d 100644\n--- a/ident.c\n+++ b/ident.c\n@@ -322,10 +322,12 @@ int split_ident_line(struct ident_split *split, const char *line, int len)\n \tif (line + len <= cp)\n \t\tgoto person_only;\n \tsplit->date_begin = cp;\n+\tif (*cp == '-')\n+\t\tcp++;\n \tspan = strspn(cp, \"0123456789\");\n \tif (!span)\n \t\tgoto person_only;\n-\tsplit->date_end = split->date_begin + span;\n+\tsplit->date_end = cp + span;\n \tfor (cp = split->date_end; cp < line + len && isspace(*cp); cp++)\n \t\t;\n \tif (line + len <= cp || (*cp != '+' && *cp != '-'))\n-- \n2.23.0.663.gbe3d117559\n\n"},{"id":"382136","messageId":"20190910163117.GB12758@sigill.intra.peff.net","threadId":"51825","inReplyTo":"003301d567f2$03550df0$09ff29d0$@nexbridge.com","subject":"Re: Patching Git to handle dates before the Unix epoch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-09-10T16:31:18Z","receivedAt":"2019-09-10T16:31:20Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 10, 2019 at 12:08:34PM -0400, Randall S. Becker wrote:\n\n> My suggestion is a new feature as a patch. See other contributions.\n> While you're at this, especially given how extensive this may be given\n> the time_t references, it might be useful to examine the end of epoch\n> concerns as well. 2036 is not that far off and some of the\n> repositories I am managing will likely last beyond the Unix date\n> rollover. There are no time64_t to get us to 2106, but even that is\n> probably not sufficient (thinking beyond my own expiry date). The\n> concept of unlimited date ranges is intriguing if we are going to\n> store broader range artifacts in git, like signatures of physical core\n> samples and use the carbon dating in git, or some Sci-Fi conceptual\n> commit on old 1701D. So if I had a preference, it would be to store an\n> extensible date range going from the Big Bang to the Big Crunch (or\n> beyond), but that is excessive.\n\nInternally we now use a uintmax_t (with a resolution of 1-second) for\nparsing and storing most timestamps, which should presumably be at least\n64-bit on most systems. We do convert to time_t at various points,\nrecognizing truncation and switching that to a sentinel value (so at\nleast you get \"Jan 1 1970\" instead of some random wrapped time).\n\nSo I think we should behave sanely in all cases (at least going forward\nfrom 1970 for now), and it's a quality-of-implementation issue whether\nany given system can pretty-print specific dates.\n\nSwitching that to intmax_t should get us pretty good coverage in the\nopposite direction, and anybody with a 64-bit signed time_t on their\nsystem will generally be happy.\n\n-Peff\n"},{"id":"382139","messageId":"7a9fdb7d-265f-a877-1a5d-619890910340@aueb.gr","threadId":"51825","inReplyTo":"20190910162629.GA12758@sigill.intra.peff.net","subject":"Re: Patching Git to handle dates before the Unix epoch","fromName":"Diomidis Spinellis","fromEmail":"dds@aueb.gr","sentAt":"2019-09-10T17:19:49Z","receivedAt":"2019-09-10T17:19:55Z","isPatch":false,"sender":{"key":"dds@aueb.gr","avatar":"https://avatars.githubusercontent.com/u/1212716?v=4"},"body":"On 10-Sep-19 19:26, Jeff King wrote:\n> On Tue, Sep 10, 2019 at 05:14:53PM +0300, Diomidis Spinellis wrote:\n>> As people use Git to create synthetic commits of code written in the past\n>> [1,2] it becomes important to handle dates before the Unix epoch (1/1/1970).\n>> I see that modern C libraries, Unix kernels, and tools can handle such\n>> dates.  However Git seems to mishandle such dates in several places, such as\n>> in date.c [3,4].  I'm planning to work on a fix, but before I embark on this\n>> I have a few questions.\n[...]\n>> - Can you think of any non-obvious issues (e.g. backward compatibility,\n>> switch to the Gregorian calendar) I should be aware of?\n> \n> The big question is: what will existing implementations do with a commit\n> object that contains a negative timestamp?\n\nNegative timestamps can already be created, because some Git libraries \ncan create such objects, and one can also create them by hand; see \nhttp://git.661346.n2.nabble.com/Back-dating-commits-way-back-for-constitution-git-tp5365303p5367657.html\n\nConsequently, I don't think that worrying about how existing clients \nwill handle such timestamps should be a big issue, as long as we don't \nbreak the handling of modern dates.\n\n > Coincidentally I was looking into this yesterday. See below.\n[...]\n> Here's how far I got working on this yesterday. It's quite messy, but\n> seems to work at least for basic tests like:\n\nAmazing!  Given that you have made significant progress, I think it's \nbest to leave it to you to complete it, right?\n\nDiomidis\n"},{"id":"382282","messageId":"20190913051203.GB28353@sigill.intra.peff.net","threadId":"51825","inReplyTo":"7a9fdb7d-265f-a877-1a5d-619890910340@aueb.gr","subject":"Re: Patching Git to handle dates before the Unix epoch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-09-13T05:12:03Z","receivedAt":"2019-09-13T05:12:06Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 10, 2019 at 08:19:49PM +0300, Diomidis Spinellis wrote:\n\n> > > - Can you think of any non-obvious issues (e.g. backward compatibility,\n> > > switch to the Gregorian calendar) I should be aware of?\n> > \n> > The big question is: what will existing implementations do with a commit\n> > object that contains a negative timestamp?\n> \n> Negative timestamps can already be created, because some Git libraries can\n> create such objects, and one can also create them by hand; see http://git.661346.n2.nabble.com/Back-dating-commits-way-back-for-constitution-git-tp5365303p5367657.html\n> \n> Consequently, I don't think that worrying about how existing clients will\n> handle such timestamps should be a big issue, as long as we don't break the\n> handling of modern dates.\n\nYes, it's easy to create lots of things with git-hash-object, but that\ndoesn't mean we should be encouraging people (yet). :) That said, I\nthink the handling here is pretty favorable: if you're not using very\nold timestamps, you won't see any problem. And if you do use them, the\nold presentation code handles it pretty well. We'd probably want to warn\npeople that older versions of Git may complain when fsck-ing.\n\n> > Here's how far I got working on this yesterday. It's quite messy, but\n> > seems to work at least for basic tests like:\n> \n> Amazing!  Given that you have made significant progress, I think it's best\n> to leave it to you to complete it, right?\n\nI was afraid you'd say that. :)\n\nI did spend a few more minutes looking into it after the last email.\nThere's a rabbit hole of handling timestamp parsing where many cases are\nnot actually checking appropriately for errors. So I think we'd want to\ndeal with those sites, too, while we're touching parse_timestamp().\n\nIt's not clear to me yet how deep the rabbit hole of little fixes goes\n(or how deep we need to go for this particular feature). So I hope to\nreturn to it when I have time, but if you or anybody else would like to\npoke at it in the meantime, be my guest.\n\n-Peff\n"}]}