{"thread":{"id":"47290","subject":"[PATCH 0/3] Improving build reproducibility","startedAt":"2017-11-21T23:29:42Z","lastAt":"2017-11-22T05:12:20Z","messageCount":10,"participants":["Jonathan Nieder","brian m. carlson","Junio C Hamano","Anders Kaseorg"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"333193","messageId":"20171121232935.GM3429@aiede.mtv.corp.google.com","threadId":"47290","inReplyTo":null,"subject":"[PATCH 0/3] Improving build reproducibility","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-11-21T23:29:35Z","receivedAt":"2017-11-21T23:29:42Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nThe reproducible builds <https://reproducible-builds.org/> project has\nbeen working on making it possible to verify that binary packages of\nopen source projects were built from the source they were claimed to\nhave been built from.\n\nTo that end, Debian has been carrying patches 1-2 for a while.  Patch\n3 is a Google-internal patch with a related but distinct goal of\nmaking builds go faster.\n\nI think these should be ready to apply.  Thoughts of all kinds welcome.\n\nSincerely,\nAnders Kaseorg (2):\n  Documentation: allow overriding timestamps of generated asciidoc\n  git-gui: sort entries in optimized tclIndex\n\nJonathan Nieder (1):\n  generate-cmdlist: avoid non-deterministic output\n\n Documentation/Makefile               | 7 +++++--\n Documentation/technical/api-index.sh | 5 +++++\n generate-cmdlist.sh                  | 2 +-\n git-gui/Makefile                     | 2 +-\n 4 files changed, 12 insertions(+), 4 deletions(-)\n"},{"id":"333194","messageId":"20171121233432.GN3429@aiede.mtv.corp.google.com","threadId":"47290","inReplyTo":"20171121232935.GM3429@aiede.mtv.corp.google.com","subject":"[PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-11-21T23:34:32Z","receivedAt":"2017-11-21T23:34:41Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"From: Anders Kaseorg <andersk@mit.edu>\nDate: Wed, 30 Nov 2016 22:21:15 -0500\n\nAllow overriding the timestamp in generated documentation by setting\nSOURCE_DATE_EPOCH to the number of seconds since 1970-01-01 00:00:00\nUTC to use.\n\nThis makes the generated documentation reproducible from the source\ncode as long as that variable is set, without losing the last-modified\ndates in the default build.\n\nWith this change, the package passes Debian's build reproducibility\ntest (https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal).\n\nThe goal is to make it easier to verify that binary packages of open\nsource projects were built from the source they were claimed to have\nbeen built from.  https://reproducible-builds.org/ has more details.\n\nSigned-off-by: Anders Kaseorg <andersk@mit.edu>\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nPerhaps this should e.g. be taking the latest timestamp of all its\ninputs.  That would be straightforward to do, but what's here is what\nwe've been running with for the past year, so I'd rather stick to it,\nat least as a starting point.\n\nAnother tweak I'd be interested in is allowing asciidoc to take the\ntimestamp as a parameter instead of inferring it from mtimes.\nAsciidoc accepts an \"--attribute footer-style=none\" parameter, but I'm\nnot aware of an \"--attribute footer-date=<foo>\" parameter to keep the\nfooter but override its date.\n\n Documentation/Makefile               | 7 +++++--\n Documentation/technical/api-index.sh | 5 +++++\n 2 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 2ab65561af..dfec29f36f 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -410,6 +410,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml\n howto-index.txt: howto-index.sh $(wildcard howto/*.txt)\n \t$(QUIET_GEN)$(RM) $@+ $@ && \\\n \t'$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \\\n+\t$(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $@+ &&) \\\n \tmv $@+ $@\n \n $(patsubst %,%.html,$(ARTICLES)) : %.html : %.txt\n@@ -420,8 +421,10 @@ WEBDOC_DEST = /pub/software/scm/git/docs\n howto/%.html: ASCIIDOC_EXTRA += -a git-relative-html-prefix=../\n $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt\n \t$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \\\n-\tsed -e '1,/^$$/d' $< | \\\n-\t$(TXT_TO_HTML) - >$@+ && \\\n+\tsed -e '1,/^$$/d' $< > $<+ && \\\n+\t$(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $<+ &&) \\\n+\t$(TXT_TO_HTML) -o $@+ $<+ && \\\n+\trm $<+ && \\\n \tmv $@+ $@\n \n install-webdoc : html\ndiff --git a/Documentation/technical/api-index.sh b/Documentation/technical/api-index.sh\nindex 9c3f4131b8..07b3909627 100755\n--- a/Documentation/technical/api-index.sh\n+++ b/Documentation/technical/api-index.sh\n@@ -20,6 +20,11 @@\n \tsed -n -e '/^\\/\\/ table of contents end/,$p' \"$skel\"\n ) >api-index.txt+\n \n+if test \"${SOURCE_DATE_EPOCH:+set}\"\n+then\n+\ttouch -d \"@$SOURCE_DATE_EPOCH\" api-index.txt+\n+fi\n+\n if test -f api-index.txt && cmp api-index.txt api-index.txt+ >/dev/null\n then\n \trm -f api-index.txt+\n-- \n2.15.0.448.gf294e3d99a\n\n"},{"id":"333196","messageId":"20171121233621.GO3429@aiede.mtv.corp.google.com","threadId":"47290","inReplyTo":"20171121232935.GM3429@aiede.mtv.corp.google.com","subject":"[PATCH 2/3] git-gui: sort entries in optimized tclIndex","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-11-21T23:36:21Z","receivedAt":"2017-11-21T23:36:29Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"From: Anders Kaseorg <andersk@mit.edu>\nDate: Wed, 16 Nov 2016 16:37:17 -0500\n\nauto_mkindex expands wildcards in directory order, which depends on\nthe underlying filesystem.  To improve build reproducibility, sort the\nlist of *.tcl files in the Makefile.\n\nThe unoptimized loading case was previously fixed in gitgui-0.21.0~14\n(git-gui: sort entries in tclIndex, 2015-01-26).\n\nSigned-off-by: Anders Kaseorg <andersk@mit.edu>\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\n Makefile | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 918a8de369..f10caedaa7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -254,7 +254,7 @@ $(ALL_MSGFILES): %.msg : %.po\n lib/tclIndex: $(ALL_LIBFILES) GIT-GUI-VARS\n \t$(QUIET_INDEX)if echo \\\n \t  $(foreach p,$(PRELOAD_FILES),source $p\\;) \\\n-\t  auto_mkindex lib '*.tcl' \\\n+\t  auto_mkindex lib $(patsubst lib/%,%,$(sort $(ALL_LIBFILES))) \\\n \t| $(TCL_PATH) $(QUIET_2DEVNULL); then : ok; \\\n \telse \\\n \t echo >&2 \"    * $(TCL_PATH) failed; using unoptimized loading\"; \\\n-- \n2.15.0.448.gf294e3d99a\n\n"},{"id":"333197","messageId":"20171121233813.GP3429@aiede.mtv.corp.google.com","threadId":"47290","inReplyTo":"20171121232935.GM3429@aiede.mtv.corp.google.com","subject":"[PATCH 3/3] generate-cmdlist: avoid non-deterministic output","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-11-21T23:38:13Z","receivedAt":"2017-11-21T23:38:21Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Date: Fri, 1 Jul 2016 17:32:00 -0700\n\nNon-determinism makes it harder for build tools to discover when a\ntarget needs to be rebuilt.\n\ngenerate-cmdlist.sh stores the full path in a comment:\n\n /* Automatically generated by /build/git-agojiD/git-2.15.0/generate-cmdlist.sh */\n\nUse the file name alone instead.\n\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\nThat's the end of the series.  Thanks for reading.\n\n generate-cmdlist.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/generate-cmdlist.sh b/generate-cmdlist.sh\nindex ab0d1b0c06..eeea4b67ea 100755\n--- a/generate-cmdlist.sh\n+++ b/generate-cmdlist.sh\n@@ -1,6 +1,6 @@\n #!/bin/sh\n \n-echo \"/* Automatically generated by $0 */\n+echo \"/* Automatically generated by generate-cmdlist.sh */\n struct cmdname_help {\n \tchar name[16];\n \tchar help[80];\n-- \n2.15.0.448.gf294e3d99a\n\n"},{"id":"333206","messageId":"20171122005433.kwv4bin6y65xuxm5@genre.crustytoothpaste.net","threadId":"47290","inReplyTo":"20171121233432.GN3429@aiede.mtv.corp.google.com","subject":"Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2017-11-22T00:54:33Z","receivedAt":"2017-11-22T00:54:46Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Tue, Nov 21, 2017 at 03:34:32PM -0800, Jonathan Nieder wrote:\n> From: Anders Kaseorg <andersk@mit.edu>\n> Date: Wed, 30 Nov 2016 22:21:15 -0500\n> \n> Allow overriding the timestamp in generated documentation by setting\n> SOURCE_DATE_EPOCH to the number of seconds since 1970-01-01 00:00:00\n> UTC to use.\n> \n> This makes the generated documentation reproducible from the source\n> code as long as that variable is set, without losing the last-modified\n> dates in the default build.\n\nThanks for this.  I had planned on either submitting this patch myself,\nor working on a similar one, but I ran into the issue I'll mention\nbelow and hadn't finished looking at it.\n\nMy research on this determined that Asciidoctor 1.5.5 and newer handle\nthis properly, because they honor SOURCE_BUILD_EPOCH.  It's only with\nAsciiDoc that this is an issue.\n\n> diff --git a/Documentation/Makefile b/Documentation/Makefile\n> index 2ab65561af..dfec29f36f 100644\n> --- a/Documentation/Makefile\n> +++ b/Documentation/Makefile\n> @@ -410,6 +410,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml\n>  howto-index.txt: howto-index.sh $(wildcard howto/*.txt)\n>  \t$(QUIET_GEN)$(RM) $@+ $@ && \\\n>  \t'$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \\\n> +\t$(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $@+ &&) \\\n\ntouch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't\nprovide it.  POSIX only allows certain fixed format, and I assume that\nnon-Linux parties (maybe OpenBSD) will want to have reproducible builds\nas well.\n\nIt's unfortunate for shell users that this variable is in seconds from\nthe epoch, since there's no portable way to format such a time in shell.\n(POSIX doesn't allow date(1) to format anything but the current time.)\n\nMy proposed solution was to use Perl to do so, and simply require that\nif you wanted a reproducible build, then you had to have Perl.  That\nwould, of course, require a separate variable in the Makefile holding\nthe formatted date.\n\nMaybe something like the following in the Makefile:\n\nifndef NO_PERL\nSOURCE_DATE_TIMESTAMP=$(shell perl -MPOSIX -e 'print strftime(\"%FT%TZ\", gmtime($ENV{SOURCE_DATE_EPOCH}));')\nendif\n\nand then:\n\n+\t$(if $(SOURCE_DATE_TIMESTAMP),touch -d '$(SOURCE_DATE_TIMESTAMP)' $@+ &&) \\\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\nhttps://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"333209","messageId":"20171122011531.GQ3429@aiede.mtv.corp.google.com","threadId":"47290","inReplyTo":"20171122005433.kwv4bin6y65xuxm5@genre.crustytoothpaste.net","subject":"Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-11-22T01:15:31Z","receivedAt":"2017-11-22T01:15:39Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nbrian m. carlson wrote:\n> On Tue, Nov 21, 2017 at 03:34:32PM -0800, Jonathan Nieder wrote:\n\n>> --- a/Documentation/Makefile\n>> +++ b/Documentation/Makefile\n>> @@ -410,6 +410,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml\n>>  howto-index.txt: howto-index.sh $(wildcard howto/*.txt)\n>>  \t$(QUIET_GEN)$(RM) $@+ $@ && \\\n>>  \t'$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \\\n>> +\t$(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $@+ &&) \\\n>\n> touch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't\n> provide it.  POSIX only allows certain fixed format, and I assume that\n> non-Linux parties (maybe OpenBSD) will want to have reproducible builds\n> as well.\n\nInteresting.  My knee-jerk preference is still to go with this patch\nas-is for now, since the non-portability only triggers when\nSOURCE_DATE_EPOCH is set.\n\n> It's unfortunate for shell users that this variable is in seconds from\n> the epoch, since there's no portable way to format such a time in shell.\n> (POSIX doesn't allow date(1) to format anything but the current time.)\n>\n> My proposed solution was to use Perl to do so, and simply require that\n> if you wanted a reproducible build, then you had to have Perl.  That\n> would, of course, require a separate variable in the Makefile holding\n> the formatted date.\n>\n> Maybe something like the following in the Makefile:\n>\n> ifndef NO_PERL\n> SOURCE_DATE_TIMESTAMP=$(shell perl -MPOSIX -e 'print strftime(\"%FT%TZ\", gmtime($ENV{SOURCE_DATE_EPOCH}));')\n> endif\n>\n> and then:\n>\n> +\t$(if $(SOURCE_DATE_TIMESTAMP),touch -d '$(SOURCE_DATE_TIMESTAMP)' $@+ &&) \\\n\nNeat.  I can play with this a little.\n\nhttp://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems\nto be getting changes at https://github.com/asciidoc/asciidoc.  I\nwonder how difficult it would be to add any required SOURCE_DATE_EPOCH\nsupport there.\n\nLonger term, I wonder what it would take to move to a markup language\nthat is more widely known, like commonmark.\n\nThanks,\nJonathan\n"},{"id":"333226","messageId":"xmqqlgizvxg4.fsf@gitster.mtv.corp.google.com","threadId":"47290","inReplyTo":"20171122011531.GQ3429@aiede.mtv.corp.google.com","subject":"Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-22T02:50:19Z","receivedAt":"2017-11-22T02:50:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n>> touch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't\n>> provide it.  POSIX only allows certain fixed format, and I assume that\n>> non-Linux parties (maybe OpenBSD) will want to have reproducible builds\n>> as well.\n>\n> Interesting.  My knee-jerk preference is still to go with this patch\n> as-is for now, since the non-portability only triggers when\n> SOURCE_DATE_EPOCH is set.\n\nAs long as this patch is kept Debian-only, that is a sensible stance\nto take.  I am not sure (note: this is different from \"I do not think\")\nif it is also OK for the wider public, though.\n\nI wondered if this affects the dirtyness of the build, regardless of\nhow file timestamps are mucked with.  It turns out that we do not\nuse \"describe --dirty\" in the GIT-VERSION-GEN script, so perhaps\nit would be OK.\n\n\n"},{"id":"333237","messageId":"alpine.DEB.2.10.1711212317440.27951@mass-toolpike.mit.edu","threadId":"47290","inReplyTo":"20171122011531.GQ3429@aiede.mtv.corp.google.com","subject":"Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc","fromName":"Anders Kaseorg","fromEmail":"andersk@mit.edu","sentAt":"2017-11-22T04:30:27Z","receivedAt":"2017-11-22T04:35:47Z","isPatch":true,"sender":{"key":"andersk@mit.edu","avatar":"https://avatars.githubusercontent.com/u/26471?v=4"},"body":"On Tue, 21 Nov 2017, Jonathan Nieder wrote:\n> http://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems\n> to be getting changes at https://github.com/asciidoc/asciidoc.  I\n> wonder how difficult it would be to add any required SOURCE_DATE_EPOCH\n> support there.\n\nIn fact I already did (https://github.com/asciidoc/asciidoc/pull/106), \nwhich is why I’d been holding off on trying to upstream this Git patch.  \nThe trouble was, the AsciiDoc developers had not been cutting new releases \n“because nobody knows how” \n(https://github.com/asciidoc/asciidoc/issues/103#issuecomment-322077321). \nHowever, it looks like AsciiDoc 8.6.10 was recently tagged and Debian got \na 8.6.10-1 package yesterday, so I guess that trouble has been quietly \nresolved.\n\nThat should make this Git patch unnecessary.  (You’re of course still \nwelcome to take it if you think build reproducibility with old AsciiDoc \nversions is worthwhile.)\n\nAnders\n"},{"id":"333242","messageId":"xmqq7euivr38.fsf@gitster.mtv.corp.google.com","threadId":"47290","inReplyTo":"alpine.DEB.2.10.1711212317440.27951@mass-toolpike.mit.edu","subject":"Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-22T05:07:39Z","receivedAt":"2017-11-22T05:07:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Anders Kaseorg <andersk@mit.edu> writes:\n\n> That should make this Git patch unnecessary.  (You’re of course still \n> welcome to take it if you think build reproducibility with old AsciiDoc \n> versions is worthwhile.)\n\nThanks.  \n\nI've queued these three only so that I won't lose track, but will\nnot hastily merge them down (yet) until I hear from people.\n\n"},{"id":"333244","messageId":"20171122051213.GB7625@aiede.mtv.corp.google.com","threadId":"47290","inReplyTo":"alpine.DEB.2.10.1711212317440.27951@mass-toolpike.mit.edu","subject":"Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-11-22T05:12:13Z","receivedAt":"2017-11-22T05:12:20Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Anders Kaseorg wrote:\n> On Tue, 21 Nov 2017, Jonathan Nieder wrote:\n\n>> http://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems\n>> to be getting changes at https://github.com/asciidoc/asciidoc.  I\n>> wonder how difficult it would be to add any required SOURCE_DATE_EPOCH\n>> support there.\n>\n> In fact I already did (https://github.com/asciidoc/asciidoc/pull/106),\n> which is why I’d been holding off on trying to upstream this Git patch.\n> The trouble was, the AsciiDoc developers had not been cutting new releases\n> “because nobody knows how”\n> (https://github.com/asciidoc/asciidoc/issues/103#issuecomment-322077321).\n> However, it looks like AsciiDoc 8.6.10 was recently tagged and Debian got\n> a 8.6.10-1 package yesterday, so I guess that trouble has been quietly\n> resolved.\n\nAh, lovely.  I'll add a build-time dependency on that version to the\nDebian package.\n\nJunio, please disregard this patch (patch 1/3).\n\nThanks,\nJonathan\n"}]}