patch, 3 partsImproving build reproducibility

Hi,

The reproducible builds <https://reproducible-builds.org/> project has been working on making it possible to verify that binary packages of open source projects were built from the source they were claimed to have been built from.

To that end, Debian has been carrying patches 1-2 for a while. Patch 3 is a Google-internal patch with a related but distinct goal of making builds go faster.

I think these should be ready to apply.  Thoughts of all kinds welcome.

Sincerely,
Anders Kaseorg (2):
  Documentation: allow overriding timestamps of generated asciidoc
  git-gui: sort entries in optimized tclIndex

Jonathan Nieder (1):
  generate-cmdlist: avoid non-deterministic output

 Documentation/Makefile               | 7 +++++--
 Documentation/technical/api-index.sh | 5 +++++
 generate-cmdlist.sh                  | 2 +-
 git-gui/Makefile                     | 2 +-
 4 files changed, 12 insertions(+), 4 deletions(-)

From: Anders Kaseorg <andersk@mit.edu>
Date: Wed, 30 Nov 2016 22:21:15 -0500

Allow overriding the timestamp in generated documentation by setting SOURCE_DATE_EPOCH to the number of seconds since 1970-01-01 00:00:00 UTC to use.

This makes the generated documentation reproducible from the source code as long as that variable is set, without losing the last-modified dates in the default build.

With this change, the package passes Debian's build reproducibility test (https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal).

The goal is to make it easier to verify that binary packages of open source projects were built from the source they were claimed to have been built from. https://reproducible-builds.org/ has more details.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Perhaps this should e.g. be taking the latest timestamp of all its
inputs.  That would be straightforward to do, but what's here is what
we've been running with for the past year, so I'd rather stick to it,
at least as a starting point.

Another tweak I'd be interested in is allowing asciidoc to take the timestamp as a parameter instead of inferring it from mtimes. Asciidoc accepts an "--attribute footer-style=none" parameter, but I'm not aware of an "--attribute footer-date=<foo>" parameter to keep the footer but override its date.

 Documentation/Makefile               | 7 +++++--
 Documentation/technical/api-index.sh | 5 +++++
 2 files changed, 10 insertions(+), 2 deletions(-)

On Tue, Nov 21, 2017 at 03:34:32PM -0800, Jonathan Nieder wrote:

Thanks for this. I had planned on either submitting this patch myself, or working on a similar one, but I ran into the issue I'll mention below and hadn't finished looking at it.

My research on this determined that Asciidoctor 1.5.5 and newer handle this properly, because they honor SOURCE_BUILD_EPOCH. It's only with AsciiDoc that this is an issue.

touch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't provide it. POSIX only allows certain fixed format, and I assume that non-Linux parties (maybe OpenBSD) will want to have reproducible builds as well.

It's unfortunate for shell users that this variable is in seconds from the epoch, since there's no portable way to format such a time in shell. (POSIX doesn't allow date(1) to format anything but the current time.)

My proposed solution was to use Perl to do so, and simply require that if you wanted a reproducible build, then you had to have Perl. That would, of course, require a separate variable in the Makefile holding the formatted date.

Maybe something like the following in the Makefile:

ifndef NO_PERL SOURCE_DATE_TIMESTAMP=$(shell perl -MPOSIX -e 'print strftime("%FT%TZ", gmtime($ENV{SOURCE_DATE_EPOCH}));') endif

and then:

+	$(if $(SOURCE_DATE_TIMESTAMP),touch -d '$(SOURCE_DATE_TIMESTAMP)' $@+ &&) \

Hi,

brian m. carlson wrote:

Interesting. My knee-jerk preference is still to go with this patch as-is for now, since the non-portability only triggers when SOURCE_DATE_EPOCH is set.

Neat.  I can play with this a little.

http://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems to be getting changes at https://github.com/asciidoc/asciidoc. I wonder how difficult it would be to add any required SOURCE_DATE_EPOCH support there.

Longer term, I wonder what it would take to move to a markup language that is more widely known, like commonmark.

Thanks, Jonathan

Jonathan Nieder <jrnieder@gmail.com> writes:

As long as this patch is kept Debian-only, that is a sensible stance to take. I am not sure (note: this is different from "I do not think") if it is also OK for the wider public, though.

I wondered if this affects the dirtyness of the build, regardless of how file timestamps are mucked with. It turns out that we do not use "describe --dirty" in the GIT-VERSION-GEN script, so perhaps it would be OK.

On Tue, 21 Nov 2017, Jonathan Nieder wrote:

In fact I already did (https://github.com/asciidoc/asciidoc/pull/106), which is why I’d been holding off on trying to upstream this Git patch. The trouble was, the AsciiDoc developers had not been cutting new releases “because nobody knows how” (https://github.com/asciidoc/asciidoc/issues/103#issuecomment-322077321). However, it looks like AsciiDoc 8.6.10 was recently tagged and Debian got a 8.6.10-1 package yesterday, so I guess that trouble has been quietly resolved.

That should make this Git patch unnecessary. (You’re of course still welcome to take it if you think build reproducibility with old AsciiDoc versions is worthwhile.)

Anders

Anders Kaseorg <andersk@mit.edu> writes:

Thanks.  

I've queued these three only so that I won't lose track, but will not hastily merge them down (yet) until I hear from people.

Anders Kaseorg wrote:

Ah, lovely. I'll add a build-time dependency on that version to the Debian package.

Junio, please disregard this patch (patch 1/3).

Thanks, Jonathan

From: Anders Kaseorg <andersk@mit.edu>
Date: Wed, 16 Nov 2016 16:37:17 -0500

auto_mkindex expands wildcards in directory order, which depends on the underlying filesystem. To improve build reproducibility, sort the list of *.tcl files in the Makefile.

The unoptimized loading case was previously fixed in gitgui-0.21.0~14 (git-gui: sort entries in tclIndex, 2015-01-26).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Date: Fri, 1 Jul 2016 17:32:00 -0700

Non-determinism makes it harder for build tools to discover when a target needs to be rebuilt.

generate-cmdlist.sh stores the full path in a comment:

 /* Automatically generated by /build/git-agojiD/git-2.15.0/generate-cmdlist.sh */

Use the file name alone instead.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
That's the end of the series.  Thanks for reading.

 generate-cmdlist.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)