threads / discuss / 38455

implement a stable 'Last updated' in Documentation

Subject: implement a stable 'Last updated' in Documentation

## tl;dr

11 messages between Jan 26, 2015 and Sep 1, 2016.

replies: 10people: 4as markdown or json

Olaf Hering· Jan 26, 2015, 17:24 UTC · lore

Several files in Documentation have an unstable 'Last updated' timestamp. The reason is that their mtime changes every time, which prevents reproducible builds.

341 technical/api-index.txt: technical/api-index-skel.txt \ 342 technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS)) 343 $(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh

388 howto-index.txt: howto-index.sh $(wildcard howto/*.txt) 389 $(QUIET_GEN)$(RM) $@+ $@ && \ 390 '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \ 391 mv $@+ $@

399 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt 400 $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \ 401 sed -e '1,/^$$/d' $< | \ 402 $(TXT_TO_HTML) - >$@+ && \ 403 mv $@+ $@

What file timestamp should be used for them? Likely "../version"? The final file, before passing it to asciidoc, should get a fixed timestamp with 'touch -r $reference_file $file'.

Olaf
Olaf Hering· Jan 27, 2015, 11:11 UTC · re: Olaf Hering · lore

Re: implement a stable 'Last updated' in Documentation

On Mon, Jan 26, Olaf Hering wrote:
Show 22 quoted lines
> Several files in Documentation have an unstable 'Last updated' timestamp. The
> reason is that their mtime changes every time, which prevents reproducible
> builds.
> 
> 341 technical/api-index.txt: technical/api-index-skel.txt \
> 342         technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS))
> 343         $(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh
> 
> 388 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
> 389         $(QUIET_GEN)$(RM) $@+ $@ && \
> 390         '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
> 391         mv $@+ $@
> 
> 399 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
> 400         $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> 401         sed -e '1,/^$$/d' $< | \
> 402         $(TXT_TO_HTML) - >$@+ && \
> 403         mv $@+ $@
> 
> What file timestamp should be used for them? Likely "../version"?
> The final file, before passing it to asciidoc, should get a fixed timestamp
> with 'touch -r $reference_file $file'.

I came up with this change. If anything happens to depend on howto-index.txt and technical/api-index.txt then this part needs rework.

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 3e39e28..25ad0bd 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -6,6 +6,7 @@ TECH_DOCS =
 ARTICLES =
 SP_ARTICLES =
 OBSOLETE_HTML =
+TIMESTAMP_FILE = ../version
 
 MAN1_TXT += $(filter-out \
 		$(addsuffix .txt, $(ARTICLES) $(SP_ARTICLES)), \
@@ -341,6 +342,7 @@ user-manual.xml: user-manual.txt user-manual.conf
 technical/api-index.txt: technical/api-index-skel.txt \
 	technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS))
 	$(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh
+	touch -r '$(TIMESTAMP_FILE)' $@
 
 technical/%.html: ASCIIDOC_EXTRA += -a git-relative-html-prefix=../
 $(patsubst %,%.html,$(API_DOCS) technical/api-index $(TECH_DOCS)): %.html : %.txt asciidoc.conf
@@ -388,6 +390,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml
 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
 	$(QUIET_GEN)$(RM) $@+ $@ && \
 	'$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
+	touch -r '$(TIMESTAMP_FILE)' $@+ && \
 	mv $@+ $@
 
 $(patsubst %,%.html,$(ARTICLES)) : %.html : %.txt
@@ -397,9 +400,12 @@ WEBDOC_DEST = /pub/software/scm/git/docs
 
 howto/%.html: ASCIIDOC_EXTRA += -a git-relative-html-prefix=../
 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
-	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
-	sed -e '1,/^$$/d' $< | \
-	$(TXT_TO_HTML) - >$@+ && \
+	$(QUIET_ASCIIDOC)$(RM) $@++ $@+ $@ && \
+	sed -e '1,/^$$/d' $< >$@++ && \
+	touch -r '$(TIMESTAMP_FILE)' $@++ && \
+	$(TXT_TO_HTML) -o $@+ $@++ && \
+	$(RM) $@++ && \
+	touch $@+ && \
 	mv $@+ $@
 
 install-webdoc : html

Olaf
Junio C Hamano· Jan 27, 2015, 23:10 UTC · re: Olaf Hering · lore

Re: implement a stable 'Last updated' in Documentation

Olaf Hering <olaf@aepfle.de> writes:
Show 18 quoted lines
> Several files in Documentation have an unstable 'Last updated' timestamp. The
> reason is that their mtime changes every time, which prevents reproducible
> builds.
>
> 341 technical/api-index.txt: technical/api-index-skel.txt \
> 342         technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS))
> 343         $(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh
>
> 388 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
> 389         $(QUIET_GEN)$(RM) $@+ $@ && \
> 390         '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
> 391         mv $@+ $@
>
> 399 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
> 400         $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> 401         sed -e '1,/^$$/d' $< | \
> 402         $(TXT_TO_HTML) - >$@+ && \
> 403         mv $@+ $@

All of them seem to have dependencies so it seems to me that two builds back to back without actually changing their input would not re-build anything. What am I missing???

> What file timestamp should be used for them? Likely "../version"?

I tend to think the "Last updated" timestamp taken from the filesystem timestamp is a bad practice inherited by these tools from the days back when nobody used any revision control systems.

If I check out v1.8.5 and build documentation now, it does not matter if the generated documentation had the timestamp of the checkout of individual files or that of a single file generated during the build process, i.e. ../version. Neither has much relevance to the time the actual contents of the documentation was prepared or what vintage of the software the documentation is for.

I am fine with branding generated documentation with the version number (i.e. "You are reading the documentation for Git version 2.2.0"). Replacing that statement with "You are reading the documentation for a version of Git that was committed on such and such time" is also fine. But using file timestamp would not help either.

And that practice of using file timestamp is doubly bad by leading misguided people to want to set timestamps of all the files by copying commit timestamp. That is backwards, I would think.

Olaf Hering· Jan 28, 2015, 08:02 UTC · re: Junio C Hamano · lore

Re: implement a stable 'Last updated' in Documentation

On Tue, Jan 27, Junio C Hamano wrote:
Show 24 quoted lines
> Olaf Hering <olaf@aepfle.de> writes:
> 
> > Several files in Documentation have an unstable 'Last updated' timestamp. The
> > reason is that their mtime changes every time, which prevents reproducible
> > builds.
> >
> > 341 technical/api-index.txt: technical/api-index-skel.txt \
> > 342         technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS))
> > 343         $(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh
> >
> > 388 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
> > 389         $(QUIET_GEN)$(RM) $@+ $@ && \
> > 390         '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
> > 391         mv $@+ $@
> >
> > 399 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
> > 400         $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> > 401         sed -e '1,/^$$/d' $< | \
> > 402         $(TXT_TO_HTML) - >$@+ && \
> > 403         mv $@+ $@
> 
> All of them seem to have dependencies so it seems to me that two
> builds back to back without actually changing their input would not
> re-build anything.  What am I missing???

Per default the files referenced do not exist, so a build from releases will generate the files. But I agree, perhaps there should be some refactoring to handle dependencies properly. In any case, if the 'Last updated' is unavoidable some reference is required, see my patch in the other mail.

Show 5 quoted lines
> > What file timestamp should be used for them? Likely "../version"?
> 
> I tend to think the "Last updated" timestamp taken from the
> filesystem timestamp is a bad practice inherited by these tools from
> the days back when nobody used any revision control systems.

I'm not sure. The bug is that such 'Last updated' line exists at all in the default output. Noone asked for it, noone really needs it. And it makes it impossible to get reproducible builds. But so far I found no way to avoid the 'Last updated' output. The docs and google suggest something like :last-updated-time!:, which did not work for me.

Olaf
Junio C Hamano· Jan 29, 2015, 06:18 UTC · re: Olaf Hering · lore

Re: implement a stable 'Last updated' in Documentation

Olaf Hering <olaf@aepfle.de> writes:
Show 11 quoted lines
> On Tue, Jan 27, Junio C Hamano wrote:
>
>> > What file timestamp should be used for them? Likely "../version"?
>> 
>> I tend to think the "Last updated" timestamp taken from the
>> filesystem timestamp is a bad practice inherited by these tools from
>> the days back when nobody used any revision control systems.
>
> I'm not sure. The bug is that such 'Last updated' line exists at all in
> the default output. Noone asked for it, noone really needs it. And it
> makes it impossible to get reproducible builds.
Amen to that ;-)
Michael J Gruber· Jan 30, 2015, 10:05 UTC · re: Junio C Hamano · lore

Re: implement a stable 'Last updated' in Documentation

Junio C Hamano schrieb am 29.01.2015 um 07:18:
Show 16 quoted lines
> Olaf Hering <olaf@aepfle.de> writes:
> 
>> On Tue, Jan 27, Junio C Hamano wrote:
>>
>>>> What file timestamp should be used for them? Likely "../version"?
>>>
>>> I tend to think the "Last updated" timestamp taken from the
>>> filesystem timestamp is a bad practice inherited by these tools from
>>> the days back when nobody used any revision control systems.
>>
>> I'm not sure. The bug is that such 'Last updated' line exists at all in
>> the default output. Noone asked for it, noone really needs it. And it
>> makes it impossible to get reproducible builds.
> 
> Amen to that ;-)
>

It's a shame one can't simply replace the [footer-text] template which asciidoc insists on.

It turns out asciidoc 8.6.9-3 and later will habe a knob to turn:
https://github.com/asciidoc/asciidoc/pull/9

I'll try and get my hands on it to see whether we can simply use that. I'm wondering though which is more useful - the version of the tree the doc is processed from, or the version of the last commit changing the corresponding doc source file. The first one changes even when the doc source is unchanged (but is stable between reruns, of course).

Michael
Jeff King· Jan 30, 2015, 15:07 UTC · re: Michael J Gruber · lore

Re: implement a stable 'Last updated' in Documentation

On Fri, Jan 30, 2015 at 11:05:36AM +0100, Michael J Gruber wrote:
Show 12 quoted lines
> It's a shame one can't simply replace the [footer-text] template which
> asciidoc insists on.
> 
> It turns out asciidoc 8.6.9-3 and later will habe a knob to turn:
> 
> https://github.com/asciidoc/asciidoc/pull/9
> 
> I'll try and get my hands on it to see whether we can simply use that.
> I'm wondering though which is more useful - the version of the tree the
> doc is processed from, or the version of the last commit changing the
> corresponding doc source file. The first one changes even when the doc
> source is unchanged (but is stable between reruns, of course).

I have 8.6.9-3 installed (it is part of Debian testing/unstable now), and confirmed that:

diff --git a/Documentation/asciidoc.conf b/Documentation/asciidoc.conf
index 2c16c53..10c777e 100644
--- a/Documentation/asciidoc.conf
+++ b/Documentation/asciidoc.conf
@@ -21,6 +21,7 @@ tilde=&#126;
 apostrophe=&#39;
 backtick=&#96;
 litdd=&#45;&#45;
+footer-style=none
 
 ifdef::backend-docbook[]
 [linkgit-inlinemacro]

drops the "last-updated" footer.

But note that this only affects the generated HTML. The manpages still
get the date in their footer. But this isn't an asciidoc-ism at all;
it's added by docbook when converting the xml to roff. I'm sure there is
a way to tweak that, too, but looking at docbook gives me nightmares.

-Peff
Olaf Hering· Feb 10, 2015, 15:17 UTC · re: Jeff King · lore

Re: implement a stable 'Last updated' in Documentation

On Fri, Jan 30, Jeff King wrote:
Show 17 quoted lines
> I have 8.6.9-3 installed (it is part of Debian testing/unstable now),
> and confirmed that:
> 
> diff --git a/Documentation/asciidoc.conf b/Documentation/asciidoc.conf
> index 2c16c53..10c777e 100644
> --- a/Documentation/asciidoc.conf
> +++ b/Documentation/asciidoc.conf
> @@ -21,6 +21,7 @@ tilde=&#126;
>  apostrophe=&#39;
>  backtick=&#96;
>  litdd=&#45;&#45;
> +footer-style=none
>  
>  ifdef::backend-docbook[]
>  [linkgit-inlinemacro]
> 
> drops the "last-updated" footer.

This does not remove "Last updated" from files like using-merge-subtree.html for me, using asciidoc-8.6.8.

Olaf
Jeff King· Feb 10, 2015, 18:43 UTC · re: Olaf Hering · lore

Re: implement a stable 'Last updated' in Documentation

On Tue, Feb 10, 2015 at 04:17:47PM +0100, Olaf Hering wrote:
Show 22 quoted lines
> On Fri, Jan 30, Jeff King wrote:
> 
> > I have 8.6.9-3 installed (it is part of Debian testing/unstable now),
> > and confirmed that:
> > 
> > diff --git a/Documentation/asciidoc.conf b/Documentation/asciidoc.conf
> > index 2c16c53..10c777e 100644
> > --- a/Documentation/asciidoc.conf
> > +++ b/Documentation/asciidoc.conf
> > @@ -21,6 +21,7 @@ tilde=&#126;
> >  apostrophe=&#39;
> >  backtick=&#96;
> >  litdd=&#45;&#45;
> > +footer-style=none
> >  
> >  ifdef::backend-docbook[]
> >  [linkgit-inlinemacro]
> > 
> > drops the "last-updated" footer.
> 
> This does not remove "Last updated" from files like
> using-merge-subtree.html for me, using asciidoc-8.6.8.
Yes, the feature is part of 8.6.9-3.
-Peff
Olaf Hering· Sep 1, 2016, 14:37 UTC · re: Jeff King · lore

Re: implement a stable 'Last updated' in Documentation

Hey, asciidoc made a move, so this patch is good to go: https://github.com/asciidoc/asciidoc/pull/9

Thanks.
Olaf
On Tue, Feb 10, Jeff King wrote:
Show 28 quoted lines
> On Tue, Feb 10, 2015 at 04:17:47PM +0100, Olaf Hering wrote:
> 
> > On Fri, Jan 30, Jeff King wrote:
> > 
> > > I have 8.6.9-3 installed (it is part of Debian testing/unstable now),
> > > and confirmed that:
> > > 
> > > diff --git a/Documentation/asciidoc.conf b/Documentation/asciidoc.conf
> > > index 2c16c53..10c777e 100644
> > > --- a/Documentation/asciidoc.conf
> > > +++ b/Documentation/asciidoc.conf
> > > @@ -21,6 +21,7 @@ tilde=&#126;
> > >  apostrophe=&#39;
> > >  backtick=&#96;
> > >  litdd=&#45;&#45;
> > > +footer-style=none
> > >  
> > >  ifdef::backend-docbook[]
> > >  [linkgit-inlinemacro]
> > > 
> > > drops the "last-updated" footer.
> > 
> > This does not remove "Last updated" from files like
> > using-merge-subtree.html for me, using asciidoc-8.6.8.
> 
> Yes, the feature is part of 8.6.9-3.
> 
> -Peff
Jeff King· Sep 1, 2016, 20:17 UTC · re: Olaf Hering · lore

Re: implement a stable 'Last updated' in Documentation

On Thu, Sep 01, 2016 at 04:37:36PM +0200, Olaf Hering wrote:
> Hey, asciidoc made a move, so this patch is good to go:
> https://github.com/asciidoc/asciidoc/pull/9

Sine this thread is 18 months old, I needed some recap to remember what we were talking about. :)

It's here:
  http://public-inbox.org/git/20150126172409.GA15204@aepfle.de/T/#u

and the gist of it is that we'd like to drop the "Last updated" footer from the HTML version of the manpages, but older versions of asciidoc did not provide a mechanism.

The patch you quoted adds "footer-style=none", which would do the trick. But I have two open questions:

  1. What does this do on older versions of asciidoc? Is it silently
     ignored (ok), or does it generate an error (bad)?
  2. This covers the HTML versions, but not the roff manpages (which
     are generated by docbook). Do we have a way to tweak the date in
     the latter?
     I don't think that's necessarily a requirement for this patch, but
     it is worth thinking about at the same time.

Assuming the answer to (1) is "ok" and (2) is "no, but it's hard because docbook is scary, so let's punt", then somebody needs to write up the commit message and send the actual patch to the list. Would you like to try that?

-Peff

← back to recent threads