v2, 7 partsDocumentation: fix warning in cat-texi.perl

There are two major processors of AsciiDoc: AsciiDoc itself, and Asciidoctor. Both have advantages and disadvantages, but traditionally the documentation has been built with AsciiDoc, leading to some surprising breakage when building with Asciidoctor. Partially, this is due to the need to specify a significant number of macros on the command line when building with Asciidoctor.

This series cleans up some issues building the documentation with Asciidoctor and provides two knobs, USE_ASCIIDOCTOR, which controls building with Asciidoctor, and ASCIIDOCTOR_EXTENSIONS_LAB, which controls the location of the Asciidoctor Extensions Lab, which is necessary to expand the linkgit macro.

The need for the extensions could be replaced with a small amount of Ruby code, if that's considered desirable. Previous opinions on doing so were negative, however.

In the process, I found several issues with cat-texi.perl, which have been fixed. It has also been modernized to use strict, warnings, and lexical file handles. I also made an attempt to produce more diffable texi files; I may follow up with additional series along this line to make the documentation build reproducibly.

Changes from v1:
* Fix a brown-paper-bag bug.

brian m. carlson (7):
  Documentation: fix warning in cat-texi.perl
  Documentation: modernize cat-texi.perl
  Documentation: remove unneeded argument in cat-texi.perl
  Documentation: sort sources for gitman.texi
  Documentation: add XSLT to fix DocBook for Texinfo
  Documentation: move dblatex arguments into variable
  Makefile: add a knob to enable the use of Asciidoctor

 Documentation/Makefile      | 22 ++++++++++++++++++----
 Documentation/cat-texi.perl | 21 ++++++++++++---------
 Documentation/texi.xsl      | 26 ++++++++++++++++++++++++++
 Makefile                    |  6 ++++++
 4 files changed, 62 insertions(+), 13 deletions(-)
 create mode 100644 Documentation/texi.xsl

Newer versions of Perl produce the warning "Unescaped left brace in regex is deprecated, passed through in regex" when an unescaped left brace occurs in a regex. Escape the brace to avoid this warning.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/cat-texi.perl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Sorting the sources makes it easier to compare the output using diff. In addition, it aids groups creating reproducible builds, as the order of the files is no longer dependent on the file system or other irrelevant factors.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

The newly-added use of the warnings pragma exposes that the $menu[0] argument to printf has long been silently ignored, since there is no format specifier for it. It doesn't appear that the argument is actually needed, either: there is no reason to insert the name of one particular documentation page anywhere in the header that's being generated.

Remove the unused argument, and since the format specification functionality is no longer needed, convert the printf to a simple print.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/cat-texi.perl | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Good style for Perl includes using the strict and warnings pragmas, and preferring lexical file handles over bareword file handles. Using lexical file handles necessitates being explicit when $_ is printed, so that Perl does not get confused and instead print the glob ref.

The benefit of this modernization is that a formerly obscured bug is now visible, which will be fixed in a followup patch.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/cat-texi.perl | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

Our dblatex invocation uses several style components from the AsciiDoc distribution, but those components are not available when building with Asciidoctor. Move the command line arguments into a variable so it can be overridden by the user or makefile configuration options.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

While Git has traditionally built its documentation using AsciiDoc, some people wish to use Asciidoctor for speed or other reasons. Add a Makefile knob, USE_ASCIIDOCTOR, that sets various options in order to produce acceptable output. For HTML output, XHTML5 was chosen, since the AsciiDoc options also produce XHTML, albeit XHTML 1.1.

Asciidoctor does not have built-in support for the linkgit macro, but it is available using the Asciidoctor Extensions Lab. Add a macro to enable the use of this extension if it is available. Without it, the linkgit macros are emitted into the output.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/Makefile | 12 ++++++++++++
 Makefile               |  6 ++++++
 2 files changed, 18 insertions(+)

On 2017-01-22 02:41:56, brian m. carlson wrote:

I applied and tested the patches on the current master, commit 787f75f0567a ("Sixth batch for 2.12"), and "make doc" with USE_ASCIIDOCTOR fails:

  $ git clean -fxd && make doc USE_ASCIIDOCTOR=1
  Removing Documentation/cmd-list.made
  Removing Documentation/cmds-ancillaryinterrogators.txt
  Removing Documentation/cmds-ancillarymanipulators.txt
  Removing Documentation/cmds-foreignscminterface.txt
  Removing Documentation/cmds-mainporcelain.txt
  Removing Documentation/cmds-plumbinginterrogators.txt
  Removing Documentation/cmds-plumbingmanipulators.txt
  Removing Documentation/cmds-purehelpers.txt
  Removing Documentation/cmds-synchelpers.txt
  Removing Documentation/cmds-synchingrepositories.txt
  Removing Documentation/doc.dep
  Removing Documentation/mergetools-diff.txt
  Removing Documentation/mergetools-list.made
  Removing Documentation/mergetools-merge.txt
  Removing GIT-VERSION-FILE
  GIT_VERSION = 2.11.0.460.g218feb5a0e89
  make -C Documentation all
  make[1]: Entering directory '/home/sunny/src/git/src-other/devel/git/git/Documentation'
      GEN mergetools-list.made
      GEN cmd-list.made
      GEN doc.dep
  make[2]: Entering directory '/home/sunny/src/git/src-other/devel/git/git'
  make[2]: 'GIT-VERSION-FILE' is up to date.
  make[2]: Leaving directory '/home/sunny/src/git/src-other/devel/git/git'
  make[2]: Entering directory '/home/sunny/src/git/src-other/devel/git/git'
  make[2]: 'GIT-VERSION-FILE' is up to date.
  make[2]: Leaving directory '/home/sunny/src/git/src-other/devel/git/git'
      ASCIIDOC git-init-db.html
  Couldn't find a view in @views for document
    Use --trace for backtrace
  Makefile:330: recipe for target 'git-init-db.html' failed
  make[1]: *** [git-init-db.html] Error 1
  make[1]: Leaving directory '/home/sunny/src/git/src-other/devel/git/git/Documentation'
  Makefile:2091: recipe for target 'doc' failed
  make: *** [doc] Error 2
  2017-01-23 03:50:05 sunny@sunbase:~/src/git/src-other/devel/git/git (tp-bmc-asciidoctor)

  $ lsb_release -d
  Description:    Debian GNU/Linux 8.7 (jessie)

  $ asciidoctor --version
  Asciidoctor 0.1.4 [http://asciidoctor.org]

I installed Asciidoctor with a standard "apt-get install asciidoctor", do I need to install more packages?

The build is broken by patch #7 ("Makefile: add a knob to enable the use of Asciidoctor"), the other commits seems to work, though I haven't tested them all individually yet. Standard "make doc" works.

Regards, Øyvind

+-| Øyvind A. Holm <sunny@sunbase.org> - N 60.37604° E 5.33339° |-+
| OpenPGP: 0xFB0CBEE894A506E5 - http://www.sunbase.org/pubkey.asc |
| Fingerprint: A006 05D6 E676 B319 55E2  E77E FB0C BEE8 94A5 06E5 |
+------------| 60bceb4e-e116-11e6-8fac-db5caa6d21d3 |-------------+

On Mon, Jan 23, 2017 at 03:57:13AM +0100, Øyvind A. Holm wrote:

I think you need a newer version of Asciidoctor. I fixed one or two issues upstream in 1.5.2, I think, that made it work properly.

You could try to do the build with the "html5" target instead of "xhtml5" and see if that works. If so, we could switch to that instead if we want to support older Asciidoctor versions.

On 2017-01-23 04:09:17, brian m. carlson wrote:

I've tried on Linux Mint 18 with Asciidoctor 1.5.4 now, and it works there, so the version is probably too old, yes.

It went a little better, but after a while it died with

  $ make doc USE_ASCIIDOCTOR=1
  [Cut 249 lines]
      GEN technical/api-index.txt
      ASCIIDOC technical/api-index.html
      ASCIIDOC git-init-db.xml
  sed "s|@@MAN_BASE_URL@@|file:///home/sunny/share/doc/git-doc/|" manpage-base-url.xsl.in > manpage-base-url.xsl
      XMLTO git-init-db.1
  xmlto: /home/sunny/src/git/src-other/devel/git/git/Documentation/git-init-db.xml does not validate (status 3)
  xmlto: Fix document syntax or use --skip-validation option
  /home/sunny/src/git/src-other/devel/git/git/Documentation/git-init-db.xml:5: element article: validity error : root and DTD name do not match 'article' and 'manpage'
  Document /home/sunny/src/git/src-other/devel/git/git/Documentation/git-init-db.xml does not validate
  Makefile:343: recipe for target 'git-init-db.1' failed
  make[1]: *** [git-init-db.1] Error 13
  make[1]: Leaving directory '/home/sunny/src/git/src-other/devel/git/git/Documentation'
  Makefile:2091: recipe for target 'doc' failed
  make: *** [doc] Error 2
  $

and that's fair enough, since the generated html isn't well-formed. Adding --skip-validation to XMLTO_EXTRA gave a slightly different result:

      GEN technical/api-index.txt
      ASCIIDOC technical/api-index.html
      ASCIIDOC git-init-db.xml
  sed "s|@@MAN_BASE_URL@@|file:///home/sunny/share/doc/git-doc/|" manpage-base-url.xsl.in > manpage-base-url.xsl
      XMLTO git-init-db.1
  Note: namesp. cut : stripped namespace before processing           git-init-db(1)
  Note: namesp. cut : processing stripped document                   git-init-db(1)
  Erro:  no refentry: No refentry elements found in "git-init-db(1)  git-init-db(1)
  Makefile:343: recipe for target 'git-init-db.1' failed
  make[1]: *** [git-init-db.1] Error 1
  make[1]: Leaving directory '/home/sunny/src/git/src-other/devel/git/git/Documentation'
  Makefile:2091: recipe for target 'doc' failed
  make: *** [doc] Error 2
  $

But frankly, this probably isn't a showstopper. Even though this is the newest stable version of Debian, Asciidoctor 0.1.4 was released 2013-09-05, 3y5m ago. USE_ASCIIDOCTOR isn't the default, so people can build the docs with asciidoc, and that works in Debian 8.7.

Regards, Øyvind

+-| Øyvind A. Holm <sunny@sunbase.org> - N 60.37604° E 5.33339° |-+
| OpenPGP: 0xFB0CBEE894A506E5 - http://www.sunbase.org/pubkey.asc |
| Fingerprint: A006 05D6 E676 B319 55E2  E77E FB0C BEE8 94A5 06E5 |
+------------| 1698e7f6-e257-11e6-bfa0-db5caa6d21d3 |-------------+

There are two ways to create a section in a reference document (i.e., manpage) in DocBook 4: refsection elements and refsect, refsect2, and refsect3 elements. Either form is acceptable as of DocBook 4.2, but they cannot be mixed. Prior to DocBook 4.2, only the numbered forms were acceptable.

docbook2texi only accepts the numbered forms, and this has not generally been a problem, since AsciiDoc produces the numbered forms. Asciidoctor, on the other hand, uses a shared backend for DocBook 4 and 5, and uses the unnumbered refsection elements instead.

If we don't convert the unnumbered form to the numbered form, docbook2texi omits section headings, which is undesirable. Add an XSLT stylesheet to transform the unnumbered forms to the numbered forms automatically, and preprocess the DocBook XML as part of the transformation to Texinfo format.

Note that this transformation is only necessary for Texinfo, since docbook2texi provides its own stylesheets. The DocBook stylesheets, which we use for other formats, provide the full range of DocBook 4 and 5 compatibility, and don't have this issue.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/Makefile |  7 ++++---
 Documentation/texi.xsl | 26 ++++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/texi.xsl

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

Thanks. We'd probably want INSTALL to talk about Asciidoctor once this matures, as it is very simple requirement for the builder to have to just set USE_ASCIIDOCTOR, but the version requirement and stuff might be still confusing.

Hi Brian,

On Sun, 22 Jan 2017, brian m. carlson wrote:

I like it.

I reviewed all the patches and think they are good (except the XSLT patch, which made me just feel incompetent because I do not know enough to have an opinion about it).

Quite frankly, it is annoying to be forced to install the extensions. I would much rather have the small amount of Ruby code in Git's repository.

Thanks, Johannes

On Wed, Jan 25, 2017 at 02:28:55PM +0100, Johannes Schindelin wrote:

Me too. Dependencies can be a big annoyance. I'd reserve judgement until I saw the actual Ruby code, though. :)

-Peff

On Wed, Jan 25, 2017 at 04:35:44PM -0500, Jeff King wrote:

I've sent the patch before, but I can send it again. It's relatively small and self-contained. I'm also happy to be responsible for maintaining it.

On Wed, Jan 25, 2017 at 11:19:26PM +0000, brian m. carlson wrote:

Ah, it's:

  http://public-inbox.org/git/1413070656-241955-5-git-send-email-sandals@crustytoothpaste.net/

(and note there is some surrounding discussion there).

The code is not _too_ bad. The main thing is that it would have to be kept up to date with changes to asciidoc.conf's version of the linkgit macro. But it's not like that changes frequently.

-Peff

On Wed, Jan 25, 2017 at 06:30:00PM -0500, Jeff King wrote:

Yes. I think I can actually simplify it some more, since we always seem to use the argument to linkgit, so I'll send out a simplified patch in a few minutes.

AsciiDoc uses a configuration file to implement macros like linkgit, while Asciidoctor uses Ruby extensions. Implement a Ruby extension that implements the linkgit macro for Asciidoctor in the same way that asciidoc.conf does for AsciiDoc. Adjust the Makefile to use it by default.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/Makefile                  |  5 +----
 Documentation/asciidoctor-extensions.rb | 28 ++++++++++++++++++++++++++++
 2 files changed, 29 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/asciidoctor-extensions.rb

On Thu, Jan 26, 2017 at 12:13:44AM +0000, brian m. carlson wrote:

Might be more readable to just leave the litdd part on its own line.

I think this looks reasonable. There's some boilerplate, but even as somebody not familiar with asciidoctor, it's all quite obvious.

The multi-line string is kind of ugly because of the indentation. Apparently Ruby has here-docs that will eat leading whitespace, but the syntax was not introduce until Ruby 2.3, which is probably more recent than we should count on.

I think you could write:

          %(<citerefentry>
            <refentrytitle>#{target}</refentrytitle><manvolnum>#{attrs[1]}</manvolnum>
            </citerefentry>
	  ).gsub(/^\s*/, "")

I don't know if that's too clever or not.

But either way, I like this better than introducing an extra dependency.

-Peff

Jeff King <peff@peff.net> wrote:

<snip>

You can use '\' to continue long lines with any Ruby version:

    "<citerefentry>" \
      "<refentrytitle>#{target}</refentrytitle>" \
      "<manvolnum>#{attrs[1]}</manvolnum>" \
    "</citerefentry>"

The above happens during the parse phase, so there's no garbage or method call overhead compared to the more-frequently seen '+' or '<<' method calls to combine strings.

Ick...

Agreed.

Hi Brian,

On Thu, 26 Jan 2017, brian m. carlson wrote:

I like it.

Thank you, Johannes

Junio C Hamano <gitster@pobox.com> wrote:

You need the '\' at the end of those strings, it's not like C since Ruby doesn't require semi-colons to terminate lines. In other words, that should be:

          "<citerefentry>\n" \
            "<refentrytitle>#{target}</refentrytitle>" \
            "<manvolnum>#{attrs[1]}</manvolnum>\n" \
          "</citerefentry>\n"

On Thu, Jan 26, 2017 at 07:18:41PM +0000, Eric Wong wrote:

This change is fine with me.

For the record, I don't have a strong opinion one way or the other. Since this code is related to Asciidoctor and Git has no existing Ruby style standards, I picked the Asciidoctor house style, which uses multi-line %(). We could pick [0] as an option, or just argue it out when someone cares, like here.

[0] https://github.com/bbatsov/ruby-style-guide

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

OK, I just squashed the final one in.  Will merge to 'next' shortly.