patchasciidoctor-extensions: provide `<refmiscinfo/>`

When we build with AsciiDoc, asciidoc.conf ensures that each xml-file we generate contains some meta-information which `xmlto` can act on, based on the following template:

  <refmeta>
  <refentrytitle>{mantitle}</refentrytitle>
  <manvolnum>{manvolnum}</manvolnum>
  <refmiscinfo class="source">Git</refmiscinfo>
  <refmiscinfo class="version">{git_version}</refmiscinfo>
  <refmiscinfo class="manual">Git Manual</refmiscinfo>
  </refmeta>

When we build with Asciidoctor, it does not honor this configuration file and we end up with only this (for a hypothetical git-foo.xml):

  <refmeta>
  <refentrytitle>git-foo</refentrytitle>
  <manvolnum>1</manvolnum>
  </refmeta>

That is, we miss out on the `<refmiscinfo/>` tags. As a result, the header of each man page doesn't say "Git Manual", but "git-foo(1)" instead. Worse, the footers don't give the Git version number and instead provide the fairly ugly "[FIXME: source]".

That Asciidoctor ignores asciidoc.conf is nothing new. This is why we implement the `linkgit:` macro in asciidoc.conf *and* in asciidoctor-extensions.rb. Follow suit and provide these tags in asciidoctor-extensions.rb, using a "postprocessor" extension.

We may consider a few alternatives:

  * Provide the `mansource` attribute to Asciidoctor. This attribute
    looks promising until one realizes that it can only be given inside
    the source file (the .txt file in our case), *not* on the command
    line using `-a mansource=foobar`. I toyed with the idea of injecting
    this attribute while feeding Asciidoctor the input on stdin, but it
    didn't feel like it was worth the complexity in the Makefile.

  * Similar to that last idea, we could inject these lines into the
    xml-files from the Makefile, e.g., using `sed`. This reduces
    repetition, but seems fairly brittle. It feels wrong to impose
    another step and another risk on the Asciidoc-processing only to
    benefit the Asciidoctor-one.

  * Considering the above abandoned ideas, it seems better to put any
    complexity inside asciidoctor-extensions.rb. It is after all
    supposed to be the "equivalent" of asciidoc.conf. I considered
    providing a "tree processor" extension and use it to set, e.g.,
    `mansource` mentioned above.

Let's instead try to stay as close as possible to what asciidoc.conf does. We'll make it fairly obvious that we aim to inject the exact same three lines of `<refmiscinfo/>` that asciidoc.conf provides. The only somewhat tricky part is that we inject them *post*-processing so we need to do the variable expansion ourselves.

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 Cc Todd and Peff who had a brief exchange [1] a while ago. Apparently
 Todd saw this "[FIXME: source]" on Fedora but Peff did not on Debian.
 I'm on Ubuntu 18.04 and used to see this also on 16.04. I'm not sure
 what might make Debian so special here.

 [1] https://public-inbox.org/git/20180627164443.GK20217@zaya.teonanacatl.net/

 I've based this on ma/asciidoctor-fixes, not because it's needed for
 the patch itself, but because it provides a15ef383e7
 ("Documentation/Makefile: add missing dependency on
 asciidoctor-extensions", 2019-02-27), which ensures that the
 documentation will be rebuilt. I'm hoping this was the right call.

 Documentation/asciidoctor-extensions.rb | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

Hi Martin,

Martin Ågren wrote:

Nice! I looked at this a long time ago and didn't figure out how to use a postprocessor extension. From my notes, I found discussions about using ruby's tilt for templating and it all seemed way too convoluted.

Your method looks quite simple and elegant.

I played with this direction before. Using Asciidoctor we can convert directly from .txt to man without docbook and xmlto. That does have some other issues which need to be worked out though. Here's what I had as a start:

-- 8< --
Subject: [PATCH] WIP: doc: improve asciidoctor manpage generation

Avoid 'FIXME: Source' by setting mansource. Skip xmlto step and render manpages directly with asciidoctor.

TODO:
    - apply to all man pages
    - fix links to html docs, like user-manual.html in git.1 (currently
      it is listed in brackets inline rather than as a footnote)

Reference:
https://lore.kernel.org/lkml/20180424150456.17353-1-tiwai@suse.de/
Signed-off-by: Todd Zullinger <tmz@pobox.com>
---
 Documentation/Makefile                  | 8 ++++++++
 Documentation/asciidoctor-extensions.rb | 2 ++
 2 files changed, 10 insertions(+)

Hi Todd,

On Sun, 17 Mar 2019 at 20:44, Todd Zullinger <tmz@pobox.com> wrote:

Your approach looks like the correct one long term, at least.

I'll try to play with this to see if I can figure out the differences. That will have to wait until tomorrow though.

Thanks for sharing your progress.

Martin

On Sun, 17 Mar 2019 at 20:44, Todd Zullinger <tmz@pobox.com> wrote:

So to be honest, I still don't understand how this works, but it does, great. I really need to improve my documentation-reading skills.

I had some more time to look at this. Thanks for getting started on this switch. A few things I noticed:

{litdd} now renders as &#x2d;&#x2d. We should find some other way to produce '--'. This should then be a simple change, as we're already providing this attribute inside an `ifdef USE_ASCIIDOCTOR`.

"+" becomes "+". I didn't immediately find where we do that.

Backticks should result in monospace.

`./doc-diff HEAD^ HEAD` shows how several "git-\nfoo" become "\ngit-foo", i.e., linkgit expansions are now treated as non-breaking. That's arguably good, but it brings some noise to the diff. Maybe one should try and see if it's possible to break that to have a nicer diff, then remove that breakage in a follow-up commit. Or, if it's possible to make "git-foo" non-breaking before the switch. Hmm, was this why you increased MANWIDTH?

A double-space between sentences turns into a single space -- at least in constructions such as "... to foobar. `git-foo` does ...". Not a problem perhaps, but noise in the diff.

And I'm sure there's more lurking in that huge diff. Whether any of that is significant or not is another matter. ;-)

Martin

Hi,

Martin Ågren wrote:

Let me know if you find any good methods for perfect retention. I've re-read enough documentation for a lifetime. ;)

I noticed that one and didn't work out a good fix, but it sounds like you have one in mind. That's great.

For this one, I was working on replacing "{plus}" with `+` (along with " " and "-"). That's probably not ideal though.

This is what I had to test:

-- 8< --
Subject: [PATCH] WIP: wrap "[ +-]" in backticks (NOT FOR SUBMISSION)

asciidoctor's manpage output html-encodes "{plus}" which seems like a
bug.  At the least it needs some option I've yet to learn.
---
 Documentation/git-add.txt | 26 +++++++++++++-------------
 Documentation/gitweb.txt  |  6 +++---
 2 files changed, 16 insertions(+), 16 deletions(-)

On Wed, 20 Mar 2019 at 19:17, Todd Zullinger <tmz@pobox.com> wrote:

The "plus" and "litdd" issues seem like they can be solved by doing:

  ASCIIDOC_EXTRA += -aplus='+'
  ASCIIDOC_EXTRA += -alitdd='\--'

:)

Using origin/ma/doc-diff-doc-vs-doctor-comparison, you can do something like

  ./doc-diff --asciidoctor --cut-header-footer foo bar --
--ignore-space-change --word-diff

to reduce the amount of noise in the diff by a great deal. Going through the result, I noted two issues which should be fixed by Asciidoctor 1.5.7, according to some initial searching (but I haven't tested):

"--" renders as an em dash followed by a ";". Looks like it's https://github.com/asciidoctor/asciidoctor/issues/2604

"&" turns into "&amp;". Seems to be known and fixed: https://github.com/asciidoctor/asciidoctor/issues/2525

Plus a few other problems. There's the odd problem here and there, but most of the diff seems to be a large number of occurences of just a few issues.

It would probably be worthwhile to try 1.5.7+ to see how much that improves things. Seems like you're already underway there.

Martin

Hi,

Martin Ågren wrote:

Hmm, setting litdd makes the html generate an em dash followed by a zero width space (in 1.5.8 and 2.0.0)

I updated my system to asciidoctor-2.0.0 last night and now I can't even generate the man pages properly, because the docbook45 converter was removed. I'll have to see if I missed some other required update. :/

Yeah, before I knew how soon 2.0.0 was coming, I updated to 1.5.8 and built the various Fedora packages which require it to see how they handled it. Almost all of the changes were fixes to bugs in previous versions. And the one which was not is likely to be fixed in 2.0.0 according to asciidoctor maintainer Dan Allen.

Have you looked at diffing the html output as well? It seems like we'll need to check it as well to be sure any fixes to the manpage output don't have a negative impact on the html output, and vice versa.

I used 'links -dump' output for comparison of the various Fedora packages which currently build with asciidoctor. It's not perfect. It could miss visual changes which might be important when viewed in a graphical browser. But it was better than trying to diff the html files directly. :)

We probably also want to check the validity of links within the docs, as one of the changes in 1.5.8 caused breakage of cross interdocument references. (This is the change I noted above which should be fixed in 2.0.0.)

It'll be quite a while until most systems with asciidoctor 1.5.x are gone. I doubt that upgrading to even 1.5.8 is suitable for the currently released Fedora versions due to incompatible changes. I am going to try and get 2.0.0 into Fedora 30, but the deadline for changes has just passed, so I may not be able to do so. If not, it'll be 6-8 months until a released version of Fedora has an asciidoctor newer than 1.5.6.1.

All that is just to say that even if newer asciidoctor fixes many of the issues we've seen it will still be a long time before we can reasonable default to asciidoctor or drop asciidoc support.

For what it's worth, the Fedora asciidoc packages moved to python3 using https://github.com/asciidoc/asciidoc-py3. That's not very active, but it should at least keep asciidoc alive beyond the approaching python2 EOL.

On Sat, Mar 23, 2019 at 03:27:56PM -0400, Todd Zullinger wrote:

I ran into that, too. Using ASCIIDOC_DOCBOOK=docbook5 worked. The output looked OK to me, but I only eyeballed it briefly. It's possible there are subtle problems.

-Peff

Hi,

Jeff King wrote:

Interesting. I did that last night as well, but it only led to more errors. I'm curious why manpage builds work for you and not for me.

On my fedora 29 test system, ASCIIDOC_DOCBOOK=docbook5 leads to a validation failure from xmlto, since docbook5 doesn't use a DTD¹. I added XMLTO_EXTRA = --skip-validation to the USE_ASCIIDOCTOR block, which can build many of the man pages, but fails when it gets to git-blame due to use of literal < > characters in the xml:

    git-blame.xml:423: parser error : StartTag: invalid element name
    <literallayout class="monospaced"><40-byte hex sha1> <sourceline> <resultline> <
                                       ^

While this may be a real issue in the formatting of git-blame.txt which we should fix, I'm suspicious of that due to the number of other files similarly affected:

    git-blame.txt
    git-diff-files.txt
    git-diff-index.txt
    git-diff-tree.txt
    git-diff.txt
    git-format-patch.txt
    git-http-fetch.txt
    git-log.txt
    git-rebase.txt
    git-rev-list.txt
    git-show.txt
    git-svn.txt

¹ Here's the failure I get without --skip-validation:
[tmz@f29 Documentation]$ make USE_ASCIIDOCTOR=1 git.1
    SUBDIR ../
make[1]: 'GIT-VERSION-FILE' is up to date.
    ASCIIDOC git.xml
    XMLTO git.1
xmlto: /home/tmz/src/git/Documentation/git.xml does not validate (status 3)
xmlto: Fix document syntax or use --skip-validation option
validity error : no DTD found!
Document /home/tmz/src/git/Documentation/git.xml does not validate
make: *** [Makefile:359: git.1] Error 13

Thanks,

On Sun, Mar 24, 2019 at 12:21:31PM -0400, Todd Zullinger wrote:

I think it's because I'm an idiot. I must have only been using 2.0.0 when I was looking at the XML output manually (for the refmiscinfo lines), and never actually rendered it to roff. I get the same problem when I try a full build.

That seems like a bug in asciidoctor, which ought to be quoting the "<". We certainly can't quote it ourselves (we don't even know that our backend output is going to a format that needs angle brackets quoted).

-Peff

Hi,

Jeff King wrote:

Ahh.  I was hoping you'd tell me that I was a fool. :)

Yep, it seems so.  I filed this upstream:

https://github.com/asciidoctor/asciidoctor/issues/3205

I updated to asciidoctor-2.0.1 this morning to test, in case it was one of the issues fixed since the 2.0.0 release. Alas, we're the first to hit it and report it.

Hi,

I wrote:

Dan Allen fixed this upstream and released 2.0.2 today. It's very good to know that asciidoctor upstream is incredibly responsive. If anyone runs into Dan at a conference, please buy him a beer. ;)

There's still the matter of 2.0 dropping docbook45. I'll try to get around to testing 1.5.x releases with docbook5 to see if they work reasonably well. If not, we can add a version check and set ASCIIDOC_DOCBOOK appropriately.

One other issue that crops up with docbook5 is that the xmlto toolchain now outputs:

    Note: namesp. cut : stripped namespace before processing

as documented at

    https://docbook.org/docs/howto/howto.html#dbxsl

It's mostly an annoyance which we either want to strip out, or pass an alternate docbook5 xsl, I think. But I'm not very familiar with the guts of the xml/docbook toolchains.

On Tue, Mar 26, 2019 at 09:06:03PM -0400, Todd Zullinger wrote:

I noticed the "Release beer" lines in the Asciidoctor relnotes... :)

In our documentation CI build jobs we check that nothing is written to Asciidoc/Asciidoctor's standard error [1]. These "Note:" lines go to stderr, and will trigger that check and cause the build to fail. So wither we should find a way to silence these notes, or filter them out in the CI builds.

[1] 505ad91304 (travis-ci: check AsciiDoc/AsciiDoctor stderr output,
    2017-04-26), though it never actually worked as intended, but that
    is about to get fixed:

    https://public-inbox.org/git/20190324215534.9495-7-szeder.dev@gmail.com/

On Tue, Mar 26, 2019 at 09:06:03PM -0400, Todd Zullinger wrote:

I'm quite familiar with this area, because I used DocBook and its stylesheets for my college papers. There are two sets of stylesheets, the namespaced ones (for DocBook 5) and the non-namespaced ones (for DocBook 4). They can be distinguished because the URLs (and typically the paths) use "xsl" (for non-namespaced) or "xsl-ns" (for namespaced).

xmlto by default uses the older ones, and assuming you have a reasonably capable XSLT processor, either set of stylesheets can be used, since they simply transform the document into the right type (although with a warning, as you've noticed).

Unfortunately, xmlto is hard-coded to use the non-namespaced stylesheets with no option to specify which to use, and it doesn't appear to be actively developed. There's an option to specify the stylesheet (-x), but it can't take a URL, since xmlto "helpfully" fully qualifies the path. That means we'd need to know the location on disk of those stylesheets in order to use them, since we can't rely on catalog resolution.

Hi,

brian m. carlson wrote:

Thanks for the very useful docbook5/xmlto details!

The hard-coded use of the non-namespaced stylesheets in xmlto is unfortunate. I hadn't gotten far enough along to run into that limitation of xmlto, so many thanks for saving me from finding it myself. I'm sure it would have taken me far longer.

If it turns out that docbook5 causes us more pain than it's worth, I suppose we could choose to use the builtin manpage backend when using asciidoctor >= 2.

Or we could see about adding a docbook45 converter, which upstream noted as a possibility in the ticket¹ which dropped docbook45 by default.

It'll be some time before we can reasonably expect to only support asciidoctor-2.x. We'll have to see what method involves the least ugly hacks to support asciidoc and the various 1.5.x and 2.x versions of asciidoctor.

¹ https://github.com/asciidoctor/asciidoctor/issues/3005

On Sat, Mar 30, 2019 at 02:00:14PM -0400, Todd Zullinger wrote:

I suspect this will be the easiest way forward. If we produce semantically equivalent manpages, then this is also a lot nicer for people who would prefer not to have a full XML toolchain installed.

Another possibility, depending on how responsive the xmlto upstream is, is to add some sort of DocBook 5 support to it. This shouldn't be terribly difficult, although we'd have to disable validation. DocBook 5 uses RELAX NG, and libxml2/xmllint's support for this has some bugs, such that it will mark some valid documents using complex interleaves as invalid. Still, if this is the direction we want to go, I have no problem adding this support.

Since we'd have a quite new Asciidoctor and a new xmlto, distros should have both around the same time.

brian m. carlson wrote:

It's a good end goal. There are a number of differences in the output when using the manpage backend directly verus docbook and then xmlto. The way links to external html documents are presented is the main one which comes to mind. Maybe we can adjust that via asciidoctor-extensions.rb or something, I don't know.

Elsewhere in this thread, Jeff made the very valid point that we're probably wise to keep using the docbook/xmlto chain as long as we're supporting both asciidoc and asciidoctor. Unless it turns out that it's more work to coax asciidoctor (and the various 1.5 and 2.0 releases in common use) to work with that same docbook/xmlto chain than it is to do it directly, that is.

Good point. It can't hurt to push for improvements across the tools. (Well, other than time being a limited resource and time you may spend on doc tools being time you can't spend on the hash conversion, which I imagine is a more interesting project.)

Thanks,

On Thu, Apr 04, 2019 at 10:17:21PM -0400, Todd Zullinger wrote:

One of my secret (maybe not so secret?) implications there was that it might be worth dropping asciidoc support sooner rather than later. I.e., if it is a burden to make it work with both old and new systems, then let's make the jump to having it work with the new system.

IMHO we can be a bit more cavalier with saying "you must have a recent-ish asciidoctor to build the docs", because it's so easy for us to provide a binary distribution of the built HTML and manpages (in fact, we already do so for the install-man-quick target).

So it doesn't really leave any platforms out in the cold; it just means they have to tweak their build procedure.

-Peff

On Tue, Mar 26, 2019 at 09:06:03PM -0400, Todd Zullinger wrote:

Cool. I've interacted a few times with the asciidoctor project due to our use on git-scm.com, and have always found them pleasant and responsive.

With the hacky patch below, I was able to doc-diff the v1.5.8 versus v2.0.2 output. It looks like there are quite a few cosmetic whitespace changes. Some probably intentional (extra blank between terms and paragraphs in a definition list) and some probably not (extra whitespace before start of content in a bulleted list).

On Wed, Mar 27, 2019 at 10:54:34PM -0400, Jeff King wrote:

Oops, this last line should be ASCIIDOC=..., of course. Which meant that I was actually just testing the docbook5 output from v1.5.8. The results with v2.0.2 seem similar, though.

-Peff

On Sun, Mar 17, 2019 at 03:47:47PM +0100, Martin Ågren wrote:

I think it was just that my version of asciidoctor had

  https://github.com/asciidoctor/asciidoctor/pull/2636

and Todd's did not. However, mine still does not do the _right_ thing, because we didn't pass the right attributes in to asciidoctor. It just didn't print an ugly "FIXME". Looking at the XML, I have:

  <refentrytitle>git-add</refentrytitle>
  <manvolnum>1</manvolnum>
  <refmiscinfo class="source">&#160;</refmiscinfo>
  <refmiscinfo class="manual">&#160;</refmiscinfo>
  </refmeta>

So it's just an nbsp instead of the real content, and the "version" field is missing entirely.

Yeah, that seems sensible overall. Some thoughts on your approach:

It does seem like "mansource" is the way asciidoctor expects us to do this. Why doesn't it work from the command line? Is it a bug in asciidoctor, or is there something more subtle going on?

I think even if it is a bug and gets fixed, though, it still wouldn't have the version field (though that seems like something we could contribute to asciidoctor).

That's more or less what your ruby code is doing on. That said, I'd just as soon do it in ruby as with a separate sed invocation. At least then the external build is the same.

This seems like the least bad option, at least for now. Your code does do a generic regex substitution. The promise of XML is that we're supposed to be able to do structured, robust transformations of the document. But my experience has been that the tooling is sufficiently difficult to work with that you just end up writing a regex.

So I'm curious if you tried to use an actual XML parser (or god forbid, XSLT) to do the transformation. But if you spent more than 5 minutes on it and got disgusted, I wouldn't ask you to look deeper than that. :)

I doubt we'd see any other refmeta tags (and any non-tag content would be quoted).

One thing that asciidoctor buys us that asciidoc does not is that we might eventually move to directly generating the manpages, without the XML / Docbook step in between. And if we do, then all of this XML hackery is going to have to get replaced with something else. I guess we can cross that bridge when we come to it.

The patch itself looks sane. Would we ever need to XML-quote the contents of git_version? I guess the asciidoc.conf version doesn't bother. Technically the user running "make" could put whatever they want into it, but I think this is a case of "if it hurts, don't do it", and we can ignore it.

-Peff

On Mon, Mar 18, 2019 at 10:46:46PM -0400, Jeff King wrote:

I see there was some discussion of that in the side-thread, too. Just to give my two cents: I think that's where we want to go eventually, but I also think that while we are maintaining a dual asciidoc/asciidoctor tool chain, it's probably crazy not to go through docbook, just because it keeps so much of the pipeline the same for both tools.

So in my mind, the endgame is that we eventually drop asciidoc in favor of asciidoctor. The repo at:

  https://github.com/asciidoc/asciidoc

says:

  NOTE: This implementation is written in Python 2, which EOLs in Jan
  2020. AsciiDoc development is being continued under @asciidoctor.

I'm not sure when is the right time to switch. If we can get the output at parity, I don't think asciidoctor is too onerous to install (and we don't have to worry about ancient platforms, as they can use pre-formatted manpages).

-Peff

Jeff King <peff@peff.net> writes:

;-)

One minor thing that bothers me abit is the continuity of the pre-formatted pages when I switch to asciidoctor to update them.

I do not mind having to see a huge diff in the "git log -p" output run in pre-formatted manpages and htmldocs repositories at the boundary due to e.g. the differences how lines are broken or folded between the formatters, but by the time we have to transition, the efforts by you, Martin and friends to allow us compare the formatted docs would have made the real differences to empty (or at least negligible). Knock knock...

On Tue, 19 Mar 2019 at 04:55, Junio C Hamano <gitster@pobox.com> wrote:

This might be a good spot to provide a bit of current status on this.

The patch under discussion here and 185f9a0ea0 ("asciidoctor-extensions: fix spurious space after linkgit", 2019-02-27), reduce the diff between asciidoc and asciidoctor considerably. Let's assume for the moment that these patches or something like them enter master...

There's one larger difference in git-checkout.txt which I'm staying away from at the moment, since Duy is doing a lot of work there at the time. (He's not making that asciidoc/tor issue any worse, and he's not spreading it to say git-switch.txt, so I'd rather just not rush to it.) Let's assume I get around to this soonish...

Let's also assume that `./doc-diff --cut-header-footer --from-asciidoc --to-asciidoctor HEAD HEAD` enters master [1]. Then what remains is a fairly small and understandable diff where most issues are "yeah, I guess that looks nicer" or even "they're just as fine", rather than "oh wow, that's ugly". (IMHO.)

Martin

[1] You could already now run it to diff "master master", but you'd trip on the Makefile thinking there's nothing to do. That'd be fixed by 9a71722b4d ("Doc: auto-detect changed build flags", 2019-03-17).

Hi Junio

On Tue, 19 Mar 2019 at 08:33, Martin Ågren <martin.agren@gmail.com> wrote:

To be clear. *This* patch has a sufficiently incorrect commit message that it really needs a makeover. You can expect a v2.

Martin

Almost half a year ago, I wrote:

Finally, here's that v2. I should probably refresh memories: The goal of the main patch here is to make the headers and footers of our manpages built by Asciidoctor look a lot more like those generated by AsciiDoc. In particular, this gets rid of the ugly "[FIXME: source]".

I spent a little bit of time trying to work on the XML as XML, and quite a lot of time procrastinating on that. In the end, I decided that the outcome of my attempts wouldn't get better and that there is some value to the stupid approach from v1 of doing a simple search-and-replace in the text. I've preserved my attempts in the commit message.

When I posted v1, it turned into quite a thread [1] on AsciiDoc vs Asciidoctor vs Asciidoctor 2.0 and differences in rendering. (I am on Asciidoctor 1.5.5.)

Among other things, the v1-thread discussed switching the rendering toolchain entirely to avoid the detour over xmlto. Doing that would render this patch obsolete. While I agree that such a switch is the correct long-term goal and that we can be fairly aggressive about it, I do also think it makes sense to first make the "softer" switch to Asciidoctor-by-default and get that particular hurdle behind us. Then, once we're ok with dropping AsciiDoc entirely, we can do the switch to an Asciidoctor-only toolchain.

(I'm preparing a second, independent series of patches that should halve the difference (sans headers/footers) between these two engines -- at least the versions of them that I'm using. The remaining differences are then mainly, but not exclusively, in favor of Asciidoctor. That series should appear on the list within the next couple of days. After that, there's user-manual.html/pdf that needs looking into...)

[1] https://public-inbox.org/git/20190317144747.2418514-1-martin.agren@gmail.com/

Martin Ågren (2):
  asciidoctor-extensions: provide `<refmiscinfo/>`
  doc-diff: replace --cut-header-footer with --cut-footer

 Documentation/asciidoctor-extensions.rb | 15 +++++++++++++++
 Documentation/doc-diff                  | 17 ++++++++---------
 2 files changed, 23 insertions(+), 9 deletions(-)

When we build with AsciiDoc, asciidoc.conf ensures that each xml-file we generate contains some meta-information which `xmlto` can act on, based on the following template:

  <refmeta>
  <refentrytitle>{mantitle}</refentrytitle>
  <manvolnum>{manvolnum}</manvolnum>
  <refmiscinfo class="source">Git</refmiscinfo>
  <refmiscinfo class="version">{git_version}</refmiscinfo>
  <refmiscinfo class="manual">Git Manual</refmiscinfo>
  </refmeta>

When we build with Asciidoctor, it does not honor this configuration file and we end up with only this (for a hypothetical git-foo.xml):

  <refmeta>
  <refentrytitle>git-foo</refentrytitle>
  <manvolnum>1</manvolnum>
  </refmeta>

That is, we miss out on the `<refmiscinfo/>` tags. As a result, the header of each man page doesn't say "Git Manual", but "git-foo(1)" instead. Worse, the footers don't give the Git version number and instead provide the fairly ugly "[FIXME: source]".

That Asciidoctor ignores asciidoc.conf is nothing new. This is why we implement the `linkgit:` macro in asciidoc.conf *and* in asciidoctor-extensions.rb. Follow suit and provide these tags in asciidoctor-extensions.rb, using a "postprocessor" extension where we just search and replace in the XML, treated as text.

We may consider a few alternatives:

  * Provide the `mansource` attribute to Asciidoctor. This attribute
    is only respected (i.e., turned into those <refmiscinfo/> tags) with
    1) the "manpage" doctype and/or converter [1], which we currently do
    not use, or 2) using Asciidoctor 1.5.7 or newer [2].

  * We could inject these lines into the xml-files from the *Makefile*,
    e.g., using `sed`. That would reduce repetition, but it feels wrong
    to impose another step and another risk on the AsciiDoc-processing
    only to benefit the Asciidoctor-one.

  * I tried providing a "docinfo processor" to inject these tags, but
    could not figure out how to "merge" the two <refmeta/> sections that
    resulted. To avoid xmlto barfing on the result, I needed need to use
    `xmlto --skip-validation ...`, which seems unfortunate.

Let's instead inject the missing tags using a postprocessor. We'll make it fairly obvious that we aim to inject the exact same three lines of `<refmiscinfo/>` that asciidoc.conf provides. We inject them in *post*-processing so we need to do the variable expansion ourselves. We do introduce the bug that asciidoc.conf already has in that we won't do any escaping, e.g., of funky versions like "some v <2.25, >2.20".

The postprocessor we add here works on the XML as raw text and doesn't really use the full potential of XML to do a more structured injection. This is actually precisely what the Asciidoctor User Manual does in its postprocessor example [3]. I looked into two other approaches:

  1. The nokogiri library is apparently the "modern" way of doing XML
     in ruby. I got it working fairly easily:
        require 'nokogiri'
        doc = Nokogiri::XML(output)
        doc.search("refmeta").each { |n| n.add_child(new_tags) }
        output = doc.to_xml
     However, this adds another dependency (e.g., the "ruby-nokogiri"
     package on Ubuntu). Using Asciidoctor is not our default, but it
     will probably need to become so soon (AsciiDoc relies on Python 2,
     which is going away). Let's avoid adding a dependency just so that
     we can say "search...add_child" rather than "sub(regex...)".

  2. The older REXML is apparently always(?) bundled with ruby, but I
     couldn't even parse the original document:
        require 'rexml/document'
        doc = REXML::Document.new(output)
        ...
     The error was "no implicit conversion of nil into String" and I
     stopped there.

Having never dabbled with ruby outside of this very file, I might be missing something obvious, but I don't think it's unlikely that doing a plain old search-and-replace will work just as fine or better compared to parsing XML and worrying about libraries and library versions.

[1] https://asciidoctor.org/docs/user-manual/#builtin-attributes [2] https://public-inbox.org/git/20190319074321.GA2189@sigill.intra.peff.net/ [3] https://asciidoctor.org/docs/user-manual/#postprocessor-example

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 Documentation/asciidoctor-extensions.rb | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

After the previous commit, AsciiDoc and Asciidoctor render the manpage headers identically, so we no longer need the "cut the header" part of our `--cut-header-footer` option. We do still need the "cut the footer" part, though. The previous commit improved the rendering of the footer in Asciidoctor by quite a bit, but the two programs still disagree on how to format the date in the footer: 01/01/1970 vs 1970-01-01.

We could keep using `--cut-header-footer`, but it would be nice if we had a slightly smaller hammer `--cut-footer` that would be less likely to hide regressions. Rather than simply adding such an option, let's also drop `--cut-header-footer`, i.e., rework it to lose the "header" part of its name and functionality.

`--cut-header-footer` is just a developer tool and it probably has no more than a handful of users, so we can afford be aggressive.

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 Documentation/doc-diff | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

On 2019-09-03 at 18:51:19, Martin Ågren wrote:

Yeah, this seems like a good approach. xmlto is a neat tool, but it's essentially unmaintained and is designed for DocBook 4. Avoiding it where possible seems like the right choice.

I looked at this series and it seems sane. I agree that adding a dependency on nokogiri isn't really desirable. It is an extremely common Ruby package, but it has native extensions, which causes problems for some people if their distro doesn't support it.

On Wed, 4 Sep 2019 at 01:16, brian m. carlson <sandals@crustytoothpaste.net> wrote:

Ok, nice to know I wasn't too far off with avoiding the added dependency. I saw statements like "oh, it's available everywhere", but also more nuanced ones like "not necessarily...". I'm glad I have a trustworthy source now. ;-)

Martin

On Tue, Sep 03, 2019 at 08:51:19PM +0200, Martin Ågren wrote:

Thanks for resuming this work. The patches both look good to me.

I think your hack is better than introducing yet another dependency. And because it's isolated and you've documented it well, it will be easy to undo later if we choose.

The doc-diff tool should tell us if we catch any false positives inside the text (though it seems quite unlikely to me).

Yes, sadly I still can't format the docs at all with 2.0.10 (which is what ships in Debian unstable).

Yeah, I do still like that as an endgame, but I like what you have here as an intermediate step in the right direction.

Yay. :)

-Peff

On Wed, 4 Sep 2019 at 05:26, Jeff King <peff@peff.net> wrote:

Hmm, so this sounds like once I am happy with replacing AsciiDoc with Asciidoctor 1(.5.5), I should rather not propose a series "let's default to Asciidoctor!!!" but instead a slightly more careful "go with Asciidoctor, but document that we work badly with v2 and that the 2nd choice after Asciidoctor 1 should be AsciiDoc". Or do you see it differently? (I wonder which Asciidoctor-version Junio would be on..)

Martin

On Thu, Sep 05, 2019 at 09:35:10PM +0200, Martin Ågren wrote:

Yeah, that seems reasonable.

TBH, if making things in the middle step work turns out to be too hard, I'm not entirely opposed to a hard switch.

The "does not work with 2.0" thing has to be a temporary step, though, I think, since using the older versions will get harder and harder as time goes on. I think it's OK to take such a temporary step as long as we understand where it leads (and presumably its to directly generating the roff with asciidoctor). The middle step of having asciidoctor+xmlto helps us understand and isolate which changes are responsible for which parts of the output.

-Peff

On Sat, 7 Sep 2019 at 08:45, Jeff King <peff@peff.net> wrote:

So of these steps:

  0. Get Asciidoctor (v1) in shape.

  1. Switch the default to Asciidoctor (v1).

  2. Drop AsciiDoc to have faster Asciidoctor-processing, avoid xmlto
     and support Asciidoctor 2. And to avoid the Python 2 EOL, too.

Step 0 is not far away, so step 1 could be done fairly soon IMHO. Step 2 would "hopefully" happen soon after -- maybe even in the same release cycle as step 1, and if not the same then the one just after. But I might be the wrong person to trust on that one. I currently don't even try to build with Asciidoctor 2. I might perhaps look into installing it, but it could also be that I'll only start using it when it happens to arrive through my distro.

So as long as I'm not looking into Asciidoctor 2, maybe I shouldn't be the one to impose "default to asciidoctor" on the world. Dunno. In any case, I should be able to bring the asciidoc/tor1 differences to a state where we trust asciidoctor 1 to be in a good shape, so that "someone else" could pick up the ball and work on asciidoctor 2 vs 1, knowing that it's ok if they regress AsciiDoc support or even drop it entirely in the process.

Martin

On Sat, Sep 07, 2019 at 04:06:48PM +0200, Martin Ågren wrote:

We could also drop xmlto for the asciidoctor codepaths, without making it the default. That requires a little bit more Makefile massaging, but here's a hacky way to do it (this is on top of brian's asciidoctor 2 patch, which I've been experimenting with, but obviously we could skip that entirely, too):

---

On 2019-09-04 at 03:26:10, Jeff King wrote:

I'll look into this.  I requested the upgrade in sid.

If it's the lack of DocBook 4.5 support, then I'll probably need to patch xmlto for that. DocBook 5 has been around for a decade now, so it's time for xmlto to support it properly.

On Fri, Sep 06, 2019 at 11:29:47PM +0000, brian m. carlson wrote:

Yes, it's the docbook45 thing. Switching to docbook5 lets asciidoc run, but then xmlto chokes.

-Peff

On 2019-09-07 at 04:40:22, Jeff King wrote:

I already have a patch for Git to fix this.  I'll send it out this weekend.

Our documentation toolchain has traditionally been built around DocBook 4.5. This version of DocBook is the last DTD-based version of DocBook. In 2009, DocBook 5 was introduced using namespaces and its syntax is expressed in RELAX-NG, which is more expressive and allows a wider variety of syntax forms.

Asciidoctor, one of the alternatives for building our documentation, dropped support for DocBook 4.5 in its recent 2.0 release and now only supports DocBook 5. This would not be a problem but for the fact that we use xmlto, which is still stuck in the DocBook 4.5 era.

xmlto performs DTD validation as part of the build process. This is not problematic for DocBook 4.5, which has a valid DTD, but it clearly cannot work for DocBook 5, since no DTD can adequately express its full syntax. In addition, even if xmlto did support RELAX-NG validation, that wouldn't be sufficient because it uses the libxml2-based xmllint to do so, which has known problems with validating interleaves in RELAX-NG.

Fortunately, there's an easy way forward: ask Asciidoctor to use its DocBook 5 backend and tell xmlto to skip validation. Asciidoctor has supported DocBook 5 since v0.1.4 in 2013 and xmlto has supported skipping validation for probably longer than that.

xmlto will still use the non-namespaced DocBook XSL stylesheets (which are designed for DocBook 4.5) for building the documentation instead of the namespaced ones (which are designed for DocBook 5). This isn't really a problem, since both forms are built from the same source and can handle both versions, but it does mean that an ugly message about the stylesheets stripping the namespace will be printed to standard error.

Overall, however, this message is a relatively minor price to pay and it means that we can continue to use the aging xmlto to drive our build process, while still supporting newer tooling like Asciidoctor 2.

The differences in output between AsciiDoc 8.6.10 on master and Asciidoctor 2.0.10 with this patch are, with one exception, all due to whitespace, wrapping, or quoting and do not affect substantive content.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

On Sat, Sep 07, 2019 at 05:07:46PM +0000, brian m. carlson wrote:

Wow, this patch turned out to be much simpler than I had feared. And the explanation turned out to be much longer and more interesting than I expected. :) Thanks for writing it all down.

The patch worked well for me.

So I think this does introduce the ugly namespace message even for people who are using older versions of asciidoctor that understand docbook45. But it's probably sensible to keep all of our asciidoctor builds using the same technique. We have enough trouble tracking the minor differences between the variants as it is.

We know already there are a lot of differences between asciidoc and asciidoctor here. What's more interesting to me is how it changes the output between two versions of asciidoctor. Of course we can't check for 2.0.10, because it doesn't build at all. But I did do a "doc-diff" with and without this patch using 1.5.5. There are quite a few whitespace changes. Some of them seem good or at least neutral, like:

  --- a/745f6812895b31c02b29bdfe4ae8e5498f776c26-asciidoctor/home/peff/share/man/man1/git-add.1
  +++ b/303729d86b69657777222bf4b3a6f95932e12648-asciidoctor/home/peff/share/man/man1/git-add.1
  [...]
  @@ -43,6 +43,7 @@ DESCRIPTION

   OPTIONS
          <pathspec>...
  +
              Files to add content from. Fileglobs (e.g.  *.c) can be given to
              add all matching files. Also a leading directory name (e.g.  dir to
              add dir/file1 and dir/file2) can be given to update the index to

Some of them seem bad, though:

  --- a/745f6812895b31c02b29bdfe4ae8e5498f776c26-asciidoctor/home/peff/share/man/man1/git-am.1
  +++ b/303729d86b69657777222bf4b3a6f95932e12648-asciidoctor/home/peff/share/man/man1/git-am.1
  [...]
  @@ -175,10 +201,10 @@ DISCUSSION

          to process. Upon seeing the first patch that does not apply, it aborts
          in the middle. You can recover from this in one of two ways:
   
  -        1. skip the current patch by re-running the command with the --skip
  +        1.  skip the current patch by re-running the command with the --skip
              option.
   
  -        2. hand resolve the conflict in the working directory, and update the
  +        2.  hand resolve the conflict in the working directory, and update the
              index file to bring it into a state that the patch should have
              produced. Then run the command with the --continue option.

I tricked doc-diff into doing a comparison against 1.5.5 without the patch and 2.0.10 with the patch, and the diff is similar.

-Peff

On 2019-09-08 at 10:48:33, Jeff King wrote:

I see the same thing in doc-diff, but this issue doesn't appear to actually render in the manual pages themselves, even with MANWIDTH=80. I'm not sure why, but it doesn't seem to misrender in the actual man output.

I also tried with the DocBook 5 (docbook-xsl-ns) stylesheets, but that doesn't appear to make a difference in doc-diff.

On Sun, Sep 08, 2019 at 05:18:07PM +0000, brian m. carlson wrote:

Hrm. But doc-diff _is_ diffing the output of man, so that implies a bug in that script. However, if I manually go into tmp-doc-diff/installed and run "man -l git-am.1", I do see the extra whitespace, which is what I'd expect. Are you sure you did your manual inspection on the right thing?

I was also curious about the root cause at the roff level, so I did this hack:

On 2019-09-08 at 21:21:22, Jeff King wrote:

Trying again, I'm able to reproduce this. I found the cause, which is in the stylesheets. XSLT stylesheets have the ability to specify elements from which whitespace should be stripped (using the xsl:strip-space directive). In the DocBook stylesheets, listitem is specified as such an element, so the whitespace there should be stripped.

However, in DocBook 5, our elements are in a namespace. Therefore, the unnamespaced stylesheets specify only "listitem", not "d:listitem", like the namespaced stylesheets do. Because this happens right after the tree has been constructed "but before it is otherwise processed by XSLT" and isn't affected by the EXSLT extension that allows re-parsing the modified tree, then we end up with the whitespace that we don't want.

There are a couple ways around this.

1. We can force xmlto to use the DocBook 5 stylesheets with the "-x"
flag, but we have to know where they are.  Debian and Fedora have them
in different places, so we'd need a knob to figure out where they are.

2. We can force xmlto to use a custom stylesheet with "-x" that merely
imports the DocBook 5 stylesheets using a URL.  If the user has the
DocBook 5 stylesheets installed and XML catalogs configured (the default
on Linux distributions), then everything will just work and the system
will resolve it to the local copy.  If, however, things are not properly
configured, this will result in multiple network downloads for each
manual page.

3. We can give up on xmlto and do things ourselves.  This has the same
problem as option 1, since we need to learn how to find the stylesheets.

4. We can send a patch to xmlto to make it use the proper stylesheets
for DocBook 5 and hope upstream does a release and everyone upgrades
shortly.  Since xmlto is not at all active upstream, this seems like it
may be an imprudent choice.

5. We can send a patch to the DocBook stylesheets and have them include
both the namespaced and unnamespaced versions of the element names in
both sets of stylesheets and hope everyone upgrades.

My personal preference is #2; I think that seems like the best choice forward. XML catalogs are well understood and well configured on Linux distributions. Homebrew supports them adequately, but you have to add an environment variable to your shell configuration to enable them. Of course, if you're doing _anything_ with XML, you'll have them enabled.

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

Sounds sensible and well reasoned.

On Sun, Sep 08, 2019 at 10:24:24PM +0000, brian m. carlson wrote:

First off, thank you again for your explanations. I dread digging into how anything related to docbook or xml works, so having you serve it up on a silver platter is a delight. :)

Isn't this already the case just with the docbook DTDs? I.e., if you don't have a catalog entry, it is up to the tool (xmlto in this case) to either fail or try to fetch it. That seems like the best we can do. And as you note, this typically just works out of the box on modern installs. Of course people may want to build on non-modern ones, but IMHO we should probably be more aggressive about dropping legacy support in the documentation and pointing people to the pre-formatted pages.

Yeah, agreed.

-Peff

On 2019-09-10 at 18:44:22, Jeff King wrote:

I'm happy to do it. I was an English major in college and virtually every assignment I wrote in DocBook, so I'm far more familiar with it than I'm sure most people ever want to be.

Yeah, that's a good point. Since we know that most people will have catalogs working (since usually fetching the DTD repeatedly over the network gets you rate-limited resulting in a failure), I think it's fair for us to do this. And anyway, DocBook 5 isn't exactly new: it's been around since 2009, IIRC, so distros will have it.

I know Homebrew doesn't have this problem, since they fetch the prebuilt docs, even for HEAD (master) builds.

I'll try to get the patch written up and sent out today or tomorrow.

On Sat, Sep 07, 2019 at 05:07:46PM +0000, brian m. carlson wrote:

These messages from 'xmlto' look like these, and there are a lot of them:

  Note: namesp. cut : stripped namespace before processing      Git User Manual
  Note: namesp. cut : stripped namespace before processing      git-sh-setup(1)
  Note: namesp. cut : stripped namespace before processing      git-get-tar-commit-id(1)

Unfortunately, these messages to standard error cause our CI builds to fail [1].

In the Documentation build job we check that there was nothing printed to standard error during 'make doc', in order to catch warnings that are potential signs of a mis-rendered documentation, but do not cause any asciidoc/asciidoctor/xmlto commands (and thus 'make doc') to fail.

Now, a few recent messages to standard error that indeed were signs of mis-rendered docs [2] looked like this:

  asciidoctor: WARNING: api-config.txt: line 232: unterminated listing block

i.e. they came from Asciidoctor and were all-caps warnings.

OTOH, these "stripped namespace" messages come from 'xmlto', are not warnings but have that "Note:" prefix, and, trusting that you did check the results thoroughly, are apparently not a sign of any rendering issues. So I think it's safe to ignore them and this patch should strip them from 'make doc's output in 'ci/test-documentation.sh'.

Related: after this patch we might want to update the CI builds to use
Asciidoctor 2 instead of sticking with 1.5.8.

[1] With Asciidoctor 1.5.8:

      https://travis-ci.org/szeder/git/jobs/582294090#L2085

    With an additional patch on top to use Asciidoctor 2:

      https://travis-ci.org/szeder/git/jobs/582294243#L2066

[2] See 'git log -3 b373e4d29b'

On Sun, Sep 08, 2019 at 04:13:08PM +0200, SZEDER Gábor wrote:

Yeah, I'd agree (though I do think there are some other issues with this approach, at least in my tests).

If we do go this route, it might even be nice to use a small xmlto wrapper that filters the stderr for us. Then the ci code doesn't have to know about this distinction, and it prevents users from seeing the ugly messages.

-Peff

Our documentation toolchain has traditionally been built around DocBook 4.5. This version of DocBook is the last DTD-based version of DocBook. In 2009, DocBook 5 was introduced using namespaces and its syntax is expressed in RELAX NG, which is more expressive and allows a wider variety of syntax forms.

Asciidoctor, one of the alternatives for building our documentation, moved support for DocBook 4.5 out of core in its recent 2.0 release and now only supports DocBook 5 in the main release. The DocBoook 4.5 converter is still available as a separate component, but this is not available in most distro packages. This would not be a problem but for the fact that we use xmlto, which is still stuck in the DocBook 4.5 era.

xmlto performs DTD validation as part of the build process. This is not problematic for DocBook 4.5, which has a valid DTD, but it clearly cannot work for DocBook 5, since no DTD can adequately express its full syntax. In addition, even if xmlto did support RELAX NG validation, that wouldn't be sufficient because it uses the libxml2-based xmllint to do so, which has known problems with validating interleaves in RELAX NG.

Fortunately, there's an easy way forward: ask Asciidoctor to use its DocBook 5 backend and tell xmlto to skip validation. Asciidoctor has supported DocBook 5 since v0.1.4 in 2013 and xmlto has supported skipping validation for probably longer than that.

We also need to teach xmlto how to use the namespaced DocBook XSLT stylesheets instead of the non-namespaced ones it usually uses. Normally these stylesheets are interchangeable, but the non-namespaced ones have a bug that causes them not to strip whitespace automatically from certain elements when namespaces are in use. This results in additional whitespace at the beginning of list elements, which is jarring and unsightly.

We can do this by passing a custom stylesheet with the -x option that simply imports the namespaced stylesheets via a URL. Any system with support for XML catalogs will automatically look this URL up and reference a local copy instead without us having to know where this local copy is located. We know that anyone using xmlto will already have catalogs set up properly since the DocBook 4.5 DTD used during validation is also looked up via catalogs. All major Linux distributions distribute the necessary stylesheets and have built-in catalog support, and Homebrew does as well, albeit with a requirement to set an environment variable to enable catalog support.

On the off chance that someone lacks support for catalogs, it is possible for xmlto (via xmllint) to download the stylesheets from the URLs in question, although this will likely perform poorly enough to attract attention. People still have the option of using the prebuilt documentation that we ship, so happily this should not be an impediment.

Finally, we need to filter out some messages from other stylesheets that when invoking dblatex in the CI job. This tool strips namespaces much like the unnamespaced DocBook stylesheets and prints similar messages. If we permit these messages to be printed to standard error, our documentation CI job will because we check standard error for unexpected output. Due to dblatex's reliance on Python 2, we may need to revisit its use in the future, in which case this problem may go away, but this can be delayed until a future patch.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/Makefile    | 4 +++-
 Documentation/manpage.xsl | 3 +++
 ci/test-documentation.sh  | 2 ++
 3 files changed, 8 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/manpage.xsl

On Fri, Sep 13, 2019 at 01:52:40AM +0000, brian m. carlson wrote:

Thanks, this fixed most of the rendering problems I saw from the earlier patch.

This did give me one minor hiccup: I had the debian docbook-xsl package installed, but not docbook-xsl-ns. The error message was pretty standard for XML: obvious if you know what catalogs are, and utterly confusing otherwise. :)

Everything worked fine after installing docbook-xsl-ns. I wonder if could/should provide some guidance somewhere (maybe in INSTALL, which discusses some catalog issues?).

s/that/that occur/ or something?

s/will/will fail/?

Running with this patch on asciidoctor 2.0.10, plus Martin's recent literal-block cleanups, plus his refmiscinfo fix, I get pretty decent output from:

  ./doc-diff --from-asciidoc --to-asciidoctor origin HEAD

The header/footer are still a little funny (but I think Martin said that he needs to update the refmiscinfo patches for later versions of asciidoctor, which is probably what's going on here):

  --- a/f1d4a28250629ae469fc5dd59ab843cb2fd68e12-asciidoc/home/peff/share/man/man1/git-add.1
  +++ b/6c08635fd1d38c83d3765ff05fabbfbd25ef4943-asciidoctor/home/peff/share/man/man1/git-add.1
  @@ -1,4 +1,4 @@
  -GIT-ADD(1)                        Git Manual                        GIT-ADD(1)
  +GIT-ADD(1)                                                          GIT-ADD(1)
   
   NAME
          git-add - Add file contents to the index
  @@ -356,4 +356,4 @@ SEE ALSO
   GIT
          Part of the git(1) suite
   
  -Git omitted                       01/01/1970                        GIT-ADD(1)
  +  omitted                         1970-01-01                        GIT-ADD(1)

One curiosity is that any ``smart-quotes'' now get two spaces between them and the period of the last sentence (whereas in asciidoc they got only one):

  -           <start> and <end> are optional. “-L <start>” or “-L <start>,” spans
  -           from <start> to end of file. “-L ,<end>” spans from start of file
  -           to <end>.
  +           <start> and <end> are optional.  “-L <start>” or “-L <start>,”
  +           spans from <start> to end of file.  “-L ,<end>” spans from start of
  +           file to <end>.

I don't think this is a big deal, but I think most of these should actually be backticks these days (the text above is from git-annotate.txt, which hasn't been touched in quite a while).

There are other miscellaneous indentation fixes. Most of them look better in asciidoctor, IMHO. For example, some lists now wrap more neatly (it looks like it's usually lists after an indented listing block? Maybe a continuation thing?):

  -           1. This step and the next one could be combined into a single step
  -           with "checkout -b my2.6.14 v2.6.14".
  +            1. This step and the next one could be combined into a single
  +               step with "checkout -b my2.6.14 v2.6.14".

Another curiosity is that single-quote `smart-quotes' are rendered as real smart-quotes by asciidoctor:

  -           The following features from ‘svn log’ are supported:
  +           The following features from “svn log” are supported:

The only other case I found was this one, where I think the asciidoctor version is better (the source has literal backticks, so there shouldn't be a visible quote; I'm guessing asciidoc got confused by the apostrophe in "variable's"):

  -           The ‘merge.*.driver` variable’s value is used to construct a
  +           The merge.*.driver variable’s value is used to construct a command

So overall, I think we're getting very close to parity.

-Peff

Jeff King <peff@peff.net> writes:

Thanks, both.  Have queued with your log message typofixes.

On Fri, 13 Sep 2019 at 07:06, Jeff King <peff@peff.net> wrote:

Yeah, I should be able to post v3 of my refmiscinfo-series this evening, which should fix this, so that the only difference that remains here is how the date is formatted.

Martin

Martin Ågren <martin.agren@gmail.com> writes:

Thanks.

On Fri, Sep 13, 2019 at 01:52:40AM +0000, brian m. carlson wrote:

Unfortunately, five out of five CI builds failed with the following:

      XMLTO git-revert.1
  I/O error : Attempt to load network entity http://docbook.sourceforge.net/release/xsl-ns/current/manpages/docbook.xsl
  warning: failed to load external entity "http://docbook.sourceforge.net/release/xsl-ns/current/manpages/docbook.xsl"
  compilation error: file /home/travis/build/git/git/Documentation/manpage.xsl line 2 element import
  xsl:import : unable to load http://docbook.sourceforge.net/release/xsl-ns/current/manpages/docbook.xsl
  Makefile:375: recipe for target 'git-revert.1' failed

https://travis-ci.org/git/git/jobs/584794387#L1552

On 2019-09-14 at 07:53:01, SZEDER Gábor wrote:

Ah, I forgot to install the packages in CI. I'll send a v3 with that fixed.

Our documentation toolchain has traditionally been built around DocBook 4.5. This version of DocBook is the last DTD-based version of DocBook. In 2009, DocBook 5 was introduced using namespaces and its syntax is expressed in RELAX NG, which is more expressive and allows a wider variety of syntax forms.

Asciidoctor, one of the alternatives for building our documentation, moved support for DocBook 4.5 out of core in its recent 2.0 release and now only supports DocBook 5 in the main release. The DocBoook 4.5 converter is still available as a separate component, but this is not available in most distro packages. This would not be a problem but for the fact that we use xmlto, which is still stuck in the DocBook 4.5 era.

xmlto performs DTD validation as part of the build process. This is not problematic for DocBook 4.5, which has a valid DTD, but it clearly cannot work for DocBook 5, since no DTD can adequately express its full syntax. In addition, even if xmlto did support RELAX NG validation, that wouldn't be sufficient because it uses the libxml2-based xmllint to do so, which has known problems with validating interleaves in RELAX NG.

Fortunately, there's an easy way forward: ask Asciidoctor to use its DocBook 5 backend and tell xmlto to skip validation. Asciidoctor has supported DocBook 5 since v0.1.4 in 2013 and xmlto has supported skipping validation for probably longer than that.

We also need to teach xmlto how to use the namespaced DocBook XSLT stylesheets instead of the non-namespaced ones it usually uses. Normally these stylesheets are interchangeable, but the non-namespaced ones have a bug that causes them not to strip whitespace automatically from certain elements when namespaces are in use. This results in additional whitespace at the beginning of list elements, which is jarring and unsightly.

We can do this by passing a custom stylesheet with the -x option that simply imports the namespaced stylesheets via a URL. Any system with support for XML catalogs will automatically look this URL up and reference a local copy instead without us having to know where this local copy is located. We know that anyone using xmlto will already have catalogs set up properly since the DocBook 4.5 DTD used during validation is also looked up via catalogs. All major Linux distributions distribute the necessary stylesheets and have built-in catalog support, and Homebrew does as well, albeit with a requirement to set an environment variable to enable catalog support.

On the off chance that someone lacks support for catalogs, it is possible for xmlto (via xmllint) to download the stylesheets from the URLs in question, although this will likely perform poorly enough to attract attention. People still have the option of using the prebuilt documentation that we ship, so happily this should not be an impediment.

Finally, we need to filter out some messages from other stylesheets that occur when invoking dblatex in the CI job. This tool strips namespaces much like the unnamespaced DocBook stylesheets and prints similar messages. If we permit these messages to be printed to standard error, our documentation CI job will fail because we check standard error for unexpected output. Due to dblatex's reliance on Python 2, we may need to revisit its use in the future, in which case this problem may go away, but this can be delayed until a future patch.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
Range-diff against v2:
1:  6f11a79d03 ! 1:  c4935af5de Documentation: fix build with Asciidoctor 2
    @@ Commit message
         documentation that we ship, so happily this should not be an impediment.
     
         Finally, we need to filter out some messages from other stylesheets that
    -    when invoking dblatex in the CI job.  This tool strips namespaces much
    -    like the unnamespaced DocBook stylesheets and prints similar messages.
    -    If we permit these messages to be printed to standard error, our
    -    documentation CI job will because we check standard error for unexpected
    -    output.  Due to dblatex's reliance on Python 2, we may need to revisit
    -    its use in the future, in which case this problem may go away, but this
    -    can be delayed until a future patch.
    +    occur when invoking dblatex in the CI job.  This tool strips namespaces
    +    much like the unnamespaced DocBook stylesheets and prints similar
    +    messages.  If we permit these messages to be printed to standard error,
    +    our documentation CI job will fail because we check standard error for
    +    unexpected output.  Due to dblatex's reliance on Python 2, we may need
    +    to revisit its use in the future, in which case this problem may go
    +    away, but this can be delayed until a future patch.
     
         Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
     
    @@ Documentation/manpage.xsl (new)
     +	<xsl:import href="http://docbook.sourceforge.net/release/xsl-ns/current/manpages/docbook.xsl" />
     +</xsl:stylesheet>
     
    + ## azure-pipelines.yml ##
    +@@ azure-pipelines.yml: jobs:
    +        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
    + 
    +        sudo apt-get update &&
    +-       sudo apt-get install -y asciidoc xmlto asciidoctor &&
    ++       sudo apt-get install -y asciidoc xmlto asciidoctor docbook-xsl-ns &&
    + 
    +        export ALREADY_HAVE_ASCIIDOCTOR=yes. &&
    +        export jobname=Documentation &&
    +
    + ## ci/install-dependencies.sh ##
    +@@ ci/install-dependencies.sh: StaticAnalysis)
    + 	;;
    + Documentation)
    + 	sudo apt-get -q update
    +-	sudo apt-get -q -y install asciidoc xmlto
    ++	sudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns
    + 
    + 	test -n "$ALREADY_HAVE_ASCIIDOCTOR" ||
    + 	gem install --version 1.5.8 asciidoctor
    +
      ## ci/test-documentation.sh ##
     @@
      filter_log () {

 Documentation/Makefile     | 4 +++-
 Documentation/manpage.xsl  | 3 +++
 azure-pipelines.yml        | 2 +-
 ci/install-dependencies.sh | 2 +-
 ci/test-documentation.sh   | 2 ++
 5 files changed, 10 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/manpage.xsl

On Sat, Sep 14, 2019 at 07:49:19PM +0000, brian m. carlson wrote:

Ok, with this package installed the build passed on Travis CI.

So, since the documentation can now be built with Asciidoctor v2, is it already time to remove this '--version 1.5.8'?

I haven't seen this latter message in the CI builds, neither with Asciidoctor v1.5.8 nor with v2. Do we really need this filter, then? Where does this message come from?

On 2019-09-15 at 09:59:52, SZEDER Gábor wrote:

I think Martin was going to send in some more patches before we did that.

I see it and it definitely fails on my system without it. It comes from libxslt, which has been patched in Debian to produce deterministic IDs. I suspect we may not have seen it on Ubuntu systems because they are running 16.04, which is likely older than the patch. If Travis updates to 18.04, we may be more likely to have a problem.

On Sun, Sep 15, 2019 at 09:26:21PM +0000, brian m. carlson wrote:

Thanks. Indeed, I kicked off a Travis CI build using their Ubuntu 18.04 image, and that "Attributed..." message was there.

I think this future-proofing is a good idea, but I also think that this should be clarified in the commit message.

On 2019-09-15 at 22:05:55, SZEDER Gábor wrote:

I can do that. I just noticed it failed on my laptop and added it, assuming it was the stylesheets. I had to search Google for the output to find out that it was libxslt.

On Sun, 15 Sep 2019 at 23:26, brian m. carlson <sandals@crustytoothpaste.net> wrote:

I've got two series floating around [1] [2]. I'll be rerolling [2] hopefully tonight and then I feel pretty good about the manpages. I've got a third series that I'll get to then, which fixes up the rendering of user-manual.pdf/html with Asciidoctor. Assuming those three series graduate, I'm not aware of any reason to hold off on the switch to "Asciidoctor by default".

As for "v2.x vs 1.5.8", that seems like a separate issue to me, though -- and one that your patch does a very good job at! My series [2] will make the rendering with v2.x /prettier/, but since your series makes the docs build /at all/, maybe it's worth switching the CI-job sooner rather than later to make sure we keep it so.

[1] https://public-inbox.org/git/cover.1567707999.git.martin.agren@gmail.com/

[2] https://public-inbox.org/git/cover.1567534373.git.martin.agren@gmail.com/

Martin

Our documentation toolchain has traditionally been built around DocBook 4.5. This version of DocBook is the last DTD-based version of DocBook. In 2009, DocBook 5 was introduced using namespaces and its syntax is expressed in RELAX NG, which is more expressive and allows a wider variety of syntax forms.

Asciidoctor, one of the alternatives for building our documentation, moved support for DocBook 4.5 out of core in its recent 2.0 release and now only supports DocBook 5 in the main release. The DocBoook 4.5 converter is still available as a separate component, but this is not available in most distro packages. This would not be a problem but for the fact that we use xmlto, which is still stuck in the DocBook 4.5 era.

xmlto performs DTD validation as part of the build process. This is not problematic for DocBook 4.5, which has a valid DTD, but it clearly cannot work for DocBook 5, since no DTD can adequately express its full syntax. In addition, even if xmlto did support RELAX NG validation, that wouldn't be sufficient because it uses the libxml2-based xmllint to do so, which has known problems with validating interleaves in RELAX NG.

Fortunately, there's an easy way forward: ask Asciidoctor to use its DocBook 5 backend and tell xmlto to skip validation. Asciidoctor has supported DocBook 5 since v0.1.4 in 2013 and xmlto has supported skipping validation for probably longer than that.

We also need to teach xmlto how to use the namespaced DocBook XSLT stylesheets instead of the non-namespaced ones it usually uses. Normally these stylesheets are interchangeable, but the non-namespaced ones have a bug that causes them not to strip whitespace automatically from certain elements when namespaces are in use. This results in additional whitespace at the beginning of list elements, which is jarring and unsightly.

We can do this by passing a custom stylesheet with the -x option that simply imports the namespaced stylesheets via a URL. Any system with support for XML catalogs will automatically look this URL up and reference a local copy instead without us having to know where this local copy is located. We know that anyone using xmlto will already have catalogs set up properly since the DocBook 4.5 DTD used during validation is also looked up via catalogs. All major Linux distributions distribute the necessary stylesheets and have built-in catalog support, and Homebrew does as well, albeit with a requirement to set an environment variable to enable catalog support.

On the off chance that someone lacks support for catalogs, it is possible for xmlto (via xmllint) to download the stylesheets from the URLs in question, although this will likely perform poorly enough to attract attention. People still have the option of using the prebuilt documentation that we ship, so happily this should not be an impediment.

Finally, we need to filter out some messages from other stylesheets that occur when invoking dblatex in the CI job. This tool strips namespaces much like the unnamespaced DocBook stylesheets and prints similar messages. If we permit these messages to be printed to standard error, our documentation CI job will fail because we check standard error for unexpected output. Due to dblatex's reliance on Python 2, we may need to revisit its use in the future, in which case this problem may go away, but this can be delayed until a future patch.

The final message we filter is due to libxslt on modern Debian and Ubuntu. The patch which they use to implement reproducible ID generation also prints messages about the ID generation. While this doesn't affect our current CI images since they use Ubuntu 16.04 which lacks this patch, if we upgrade to Ubuntu 18.04 or a modern Debian, these messages will appear and, like the above messages, cause a CI failure.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
Changes since v3:
* Further expanded the commit message to include information about
  libxslt.  It is now just over a hundred words away from classification
  as "sudden nonfiction".

 Documentation/Makefile     | 4 +++-
 Documentation/manpage.xsl  | 3 +++
 azure-pipelines.yml        | 2 +-
 ci/install-dependencies.sh | 2 +-
 ci/test-documentation.sh   | 2 ++
 5 files changed, 10 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/manpage.xsl

Here's v3 of ma/asciidoctor-refmiscinfo to align the headers and footers of Asciidoctor-rendered manpages with those from AsciiDoc. Patch 1 is new.

It turns out that with newer (>=1.5.7) versions of Asciidoctor, we get default values of these refmiscinfo attributes put into the xml. The hack in patch 2/3 (previously patch 1/2) happened to inject these attributes after the default ones and it seems like the earlier ones happen to win when xmlto processes the xml. This explains what Peff noticed in [1], where the attribute values introduced by the hacky patch 2 are effectively ignored and blank values are used instead.

Rather than trying to game the order of those attributes, I decided to do a patch 1 which uses a saner way of providing these attributes to Asciidoctor. This allowed me to reframe patch 2 from "fix all of these attributes with Asciidoctor" to "fix some of them with some Asciidoctor versions".

My test results:

  AsciiDoc (8.6.10): Each patch in this series is a no-op.

  Asciidoctor 1.5.5 and 1.5.6: Patch 1/3 is a no-op. Patch 2/3 fixes the
  header/footer to be as with AsciiDoc, except for how the date is
  rendered.

  Aciidoctor 1.5.7 and 2.0.10 [2]: Patch 1/3 fixes the header/footer
  partially -- the version number is still missing. Patch 2/3 fixes the
  version number, after which the header/footer are just as with
  AsciiDoc, except for how the date is rendered.

  So in short, "all" versions of Asciidoctor now render the headers and
  footers like AsciiDoc does, except for the date formatting.

[1] https://public-inbox.org/git/20190913050634.GB21172@sigill.intra.peff.net/

[2] The tests with 2.0.10 were done on top of brian's recent patch to build at all with Asciidoctor 2: https://public-inbox.org/git/20190915224332.103930-1-sandals@crustytoothpaste.net/

Martin Ågren (3):
  Doc/Makefile: give mansource/-version/-manual attributes
  asciidoctor-extensions: provide `<refmiscinfo/>`
  doc-diff: replace --cut-header-footer with --cut-footer

 Documentation/Makefile                  |  3 ++-
 Documentation/asciidoc.conf             |  6 +++---
 Documentation/asciidoctor-extensions.rb | 17 +++++++++++++++++
 Documentation/doc-diff                  | 17 ++++++++---------
 4 files changed, 30 insertions(+), 13 deletions(-)

Rather than hardcoding "Git Manual" and "Git" as the manual and source in asciidoc.conf, provide them as attributes `manmanual` and `mansource`. Rename the `git_version` attribute to `manversion`.

These new attribute names are not arbitrary, see, e.g., [1].

For AsciiDoc (8.6.10) and Asciidoctor <1.5.7, this is a no-op. Starting with Asciidoctor 1.5.7, `manmanual` and `mansource` actually end up in the xml-files and eventually in the rendered manpages. In particular, the manpage headers now render just as with AsciiDoc.

No versions of Asciidoctor pick up the `manversion` [2], and older versions don't pick up any of these attributes. -- We'll fix that with a bit of a hack in the next commit.

[1] https://asciidoctor.org/docs/user-manual/#man-pages

[2] Note how [1] says "Not used by Asciidoctor".

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 Documentation/Makefile      | 3 ++-
 Documentation/asciidoc.conf | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)

As can be seen from the previous commit, there are three attributes that we provide to AsciiDoc through asciidoc.conf. Asciidoctor ignores that file. After that patch, newer versions of Asciidoctor pick up the `manmanual` and `mansource` attributes as we invoke `asciidoctor`, but they don't pick up `manversion`. ([1] says: "Not used by Asciidoctor.") Older versions (<1.5.7) don't handle these attributes at all. As a result, we are missing one or three `<refmiscinfo/>` tags in each xml-file produced when we build with Asciidoctor.

Because of this, xmlto doesn't include the Git version number in the rendered manpages. And in particular, with versions <1.5.7, the manpage footers instead contain the fairly ugly "[FIXME: source]".

That Asciidoctor ignores asciidoc.conf is nothing new. This is why we implement the `linkgit:` macro in asciidoc.conf *and* in asciidoctor-extensions.rb. Follow suit and provide these tags in asciidoctor-extensions.rb, using a "postprocessor" extension where we just search and replace in the XML, treated as text.

We may consider a few alternatives:

  * Inject these lines into the xml-files from the *Makefile*, e.g.,
    using `sed`. That would reduce repetition, but it feels wrong to
    impose another step and another risk on the AsciiDoc-processing only
    to benefit the Asciidoctor-one.

  * I tried providing a "docinfo processor" to inject these tags, but
    could not figure out how to "merge" the two <refmeta/> sections that
    resulted. To avoid xmlto barfing on the result, I needed to use
    `xmlto --skip-validation ...`, which seems unfortunate.

Let's instead inject the missing tags using a postprocessor. We'll make it fairly obvious that we aim to inject the exact same three lines of `<refmiscinfo/>` that asciidoc.conf provides. We inject them in *post*-processing so we need to do the variable expansion ourselves. We do introduce the bug that asciidoc.conf already has in that we won't do any escaping, e.g., of funky versions like "some v <2.25, >2.20".

The postprocessor we add here works on the XML as raw text and doesn't really use the full potential of XML to do a more structured injection. This is actually precisely what the Asciidoctor User Manual does in its postprocessor example [2]. I looked into two other approaches:

  1. The nokogiri library is apparently the "modern" way of doing XML
     in ruby. I got it working fairly easily:
        require 'nokogiri'
        doc = Nokogiri::XML(output)
        doc.search("refmeta").each { |n| n.add_child(new_tags) }
        output = doc.to_xml
     However, this adds another dependency (e.g., the "ruby-nokogiri"
     package on Ubuntu). Using Asciidoctor is not our default, but it
     will probably need to become so soon. Let's avoid adding a
     dependency just so that we can say "search...add_child" rather than
     "sub(regex...)".

  2. The older REXML is apparently always(?) bundled with ruby, but I
     couldn't even parse the original document:
        require 'rexml/document'
        doc = REXML::Document.new(output)
        ...
     The error was "no implicit conversion of nil into String" and I
     stopped there.

I don't think it's unlikely that doing a plain old search-and-replace will work just as fine or better compared to parsing XML and worrying about libraries and library versions.

[1] https://asciidoctor.org/docs/user-manual/#builtin-attributes

[2] https://asciidoctor.org/docs/user-manual/#postprocessor-example

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 Documentation/asciidoctor-extensions.rb | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

After the previous commit, AsciiDoc and Asciidoctor render the manpage headers identically, so we no longer need the "cut the header" part of our `--cut-header-footer` option. We do still need the "cut the footer" part, though. The previous commits improved the rendering of the footer in Asciidoctor by quite a bit, but the two programs still disagree on how to format the date in the footer: 01/01/1970 vs 1970-01-01.

We could keep using `--cut-header-footer`, but it would be nice if we had a slightly smaller hammer `--cut-footer` that would be less likely to hide regressions. Rather than simply adding such an option, let's also drop `--cut-header-footer`, i.e., rework it to lose the "header" part of its name and functionality.

`--cut-header-footer` is just a developer tool and it probably has no more than a handful of users, so we can afford to be aggressive.

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 Documentation/doc-diff | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

On Mon, Mar 18, 2019 at 10:46:45PM -0400, Jeff King wrote:

I just tried with asciidoc 2.0.0.rc.2, which came out last week. It does seem to work from the command line:

  $ make USE_ASCIIDOCTOR=Yes \
         ASCIIDOC_DOCBOOK=docbook5 \
         ASCIIDOC='asciidoctor -amansource=Git -amanmanual="Git Manual"' \
         git-add.xml
  $ sed -n '/refmeta/,/refmeta/p' git-add.xml
  <refmeta>
  <refentrytitle>git-add</refentrytitle>
  <manvolnum>1</manvolnum>
  <refmiscinfo class="source">Git</refmiscinfo>
  <refmiscinfo class="manual">Git Manual</refmiscinfo>
  </refmeta>

Still no "manversion" attribute, though.

-Peff

On Tue, 19 Mar 2019 at 04:30, Jeff King <peff@peff.net> wrote:

No such luck with asciidoctor 1.5.5. Seems like it really wants "manpage" before it considers these attributes.

(That's still me holding the tool, so factor that into it.)

Martin

On Tue, Mar 19, 2019 at 08:12:54AM +0100, Martin Ågren wrote:

The refmiscinfo stuff didn't arrive until asciidoctor v1.5.7.

-Peff

Jeff King wrote:

Now I don't feel as bad that I didn't find any good way to handle refmiscinfo when I first tried to use asciidoctor in November 2017 at least.

This made me look closer at the fedora asciidoctor packages. They have been stuck on 1.5.6.1. So all my recent testing has been with 1.5.6.1.

I spent some time yesterday working toward getting the fedora packages updated to 1.5.8. With luck that will reach the stable updates channels before too long.

(I'm guessing the upcoming 2.0 release won't be suitable as an update for released fedora versions, being a major release with potentially backwards-incompatible changes.)

There are differences in the output from 1.5.6.1 and 1.5.8, but I haven't looked closely yet.

On Tue, 19 Mar 2019 at 03:46, Jeff King <peff@peff.net> wrote:

Huh, yeah, that's a big improvement already.

The bug is in my docs-reading, see below.

Well, I didn't spend 5 minutes on it, but my experience told me something like that would happen. ;-)

Now I realize that I'm wrong in my "it doesn't work from the command line". Somehow, I read the following in the user manual: "Many attributes can only be defined in the document header (or via the API or CLI)." And *repeatedly* read is as "(not via the API or CLI)", somehow always expecting a "not" to follow an "only". Oh well.

But of course, I did try it out before reaching for the docs, like anyone would. The true reason it doesn't work for me is probably this header from the listing that contains "mansource": "Manpage attributes (relevant only when using the manpage doctype and/or converter)".

Todd has made a promising start in another part of this thread. There seems to be a few wrinkles that need some care, but hopefully nothing impossible (famous last words).

Good point. Hadn't thought of it. You're right that the asciidoc.conf version has the same problem and a version string like "<>" goes unescaped into the xml.

:-)

Martin