threads / rfc / 55640

[RFC suggestion] Generate manpage directly with Asciidoctor

Subject: [RFC suggestion] Generate manpage directly with Asciidoctor

## tl;dr

38 messages between May 7, 2021 and May 12, 2021.

replies: 37people: 9as markdown or json

Bagas Sanjaya· May 7, 2021, 06:06 UTC · lore
Hi,
Asciidoctor has support for directly generating manpage, see [1].

We support using Asciidoctor as drop-in replacement for original Asciidoc, but currently we need to use xmlto together with Asciidoc(tor) to produce manpages. However, most users don't inclined to install xmlto toolchain, partly because they had to download more than 300 MB of data just to install xmlto and its dependencies (including dblatex and texlive).

So completely migrating to Asciidoctor can eliminate xmlto requirement for generating manpage.

What do you think about above?
[1]: https://docs.asciidoctor.org/asciidoctor/latest/manpage-backend/
-- 
An old man doll... just what I always wanted! - Clara
Randall S. Becker· May 7, 2021, 12:02 UTC · re: Bagas Sanjaya · lore

RE: [RFC suggestion] Generate manpage directly with Asciidoctor

On May 7, 2021 2:07 AM, Bagas Sanjaya wrote:
Show 14 quoted lines
>To: Git Users <git@vger.kernel.org>
>Subject: [RFC suggestion] Generate manpage directly with Asciidoctor
>Asciidoctor has support for directly generating manpage, see [1].
>
>We support using Asciidoctor as drop-in replacement for original Asciidoc, but
>currently we need to use xmlto together with Asciidoc(tor) to produce
>manpages. However, most users don't inclined to install xmlto toolchain, partly
>because they had to download more than 300 MB of data just to install xmlto
>and its dependencies (including dblatex and texlive).
>
>So completely migrating to Asciidoctor can eliminate xmlto requirement for
>generating manpage.
>
>What do you think about above?
Our toolchain does not support asciidoctor itself because of porting issues. I am not sure it is available everywhere.
-Randall

-- Brief whoami: NonStop developer since approximately 211288444200000000 UNIX developer since approximately 421664400 -- In my real life, I talk too much.

Felipe Contreras· May 7, 2021, 22:55 UTC · re: Randall S. Becker · lore

RE: [RFC suggestion] Generate manpage directly with Asciidoctor

Randall S. Becker wrote:
Show 7 quoted lines
> On May 7, 2021 2:07 AM, Bagas Sanjaya wrote:
> >To: Git Users <git@vger.kernel.org>
> >Subject: [RFC suggestion] Generate manpage directly with Asciidoctor
> >What do you think about above?
> 
> Our toolchain does not support asciidoctor itself because of porting
> issues. I am not sure it is available everywhere.
Then don't use it.

If I understand correctly the proposal affects people that build with USE_ASCIIDOCTOR=YesPlease only.

-- 
Felipe Contreras
brian m. carlson· May 7, 2021, 22:57 UTC · re: Randall S. Becker · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 2021-05-07 at 12:02:13, Randall S. Becker wrote:
Show 17 quoted lines
> On May 7, 2021 2:07 AM, Bagas Sanjaya wrote:
> >To: Git Users <git@vger.kernel.org>
> >Subject: [RFC suggestion] Generate manpage directly with Asciidoctor
> >Asciidoctor has support for directly generating manpage, see [1].
> >
> >We support using Asciidoctor as drop-in replacement for original Asciidoc, but
> >currently we need to use xmlto together with Asciidoc(tor) to produce
> >manpages. However, most users don't inclined to install xmlto toolchain, partly
> >because they had to download more than 300 MB of data just to install xmlto
> >and its dependencies (including dblatex and texlive).
> >
> >So completely migrating to Asciidoctor can eliminate xmlto requirement for
> >generating manpage.
> >
> >What do you think about above?
> 
> Our toolchain does not support asciidoctor itself because of porting issues. I am not sure it is available everywhere.

I think Asciidoctor is pure Ruby, since it also uses JRuby and Opal to run in Java and JavaScript environments. Ruby is relatively portable to most architectures and systems (for example, it runs on 16 Debian architectures).

Is the problem in your case Ruby or is it Asciidoctor itself? I'm happy to send portability patches to Asciidoctor if necessary, but I'm not sure I'm a good candidate to send patches to Ruby itself, especially for an OS I don't use.

-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US
Randall S. Becker· May 8, 2021, 01:42 UTC · re: brian m. carlson · lore

RE: [RFC suggestion] Generate manpage directly with Asciidoctor

On May 7, 2021 6:58 PM, brian m. carlson wrote:
Show 30 quoted lines
>Subject: Re: [RFC suggestion] Generate manpage directly with Asciidoctor
>
>On 2021-05-07 at 12:02:13, Randall S. Becker wrote:
>> On May 7, 2021 2:07 AM, Bagas Sanjaya wrote:
>> >To: Git Users <git@vger.kernel.org>
>> >Subject: [RFC suggestion] Generate manpage directly with Asciidoctor
>> >Asciidoctor has support for directly generating manpage, see [1].
>> >
>> >We support using Asciidoctor as drop-in replacement for original
>> >Asciidoc, but currently we need to use xmlto together with
>> >Asciidoc(tor) to produce manpages. However, most users don't inclined
>> >to install xmlto toolchain, partly because they had to download more
>> >than 300 MB of data just to install xmlto and its dependencies (including
>dblatex and texlive).
>> >
>> >So completely migrating to Asciidoctor can eliminate xmlto
>> >requirement for generating manpage.
>> >
>> >What do you think about above?
>>
>> Our toolchain does not support asciidoctor itself because of porting issues. I
>am not sure it is available everywhere.
>
>I think Asciidoctor is pure Ruby, since it also uses JRuby and Opal to run in Java
>and JavaScript environments.  Ruby is relatively portable to most architectures
>and systems (for example, it runs on 16 Debian architectures).
>
>Is the problem in your case Ruby or is it Asciidoctor itself?  I'm happy to send
>portability patches to Asciidoctor if necessary, but I'm not sure I'm a good
>candidate to send patches to Ruby itself, especially for an OS I don't use.
In this case, Ruby is not 100% portable to all platforms. I have an old port, but the latest version is not portable because of unportable dependencies - not for a lack of a large amount of trying. So the assumption is not entirely workable. My only option is to take the pre-generated pages from the manpages repo.
-Randall
Đoàn Trần Công Danh· May 7, 2021, 12:27 UTC · re: Bagas Sanjaya · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 2021-05-07 13:06:31+0700, Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> Hi,
> 
> Asciidoctor has support for directly generating manpage, see [1].
While I'm impressed with this news.
Show 6 quoted lines
> We support using Asciidoctor as drop-in replacement for original
> Asciidoc, but currently we need to use xmlto together with Asciidoc(tor)
> to produce manpages. However, most users don't inclined to install
> xmlto toolchain, partly because they had to download more than 300 MB
> of data just to install xmlto and its dependencies (including dblatex
> and texlive).
I'm pretty sure xmlto doesn't depend on dblatex and/or texlive.

I'm building Git and Git's documentation on different Linux boxes with asciidoc, most of them don't have dblatex and/or texlive. In fact, I only have texlive installed in my laptop.

I guess you're using Debian or one of its derived distribution? Debian puts dblatex as xmlto's rec. [2]

I think you can use:
	apt install --no-install-recommends xmlto
to avoid those recommendations.
> So completely migrating to Asciidoctor can eliminate xmlto requirement
> for generating manpage.
> 
> What do you think about above?

Hm, I'm pretty sure Ruby (asciidoctor's language) is very hard to port. Last I heard, it's buggy on some platforms.

I think we're better to do like this:
* If we're using asciidoc, we will use xmlto
* If we're using asciidoctor, we will generate man-pages with
  asciidoctor directly
> [1]: https://docs.asciidoctor.org/asciidoctor/latest/manpage-backend/
[2]: https://packages.debian.org/sid/xmlto
-- 
Danh
Bagas Sanjaya· May 7, 2021, 12:47 UTC · re: Đoàn Trần Công Danh · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 07/05/21 19.27, Đoàn Trần Công Danh wrote:
Show 15 quoted lines
> I'm pretty sure xmlto doesn't depend on dblatex and/or texlive.
> 
> I'm building Git and Git's documentation on different Linux boxes with
> asciidoc, most of them don't have dblatex and/or texlive.
> In fact, I only have texlive installed in my laptop.
> 
> I guess you're using Debian or one of its derived distribution?
> Debian puts dblatex as xmlto's rec. [2]
> 
> I think you can use:
> 
> 	apt install --no-install-recommends xmlto
> 
> to avoid those recommendations.
>
Yes, I'm using Ubuntu (derived from Debian).
Show 9 quoted lines
> Hm, I'm pretty sure Ruby (asciidoctor's language) is very hard to port.
> Last I heard, it's buggy on some platforms.
> 
> I think we're better to do like this:
> 
> * If we're using asciidoc, we will use xmlto
> * If we're using asciidoctor, we will generate man-pages with
>    asciidoctor directly
> 
Nice for your suggestion.

But will the resulting manpage from asciidoc+xmlto and from asciidoctor syntatically and semantically same?

-- 
An old man doll... just what I always wanted! - Clara
Felipe Contreras· May 7, 2021, 23:03 UTC · re: Đoàn Trần Công Danh · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

Đoàn Trần Công Danh wrote:
Show 8 quoted lines
> On 2021-05-07 13:06:31+0700, Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> > So completely migrating to Asciidoctor can eliminate xmlto requirement
> > for generating manpage.
> > 
> > What do you think about above?
> 
> Hm, I'm pretty sure Ruby (asciidoctor's language) is very hard to port.
> Last I heard, it's buggy on some platforms.
And I'm pretty sure it isn't.

Ruby is a mainstream language. At some point it was #8 in the TIOBE index, above PHP and Perl. Right now it's #11.

A lot of effort has been put to make Ruby work on all platforms.
Show 5 quoted lines
> I think we're better to do like this:
> 
> * If we're using asciidoc, we will use xmlto
> * If we're using asciidoctor, we will generate man-pages with
>   asciidoctor directly
This certainly is the way to go.

I'm not sure if Bagas suggested to make USE_ASCIIDOCTOR=YesPlease the default, but first we would need to make sure the output without xmlto is desirable, which we don't at the moment.

So, even if we were to consider this, *first* the option to use pure asciidoctor needs to be in place.

Cheers.
-- 
Felipe Contreras
Bagas Sanjaya· May 8, 2021, 04:27 UTC · re: Felipe Contreras · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 08/05/21 06.03, Felipe Contreras wrote:
  >> I think we're better to do like this:
Show 10 quoted lines
>>
>> * If we're using asciidoc, we will use xmlto
>> * If we're using asciidoctor, we will generate man-pages with
>>    asciidoctor directly
> 
> This certainly is the way to go.
> 
> I'm not sure if Bagas suggested to make USE_ASCIIDOCTOR=YesPlease the
> default, but first we would need to make sure the output without xmlto
> is desirable, which we don't at the moment.

I'm not aiming for defaulting to AsciiDoctor, but I agree that we would like to migrate away from xmlto by making sure the output sans it is desirable.

> So, even if we were to consider this, *first* the option to use pure
> asciidoctor needs to be in place.
Don't you know there is USE_ASCIIDOCTOR=YesPlease flag which do the job?
-- 
An old man doll... just what I always wanted! - Clara
brian m. carlson· May 7, 2021, 20:25 UTC · re: Bagas Sanjaya · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 2021-05-07 at 06:06:31, Bagas Sanjaya wrote:
Show 10 quoted lines
> Hi,
> 
> Asciidoctor has support for directly generating manpage, see [1].
> 
> We support using Asciidoctor as drop-in replacement for original
> Asciidoc, but currently we need to use xmlto together with Asciidoc(tor)
> to produce manpages. However, most users don't inclined to install
> xmlto toolchain, partly because they had to download more than 300 MB
> of data just to install xmlto and its dependencies (including dblatex
> and texlive).

If you're installing on Debian, you want to install with --no-install-recommends to avoid the dependency on dblatex or fop, libpaper-utils, and zip. That should avoid the large downloads.

> So completely migrating to Asciidoctor can eliminate xmlto requirement
> for generating manpage.
> 
> What do you think about above?

I didn't do this because it makes things very complicated in the Makefile and my goal was to minimize the needed changes and divergence. I'm not opposed to someone else doing it, but I expect it will be a bunch of work.

It also doesn't help if you're generating more than just the manual pages, since generating the PDFs will definitely require xmlto and either dblatex or fop. I personally prefer fop over dblatex, but folks have different opinions.

-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US
Jeff King· May 7, 2021, 22:19 UTC · re: brian m. carlson · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Fri, May 07, 2021 at 08:25:36PM +0000, brian m. carlson wrote:
Show 9 quoted lines
> > So completely migrating to Asciidoctor can eliminate xmlto requirement
> > for generating manpage.
> > 
> > What do you think about above?
> 
> I didn't do this because it makes things very complicated in the
> Makefile and my goal was to minimize the needed changes and divergence.
> I'm not opposed to someone else doing it, but I expect it will be a
> bunch of work.

Yeah. In the period where we support both asciidoc and asciidoctor, using the same flow for both (even if it's suboptimal for asciidoctor) kept the complexity down.

I _thought_ the original asciidoc was marked as deprecated / unmaintained at some point. But it does seem to have gotten a few releases in the last year (it looks like maybe the python 2 version was EOL, but somebody decided to make the effort to port it to python 3?).

But I wouldn't be at all sad to just standardize on asciidoctor. I think we're at parity in terms of the output (thanks to lots of work from you and Martin over the past couple of years), and I've generally found it nicer to work with.

The only downside is that it may be available in fewer places (though I'd think that python vs ruby is not so different). IMHO it's OK to be aggressive about the doc toolchain requirements, because the fallback is always grabbing the preformatted roff or HTML pages that were generated on a different system.

-Peff
Bagas Sanjaya· May 8, 2021, 04:22 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 08/05/21 05.19, Jeff King wrote:
> I _thought_ the original asciidoc was marked as deprecated /
> unmaintained at some point. But it does seem to have gotten a few
> releases in the last year (it looks like maybe the python 2 version was
> EOL, but somebody decided to make the effort to port it to python 3?).

Asciidoc.py [1] now supports Python 3, however it only handles legacy AsciiDoc syntax.

[1]: https://github.com/asciidoc-py/asciidoc-py
-- 
An old man doll... just what I always wanted! - Clara
Martin Ågren· May 9, 2021, 08:20 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Sat, 8 May 2021 at 00:19, Jeff King <peff@peff.net> wrote:
Show 10 quoted lines
>
> I _thought_ the original asciidoc was marked as deprecated /
> unmaintained at some point. But it does seem to have gotten a few
> releases in the last year (it looks like maybe the python 2 version was
> EOL, but somebody decided to make the effort to port it to python 3?).
>
> But I wouldn't be at all sad to just standardize on asciidoctor. I think
> we're at parity in terms of the output (thanks to lots of work from you
> and Martin over the past couple of years), and I've generally found it
> nicer to work with.

I tend to think asciidoctor even renders our manpages *better* than asciidoc does. Not by a huge margin, but a few things here and there. Some time around the Python 2 EOL, I was about to propose flipping the default, but then I went to look up the asciidoc EOL schedule, and like you, I noticed that it was a lot more alive and kicking than I thought it was. So it's not so much "we should flip to avoid a bitrotting dependency" as it is "asciidoctor is arguably nicer" or "it's the way forward".

I've done some working-around in the past to try to make something look non-broken in both of asciidoc and asciidoctor. Working with *three* toolchains, we should probably be clear about which one it is we're going to leave behind, and that shouldn't be the default. I'll be happy to make a patch. I can't use the EOL argument, it seems, but I do think it's the only thing that will allow us to reach an xmlto-less generation of the manpages without losing our minds.

Show 5 quoted lines
> The only downside is that it may be available in fewer places (though
> I'd think that python vs ruby is not so different). IMHO it's OK to be
> aggressive about the doc toolchain requirements, because the fallback is
> always grabbing the preformatted roff or HTML pages that were generated
> on a different system.

In general, I agree. I do think it's important that "most people contributing to Git", whatever that means, can build the documentation to check the part they're adding/modifying and not find it broken left and right. They would then (quite rightly) not even bother building it.

When we looked at xmlto-less rendering around two years ago [1], we found various asciidoctor bugs up to and around version 2.0. We would likely need to require some >=2.0.x. The exact requirements will probably only become clear when someone really does the work.

I think what I'm arguing for is
  1) switch the default to asciidoctor,
  2) enable optionally using it without xmlto,
  3) figure out what broke and fix it, and document which is the minimum
     asciidoctor version we're going to bother with for (2),
  4) lather, rinse, repeat (3),
  5) switch the default to not using xmlto,
  6) drop the xmlto way of generating the manpages(?).
[1] https://lore.kernel.org/git/20190317194431.GY31362@pobox.com/
Martin
Felipe Contreras· May 9, 2021, 18:46 UTC · re: Martin Ågren · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Sun, May 9, 2021 at 3:26 AM Martin Ågren <martin.agren@gmail.com> wrote:
Show 9 quoted lines
> I think what I'm arguing for is
>
>   1) switch the default to asciidoctor,
>   2) enable optionally using it without xmlto,
>   3) figure out what broke and fix it, and document which is the minimum
>      asciidoctor version we're going to bother with for (2),
>   4) lather, rinse, repeat (3),
>   5) switch the default to not using xmlto,
>   6) drop the xmlto way of generating the manpages(?).

Are there some minimal requirements to say; this documentation was built correctly (as far as we know)? If so, maybe we can add a checker, or perhaps add a test under t/.

Cheers.
-- 
Felipe Contreras
Martin Ågren· May 10, 2021, 18:43 UTC · re: Felipe Contreras · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Sun, 9 May 2021 at 20:46, Felipe Contreras <felipe.contreras@gmail.com> wrote:

>
> Are there some minimal requirements to say; this documentation was
> built correctly (as far as we know)? If so, maybe we can add a
> checker, or perhaps add a test under t/.

There's `make check-doc` which does some linting. But that's about checking that there's a manpage at all for each builtin, that the doc sources list sections in the right order, and such things. It doesn't actually build the docs.

ci/test-documentation.sh runs `make check-doc` (see above), then goes on to actually build the docs using both asciidoc and asciidoctor. It checks the exit status, but also that stderr is empty after filtering out some expected, harmless output.

I think that's about the right balance. We could perhaps do something under t/, but it probably shouldn't be to actually build the stuff. More like, "oh, you've built the documentation -- let me sanity-check it for you". One way to detect that it makes sense to check it might be to see if the Git version in the manpage footer(s) matches the version under test. But I don't know what to check next. (Actually, if we do find the version number there, we have verified a fairly tricky piece of content injection. At least "tricky" as in "we have more than one way of doing it, because we support more than one toolchain" [1]. And if that stuff broke, the test wouldn't notice, if we relied on detecting a version match to even start testing...)

[1] See 7a30134358 ("asciidoctor-extensions: provide `<refmiscinfo/>`", 2019-09-16) and its parent commit.

Martin
Jeff King· May 10, 2021, 22:24 UTC · re: Martin Ågren · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Sun, May 09, 2021 at 10:20:37AM +0200, Martin Ågren wrote:
Show 7 quoted lines
> > But I wouldn't be at all sad to just standardize on asciidoctor. I think
> > we're at parity in terms of the output (thanks to lots of work from you
> > and Martin over the past couple of years), and I've generally found it
> > nicer to work with.
> 
> I tend to think asciidoctor even renders our manpages *better* than
> asciidoc does. Not by a huge margin, but a few things here and there.
Yeah, I'd agree with that.
Show 6 quoted lines
> Some time around the Python 2 EOL, I was about to propose flipping the
> default, but then I went to look up the asciidoc EOL schedule, and like
> you, I noticed that it was a lot more alive and kicking than I thought
> it was. So it's not so much "we should flip to avoid a bitrotting
> dependency" as it is "asciidoctor is arguably nicer" or "it's the way
> forward".
I'm OK with those arguments, too. :)
Show 10 quoted lines
> > The only downside is that it may be available in fewer places (though
> > I'd think that python vs ruby is not so different). IMHO it's OK to be
> > aggressive about the doc toolchain requirements, because the fallback is
> > always grabbing the preformatted roff or HTML pages that were generated
> > on a different system.
> 
> In general, I agree. I do think it's important that "most people
> contributing to Git", whatever that means, can build the documentation
> to check the part they're adding/modifying and not find it broken left
> and right. They would then (quite rightly) not even bother building it.

Agreed. But I think that is mostly the case (asciidoctor seems no harder to acquire on most modern systems than asciidoc; there are system packages in most cases, and decent binary-package systems for both ruby and python if you really need it).

It does create a situation where people like Randall on NonStop might need to do part of their dev work on another, more mainstream platform. But I suspect that is already the case.

> When we looked at xmlto-less rendering around two years ago [1], we
> found various asciidoctor bugs up to and around version 2.0. We would
> likely need to require some >=2.0.x. The exact requirements will
> probably only become clear when someone really does the work.

That does make things a little less convenient; Debian stable, for instance, still has 1.5.8. It's not too hard to install an updated gem, but not quite as nice as using the system package (it also makes things weird for building the stable Debian package itself, which would want to rely only on other packages; but of course any proposed change to the doc toolchain would be for new versions, and would not get backported there anyway).

Show 9 quoted lines
> I think what I'm arguing for is
> 
>   1) switch the default to asciidoctor,
>   2) enable optionally using it without xmlto,
>   3) figure out what broke and fix it, and document which is the minimum
>      asciidoctor version we're going to bother with for (2),
>   4) lather, rinse, repeat (3),
>   5) switch the default to not using xmlto,
>   6) drop the xmlto way of generating the manpages(?).

I'm unclear when support for python asciidoc goes away here. Is it part of step 6 (because it does not have another way of generating them)? Or does it live on forever as a non-default legacy system? I'd prefer not, but as long as we are clear about the primary target and leave it up to people interested in the legacy to do the compat fixes, that might be OK.

-Peff
Felipe Contreras· May 11, 2021, 04:27 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

Jeff King wrote:
Show 14 quoted lines
> On Sun, May 09, 2021 at 10:20:37AM +0200, Martin Ågren wrote:
> > In general, I agree. I do think it's important that "most people
> > contributing to Git", whatever that means, can build the documentation
> > to check the part they're adding/modifying and not find it broken left
> > and right. They would then (quite rightly) not even bother building it.
> 
> Agreed. But I think that is mostly the case (asciidoctor seems no harder
> to acquire on most modern systems than asciidoc; there are system
> packages in most cases, and decent binary-package systems for both ruby
> and python if you really need it).
> 
> It does create a situation where people like Randall on NonStop might
> need to do part of their dev work on another, more mainstream platform.
> But I suspect that is already the case.
Or use distributed tarballs with already built documentation.
I don't see any big issue with cross-compilation though.
Show 7 quoted lines
> > When we looked at xmlto-less rendering around two years ago [1], we
> > found various asciidoctor bugs up to and around version 2.0. We would
> > likely need to require some >=2.0.x. The exact requirements will
> > probably only become clear when someone really does the work.
> 
> That does make things a little less convenient; Debian stable, for
> instance, still has 1.5.8.
And it has git 2.20.1, released at the end of 2018.

I've never understood developers worried about how the bleeding edge would build in ancient platforms, when ancient platforms don't care about the bleeding edge.

Show 5 quoted lines
> It's not too hard to install an updated gem, but not quite as nice as
> using the system package (it also makes things weird for building the
> stable Debian package itself, which would want to rely only on other
> packages; but of course any proposed change to the doc toolchain would
> be for new versions, and would not get backported there anyway).

Anyone trying to build git master on top of Debian stable 1. probably can live with the output of the current doc toolchain, and 2. probably doesn't exist.

I'm not sure how much the git project gains by worrying about such a hypothetical person.

Show 16 quoted lines
> > I think what I'm arguing for is
> > 
> >   1) switch the default to asciidoctor,
> >   2) enable optionally using it without xmlto,
> >   3) figure out what broke and fix it, and document which is the minimum
> >      asciidoctor version we're going to bother with for (2),
> >   4) lather, rinse, repeat (3),
> >   5) switch the default to not using xmlto,
> >   6) drop the xmlto way of generating the manpages(?).
> 
> I'm unclear when support for python asciidoc goes away here. Is it part
> of step 6 (because it does not have another way of generating them)? Or
> does it live on forever as a non-default legacy system? I'd prefer not,
> but as long as we are clear about the primary target and leave it up to
> people interested in the legacy to do the compat fixes, that might be
> OK.

How about we leave the legacy system in place as an alternative, and decide later what to do with it?

-- 
Felipe Contreras
Jeff King· May 11, 2021, 06:13 UTC · re: Felipe Contreras · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Mon, May 10, 2021 at 11:27:54PM -0500, Felipe Contreras wrote:
Show 17 quoted lines
> Jeff King wrote:
> > On Sun, May 09, 2021 at 10:20:37AM +0200, Martin Ågren wrote:
> > > In general, I agree. I do think it's important that "most people
> > > contributing to Git", whatever that means, can build the documentation
> > > to check the part they're adding/modifying and not find it broken left
> > > and right. They would then (quite rightly) not even bother building it.
> > 
> > Agreed. But I think that is mostly the case (asciidoctor seems no harder
> > to acquire on most modern systems than asciidoc; there are system
> > packages in most cases, and decent binary-package systems for both ruby
> > and python if you really need it).
> > 
> > It does create a situation where people like Randall on NonStop might
> > need to do part of their dev work on another, more mainstream platform.
> > But I suspect that is already the case.
> 
> Or use distributed tarballs with already built documentation.

For users, yes. But the context above is about people who are contributing to Git, and writing their own new documentation. Presumably they'd like to build it to see the output.

Show 8 quoted lines
> > That does make things a little less convenient; Debian stable, for
> > instance, still has 1.5.8.
> 
> And it has git 2.20.1, released at the end of 2018.
> 
> I've never understood developers worried about how the bleeding edge
> would build in ancient platforms, when ancient platforms don't care
> about the bleeding edge.

Again, this is about developers. Are there people contributing new documentation to Git who are doing so on Debian stable, and would be inconvenienced by needing to upgrade their toolchain?

Show 19 quoted lines
> > > I think what I'm arguing for is
> > > 
> > >   1) switch the default to asciidoctor,
> > >   2) enable optionally using it without xmlto,
> > >   3) figure out what broke and fix it, and document which is the minimum
> > >      asciidoctor version we're going to bother with for (2),
> > >   4) lather, rinse, repeat (3),
> > >   5) switch the default to not using xmlto,
> > >   6) drop the xmlto way of generating the manpages(?).
> > 
> > I'm unclear when support for python asciidoc goes away here. Is it part
> > of step 6 (because it does not have another way of generating them)? Or
> > does it live on forever as a non-default legacy system? I'd prefer not,
> > but as long as we are clear about the primary target and leave it up to
> > people interested in the legacy to do the compat fixes, that might be
> > OK.
> 
> How about we leave the legacy system in place as an alternative, and
> decide later what to do with it?
That's what I was asking.

Leaving it forever does mean supporting xmlto forever, which complicates the Makefile (and that support will bitrot if people are not actually building it).

-Peff
Felipe Contreras· May 11, 2021, 08:03 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

Jeff King wrote:
Show 11 quoted lines
> On Mon, May 10, 2021 at 11:27:54PM -0500, Felipe Contreras wrote:
> > Jeff King wrote:
> > > It does create a situation where people like Randall on NonStop might
> > > need to do part of their dev work on another, more mainstream platform.
> > > But I suspect that is already the case.
> > 
> > Or use distributed tarballs with already built documentation.
> 
> For users, yes. But the context above is about people who are
> contributing to Git, and writing their own new documentation. Presumably
> they'd like to build it to see the output.

I'm a developer, I've added 261 lines and removed 212 lines to the documentation, and I've very rarely built it. Why? Because it takes too long.

But you were talking about Randall, who I don't know what his role is, but my bet is that he is a packager. It's not the same thing. Randall has contributed only one patch to the Documentation, and it was to install pre-formatted documentation. Precisely what I thought he would be interested in.

In fact, the bulk of his contribution was to install-doc-quick.sh, which needs git-htmldocs and git-manpages. Two repositories I didn't even know existed. So that's yet another option for him.

Of course Randall can say if he sees himself as a developer.
Show 12 quoted lines
> > > That does make things a little less convenient; Debian stable, for
> > > instance, still has 1.5.8.
> > 
> > And it has git 2.20.1, released at the end of 2018.
> > 
> > I've never understood developers worried about how the bleeding edge
> > would build in ancient platforms, when ancient platforms don't care
> > about the bleeding edge.
> 
> Again, this is about developers. Are there people contributing new
> documentation to Git who are doing so on Debian stable, and would be
> inconvenienced by needing to upgrade their toolchain?

Developers don't need to create (or use) debian packages. They can simply do `gem install asciidoctor` and be done with it. Some may even create a docker container to install all the doc toolchain in order to avoid polluting their main environment.

I for one would start building the documentation more if all I needed is one dependency.

Show 15 quoted lines
> > > I'm unclear when support for python asciidoc goes away here. Is it part
> > > of step 6 (because it does not have another way of generating them)? Or
> > > does it live on forever as a non-default legacy system? I'd prefer not,
> > > but as long as we are clear about the primary target and leave it up to
> > > people interested in the legacy to do the compat fixes, that might be
> > > OK.
> > 
> > How about we leave the legacy system in place as an alternative, and
> > decide later what to do with it?
> 
> That's what I was asking.
> 
> Leaving it forever does mean supporting xmlto forever, which complicates
> the Makefile (and that support will bitrot if people are not actually
> building it).

Indeed. If and when it's clear the xmlto part has bitrotten, and people are happy with the asciidoc toolchain and output, then it can be obsoleted.

That would be my vote (I don't think there will be a strong need to main the xmlto parts).

It doesn't need to be decided today though.
-- 
Felipe Contreras
Ævar Arnfjörð Bjarmason· May 11, 2021, 12:44 UTC · re: Felipe Contreras · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Tue, May 11 2021, Felipe Contreras wrote:
Show 19 quoted lines
> Jeff King wrote:
>> On Mon, May 10, 2021 at 11:27:54PM -0500, Felipe Contreras wrote:
>> > Jeff King wrote:
>> [...]
>> > I've never understood developers worried about how the bleeding edge
>> > would build in ancient platforms, when ancient platforms don't care
>> > about the bleeding edge.
>> 
>> Again, this is about developers. Are there people contributing new
>> documentation to Git who are doing so on Debian stable, and would be
>> inconvenienced by needing to upgrade their toolchain?
>
> Developers don't need to create (or use) debian packages. They can
> simply do `gem install asciidoctor` and be done with it. Some may even
> create a docker container to install all the doc toolchain in order to
> avoid polluting their main environment.
>
> I for one would start building the documentation more if all I needed is
> one dependency.

Just because I'm developing the latest git.git revision on Debian stable that doesn't mean that I'm keen to install the very latest openssl, libcurl, asciidoc, C compiler, or whatever other thing we depend on.

I'm not disagreeing with bumping the dependency in this case (I haven't looked into it). I'm just pointing out that in general there's a lot of use-cases for e.g. building a latest git on an N year old OS.

Of course we can ask these people to just build their dependencies too, as I noted in [1] in a past discussion. Whether we bump our required dependencies is a trade-off between our own convenience and these sorts of in-the-wild builds.

I'm just saying we should keep this use-case in mind, it's not an all or nothing where you either have ancient deps + ancient git or bleeding edge deps + bleeding edge git. A lot of people build ancient deps + bleeding edge git.

The "just use the built doc tarballs" is only a partial solution, and e.g. won't work for someone who's interested in building "next" or otherwise applying local patches that have doc changes.

1. https://lore.kernel.org/git/874ltg2tvo.fsf@gmail.com/
Felipe Contreras· May 11, 2021, 19:00 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

Ævar Arnfjörð Bjarmason wrote:
Show 29 quoted lines
> On Tue, May 11 2021, Felipe Contreras wrote:
> 
> > Jeff King wrote:
> >> On Mon, May 10, 2021 at 11:27:54PM -0500, Felipe Contreras wrote:
> >> > Jeff King wrote:
> >> [...]
> >> > I've never understood developers worried about how the bleeding edge
> >> > would build in ancient platforms, when ancient platforms don't care
> >> > about the bleeding edge.
> >> 
> >> Again, this is about developers. Are there people contributing new
> >> documentation to Git who are doing so on Debian stable, and would be
> >> inconvenienced by needing to upgrade their toolchain?
> >
> > Developers don't need to create (or use) debian packages. They can
> > simply do `gem install asciidoctor` and be done with it. Some may even
> > create a docker container to install all the doc toolchain in order to
> > avoid polluting their main environment.
> >
> > I for one would start building the documentation more if all I needed is
> > one dependency.
> 
> Just because I'm developing the latest git.git revision on Debian stable
> that doesn't mean that I'm keen to install the very latest openssl,
> libcurl, asciidoc, C compiler, or whatever other thing we depend on.
> 
> I'm not disagreeing with bumping the dependency in this case (I haven't
> looked into it). I'm just pointing out that in general there's a lot of
> use-cases for e.g. building a latest git on an N year old OS.

I'm not disagreeing with that. The reason I mentioned it is what Jeff said next:

Show 6 quoted lines
> > > It's not too hard to install an updated gem, but not quite as nice
> > > as using the system package (it also makes things weird for
> > > building the stable Debian package itself, which would want to
> > > rely only on other packages; but of course any proposed change to
> > > the doc toolchain would be for new versions, and would not get
> > > backported there anyway).

Doing `gem install` solves the problem for whomever wants to build the latest git in Debian stable.

Building Debian stable packages is something else.
Show 9 quoted lines
> Of course we can ask these people to just build their dependencies too,
> as I noted in [1] in a past discussion. Whether we bump our required
> dependencies is a trade-off between our own convenience and these sorts
> of in-the-wild builds.
> 
> I'm just saying we should keep this use-case in mind, it's not an all or
> nothing where you either have ancient deps + ancient git or bleeding
> edge deps + bleeding edge git. A lot of people build ancient deps +
> bleeding edge git.
But what is "this"?
 1) Building the latest documentation in Debian stable
 2) Packaging the latest git in Debian stable

If it's 1) then there's no problem: `gem install` does the trick, and in fact in their CI builds [1] they test with versions of Ruby as old as 2.3, and Debian stable ships with 2.5.

I test my own Ruby code with versions of Ruby as old as 2.0 and there's rarely any issue. What works with Ruby 2.3 works with Ruby 2.0. Ruby is not like python (where 3.0 is completely different from 2.0).

If it's 2) then that's where I say: who cares?
> The "just use the built doc tarballs" is only a partial solution, and
> e.g. won't work for someone who's interested in building "next" or
> otherwise applying local patches that have doc changes.

Right, but "just use the built doc tarballs" is not the suggestion for people trying to do 1).

Cheers.
[1] https://github.com/asciidoctor/asciidoctor/actions/runs/830132862
-- 
Felipe Contreras
Jeff King· May 11, 2021, 19:09 UTC · re: Felipe Contreras · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Tue, May 11, 2021 at 02:00:55PM -0500, Felipe Contreras wrote:
Show 11 quoted lines
> > > > It's not too hard to install an updated gem, but not quite as nice
> > > > as using the system package (it also makes things weird for
> > > > building the stable Debian package itself, which would want to
> > > > rely only on other packages; but of course any proposed change to
> > > > the doc toolchain would be for new versions, and would not get
> > > > backported there anyway).
> 
> Doing `gem install` solves the problem for whomever wants to build the
> latest git in Debian stable.
> 
> Building Debian stable packages is something else.

Perhaps I wasn't clear with the "but of course" part of my statement. It was meant to rebut the earlier half of the parenthetical. I.e., drawing that same distinction as "we don't have to worry about packaging Debian stable here".

-Peff
Felipe Contreras· May 11, 2021, 20:22 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

Jeff King wrote:
Show 18 quoted lines
> On Tue, May 11, 2021 at 02:00:55PM -0500, Felipe Contreras wrote:
> 
> > > > > It's not too hard to install an updated gem, but not quite as nice
> > > > > as using the system package (it also makes things weird for
> > > > > building the stable Debian package itself, which would want to
> > > > > rely only on other packages; but of course any proposed change to
> > > > > the doc toolchain would be for new versions, and would not get
> > > > > backported there anyway).
> > 
> > Doing `gem install` solves the problem for whomever wants to build the
> > latest git in Debian stable.
> > 
> > Building Debian stable packages is something else.
> 
> Perhaps I wasn't clear with the "but of course" part of my statement. It
> was meant to rebut the earlier half of the parenthetical. I.e., drawing
> that same distinction as "we don't have to worry about packaging Debian
> stable here".
Ah, I undesrstood wrong there, it seems we are in agreement.

Semicolons in English are similar to semicolons in computer languages: when I read ($a; $b), $a needs to stand on its own. If $b is really important I would do ($a, but $b). But this is like splitting hairs of splitted hairs.

I've seen too much discussion and too few patches, I'll send the mockup I did a few days ago to see if it sticks.

Cheers.
-- 
Felipe Contreras
brian m. carlson· May 11, 2021, 23:14 UTC · re: Felipe Contreras · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 2021-05-11 at 04:27:54, Felipe Contreras wrote:
> I've never understood developers worried about how the bleeding edge
> would build in ancient platforms, when ancient platforms don't care
> about the bleeding edge.

Debian stable is a common environment to do development on. I know people who do use it for Git development, so I suspect we'll want to continue to support it. In fact, most of my colleagues who use a Debian system for development use Debian stable, I'd say.

Moreover, in some cases, the distros one can use for development are restricted due to requirements for software that runs on corporate machines, so making things work nicely on the latest stable versions of Debian and Ubuntu is generally kind.

Yes, people _can_ run "gem install asciidoctor", but people who are not Ruby developers generally would prefer a distro package over installing one-off gems, especially since getting the binaries into PATH is tricky with gem.

Show 9 quoted lines
> > It's not too hard to install an updated gem, but not quite as nice as
> > using the system package (it also makes things weird for building the
> > stable Debian package itself, which would want to rely only on other
> > packages; but of course any proposed change to the doc toolchain would
> > be for new versions, and would not get backported there anyway).
> 
> Anyone trying to build git master on top of Debian stable 1. probably
> can live with the output of the current doc toolchain, and 2. probably
> doesn't exist.
I believe I have just demonstrated that 2 is false above.
Show 19 quoted lines
> > > I think what I'm arguing for is
> > > 
> > >   1) switch the default to asciidoctor,
> > >   2) enable optionally using it without xmlto,
> > >   3) figure out what broke and fix it, and document which is the minimum
> > >      asciidoctor version we're going to bother with for (2),
> > >   4) lather, rinse, repeat (3),
> > >   5) switch the default to not using xmlto,
> > >   6) drop the xmlto way of generating the manpages(?).
> > 
> > I'm unclear when support for python asciidoc goes away here. Is it part
> > of step 6 (because it does not have another way of generating them)? Or
> > does it live on forever as a non-default legacy system? I'd prefer not,
> > but as long as we are clear about the primary target and leave it up to
> > people interested in the legacy to do the compat fixes, that might be
> > OK.
> 
> How about we leave the legacy system in place as an alternative, and
> decide later what to do with it?

I think it would be fine to just leave it in place for now and let people decide which toolchain they'd like to use.

-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US
Felipe Contreras· May 12, 2021, 01:44 UTC · re: brian m. carlson · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

brian m. carlson wrote:
Show 8 quoted lines
> On 2021-05-11 at 04:27:54, Felipe Contreras wrote:
> > I've never understood developers worried about how the bleeding edge
> > would build in ancient platforms, when ancient platforms don't care
> > about the bleeding edge.
> 
> Debian stable is a common environment to do development on.  I know
> people who do use it for Git development, so I suspect we'll want to
> continue to support it.
I didn't mean to suggest otherwise.
> Yes, people _can_ run "gem install asciidoctor", but people who are not
> Ruby developers generally would prefer a distro package over installing
> one-off gems, especially since getting the binaries into PATH is tricky
> with gem.
Yes, but there's only so much we can hold hands with our users.
If a user:
 1. Uses an acient distribution
 2. Wants to build the documentation
 3. Enables USE_ASCIIDOCTOR
 4. Doesn't know Ruby
 5. Wants to use distribution packages
 6. Is bothered by the output

I think it's valid for the project to say "you are on your own". In fact, not really that because if they contact the mailing list we would help them.

The only thing we could do is print a warning if they try to build with versions of asciidoctor that we know are problematic.

That being said; it's not "tricky" to get binaries into your PATH:
  export PATH="$GEM_HOME/bin:$PATH"
And you don't need to get gem binaries into your PATH:
  export GEM_HOME=/tmp/gems
  gem install asciidoctor
  make USE_ASCIIDOCTOR=YesPlease ASCIIDOC=$GEM_HOME/bin/asciidoctor doc
Works just fine without modifying PATH.
Show 11 quoted lines
> > > It's not too hard to install an updated gem, but not quite as nice as
> > > using the system package (it also makes things weird for building the
> > > stable Debian package itself, which would want to rely only on other
> > > packages; but of course any proposed change to the doc toolchain would
> > > be for new versions, and would not get backported there anyway).
> > 
> > Anyone trying to build git master on top of Debian stable 1. probably
> > can live with the output of the current doc toolchain, and 2. probably
> > doesn't exist.
> 
> I believe I have just demonstrated that 2 is false above.
I meant trying to build the documentation of git on git master.
Show 22 quoted lines
> > > > I think what I'm arguing for is
> > > > 
> > > >   1) switch the default to asciidoctor,
> > > >   2) enable optionally using it without xmlto,
> > > >   3) figure out what broke and fix it, and document which is the minimum
> > > >      asciidoctor version we're going to bother with for (2),
> > > >   4) lather, rinse, repeat (3),
> > > >   5) switch the default to not using xmlto,
> > > >   6) drop the xmlto way of generating the manpages(?).
> > > 
> > > I'm unclear when support for python asciidoc goes away here. Is it part
> > > of step 6 (because it does not have another way of generating them)? Or
> > > does it live on forever as a non-default legacy system? I'd prefer not,
> > > but as long as we are clear about the primary target and leave it up to
> > > people interested in the legacy to do the compat fixes, that might be
> > > OK.
> > 
> > How about we leave the legacy system in place as an alternative, and
> > decide later what to do with it?
> 
> I think it would be fine to just leave it in place for now and let
> people decide which toolchain they'd like to use.
Agreed.
-- 
Felipe Contreras
Martin Ågren· May 11, 2021, 18:45 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Tue, 11 May 2021 at 00:24, Jeff King <peff@peff.net> wrote:
Show 8 quoted lines
>
> On Sun, May 09, 2021 at 10:20:37AM +0200, Martin Ågren wrote:
>
> > it was. So it's not so much "we should flip to avoid a bitrotting
> > dependency" as it is "asciidoctor is arguably nicer" or "it's the way
> > forward".
>
> I'm OK with those arguments, too. :)
Excellent. :)
Show 12 quoted lines
> > When we looked at xmlto-less rendering around two years ago [1], we
> > found various asciidoctor bugs up to and around version 2.0. We would
> > likely need to require some >=2.0.x. The exact requirements will
> > probably only become clear when someone really does the work.
>
> That does make things a little less convenient; Debian stable, for
> instance, still has 1.5.8. It's not too hard to install an updated gem,
> but not quite as nice as using the system package (it also makes things
> weird for building the stable Debian package itself, which would want to
> rely only on other packages; but of course any proposed change to the
> doc toolchain would be for new versions, and would not get backported
> there anyway).

Right. And 1.5.8 is perfectly fine for ascidoctor *with* xmlto, i.e., as long as we're discussing moving away from asciidoc, not moving away from xmlto entirely. And soon enough, Debian stable should be at 2.12. ;-) (I realize Debian stable was just an example.)

Show 16 quoted lines
> > I think what I'm arguing for is
> >
> >   1) switch the default to asciidoctor,
> >   2) enable optionally using it without xmlto,
> >   3) figure out what broke and fix it, and document which is the minimum
> >      asciidoctor version we're going to bother with for (2),
> >   4) lather, rinse, repeat (3),
> >   5) switch the default to not using xmlto,
> >   6) drop the xmlto way of generating the manpages(?).
>
> I'm unclear when support for python asciidoc goes away here. Is it part
> of step 6 (because it does not have another way of generating them)? Or
> does it live on forever as a non-default legacy system? I'd prefer not,
> but as long as we are clear about the primary target and leave it up to
> people interested in the legacy to do the compat fixes, that might be
> OK.

I think I meant it to happen somewhere in step 3 or 4, I just didn't call it out. If it happens super-early in step 3, that would perhaps be a bit unfortunate. But if fixing up the last few bits of xmlto-vs-no-xmlto with asciidoctor involves giving up on asciidoc, then so be it... If nothing else, we might all of a sudden find that our asciidoc-processed docs look awful and that nobody seems to have noticed.

Martin
Jeff King· May 11, 2021, 19:07 UTC · re: Martin Ågren · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Tue, May 11, 2021 at 08:45:10PM +0200, Martin Ågren wrote:
Show 12 quoted lines
> > That does make things a little less convenient; Debian stable, for
> > instance, still has 1.5.8. It's not too hard to install an updated gem,
> > but not quite as nice as using the system package (it also makes things
> > weird for building the stable Debian package itself, which would want to
> > rely only on other packages; but of course any proposed change to the
> > doc toolchain would be for new versions, and would not get backported
> > there anyway).
> 
> Right. And 1.5.8 is perfectly fine for ascidoctor *with* xmlto, i.e., as
> long as we're discussing moving away from asciidoc, not moving away from
> xmlto entirely. And soon enough, Debian stable should be at 2.12. ;-)
> (I realize Debian stable was just an example.)

Debian stable is just an example, but I also consider it a bit of a benchmark for "reasonable". Surely there are people running RHEL6 somewhere in this world, but it's hard to care too much about them.

I think the transition you're proposing would probably take a while to do, too. If we don't drop the python asciidoc support until close to the end, then that buys a bit more time. Likewise, this isn't a hard limit for OS support for users. The worst case is just making things slightly more inconvenient for Git developers on older systems, because because they might have to install an updated gem rather than using the system package (you sometimes can end up in dependency hell for a gem upgrade with versions of ruby, system libraries, etc, but I haven't found asciidoctor particularly needy in that respect).

So I dunno. I certainly don't have a big complaint about _starting_ the transition. If we can hold on to python asciidoc support (or even old-asciidoctor + xmlto) for a while as a fallback, even if we know it's slowly bitrotting, then it's possible nobody would even complain.

-Peff
Martin Ågren· May 11, 2021, 19:11 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Tue, 11 May 2021 at 21:07, Jeff King <peff@peff.net> wrote:
Show 5 quoted lines
>
> So I dunno. I certainly don't have a big complaint about _starting_ the
> transition. If we can hold on to python asciidoc support (or even
> old-asciidoctor + xmlto) for a while as a fallback, even if we know it's
> slowly bitrotting, then it's possible nobody would even complain.

You made several good points; I just quoted the last few here. Thanks for your thoughts on this.

Martin
Felipe Contreras· May 11, 2021, 20:14 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

Jeff King wrote:
Show 6 quoted lines
> The worst case is just making things slightly more inconvenient for
> Git developers on older systems, because because they might have to
> install an updated gem rather than using the system package (you
> sometimes can end up in dependency hell for a gem upgrade with
> versions of ruby, system libraries, etc, but I haven't found
> asciidoctor particularly needy in that respect).

The asciidoctor gem has *zero* dependencies. If you end up in a dependency hell, it wasn't the fault of asciidoctor, and it won't be affected.

I've been using Ruby for at least 15 years, and I've never found myself in a dependency hell, so it would be interesting to see how anybody could end up there. Sometimes the mix between system and user gems is confusing, but they work fine. I wouldn't call that a "dependency hell".

And of course you can always just `rm -r $GEM_HOME` and start fresh. Most gems install rather quickly.

-- 
Felipe Contreras
Jean-Noël Avila· May 11, 2021, 09:04 UTC · re: Martin Ågren · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 09/05/2021 at 10:20, Martin Ågren wrote :
Show 9 quoted lines
>
> I tend to think asciidoctor even renders our manpages *better* than
> asciidoc does. Not by a huge margin, but a few things here and there.
> Some time around the Python 2 EOL, I was about to propose flipping the
> default, but then I went to look up the asciidoc EOL schedule, and like
> you, I noticed that it was a lot more alive and kicking than I thought
> it was. So it's not so much "we should flip to avoid a bitrotting
> dependency" as it is "asciidoctor is arguably nicer" or "it's the way
> forward".

If we start to change the documentation format to "the way  forward", we may soon end up with a format which is no longer handled by the legacy asciidoc.py

As stated on https://github.com/asciidoc-py/asciidoc-py :

"AsciiDoc.py is a legacy processor for this syntax, handling an older rendition of AsciiDoc. As such, this will not properly handle the current AsciiDoc specification. It is suggested that unless you specifically require the AsciiDoc.py toolchain, you should find a processor that handles the modern AsciiDoc syntax."

So, as soon as the asciidoc format formal specification will gain traction in the public, we can expect asciidoc to be abandoned for new projects and receive minimal maintenance only for compatibility with legacy documentation.

One argument in favor of Asciidoctor is that it's delivered "with batteries", meaning that you can generate manpages, html and even pdf with the same tool, without requiring secondary or even tertiary toolchains, which should ease usage on a broader range of platforms.

FWIW, we are already using Asciidoctor for publishing the manpages to https://git-scm.com

JN
Martin Ågren· May 11, 2021, 18:54 UTC · re: Jean-Noël Avila · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Tue, 11 May 2021 at 11:04, Jean-Noël Avila <avila.jn@gmail.com> wrote:
Show 15 quoted lines
>
> On 09/05/2021 at 10:20, Martin Ågren wrote :
> >
> > I tend to think asciidoctor even renders our manpages *better* than
> > asciidoc does. Not by a huge margin, but a few things here and there.
> > Some time around the Python 2 EOL, I was about to propose flipping the
> > default, but then I went to look up the asciidoc EOL schedule, and like
> > you, I noticed that it was a lot more alive and kicking than I thought
> > it was. So it's not so much "we should flip to avoid a bitrotting
> > dependency" as it is "asciidoctor is arguably nicer" or "it's the way
> > forward".
>
> If we start to change the documentation format to "the way  forward", we
> may soon end up with a format which is no longer handled by the legacy
> asciidoc.py

We used to be in a situation where Asciidoctor looked worse and the rendered versions were quite different. We've fixed up quite a few discrepancies by making some change that "happens" to be a noop with one engine but an improvement with the other. (That it just "happens" has sometimes been my feeling anyway.) Sometimes, we've been able to improve both by spotting a difference, so that's good.

I would also expect that with more eyes on asciidoctor-built docs (because default) and fewer on the other, the non-default will start to degrade.

Show 7 quoted lines
> As stated on https://github.com/asciidoc-py/asciidoc-py :
>
> "AsciiDoc.py is a legacy processor for this syntax, handling an older
> rendition of AsciiDoc. As such, this will not properly handle the
> current AsciiDoc specification. It is suggested that unless you
> specifically require the AsciiDoc.py toolchain, you should find a
> processor that handles the modern AsciiDoc syntax."
Thanks for that quote. It's very enlightening.
> FWIW, we are already using Asciidoctor for publishing the manpages to
> https://git-scm.com
Thank you for all your work on that site!
Martin
Felipe Contreras· May 7, 2021, 23:35 UTC · re: brian m. carlson · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

brian m. carlson wrote:
Show 10 quoted lines
> On 2021-05-07 at 06:06:31, Bagas Sanjaya wrote:
> > So completely migrating to Asciidoctor can eliminate xmlto requirement
> > for generating manpage.
> > 
> > What do you think about above?
> 
> I didn't do this because it makes things very complicated in the
> Makefile and my goal was to minimize the needed changes and divergence.
> I'm not opposed to someone else doing it, but I expect it will be a
> bunch of work.
I don't think it would be that complicated.
This patch seems to do the trick here:
diff --git a/Documentation/Makefile b/Documentation/Makefile
index c2baad0bd8..8fa4896d22 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -189,6 +189,7 @@ ASCIIDOC_EXTRA += -alitdd='&\#x2d;&\#x2d;'
 DBLATEX_COMMON =
 XMLTO_EXTRA += --skip-validation
 XMLTO_EXTRA += -x manpage.xsl
+TXT_TO_MAN = $(ASCIIDOC_COMMON) -b manpage
 endif
 
 SHELL_PATH ?= $(SHELL)
@@ -360,9 +361,16 @@ $(OBSOLETE_HTML): %.html : %.txto asciidoc.conf asciidoctor-extensions.rb GIT-AS
 manpage-base-url.xsl: manpage-base-url.xsl.in
 	$(QUIET_GEN)sed "s|@@MAN_BASE_URL@@|$(MAN_BASE_URL)|" $< > $@
 
+ifdef TXT_TO_MAN
+%.1 %.5 %.7 : %.txt asciidoc.conf asciidoctor-extensions.rb GIT-ASCIIDOCFLAGS
+	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
+	$(TXT_TO_MAN) -o $@+ $< && \
+	mv $@+ $@
+else
 %.1 %.5 %.7 : %.xml manpage-base-url.xsl $(wildcard manpage*.xsl)
 	$(QUIET_XMLTO)$(RM) $@ && \
 	$(XMLTO) -m $(MANPAGE_XSL) $(XMLTO_EXTRA) man $<
+endif
 
 %.xml : %.txt asciidoc.conf asciidoctor-extensions.rb GIT-ASCIIDOCFLAGS
 	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \

I'm sure this can be improved with some knowledge about intermediary
rules.

> It also doesn't help if you're generating more than just the manual
> pages, since generating the PDFs will definitely require xmlto and
> either dblatex or fop.  I personally prefer fop over dblatex, but folks
> have different opinions.

Huh? user-manual.pdf seems to require DBLATEX, not XMLTO.

Cheers.
-- 
Felipe Contreras
brian m. carlson· May 7, 2021, 23:57 UTC · re: Felipe Contreras · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 2021-05-07 at 23:35:21, Felipe Contreras wrote:
Show 36 quoted lines
> I don't think it would be that complicated.
> 
> This patch seems to do the trick here:
> 
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index c2baad0bd8..8fa4896d22 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -189,6 +189,7 @@ ASCIIDOC_EXTRA += -alitdd='&\#x2d;&\#x2d;'
>  DBLATEX_COMMON =
>  XMLTO_EXTRA += --skip-validation
>  XMLTO_EXTRA += -x manpage.xsl
> +TXT_TO_MAN = $(ASCIIDOC_COMMON) -b manpage
>  endif
>  
>  SHELL_PATH ?= $(SHELL)
> @@ -360,9 +361,16 @@ $(OBSOLETE_HTML): %.html : %.txto asciidoc.conf asciidoctor-extensions.rb GIT-AS
>  manpage-base-url.xsl: manpage-base-url.xsl.in
>  	$(QUIET_GEN)sed "s|@@MAN_BASE_URL@@|$(MAN_BASE_URL)|" $< > $@
>  
> +ifdef TXT_TO_MAN
> +%.1 %.5 %.7 : %.txt asciidoc.conf asciidoctor-extensions.rb GIT-ASCIIDOCFLAGS
> +	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> +	$(TXT_TO_MAN) -o $@+ $< && \
> +	mv $@+ $@
> +else
>  %.1 %.5 %.7 : %.xml manpage-base-url.xsl $(wildcard manpage*.xsl)
>  	$(QUIET_XMLTO)$(RM) $@ && \
>  	$(XMLTO) -m $(MANPAGE_XSL) $(XMLTO_EXTRA) man $<
> +endif
>  
>  %.xml : %.txt asciidoc.conf asciidoctor-extensions.rb GIT-ASCIIDOCFLAGS
>  	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> 
> I'm sure this can be improved with some knowledge about intermediary
> rules.

It's not quite that simple. It requires changes to asciidoctor-extensions.rb and we also need to consider NO_MAN_BOLD_LITERAL and GNU_ROFF. The latter especially is likely to be inconvenient, since according to the Makefile, it's impossible to write man output that works across troff versions.

We will also want to preserve the ability to use xmlto because we currently have doc-diff which helps us find divergences in the behavior of AsciiDoc and Asciidoctor, many of which affect all formats, and that will be completely useless if we use two different rendering toolchains.

So, yes, this is a good start, but it's not quite as trivial as it seems.
Show 6 quoted lines
> > It also doesn't help if you're generating more than just the manual
> > pages, since generating the PDFs will definitely require xmlto and
> > either dblatex or fop.  I personally prefer fop over dblatex, but folks
> > have different opinions.
> 
> Huh? user-manual.pdf seems to require DBLATEX, not XMLTO.
Ah, yes, because we do it manually instead of using xmlto.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US
Jeff King· May 8, 2021, 03:10 UTC · re: brian m. carlson · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Fri, May 07, 2021 at 11:57:32PM +0000, brian m. carlson wrote:
Show 10 quoted lines
> It's not quite that simple.  It requires changes to
> asciidoctor-extensions.rb and we also need to consider
> NO_MAN_BOLD_LITERAL and GNU_ROFF.  The latter especially is likely to be
> inconvenient, since according to the Makefile, it's impossible to write
> man output that works across troff versions.
> 
> We will also want to preserve the ability to use xmlto because we
> currently have doc-diff which helps us find divergences in the behavior
> of AsciiDoc and Asciidoctor, many of which affect all formats, and that
> will be completely useless if we use two different rendering toolchains.

Speaking of doc-diff: with that patch applied, try "./doc-diff --asciidoctor HEAD^ HEAD", which shows quite a few of the complications (another one seems to be that all of the linkgit macros are broken).

-Peff
Jeff King· May 8, 2021, 03:23 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Fri, May 07, 2021 at 11:10:50PM -0400, Jeff King wrote:
Show 16 quoted lines
> On Fri, May 07, 2021 at 11:57:32PM +0000, brian m. carlson wrote:
> 
> > It's not quite that simple.  It requires changes to
> > asciidoctor-extensions.rb and we also need to consider
> > NO_MAN_BOLD_LITERAL and GNU_ROFF.  The latter especially is likely to be
> > inconvenient, since according to the Makefile, it's impossible to write
> > man output that works across troff versions.
> > 
> > We will also want to preserve the ability to use xmlto because we
> > currently have doc-diff which helps us find divergences in the behavior
> > of AsciiDoc and Asciidoctor, many of which affect all formats, and that
> > will be completely useless if we use two different rendering toolchains.
> 
> Speaking of doc-diff: with that patch applied, try "./doc-diff
> --asciidoctor HEAD^ HEAD", which shows quite a few of the complications
> (another one seems to be that all of the linkgit macros are broken).

Hmph. Actually, this seems to give inconsistent results. There definitely is a problem. Try:

  cd Documentation
  make USE_ASCIIDOCTOR=1 git.1
  man -l git.1 >old
  git am .../the_patch
  make USE_ASCIIDOCTOR=1 git.1
  man -l git.1 >new
  diff -u old new

which shows it. I think the patch does not specify the dependencies to make correctly (and doc-diff reuses the worktree to avoid work). Doing this:

  git am .../the_patch
  ./doc-diff --asciidoctor HEAD HEAD
  ./doc-diff --asciidoctor HEAD^ HEAD
works around it and shows the extent of the problem.

(Not really a critique on the patch, which I know was meant to be a quick illustration, but I thought it worth pointing out a doc-diff command which actually works).

-Peff
brian m. carlson· May 9, 2021, 00:22 UTC · re: Jeff King · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On 2021-05-08 at 03:10:50, Jeff King wrote:
Show 16 quoted lines
> On Fri, May 07, 2021 at 11:57:32PM +0000, brian m. carlson wrote:
> 
> > It's not quite that simple.  It requires changes to
> > asciidoctor-extensions.rb and we also need to consider
> > NO_MAN_BOLD_LITERAL and GNU_ROFF.  The latter especially is likely to be
> > inconvenient, since according to the Makefile, it's impossible to write
> > man output that works across troff versions.
> > 
> > We will also want to preserve the ability to use xmlto because we
> > currently have doc-diff which helps us find divergences in the behavior
> > of AsciiDoc and Asciidoctor, many of which affect all formats, and that
> > will be completely useless if we use two different rendering toolchains.
> 
> Speaking of doc-diff: with that patch applied, try "./doc-diff
> --asciidoctor HEAD^ HEAD", which shows quite a few of the complications
> (another one seems to be that all of the linkgit macros are broken).

Yes, that's because the linkgit macros are specified on a format-specific basis by the extensions file and without updating they won't understand how to generate output for the manpage formatter.

-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US
Martin Ågren· May 9, 2021, 08:29 UTC · re: Felipe Contreras · lore

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

On Sat, 8 May 2021 at 01:40, Felipe Contreras <felipe.contreras@gmail.com> wrote:

Show 16 quoted lines
>
> brian m. carlson wrote:
> > On 2021-05-07 at 06:06:31, Bagas Sanjaya wrote:
> > > So completely migrating to Asciidoctor can eliminate xmlto requirement
> > > for generating manpage.
> > >
> > > What do you think about above?
> >
> > I didn't do this because it makes things very complicated in the
> > Makefile and my goal was to minimize the needed changes and divergence.
> > I'm not opposed to someone else doing it, but I expect it will be a
> > bunch of work.
>
> I don't think it would be that complicated.
>
> This patch seems to do the trick here:

This looks similar to Todd's wip [1]. Here's something whitespace-damaged on top that improves some things, but I'm sure there's more to do. I think in order to work on this, it would need to be a separate switch so that we can use asciidoctor both the old-school and the new-school way for a while.

[1] https://lore.kernel.org/git/20190317194431.GY31362@pobox.com/
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 0cfa88a92b..9f73d2d033 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -192,7 +192,9 @@ ASCIIDOC_HTML = xhtml5
 ASCIIDOC_DOCBOOK = docbook5
 ASCIIDOC_EXTRA += -acompat-mode -atabsize=8
 ASCIIDOC_EXTRA += -I. -rasciidoctor-extensions
-ASCIIDOC_EXTRA += -alitdd='&\#x2d;&\#x2d;'
+ASCIIDOC_EXTRA += -alitdd='\--'
+ASCIIDOC_EXTRA += -aplus='+'
+ASCIIDOC_COMMON += -a mansource="Git $(GIT_VERSION)" -a manmanual="Git Manual"
 DBLATEX_COMMON =
 XMLTO_EXTRA += --skip-validation
 XMLTO_EXTRA += -x manpage.xsl
diff --git a/Documentation/asciidoctor-extensions.rb
b/Documentation/asciidoctor-extensions.rb
index d906a00803..fb74443241 100644
--- a/Documentation/asciidoctor-extensions.rb
+++ b/Documentation/asciidoctor-extensions.rb
@@ -10,11 +10,16 @@ module Git

       def process(parent, target, attrs)
         prefix = parent.document.attr('git-relative-html-prefix')
+        s = "#{target}(#{attrs[1]})"
         if parent.document.doctype == 'book'
           "<ulink url=\"#{prefix}#{target}.html\">" \
           "#{target}(#{attrs[1]})</ulink>"
+        elsif parent.document.basebackend? 'manpage'
+          # FIXME: boldness should respect NO_MAN_BOLD_LITERAL?
+          # -- should be doable using an attribute, similar
+          # to "prefix" above
+          create_inline(parent, :quoted, s, type: :strong).convert
         elsif parent.document.basebackend? 'html'
           %(<a href="#{prefix}#{target}.html">#{target}(#{attrs[1]})</a>)
         elsif parent.document.basebackend? 'docbook'
           "<citerefentry>\n" \
             "<refentrytitle>#{target}</refentrytitle>" \
Felipe Contreras· May 7, 2021, 22:48 UTC · re: Bagas Sanjaya · lore

RE: [RFC suggestion] Generate manpage directly with Asciidoctor

Bagas Sanjaya wrote:
Show 13 quoted lines
> Asciidoctor has support for directly generating manpage, see [1].
> 
> We support using Asciidoctor as drop-in replacement for original
> Asciidoc, but currently we need to use xmlto together with Asciidoc(tor)
> to produce manpages. However, most users don't inclined to install
> xmlto toolchain, partly because they had to download more than 300 MB
> of data just to install xmlto and its dependencies (including dblatex
> and texlive).
> 
> So completely migrating to Asciidoctor can eliminate xmlto requirement
> for generating manpage.
> 
> What do you think about above?

Makes complete sense. If only we had some patches to go along with that proposal.

-- 
Felipe Contreras

← back to recent threads