podlators v6.0.0 released

[Russ Allbery]

I'm pleased to announce release v6.0.0 of podlators.

Module authors, note that this release increases the minimum supported
Perl version to 5.12. If you have an explicit dependency on podlators or
its constituent modules, directly or indirectly, be aware that this will
now only be satisfied with Perl 5.12 or later.

podlators contains Pod::Man and Pod::Text modules which convert POD input
to *roff source output, suitable for man pages, or plain text. It also
includes several subclasses of Pod::Text for formatted output to terminals
with various capabilities. It is the source package for the Pod::Man and
Pod::Text modules included with Perl.

Changes from previous release:

- Drop support for Perl 5.10. podlators now requires Perl 5.12 or later.

- podlators now uses semantic versioning for the package and module
versions, with a v prefix to work with Perl's packaging system.

- Pod::Man now translates all "-" characters in the input into *roff "\-"
escapes (normally rendered as an ASCII hyphen-minus, U+002D) rather
than using fragile heuristics to decide which characters represent true
hyphens and which represent ASCII hyphen-minus. The previous
heuristics misrendered command names such as apt-get, causing search
and cut-and-paste issues. This change may cause line-break issues with
long hyphenated phrases. In cases where the intent is a true hyphen,
consider using UTF-8 as the POD character set (declared with =encoding)
and using true Unicode hyphens instead of the ASCII "-" character.

- Pod::Man now disables the special *roff interpretation of "`" and "'"
characters as paired quotes everywhere, not just in verbatim text, thus
forcing them to be interpreted as the regular ASCII characters. This
also disables the use of "``" and "''" for paired double-quotes. The
rationale is similar to that for hyphens: there is no way to tell from
the POD source that the special interpretation as quotes is intended.
To produce paired typographic quotes in the output, use UTF-8 and
Unicode paired quote characters.

- Man page references in L<> that are detected as such by Pod::Simple are
now always formatted as man page references even if our normal
heuristic would not detect them. This fixes the formatting of
constructions such as @@RXVT_NAME@@perl(3), which are used by packages
that format a man page with POD and then substitute variables into it
at build time. Thanks to Marco Sirabella for the analysis and an
initial patch. (GitHub #21)

- Add a workaround to Pod::Man to force persistent ragged-right
justification under nroff with groff 1.23.0. Thanks to Guillem Jover
for the report and G. Branden Robinson for the analysis. (GitHub #23)

- Fix wrapping of text with S<> markup in all subclasses of Pod::Text.
Thanks to Jim Avera for the report. (GitHub #24)

- Pod::Man now forces a blank line after a nested list contains only
=item tags without bodies. In previous versions, the blank line before
the next item in the surrounding =over block was not included. Thanks
to Julien ÉLIE for the report. (GitHub #26)

- Import PerlIO before checking for layers so that PerlIO::F_UTF8 is
available, which fixes double-encoding of output when a :utf8 layer is
in place and PerlIO is not imported. Thanks to youpong for the bug
report, James Keenan for the elaboration, and Graham Knop for the fix.
(GitHub #25)

- pod2text --help now exits with status 0, not 1, matching normal UNIX
command behavior and the behavior of pod2man. (GitHub #19)

- Fix tests when NO_COLOR is set in the environment. (GitHub #20)

You can download it from CPAN or from:

<https://www.eyrie.org/~eagle/software/podlators/>

This package is maintained using Git; see the instructions on the above
page to access the Git repository.

Please let me know of any problems or feature requests not already listed
in the TODO file.

[p***@tux.freedom.nl]

Post by Russ Allbery
I'm pleased to announce release v6.0.0 of podlators.
Module authors, note that this release increases the minimum supported
Perl version to 5.12. If you have an explicit dependency on podlators or
its constituent modules, directly or indirectly, be aware that this will
now only be satisfied with Perl 5.12 or later.
podlators contains Pod::Man and Pod::Text modules which convert POD input
to *roff source output, suitable for man pages, or plain text. It also
includes several subclasses of Pod::Text for formatted output to terminals
with various capabilities. It is the source package for the Pod::Man and
Pod::Text modules included with Perl.
- Drop support for Perl 5.10. podlators now requires Perl 5.12 or later.
- podlators now uses semantic versioning for the package and module
versions, with a v prefix to work with Perl's packaging system.
- Pod::Man now translates all "-" characters in the input into *roff "\-"
escapes (normally rendered as an ASCII hyphen-minus, U+002D) rather
than using fragile heuristics to decide which characters represent true
hyphens and which represent ASCII hyphen-minus. The previous
heuristics misrendered command names such as apt-get, causing search
and cut-and-paste issues. This change may cause line-break issues with
long hyphenated phrases. In cases where the intent is a true hyphen,
consider using UTF-8 as the POD character set (declared with =encoding)
and using true Unicode hyphens instead of the ASCII "-" character.
- Pod::Man now disables the special *roff interpretation of "`" and "'"
characters as paired quotes everywhere, not just in verbatim text, thus
forcing them to be interpreted as the regular ASCII characters. This
also disables the use of "``" and "''" for paired double-quotes. The
rationale is similar to that for hyphens: there is no way to tell from
the POD source that the special interpretation as quotes is intended.
To produce paired typographic quotes in the output, use UTF-8 and
Unicode paired quote characters.
- Man page references in L<> that are detected as such by Pod::Simple are
now always formatted as man page references even if our normal
heuristic would not detect them. This fixes the formatting of
that format a man page with POD and then substitute variables into it
at build time. Thanks to Marco Sirabella for the analysis and an
initial patch. (GitHub #21)
- Add a workaround to Pod::Man to force persistent ragged-right
justification under nroff with groff 1.23.0. Thanks to Guillem Jover
for the report and G. Branden Robinson for the analysis. (GitHub #23)

Is this the reason why my carefully aligned text changes to ragged in the
generated man pages? I really do not like that! I see no option to prevent
this new behavior

- • Allowable characters within a "CSV" field include 0x09 ("TAB") and
- the inclusive range of 0x20 (space) through 0x7E (tilde). In binary
+ • Allowable characters within a "CSV" field include 0x09 ("TAB") and
+ the inclusive range of 0x20 (space) through 0x7E (tilde). In binary

What can I do to get the old behavior back?

🐧 git diff doc/CSV_XS.man | wc -l
2473
🐧 git diff doc/CSV_XS.man | grep '^- ' | wc -l
517

Post by Russ Allbery
- Fix wrapping of text with S<> markup in all subclasses of Pod::Text.
Thanks to Jim Avera for the report. (GitHub #24)
- Pod::Man now forces a blank line after a nested list contains only
=item tags without bodies. In previous versions, the blank line before
the next item in the surrounding =over block was not included. Thanks
to Julien ÉLIE for the report. (GitHub #26)
- Import PerlIO before checking for layers so that PerlIO::F_UTF8 is
available, which fixes double-encoding of output when a :utf8 layer is
in place and PerlIO is not imported. Thanks to youpong for the bug
report, James Keenan for the elaboration, and Graham Knop for the fix.
(GitHub #25)
- pod2text --help now exits with status 0, not 1, matching normal UNIX
command behavior and the behavior of pod2man. (GitHub #19)
- Fix tests when NO_COLOR is set in the environment. (GitHub #20)
<https://www.eyrie.org/~eagle/software/podlators/>
This package is maintained using Git; see the instructions on the above
page to access the Git repository.
Please let me know of any problems or feature requests not already listed
in the TODO file.

[p***@tux.freedom.nl]

Post by p***@tux.freedom.nl

Post by Russ Allbery
- Add a workaround to Pod::Man to force persistent ragged-right
justification under nroff with groff 1.23.0. Thanks to Guillem Jover
for the report and G. Branden Robinson for the analysis. (GitHub #23)

Is this the reason why my carefully aligned text changes to ragged in the
generated man pages? I really do not like that! I see no option to prevent
this new behavior
- • Allowable characters within a "CSV" field include 0x09 ("TAB") and
- the inclusive range of 0x20 (space) through 0x7E (tilde). In binary
+ • Allowable characters within a "CSV" field include 0x09 ("TAB") and
+ the inclusive range of 0x20 (space) through 0x7E (tilde). In binary

Ooh, is that why the Text::CSV_XS docs look so weird, with random extra spaces everywhere? See the attached screenshot for what I mean; I've marked the extra spaces in red. Reading this stuff feels like listening to someone who adds random pauses between words.
If this is fixed now, I love it.

Post by p***@tux.freedom.nl
What can I do to get the old behavior back?

I'd rather you didn't. :-(

I understand, as in your screenshot the right margin is not aligned,
which makes the spacing unnatural and weird

I personally find aligned paragraphs *MUCH* easier to read than ragged
text. Which makes me wonder if `perldoc` would be able to implement an
option to make the manual pages aligned if the end-user prefers that.
I'd put that in my ~/.perdocrc or or ~/.config/perldoc or whatever to
do so

For both `man` and `perldoc` on screens wider than 80 characters, you
just proved the point of not doing it the old way, but please also
keep in mind the other type of reader. Having the docs aligned makes
it easier for myself to proofread and improve the content.

[Paul "LeoNerd" Evans]

On Fri, 9 Aug 2024 11:54:13 +0200

Post by p***@tux.freedom.nl
I personally find aligned paragraphs *MUCH* easier to read than ragged
text. Which makes me wonder if `perldoc` would be able to implement an
option to make the manual pages aligned if the end-user prefers that.
I'd put that in my ~/.perdocrc or or ~/.config/perldoc or whatever to
do so
For both `man` and `perldoc` on screens wider than 80 characters, you
just proved the point of not doing it the old way, but please also
keep in mind the other type of reader. Having the docs aligned makes
it easier for myself to proofread and improve the content.

If anyone wanted a "justify full lines" option in App::sdview, I could
easily add such a thing. ;)

[Michael Conrad]

Post by p***@tux.freedom.nl
Is this the reason why my carefully aligned text changes to ragged in the
generated man pages? I really do not like that! I see no option to prevent
this new behavior
-       • Allowable  characters  within  a "CSV" field include 0x09
("TAB") and
-         the inclusive range of 0x20 (space) through 0x7E (tilde). 
In  binary
+       • Allowable characters within a "CSV" field include 0x09
("TAB") and
+         the inclusive range of 0x20 (space) through 0x7E (tilde). 
In binary

Ooh, is that why the Text::CSV_XS docs look so weird, with random
extra spaces everywhere?

I've always had full-justification on every man page.  How have you not
seen that before?

But also, why does pod2man need to have a different default
justification than the rest of the system?

[Russ Allbery]

Post by p***@tux.freedom.nl
Is this the reason why my carefully aligned text changes to ragged in
the generated man pages? I really do not like that! I see no option to
prevent this new behavior

Pod::Man has disabled fully-aligned text since the 2.0.5 release in 2006.
There was a regression with groff 1.23.0 due to (in my opinion) a bug in
groff that caused alignment to be restored following .TD, which meant that
the man page would be rendered ragged-right up until the first =over/=item
block and then switch to fully justified. This was fixed in podlators
v6.0.0 to force ragged-right for the entire man page as intended.

groff supports a way for the user to override the justification at render
time, but unfortunately it doesn't work in combination with a desire to
change the default to ragged-right. To enable it, I have to leave the
default at fully justified, which I consider a regression. I'm hopeful
that can eventually be fixed so that I can change the default but the user
can still override at render time if they desire.

I personally feel fairly strongly about ragged right as a default, but
somewhat more to the point it's been the default for nearly twenty years,
so I'd be reluctant to change it.

I could add an option to Pod::Man and pod2man to override the default at
the *roff generation level, but I haven't done that because such options
are usually not all that useful given how deep man page generation tends
to be inside the build system. There often isn't a straightforward way to
pass options down to that level. However, if anyone would find that
useful, I can add it.

[Russ Allbery]

Post by Russ Allbery
Pod::Man has disabled fully-aligned text since the 2.0.5 release in 2006.
There was a regression with groff 1.23.0 due to (in my opinion) a bug in
groff that caused alignment to be restored following .TD, which meant that
the man page would be rendered ragged-right up until the first =over/=item
block and then switch to fully justified.

That was supposed to be .TP, not .TD.

Apologies, one other correction: podlators has attempted to configure
ragged-right justification for nroff since the very first release of
Pod::Man in 1999. I no longer have the history to check, but that makes
me suspect that this is a default that I copied from the preamble emitted
by Larry Wall and Tom Christiansen's original pod2man script.

The fix in 2.0.5 in 2006 was specific to groff: previously, the
configuration of ragged right was in the preamble of the document, but
groff sets the hypenation in the .TH macro, so the setting was ignored for
groff. I hadn't noticed until I started using Linux regularly around that
time, so 2.0.5 moved the setting to after .TH so that it would be uniform
for both groff and older nroff implementations.

(The change in defaults is only for nroff, not for troff, where full
justification can be done more smoothly.)

I know it's a matter of taste and some people prefer it, but for the
record, my position is that full justification only works well when
flexible whitespace is possible so the spacing for the justification can
be nicely distributed across the line. In nroff, where insertion of full
spaces are required, the necessary variation of interword spacing is more
intrusive. Based on the discussion of this on the groff list, my
understanding is that the original nroff authors agreed with me and used
ragged-right formatting for the terminal, and fully-justified formatting
in nroff is something groff added.

[Karl Williamson]

Post by Russ Allbery
Based on the discussion of this on the groff list, my
understanding is that the original nroff authors agreed with me and used
ragged-right formatting for the terminal, and fully-justified formatting
in nroff is something groff added.

The nroff manual written in 1976 by its original developer, Joseph
Ossanna, includes the .ad command that allows adjustment to be left,
right, both, or centered.

Few people back then (or now) directly interacted with either nroff or
troff. Instead, various macro packages were developed that hid the many
intricate details from the average user. -man for man pages being just
one. I typically used the -mm package, which stood for memorandum
macros. It had the .SA macro that set the adjustment. I remember
including that in the documents I wrote, since the default for that
macro package wasn't to have right adjustment.

I'm not sure that non-ragged is simply a matter of taste. A manager at
$work once told me that serifs were invented centuries ago, not for a
rococo jazzing up of text, as I had theretofore thought, but to help
people like him who had a condition where his eyes otherwise found it
hard to stick to a line of text. Such people sometimes use a ruler to
help them read. Sans-serif fonts are hard for such people. This came
up when I was developing a new set of macros for [nt]roff for making
presentations. Serifs are not great for overhead projections, but an
audience member with this condition is kind of left out. I had never
heard of this before. (It probably led him to his specialty in
computers, which was then called human factors.) So, I can imagine that
non-ragged edges might also be more than taste.

Karl Williamson

[Russ Allbery]

Post by Karl Williamson

Based on the discussion of this on the groff list, my understanding is
that the original nroff authors agreed with me and used ragged-right
formatting for the terminal, and fully-justified formatting in nroff is
something groff added.

The nroff manual written in 1976 by its original developer, Joseph
Ossanna, includes the .ad command that allows adjustment to be left,
right, both, or centered.

Yes, it was always part of the language. I'm only talking about the nroff
defaults if one doesn't explicitly set a justification mode. Those who
are curious can see the whole discussion here:

https://lists.gnu.org/archive/html/groff/2024-03/msg00079.html

Post by Karl Williamson
I'm not sure that non-ragged is simply a matter of taste. A manager at
$work once told me that serifs were invented centuries ago, not for a
rococo jazzing up of text, as I had theretofore thought, but to help
people like him who had a condition where his eyes otherwise found it
hard to stick to a line of text. Such people sometimes use a ruler to
help them read. Sans-serif fonts are hard for such people. This came
up when I was developing a new set of macros for [nt]roff for making
presentations. Serifs are not great for overhead projections, but an
audience member with this condition is kind of left out. I had never
heard of this before. (It probably led him to his specialty in
computers, which was then called human factors.) So, I can imagine that
non-ragged edges might also be more than taste.

I would love to let the user be able to choose. I'm just not sure that
changing the default to fully-justified in order to let the user choose on
man-db/groff systems (by setting MANROFFOPT to -dAD=l, which is rather
underdocumented but does work if you have the right mix of software) is
worth it, given that I got a bug report as soon as this changed and
Pod::Man has been consistent about using ragged-right for well over 20
years.

[Arne Johannessen]

[serifs ... to help people like him who had a condition where his eyes otherwise found it hard to stick to a line of text.]
So, I can imagine that non-ragged edges might also be more than taste.

Yes: Justified text (= non-ragged egdes) can result in large or uneven gaps between words. Such gaps can make text considerably harder to read.

The human eye perceives words as complete visual shapes. Uneven gaps between words are thus similar to
u n ev e n g aps between letters. Sure, it's still readable, but the brain has to do a wee bit more work, which isn't equally easy for everybody.

With a monospaced font, uneven gaps will almost always happen. That's true even with good automatic hyphenation, which is a requirement for properly justified text because it avoids unnecessary large gaps. Therefore, for man pages displayed in a monospaced terminal font, justified text should probably not be the default setting.

The advantage of justified text is the clear delineation of text columns, like in a newspaper. Beyond that, to my knowledge, it's just aesthetics.

[p***@tux.freedom.nl]

Post by Arne Johannessen

[serifs ... to help people like him who had a condition where his eyes otherwise found it hard to stick to a line of text.]
So, I can imagine that non-ragged edges might also be more than taste.

Yes: Justified text (= non-ragged egdes) can result in large or
uneven gaps between words. Such gaps can make text considerably
harder to read.
The human eye perceives words as complete visual shapes. Uneven gaps
between words are thus similar to u n ev e n g aps between letters.
Sure, it's still readable, but the brain has to do a wee bit more
work, which isn't equally easy for everybody.
With a monospaced font, uneven gaps will almost always happen. That's
true even with good automatic hyphenation, which is a requirement for
properly justified text because it avoids unnecessary large gaps.
Therefore, for man pages displayed in a monospaced terminal font,
justified text should probably not be the default setting.
The advantage of justified text is the clear delineation of text
columns, like in a newspaper. Beyond that, to my knowledge, it's just
aesthetics.

As noted by many now, it is probably not just taste. People that has
problems reading might be helped with specialized fonts, for some even
Comic Sans is the best font for their brain (or brain condition).

Personally I refuse to read books, papers, and web-sites that use
serif-fonts. I hate it so much that when https was not common yet, I
wrote a proxy to replace each and every CSS provided by a site was
"corrected" to use sans-serif fonts. After https, I still quite often
got to web-properties to disable the site specific font choices.

Same for ragged vs non-ragged. My brain has more problems to keep track
of position on ragged paragraphs than on justified paragraphs, causing
my brain to read the same line twice or more, which is very annoying. I
rather have the weird spacing than the time lost.

I am willing, given the background posted in this thread, to remove the
alignments in my documentation, but I would be over the moon when
Pod::Man would allow me to set alignment for all screen-oriented
documentation.