% language=us runpath=texruns:manuals/ontarget

% timestamp: Scary Pockets with Leland Sklar: https://www.youtube.com/watch?v=1UOVLdPslmo
% around 19 minutes in ... interesting! Hopefully I once see LS live!

% we can yscale too

\usebodyfont[modern,10pt]
\usebodyfont[lucida,10pt]
\usebodyfont[pagella,10pt]
\usebodyfont[bonum,10pt]
\usebodyfont[antykwa,10pt]

\startcomponent ontarget-profiles

\environment ontarget-style

\startchapter[title={Profiles}]

\startsection[title=Introduction]

Among the typesetting problems that relate to math are inline formulas that have
a bit too much height or depth but not so much as to justify some additional
interline space. For that reason in \CONTEXT\ \MKII\ we have some snapping
features that can be enabled that limit the dimensions. In \MKIV\ a more
extensive profile feature was written (we talked about it at meetings in 2015)
that look at the bottom and top of lines in order to determine if lines can be
moved closer, but in practice snapping and profiling are never really used. In
the end it was more an academic exercise which is not uncommon when it comes to
\TEX\ user demands and practice.

As part of exploring math micro typography these features surfaced again during
some discussion about weird mechanisms and we actually wondered if we could revive
this now that we also control other aspects of math typesetting in more detail.
One condition is that the overhead is not that high. Users accept some overhead
for protrusion and expansion that relate to horizontal optimization so a little
extra overhead for vertical optimization should not be a problem.

Of course, as with protrusion and especially expansion, the question is if
readers will notice it. Best would be to set up some experiments but, although
one can argue that research is important, in practice it always boils down to a
visual impression, feel good and, like it or not, exploration, trial and error.
And so a simplified variant of profiling was implemented and applied to a math
intensive math book used in academia. Instead of proper experiments some unaware
bystanders were asked if they noticed a difference and to our surprise that was
the case! And these were not even math students but kids who were more familiar
with children books and phones. That convinced us that we were on the right
track, that we need to explain a little about what we actually do, and that we
should tell users what to look at when this gets applied.

With an introduction like this, mentioning \quote {research}, \quote {academia},
\quote {students} and \quote {typography} we're sure that future generations will
be convinced that what  is discussed next has a strong fundament so here we go!

\stopsection

\startsection[title=A first example]

We start with a simple example. Because in practice profiling always kicks in
when possible one really need to handcraft an example that can be used for
demonstration: \in {figure} [fig:profile].

\definelineprofile[MyLineProfileA][step=1pt,factor=0.125]
\definelineprofile[MyLineProfileB][step=1pt,factor=0.250]
\definelineprofile[MyLineProfileC][step=1pt,factor=1.000]

\startbuffer
    \hsize 7cm
    no no no no  no  no no no no no no no no no no no no \im{x^{2^{A^P}}+x_{2_g}}. no no no no no no no no no  no  no no no no no no no no no no no no no \im{\frac{1+A^{2^x}}{2+B_2}} no no no no no no no no no no no no no no no no no.\blank
    no no no yes yes no no no no no no no no no no no    \im{x^{2^{A^P}}+x_{2_g}}. no no no no no no no no yes yes no no no no no no no no no no no no \im{\frac{1+A^{2^x}}{2+B_2}} no no no no no no no no no no no no no no no no no.\par
\stopbuffer

\startbuffer[set]
\showmakeup[line]
\enabletrackers[profiling.lines.show]
\stopbuffer

\startbuffer[reset]
\disabletrackers[profiling.lines.show]
\stopbuffer

\startplacefigure[reference=fig:profile]
\startcombination[2*2]
    {\getbuffer[set]\vbox{\hsize7cm\resetlineprofile                \getbuffer}\getbuffer[reset]}{\type{no profile}}
    {\getbuffer[set]\vbox{\hsize7cm\setlineprofile  [MyLineProfileA]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=0.125}}
    {\getbuffer[set]\vbox{\hsize7cm\setlineprofile  [MyLineProfileB]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=0.250}}
    {\getbuffer[set]\vbox{\hsize7cm\setlineprofile  [MyLineProfileC]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=1.000}}
\stopcombination
\stopplacefigure

In order to see what happens it is important to understand how \TEX\ sees lines.
Actually, the concept of lines in \TEX\ is rather limited: lines are just
horizontal boxes where the baselines are separated by \type {\baselineskip} and
when the distance is larger than that dimensions a \type {\lineskip} gets added.

\appendtoks\showmakeup[reset]\to\everybeforeoutput

\startlines \showmakeup[line]
these are a few
lines of text
where lines have depths
or no real
height at all
\stoplines

Between all these lines some skip needs to be added: the \type {\baselineskip}
minus the height and depth. If we add struts the lines get the optimal height and
depth so then no skips are inserted:

\startlines \showmakeup[line] \showstruts
\strut these are a few
\strut lines of text
\strut where lines have depths
\strut or no real
\strut height at all
\stoplines

When we increase the depth a little, for instance 1.2 times the normal strut depth,
we see that some additional space, the \type {\lineskip} gets added:

\startlines \showmakeup[line] \showstruts
\strut these {\vrule depth 1.2\strutdp width .25em \relax} are a few
\strut lines of {\vrule depth 1.2\strutdp width .25em \relax} text
\strut where lines have {\vrule depth 1.2\strutdp width .25em \relax} depths
\strut or {\vrule depth 1.2\strutdp width .25em \relax} no real
\strut height at {\vrule depth 1.2\strutdp width .25em \relax} all
\stoplines

However, there is no real reason to do that here because the
larger rules don't clash with text or other content. So this is what we get when
we pass the \type {profile} option to \type {\setupalign}:

\def\MyStrut{\vrule depth 1.2\strutdp width .25em \relax}

\enabletrackers[profiling.lines.show]
\startpacked \setupalign[profile]
\strut these {\MyStrut} are a few \crlf
\strut lines of {\MyStrut} text \crlf
\strut where lines have {\MyStrut} depths \crlf
\strut or {\MyStrut} no real \crlf
\strut height at {\MyStrut} all
\stoppacked
\disabletrackers[profiling.lines.show]

Profiling works per paragraph, so when we add a \type {\par} in the middle we get
this:

\enabletrackers[profiling.lines.show]
\startpacked \setupalign[profile]
\strut these {\MyStrut} are a few \crlf
\strut lines of {\MyStrut} text  \crlf
\strut where lines have {\MyStrut} depths \par
\strut or {\MyStrut} no real  \crlf
\strut height at {\MyStrut} all
\stoppacked
\disabletrackers[profiling.lines.show]

But, we can actually setup the profiler to look back. Setting up the main
(document) profiler happens with:

\starttyping
\setuplineprofile
  [factor=0.125,      % default
   paragraph=yes,     % default: no
   step=0.5\emwidth]  % default
\stoptyping

but as with most \CONTEXT\ mechanisms you can define your own profiler. The step
tells what granularity to use when comparing positions in a line. The factor sets
the threshold for the interline skip. We saw these two differ in the first
example we gave.

\stopsection

\startsection[title=Profiled math]

We will give several examples of math profiling.
In the examples we will switch font to Latin Modern,
since the effect is more visible for that font.
Most of our examples will be \quotation{real} (slightly modified) ones,
but we start with a rather artificial example. Below we have
two occurrences of a fraction. Note that the profiling only kicks
in for the second one. The reason is that on the line above the first
one we only have letters (x) with no depth, while in the second one,
we have added one letter (g) that has depth.

\startbuffer \hsize 8cm
xxx xxxx xxxx xx xxx xx xx xxx xxxx xxx xx xxx xxx xx xx xxx x xxxxx xxxxx
xxx xxxx xxxx xx xxx xx xx xxx xxxx xxx xx xxx xxx xx xx xxx x xxxxx xxxxx
\im{\frac{1}{2}} xxx xxx xx xx xxx xxxx xxxxxxx xxxx xxxx xxxxx xxx xxxx xxx
xxx xxx xxx xxx xxx xxxx xxxxxx xxxxxxx xxxgxxx xxx xxx xxx xx x xxxxxxx xxx
\im{\frac{1}{2}} xxx xxxx xxxx xx xxx xx xx xxx xxxx xxx xx xxx xxx xx xx
xxx x xxxxx xxxxx.\par
\stopbuffer

\starttextdisplay
\switchtobodyfont[modern,9pt]
\startcombination[nx=2,ny=1,distance=1cm]
    {\getbuffer[set]\vtop{\hsize8cm\resetlineprofile                \getbuffer}\getbuffer[reset]}{\type{no profile}           }
    {\getbuffer[set]\vtop{\hsize8cm\setlineprofile  [MyLineProfileA]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=0.125}}
\stopcombination
\stoptextdisplay

We next show a simple paragraph where the mechanism gets applied in three out of five line breaks.

\startbuffer \hsize 8cm
The results of Section 6.3 show that the same phenomenon is encountered when
treating the norms of inverses: \im{\fenced[doublebar][size=0]{B_n^{-1}}_p}
converges to \im{N_p} very fast if \im{N_p > N_p^0}, while the convergence
may be slow if \im{N_p = N_p^0}. As the following proposition reveals, at
least for \im{p = 2} the strict inequality \im{N_2 > N_2^0} is the generic
case.\par
\stopbuffer

\starttextdisplay
\startcombination[nx=2,ny=1,distance=1cm]
    {\switchtobodyfont[modern,9pt]\getbuffer[set]\vtop{\hsize8cm\resetlineprofile                \getbuffer}\getbuffer[reset]}{\type{no profile}           }
    {\switchtobodyfont[modern,9pt]\getbuffer[set]\vtop{\hsize8cm\setlineprofile  [MyLineProfileA]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=0.125}}
\stopcombination
\stoptextdisplay

We have shown the lines and used the helper to show where the profiling
is applied. We show the same example but without these helpers. After
all, this is how we usually see it.

\starttextdisplay
\startcombination[nx=2,ny=1,distance=1cm]
    {\switchtobodyfont[modern,9pt]\vtop{\hsize8cm\resetlineprofile                \getbuffer}\getbuffer[reset]}{\type{no profile}           }
    {\switchtobodyfont[modern,9pt]\vtop{\hsize8cm\setlineprofile  [MyLineProfileA]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=0.125}}
\stopcombination
\stoptextdisplay

If the paragraph is slightly reformulated, the profiling might change.
Below we show an example where the subscript (p) on the fourth line gets
too close to the superscript (0) on the last line.

\startbuffer \hsize 8cm
The results of Section 6.3 show that the same phenomenon is encountered when
treating the norms of inverses: \im{\fenced[doublebar][size=0]{B_n^{-1}}_p}
converges to \im{N_p} very fast if \im{N_p > N_p^0}, while the convergence
may be slow if \im{N_p = N_p^0}.
At least for \im{p = 2} the strict inequality \im{N_2 > N_2^0} is the generic
case.\par
\stopbuffer

\starttextdisplay
\startcombination[nx=2,ny=1,distance=1cm]
    {\switchtobodyfont[modern,9pt]\getbuffer[set]\vtop{\hsize8cm\resetlineprofile                \getbuffer}\getbuffer[reset]}{\type{no profile}           }
    {\switchtobodyfont[modern,9pt]\getbuffer[set]\vtop{\hsize8cm\setlineprofile  [MyLineProfileA]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=0.125}}
\stopcombination
\stoptextdisplay

We can configure the amount of space that shall be added with the \type{factor} key.

% MS: Fixing a shift issue in the one below took one timestamp loop:

\starttextdisplay
\startcombination[nx=2,ny=1,distance=1cm]
    {\switchtobodyfont[modern,9pt]\getbuffer[set]\vtop{\hsize8cm\setlineprofile  [MyLineProfileA]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=0.125}    }
    {\switchtobodyfont[modern,9pt]\getbuffer[set]\vtop{\hsize8cm\setlineprofile  [MyLineProfileC]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=1}}
\stopcombination
\stoptextdisplay

\startbuffer \hsize 8cm
This shows that the sequence \im{(f_n)} is a Cauchy sequence in \im{L^1(\reals)}
and therefore it converges in norm to some \im{\sum_{k=1}^n \hat{f} \in L^1(\reals)}, by
Thereom~2.8.1. On the other hand, by Theorem~2.8.2, there exists an increasing
sequence of positive integers \im{q_n} such that \im{f_{q_n} \to \tilde{f}}
almost everywhere.\par
\stopbuffer

\starttextdisplay
\startcombination[nx=2,ny=2,distance=1cm]
    {\switchtobodyfont[modern,9pt]\getbuffer[set]\vtop{\hsize8cm\resetlineprofile                \getbuffer}\getbuffer[reset]}{\type{no profile}           }
    {\switchtobodyfont[modern,9pt]\getbuffer[set]\vtop{\hsize8cm\setlineprofile  [MyLineProfileA]\getbuffer}\getbuffer[reset]}{\type{step=1pt,factor=0.125}}
\stopcombination
\stoptextdisplay

\stopsection

\startsection[title=Line spacing]

When we enable the line profiler on a 300 page math course with plenty inline
formulas, the number of \quote {corrections} varies a lot with the fonts. Some
simple tests show that Latin Modern, Bonum and EBGaramond get quite some applied,
while Lucida, Dejavu, Antykwa, Erewhon and   Libertinus only see a few corrections.
Pagella, Termes and StixTwo end up in the middle.

The trigger is not always text or math. The course material has quite some
structure, like numbered descriptions. In \CONTEXT\ we use plenty of struts to
make sure that spacing is consistent and the keyword that starts a description
therefore gets them. Normally that is not an issue but when the height of the
next line exceeds the strut height we get a clash and line skip will be added.
One can argue that the strut spoils the typesetting but in general it does more
good than harm, at least in \CONTEXT. It looks like the profiler is quite capable
of getting rid of the cases where it interferes (or more precisely: where it
doesn't run into the next line).

The reason why we get a line skip added is simple: when the depth of the first
line equals strut depth and the height of the second one equals strut height
we're okay. When one of them is less we're also okay because \TEX\ will adapt the
baseline skip so that it compensated the difference. However, when the first line
has strut depth (due to the present strut) and the second line more than strut
height (resulting for instance from a formula) the lines are considered
overflowing in each other and therefore interline skip gets added.

When we end up in this situation the profiler can bring down the line skip when
it concludes that the strut is not running into the next line. However when the
formula sits directly below the strut we cannot really determine what is right so
then we just keep the skip. This situation occurs seldom. In many cases struts
are optional so one can always disable them (locally).

As the mentioned test document uses Lucida as body font, in the three cases where
we actually get a clash, one definitely relates to the strut: the overflow in the
second line occurs close to the right margin and the strut in the first line sits
at the left margin so we can get rid of the line skip, which leave us with only
two cases. However, there is another observation, one that involves the baseline
distance or line height.

In \CONTEXT\ the ratio between the strut height and depth is 72:28 which works
quite well for most fonts. If we look at Lucida shapes we see that the depth is
normally small so we can actually decide to change that ratio. It is however not
clear how that will influence decisions. Assuming more height will help with for
instance formulas that have superscripts, but an inline integral with subscript
might suffer. For other fonts, like EBGaramond, that have some extremely deep
shapes changing rations won't help anyway. We win here and loose there.

\startbuffer[text-001]
    abcdefghijklmnopqrstuvwxyz
    ABCDEFGHIJKLMNOPQRSTUVWXYZ
    1234567890
    ()[]/\{\}\|
    .,!?@\#\$\%^&*_-+=
\stopbuffer

\startbuffer[math-001]
     $\int\sum\sqrt{}$
\stopbuffer

\startbuffer[math-002]
     x_2^2
\stopbuffer

\newbox\MyHtDpBoxA
\newbox\MyHtDpBoxB
\newbox\MyHtDpBoxC

\def\CheckThemA#1%
  {\begingroup
   \switchtobodyfont[#1,10pt]%
   \starttabulate[||r|]
     \FL
     \BC \rlap{#1} \NC \NC \NR
     \FL
     \NC strut ht \EQ \the\strutht \NC \NR
     \NC strut dp \EQ \the\strutdp \NC \NR
   % \NC height   \EQ \meaningless\strutheightfactor \NC \NR
   % \NC depth    \EQ \meaningless\strutdepthfactor \NC \NR
     \NC ex       \EQ \the\exheight \NC \NR
     \LL
   \stoptabulate
   \endgroup}

\def\CheckThemB#1%
  {\begingroup
   \switchtobodyfont[#1,10pt]%
   \setbox\MyHtDpBoxA\ruledhpack\bgroup\inlinebuffer[text-001]\egroup
   \setbox\MyHtDpBoxB\ruledhpack\bgroup\inlinebuffer[math-001]\egroup
   \setbox\MyHtDpBoxC\ruledhpack\bgroup\inlinebuffer[math-002]\egroup
   \def\ShowThem##1##2%
     {\cldcontext{"\letterpercent 0.2f",\number##1##2/\number\htdp##2}
      \NC
      \the##1##2%
      \NC}
   \starttabulate[|r|r|i2r|r|i2r|r|]
     \FL
        \NS[1][c] \bf text-001
        \NS[1][c] \bf math-001
        \NS[1][c] \bf math-002
%         \rlap{\quad\quad#1}
     \NR
     \FL
     \NC
        \ShowThem\ht\MyHtDpBoxA
        \ShowThem\ht\MyHtDpBoxB
        \ShowThem\ht\MyHtDpBoxC
     \NR
     \NC
        \ShowThem\dp\MyHtDpBoxA
        \ShowThem\dp\MyHtDpBoxB
        \ShowThem\dp\MyHtDpBoxC
     \NR
     \LL
   \stoptabulate
   \endgroup}

We will look into a few fonts to get a better impression how all this relates. We
will use 10pt sizes. When we compare Lucida, Latin Modern, Bonum and Pagella we
notice that we start out with design sizes that are quite different.

\CheckThemA{lucida}
\CheckThemA{modern}
\CheckThemA{bonum}
\CheckThemA{pagella}

This is why we always use ratios (the 0.72 and 0.28) as well as abstract
dimensions like \type {ex} and \type {em} so that we adapt to what the font
provides. Because the default (total) line height is set to \type {2.8ex} we get
larger values in for instance Lucida.

In the next tables we use three samples, with \type {text-001} being:

\typebuffer[text-001]

The two math samples are:

\typebuffer[math-001]

and

\typebuffer[math-002]

For text the ratios are not that far off the defaults, but for math they start
to differ and the distance becomes larger. For Lucida we get:

\CheckThemB{lucida}

Modern gives:

\CheckThemB{modern}

And bonum moves in the other direction:

\CheckThemB{bonum}

Pagella also differs:

\CheckThemB{pagella}

Changing the ratios for the sake of math makes not that much sense because
profiling depends a lot on what math ends up inline. When we looked around a bit
for realistic examples we got the impression that seeing some clash (read:
getting uneven line spacing) might be a reason why small formulas eventually end
up as display. Without mentioning names, we noticed that a reprint of a book
actually got reformatted and when we looked for the slashing formulas in the
original they had become display instead. With proper profiling there is no need
for that. On top of that one can argue that some inline rendering can be done
better anyway, like using skewed fraction instead of ruled ones. Can we predict
that math with many superscripts goes well with a font with relatively high
shapes? As soon as some parenthesis are used we get depth anyway and how likely
is it that math without these is used inline? It also depends on the amount of
math: two lines with superscripted math will drive \TEX\ to use line skip so we
need to profile anyway. For that reason we will not adapt the ratios, just like
we keep the default line spacing. There are of course fonts with extreme heights
(like the Computer Modern Dunhill variant) but no one will use those artistic
variants in a math document. If you want to go fancy and distinctive, Antykwa is
a good choice and that one actually scores pretty good with the defaults!

\CheckThemA{antykwa}

\CheckThemB{antykwa}

\stopsection

\startsection[title=Conclusion]

So, should we enable profiling on mixed text|-|math documents or not? One
possible reason for not doing it is that it adds overhead, but in practice it's
not that much compared to processing the rest. It is no problem to find
complaints on the internet about \LUATEX\ performing worse than its ancestors so
if you're in that category: don't use profiling because it sets you back a few
percent runtime. However, when you're a demanding \CONTEXT\ user who mixes in a
lot of math, you might give it a try. It will intercept a couple of cases where
struts (assuming structure is used) trigger a line skip, and it might also catch
a couple of cases where \TEX\ found lines getting too close. Tweaking the \type
{factor} and \type {step} can actually be fun. Because it does influence page
breaks it is not something to be applied last minute. And, talking performance,
this kind of vertical optimization comes cheaper than horizontal optimization
using expansion (hz) and protrusion.

\stopsection

\stopchapter

\stopcomponent

% \framed[width=7cm,offset=overlay,align={lohi,normal}]{\getbuffer}
% \framed[width=7cm,offset=overlay,align={lohi,normal,profile}]{\getbuffer}

% maybe add an interline callback ...

% Test different fonts
% lots: modern, bonum, ebgaramond,
% some: pagella, termes, stixtwo
% few:  lucida, dejavu, antykwa, kurier, erewhon, libertinus

% Strut can force lineheight and depth.

% antykwa size of vertical bar is maybe too big. Change step?

% antykwa \Biggl(1+\frac{1}{n}\Biggr)^n

% cambria, superscripts sit a bit high (we did not change it)

% the role of font parameters (superscripts... not enough steps in i.e. square roots)

% dp / ht relation and struts
