% language=us runpath=texruns:manuals/lowlevel

\environment lowlevel-style

\startdocument
  [title=lines,
   coauthor=Mikael Sundqvist,
   color=middleorange]

\startsectionlevel[title=Introduction]

There is no doubt that \TEX\ does an amazing job of \quotation {breaking
paragraphs into lines} where a paragraph is a sequence of words in the input
separated by spaces or its equivalents (single line endings turned space). The
best descriptions of how that is done can be found in Don Knuths \quotation {The
\TEX\ Book}, \quotation {\TEX\ The Program} and \quotation {Digital Typography}.
Reading and rereading the relevant portions of those texts is a good exercise in
humility.

That said, whatever follows here builds upon what Knuth gave us and in no way we
pretend to do better than that. It started out as a side track of improving
rendering math in combination with more control over breaking inline math. It
pretty much about having fun with the par builder but in the end can also help
make your results look better. This is especially true for proze.

Trying to describe the inner working of the par builder makes no sense. Not only
is it kind of complex, riddled with magic constants and heuristics, but there is
a good chance for us to talk nonsense thanks to misunderstanding. However, some
curious aspects will be brought up. Consider what follows a somewhat naive
approach and whatever goes wrong, blame the authors, not \TEX.

If you're one of those reader who love to complain about the bad manuals, you can
stop reading here. There is plenty said in the mentioned books but you can also
consult Viktor Eijkhouts excellent \quotation {\TEX\ by Topic} (just search the
web for how the get these books). If you're curious and in for some adventure,
keep reading.

\stopsectionlevel

\startsectionlevel[title=Warning]

This is a first version. What is described here will stay but is still
experimental and how it evolves also depends on what demands we get from the
users. We have defined some experimental setups in \CONTEXT. We wil try to
improve the explanations in ways that (we hope) makes clear what happens deep
down but that takes time. These might change depending on feedback. We assume
that we're in granular mode:

\starttyping
\setupalign[granular]
\stoptyping

We will explain below what that means, but let us already now make clear that
this will likely become the default! As far as we can see, due to the larger
solution space, the inter-word spacing is more even but that also means that some
paragraphs can become one line less or more.

\stopsectionlevel

\startsectionlevel[title=Constructing paragraphs]

There are several concepts at work when \TEX\ breaks a paragraph into lines. Here
we assume that we talk about text: words separated by spaces. We also assume that
the text starts at the left edge and nicely runs till the right edge, with the
exception of the last line.

\startitemize
\startitem
    The spaces between words can stretch or shrink. We don't want that to be too
    inconsistent (visible) between two lines. This is where the terms loose and
    tight come into play.
\stopitem
\startitem
    Words can be hyphenated but we don't want that to happen too often. We also
    discourage neighboring lines to have hyphens. Hyphenating the (pre) final line
    is also sort of bad.
\stopitem
\startitem
    We definitely don't want words to stick out in the margin. If we have to choose,
    stretching is preferred over shrinking. If spaces become too small words,
    start to blur.
\stopitem
\startitem
    If needed glyphs can stretch or shrink a little in order to get rid of
    excessive spacing. But we really want to keep it minimal, and avoid it when
    possible. Usually we permit more stretch than shrink. Not all scripts (and
    fonts for that matter) might work well with this feature.
\stopitem
\startitem
    As a last resort we can stretch spaces so that we get rid of any still
    sticking out word. When \TEX\ reports an overfull box (often a line) you have
    to pay attention!
\stopitem
\stopitemize

When \TEX\ decides where to break and when to finish doing so it uses a system of
penalties and demerits and at some point makes decisions with regards to how bad
a breakpoint (and eventually a paragraph) is. The penalties are normally
relatively small unless we really want to penalize. When \TEX\ is in the process
of breaking a paragraph it calculates badness values for each line. This can be
seen as a measure on how bad looking a line is; a badness of zero is good, but
the larger the badness becomes, the worse the line is.

Here we shortly summarize the parameters that play a role in calculating what
\TEX\ calls the costs of breaking a line at some point: it's a combination of
weighting penalties as well as over- or undershooting the line with, where the
amount (dimension) and kind of (fillers) stretch and shrink determien the final
verdict.

\startbuffer
\ruledhbox to 20 ts{left \hss right}
\ruledhbox to 40 ts{left \hss right}
\ruledhbox to  5 ts{left \hss right}
\ruledhbox to  5 ts{left      right}
\ruledhbox to  5 es{%
    left
    \hskip 1ts plus 0.5ts\relax
    middle
    \hskip 1ts plus 1.5ts\relax
    right%
}
\stopbuffer

\typebuffer

These boxes show a bit what happens with spacing that can stretch of shrink. The
first three cases are not bad because it's what we ask for with the wildcard
\type {\hss}. \footnote {We use this opportunity to promote the new \type {ts} and
\type {es} units.}

\startlines \getbuffer \stoplines

\TEX\ will run over each paragraph at most three times. On each such run, it will
choose different breakpoints, calculate badness of each possible line, combine
that with eventual penalties, and calculate a certain demerit value for each
possible paragraph. It creats a set of solutions as it progresses, discards the
worse cases so far and eventually ends of what it thinks is best.

The process is primarily controlled by these parameters:

\startitemize
\startitem
    \type {\pretolerance}: This number determines the success of the first, not
    hyphenated pass. Often the value is set to the plain \TEX\ value of 100.
    If \TEX\ finds a possible division of a paragraph such that no line has
    a badness higher than \type {\pretolerance}, the algorithm quits here and
    that line is chosen.
\stopitem
\startitem
    \type {\tolerance}: This number determines the success of the second,
    hyphenated pass. Often the value is set to the plain \TEX\ value of 200.
\stopitem
\startitem
    \type {\emergencystretch}: This dimension kicks in when the second pass is
    not successful. In \CONTEXT\ we often set it to \type {2\bodyfontsize}.
\stopitem
\stopitemize

When we are (in \CONTEXT\ speak) \type {tolerant}, we have a value of 3000, while
\type {verytolerant} bumps it to 4500. These are pretty large values compared to
the default 100 and 200 that seem to cover most cases well, especially when we
have short words, a reasonable width and lots of opportunities for hyphenation.
Keep in mind that a macro package has to default to values that make sense for
the average case.

We now come to the other relevant parameters. You need to keep in mind that the
demerits are made from penalty values that get squared which is why parameters
with demerits in their name have high values: a penalty of $50$ squared has to
relate to a demerit of $5000$, so we might have $2500 + 5000$ at some point.

The formula (most often) used to calculate the demerits \im{d} is

\startformula
    d = (l + b + p)^2 + e
\stopformula

Here \im {l} is the \type {\linepenalty}, set to \im {10} in plain, \im {b} is the
badness of the line, and \im {p} is the penalty of the current break (for example,
added by hyphenation, or by breaking an inline formula). The \im {e} stands for
extra non-local demerits, that do not depend on only the current line, like
the \type {\doublehyphendemerits} that is added if two lines in a row are
hyphenated.

The badness reflects how the natural linewidth relates to the target width and
uses a cubic function. A badness of zero is of course optimal, but a badness of
99 is pretty bad. A magic threshold is 12 (around that value a line is considered
decent). If you look at the formula above you can now understand why the line
penalty defaults to the low value of 10.

\startitemize
\startitem
    \type {\hyphenpenalty}: When a breakpoint occurs at a discretionary this one
    gets added. In \LUAMETATEX\ we store penalties in the discretionary nodes but
    user defined \typ {\discretionary}'s can carry dedicated penalties. This
    value is set to 50, which is not that much. Large values reduce the solution
    space so best keep this one reasonable.
\stopitem
\startitem
    \type {\linepenalty}: Normally this is set to 10 and it is the baseline for a
    breakpoint. This is again a small value compared to for instance the
    penalties that you find in inline math. There we need some breakpoints and
    after binary and relation symbols such an opportunity is created. The
    specific penalties are normally 500 and 700. One has to keep in mind, as
    shown in the formula above, that the penalties are not acting on a linear
    scale when the demerits are calculated. Math spacing and penalty control is
    discussed in the (upcoming) math manual.
\stopitem
\startitem
    \type {\doublehyphendemerits}: Because it is considered bad to have two
    hyphens in a row this is often set pretty high, many thousands. These are
    treated as demerits (so outside of the squared part of the above formula).
\stopitem
\startitem
    \type {\finalhyphendemerits}: The final (pre last) line having a hyphen is
    also considered bad. The last line is handled differently anyway, just
    because it gets normally flushed left.
\stopitem
\startitem
    \type {\adjdemerits}: lines get rated in terms of being loose, decent, tight,
    etc. When two lines have a different rating we bump the total demerits.
\stopitem
\startitem
    \type {\looseness}: it is possible to force less or more lines but to what
    extend this request is honored depends on for instance the possible
    (emergency) stretch in the spaces (or any glue for that matter). `
    % Needs an explanation
\stopitem
\stopitemize

It is worth noticing that you can set \typ {\lastlinefit} such that the spaces in
the last line will be comparable to those in the preceding line. This is a
feature that \ETEX\ brought us. Anyways, keep in mind normally penalties are
either small, or when we want to be tough, pretty high. Demerits are often
relatively large.

The next one is a flag that triggers expansion (or compression) of glyphs to kick
in. Those get added to the available stretch and/or shrink of a line:

\startitemize[continue]
\startitem
    \type {\adjustspacing}: Its value determines if expansion kicks in: glyphs
    basically get a stretch and shrink value, something that helps filling our
    lines. We only have zero, two and three (and not the \PDFTEX\ value of two):
    three means \quote {only glyphs} and two means \quote {font kerns and
    glyphs}.
\stopitem
\stopitemize

In \LUAMETATEX\ we also have:

\startitemize
\startitem
    \typ {\linebreakcriterion}: The normal distinction between loose, decent and
    tight in \TEX\ uses 12 for 0.5 and 99 for about 1.0, but because we have more
    granularity (.25) we can set four values instead. The default of zero (\type
    {"0C0C0C63}) then becomes \type {"020C2A63}. When set that way the default
    \typ {\adjdemerits} has to be halved 5000 so that we compare the more
    granular distances. Don't worry if you \quote {don't get it}, hardly any user
    will change these values. One can think of the 100 squared becomes a 10000
    (at least this helps relating these numbers) and 10000 is pretty bad in \TEX s
    perception.
\stopitem
\startitem
    \type {\adjustspacingstep}: When set this one is are used instead of the font
    bound value which permits local control without defining a new font instance.
\stopitem
\startitem
    \type {\adjustspacingstretch}: idem.
\stopitem
\startitem
    \type {\adjustspacingshrink}: idem.
\stopitem
% \startitem
%     \type {\extrahyphenpenalty}:
% \stopitem
\startitem
    \type {\orphanpenalty}: This penalty will be injected before the last word of a paragraph.
\stopitem
\startitem
    \type {\orphanpenalties}: Alternatively a series of penalties can be defined.
    This primitive expects a count followed by that number of penalties. These
    will be injected starting from the end.
\stopitem
\stopitemize

The shape of a paragraph is determined by \typ {\hangindent}, \type {\hangafter},
\typ {\parshape} and \typ {\parindent}. The width is controlled by \typ {\hsize},
\typ {\leftskip}, \typ {\rightskip}. In addition there are \typ
{\parinitleftskip}, \typ {\parinitrightskip}, \typ {\parfillleftskip} and \typ
{\parfillrightskip} that control first and last lines.

We also have these:

\startitemize
\startitem
    \type {\linebreakpasses}: When set to one, the currently set \type {\parpasses}
    will be applied.
\stopitem
\startitem
    \type {\parpasses}: This primitive defined a set of sub passes that kick in
    when the second pass is finished. This basically opens up the par builder. It
    is still experimental and will be improved based upon user feedback. Although
    it is a side effect of improving the breaking of extensive mixes of math and
    text, it is also quite useful for text only (think novels).
\stopitem
\stopitemize

In the next sections we will explain how these can improve the look and feel of
what you typeset.

\stopsectionlevel

\startsectionlevel[title=Subpasses]

In \LUATEX\ and therefore also in \LUAMETATEX\ a paragraph is constructed in steps:

\startitemize
\startitem
    The list of nodes that makes the paragraph is hyphenated: words become a
    mixture of glyphs and discretionaries.
\stopitem
\startitem
    That list is processed by a font handler that can remove, add or change glyphs
    depending on how glyphs interact. This depends on the language and scripts used.
\stopitem
\startitem
    The result is fed into the parbuilder that applies upto three passes as mentioned
    before.
\stopitem
\stopitemize

In traditional \TEX\ these three actions are combined into one and the overhead
is shared. In the split case the processing time gets distributed and in practice
the last action is not the one that takes most time. This is why the mechanism
that we discuss next has little impact on the run: calling the par builder a few
times more seldom results in more runtime. This is why in we support so called
sub passes between the second and third one.

Here is an example of a setup. We set a low tolerance for the first pass and second
pass. We can do that because we don't  need to play safe nor need to compromise.

\starttyping
\pretolerance  75
\tolerance    150
\parpasses      3
    threshold            0.025pt
    classes              \indecentparpassclasses
    tolerance            150
  next
    threshold            0.025pt
    classes              \indecentparpassclasses
    tolerance            200
    emergencystretch     .25\bodyfontsize
  next
    threshold            0.025pt
    classes              \indecentparpassclasses
    tolerance            200
    optional             1
    emergencystretch     .5\bodyfontsize
\relax
\linebreakpasses 1
\stoptyping

Because we want to retain performance we need to test efficiently if we really
need the (here upto three) additional passes, so let's see how it is done. When a
pass list is defined, and line break passes are enabled, the engine will check {\em after}
the second pass if some more work is needed. For that it will do a quick analysis and
calculate four values:

\startitemize[packed]
\startitem overflow   : the maximum value found, this is something really bad. \stopitem
\startitem underflow  : the maximum value found, this is something we can live with. \stopitem
\startitem verdict    : what is the worst badness of lines in this paragraph. \stopitem
\startitem classified : what classes are assigned to lines, think looseness, decent and tight. \stopitem
\stopitemize

There are two cases where the engine will continue with the applying passes:
there is an overflow or there is a verdict (max badness) larger than zero. When we
tested this on some large documents we noticed that this is nearly always true,
but by checking we save a few unnecessary passes.

Next we test if a pass is really needed, and if not we check the next pass. When
a pass is done, we pick up where we left, but we test for the overflow or badness
every sub pass. The next checks make us run a pass:

\startitemize[packed]
\startitem overfull   exceeds  threshold \stopitem
\startitem verdict    exceeds  badness   \stopitem
\startitem classified overlaps classes   \stopitem
\stopitemize

Here \typ {threshold}, \typ {badness} and \typ {classes} are options in a pass
section. Which test makes sense depends a bit on how \TEX\ sees the result.
Internally \TEX\ uses numbers for its classification (0..5) but we map that onto
a bitset because we want an overview:

\starttabulate[|r|l|c|c|c|c|]
\NC      \NC             \BC indecent \BC almostdecent \BC loose \BC tight \NC \NR
\NC   1  \BC  veryloose  \NC    +     \NC      +       \NC   +   \NC       \NC \NR
\NC   2  \BC  loose      \NC    +     \NC      +       \NC   +   \NC       \NC \NR
\NC   4  \BC  semiloose  \NC    +     \NC              \NC   +   \NC       \NC \NR
\NC   8  \BC  decent     \NC          \NC              \NC       \NC       \NC \NR
\NC  16  \BC  semitight  \NC    +     \NC              \NC       \NC   +   \NC \NR
\NC  32  \BC  tight      \NC    +     \NC      +       \NC       \NC   +   \NC \NR
\stoptabulate

The semiloose and semitight values are something \LUAMETATEX. In \CONTEXT\ we
have these four variants predefined as \typ {\indecentparpassclasses} and such.

The sections in a par pass setup are separated by \type {next}. For testing
purposes you can add \type {skip} and \type {quit}. The \type {threshold} tests
against the overfull value, the \type {badness} against the verdict and \type
{classes} checks for overlap with encountered classes, the classification.

You can specify an \typ {identifier} in the first segment that then will be used
in tracing but it is also passed to callbacks that relate to this feature.
Discussing these callback is outside the scope fo this wrapup.

You need to keep in mind that parameters are not reset to their original values
between two subpasses of a paragraph.
We have \typ {tolerance} and \typ {emergencystretch} which are handy for simple
setups. When we start with a small tolerance we often need to bump that one. The
stretch is likely a last resort. The usual demerits can be set too: \typ
{doublehyphendemerits}, \typ {finalhyphendemerits} and \typ {adjdemerits}. We
have \typ {extrahyphenpenalty} that gets added to the penalty in a discretionary.
You can also set \typ {linepenalty} to a different value than it normally gets.

The \typ {looseness} can be set but keep in mind that this only makes sense in
very special cases. It's hard to be loose when there is not much stretch or shrink
available. The \typ {linebreakcriterion} parameter can best be left untouched and is
mostly there for testing purposes.

The \LUAMETATEX\ specific \typ {orphanpenalty} gets injected before the last word
in a paragraph. High values can lead to overfull boxes but when used in text that
hyphenate well or with languages that have short words it might work out well.

The next four parameters are related to expansion: \typ {adjustspacing}, \typ
{adjustspacingstep}, \typ {adjustspacingshrink} and \typ {adjustspacingstretch}.
Here we have several scenarios.

\startitemize
\startitem
    Fonts are set up for expansion (in \CONTEXT\ for instance with the quality
    specifier). When \type {hz} is then enabled it will always kick in.
\stopitem
\startitem
    When we don't enable it, the par pass can do it by setting \typ {adjustspacing} (to 3).
\stopitem
\startitem
    When the other parameters are set these will overload the ones in the font,
    but used with the factors in there, so different characters get scaled
    differently. You can set the step to one to get more granular results.
\stopitem
\startitem
    When expansion is {\em not} set on the font, setting the options in a pass will activate
    expansion but with the factors set to 1000. This means all characters are treated equal,
    which is less subtle.
\stopitem
\stopitemize

When a font is not set up to use expansion, you can do something like this:

\starttyping
\parpasses    6
    classes              \indecentparpassclasses
    threshold            0.025pt
    tolerance             250
    extrahyphenpenalty     50
    orphanpenalty        5000
  % font driven
  next ifadjustspacing
    threshold            0.025pt
    classes              \tightparpassclasses
    tolerance             300
    adjustspacing           3
    orphanpenalty        5000
  next ifadjustspacing
    threshold            0.025pt
    tolerance            350
    adjustspacing           3
    adjustspacingstep       1
    adjustspacingshrink    20
    adjustspacingstretch   40
    orphanpenalty        5000
    emergencystretch     .25\bodyfontsize
  % otherwise, factors 1000
  next
    threshold            0.025pt
    classes              \tightparpassclasses
    tolerance             300
    adjustspacing           3
    adjustspacingstep       1
    adjustspacingshrink    10
    adjustspacingstretch   15
    orphanpenalty        5000
  next
    threshold            0.025pt
    tolerance             350
    adjustspacing           3
    adjustspacingstep       1
    adjustspacingshrink    20
    adjustspacingstretch   40
    orphanpenalty        5000
    emergencystretch     .25\bodyfontsize
  % whatever
  next
    threshold            0.025pt
    tolerance            3000
    orphanpenalty        5000
    emergencystretch     .25\bodyfontsize
\relax
\stoptyping

With \typ {ifadjustspacing} you ignore steps that expect the font to be setup, so
you don't waste time if that is not the case.

There is also a \typ {callback} parameter but that one is experimental and used
for special purposes and testing. We don't expect users to mess with that.

A really special feature is optional content. Here we use as example a quote from
Digital Typography:

\starttyping[paragraph=yes,align=flushleft]
Many readers will skim over formulas on their first reading
of your exposition. Therefore, your sentences should flow
smoothly when all but the simplest formulas are replaced by
\quotation {blah} or some other \optionalword {1} {grunting }noise.
\stoptyping

Here the \type {grunting} (with embedded space) is considered optional. When you
set \typ {\linebreakoptional} to~1 this word will be typeset. However, when you
set the pass parameter \typ {linebreakoptional} to~0 it will be skipped. There
can be multiple optional words with different numbers. The numbers are actually
bits in a bit set so plenty is possible. However, normally these two values are
enough, if used at all.

\stopsectionlevel

\startsectionlevel[title=Definitions]

The description above is rather low level and in practice users will use a bit
higher level interface. Also, in practice only a subset of the parameters makes
sense in general usage. It is not that easy to decide on what parameter subset
will work out well but it can be fun to play with variants. After all, this is
also what \TEX\ is about: look, feel and fun.

Some users praise the ability of recent \TEX\ engines to provide expansion and
protrusion. This feature is a bit demanding because not only does it add to
runtime (although in \LUAMETATEX\ that normally can be neglected), it also makes
the output files larger. Some find it hard to admit, but it even can result in
bad looking documents when applied with extremes.

The traditional (\MKIV) way to set up expansion is to add this to the top of the
document, or at least before fonts get loaded.

\startbuffer[pass-a]
\definefontfeature
  [default]
  [default]
  [expansion=quality,protrusion=quality]
\stopbuffer

\typebuffer[a]

and later on to enable it with:

\starttyping
\setupalign[hz]
\stoptyping

However, par passes make it possible to be more selective. Take the following two
definitions:

\startbuffer[pass-b]
\startsetups align:pass:quality:1
    \pretolerance 50
    \tolerance    150
    \parpasses    6
        identifier           \parpassidentifier{quality:1}
        threshold            0.025pt
        tolerance            175
      next
        threshold            0.025pt
        tolerance            200
      next
        threshold            0.025pt
        tolerance            250
      next
        classes              \almostdecentparpassclasses
        tolerance            300
        emergencystretch     .25\bodyfontsize
      next ifadjustspacing
        classes              \indecentparpassclasses
        tolerance            300
        adjustspacing          3
        emergencystretch     .25\bodyfontsize
      next
        threshold            0.025pt
        tolerance            3000
        emergencystretch     2\bodyfontsize
    \relax
\stopsetups

\startsetups align:pass:quality:2
    \pretolerance 50
    \tolerance    150
    \parpasses    5
        identifier           \parpassidentifier{quality:2}
        threshold            0.025pt
        tolerance            175
      next
        threshold            0.025pt
        tolerance            200
      next
        threshold            0.025pt
        tolerance            250
      next ifadjustspacing
        classes              \indecentparpassclasses
        tolerance            300
        adjustspacing          3
        emergencystretch     .25\bodyfontsize
      next
        threshold            0.025pt
        tolerance            3000
        emergencystretch     2\bodyfontsize
    \relax
\stopsetups
\stopbuffer

\typebuffer[pass-b]

You can now enable one of these:

\starttyping
\setupalignpass[quality:1]
\stoptyping

\startbuffer[pass-c]
\starttext
    \showmakeup[expansion,space]
    \startTEXpage[offset=1ts]
        \startcombination[3*3]
            {\vtop{\hsize 8cm\setupalignpass[none]     \samplefile{tufte}}} {none}
            {\vtop{\hsize 9cm\setupalignpass[none]     \samplefile{tufte}}} {none}
            {\vtop{\hsize12cm\setupalignpass[none]     \samplefile{tufte}}} {none}
            {\vtop{\hsize 8cm\setupalignpass[quality:1]\samplefile{tufte}}} {quality:1}
            {\vtop{\hsize 9cm\setupalignpass[quality:1]\samplefile{tufte}}} {quality:1}
            {\vtop{\hsize12cm\setupalignpass[quality:1]\samplefile{tufte}}} {quality:1}
            {\vtop{\hsize 8cm\setupalignpass[quality:2]\samplefile{tufte}}} {quality:2}
            {\vtop{\hsize 9cm\setupalignpass[quality:2]\samplefile{tufte}}} {quality:2}
            {\vtop{\hsize12cm\setupalignpass[quality:2]\samplefile{tufte}}} {quality:2}
        \stopcombination
    \stopTEXpage
\stoptext
\stopbuffer

The result is shown in \in {figure} [fig:passes:expansion] where you can see that
expansion is applied selectively; you have to zoom in to see where.

\startplacefigure[location=page,reference=fig:passes:expansion,title={Two different passes applied to \type {tufte.tex}.}]
    \typesetbuffer[pass-a,pass-b,pass-c][width=\textwidth]
\stopplacefigure

\stopsectionlevel

\startsectionlevel[title=Tracing]

There are several ways to see what goes on. The engine has a tracing option that
is set with \type {\tracingpasses}. Setting it to \type {1} reports the passes on
the console, and a value of \type {2} also gives some details.

There is a also a tracker, \type {paragraphs.passes} that can be enabled. This gives
a bit more information:

\starttyping
\enabletrackers[paragraphs.passes]
\enabletrackers[paragraphs.passes=summary]
\enabletrackers[paragraphs.passes=details]
\stoptyping

If you want to see where expansion kicks in, you can use:

\starttyping
\showmakeup[expansion]
\stoptyping

This is just one of the options, \type {spaces}, \type {penalties}, \type {glue}
are are useful when you play with passes, but if you are really into the low level
details, this is what you want:

\startbuffer
\startnarrower[5*right]
\startshowbreakpoints[option=margin,offset=\dimexpr{.5\emwidth-\rightskip}]
\samplefile{tufte}
\stopshowbreakpoints
\stopnarrower
\stopbuffer

\typebuffer \getbuffer

You can see the chosen solutions with

\startbuffer
\showbreakpoints[n=1]
\stopbuffer

\typebuffer \getbuffer

When we started playing with the par builder in the perspective of
math, we side tracked and ended up with a feature that can ge used
in controlled situations. Currently we only have a low level
\CONTEXT\ interface for this (see \in {figure} [fig:passes:lousiness]).

\startbuffer[lousiness]
  \startTEXpage[offset=1ts]
    \startcombination[3*1]
        \bgroup \vtop\bgroup
            \hsize8cm
            \setupalign[verytolerant]
            \tracinglousiness 1
            \lousiness 0
            \samplefile{ward}
        \egroup \egroup
        {\type {\tracinglousiness 1}
         \type {\lousiness 0}}
        \bgroup \vtop\bgroup
            \hsize8cm
            \setupalign[verytolerant]
            \lousiness 1 11 0
            \samplefile{ward}
        \egroup \egroup
        {\type {\lousiness 1 11 0}}
        \bgroup \vtop\bgroup
            \hsize8cm
            \setupalign[verytolerant]
            \silliness 11
            \samplefile{ward}
        \egroup \egroup
        {\type {\silliness 11}}
    \stopcombination
\stopTEXpage
\stopbuffer

\startplacefigure[location=here,reference=fig:passes:lousiness,title={Influencing the way \TEX\ breaks lines applied to \type {ward.tex}.}]
    \typesetbuffer[lousiness][width=\textwidth]
\stopplacefigure

\stopsectionlevel

\startsectionlevel[title=Criterion]

The \type {granular} alignment option will configure the linebreakcriterion to
work with $0.25$ steps instead of $0.50$ steps which means that successive lines
can become a bit closer in spacing. There is no real impact on performance
because testing happens anyway. In \in {figure}[fig:criterion] you see some
examples, where in some it indeed makes a difference.

\startbuffer[criterion]
\starttext
\definecolor[ttest][a=1,t=.5]
\definecolor[rtest][a=1,t=.5,r=1]
\dostepwiserecurse{80}{120}{1}{
    \startTEXpage[offset=1ts]
        \startcombination[3*1]
            {\startoverlay
                {\vtop{\showmakeup[space]\hsize #1mm\ttest                      \samplefile{tufte}}}
                {\vtop{\showmakeup[space]\hsize #1mm\rtest \setupalign[granular]\samplefile{tufte}}}
            \stopoverlay} {}
            {\vtop{\showmakeup[space]\hsize #1mm\ttest                      \samplefile{tufte}}} {}
            {\vtop{\showmakeup[space]\hsize #1mm\rtest \setupalign[granular]\samplefile{tufte}}} {}
        \stopcombination
    \stopTEXpage
}
\stoptext
\stopbuffer

\startplacefigure[location=here,reference=fig:criterion,title={More granular interline criteria.}]
    \startcombination[1*4]
        {\typesetbuffer[criterion][width=\textwidth,page=35]} {}
        {\typesetbuffer[criterion][width=\textwidth,page=36]} {}
        {\typesetbuffer[criterion][width=\textwidth,page=37]} {}
        {\typesetbuffer[criterion][width=\textwidth,page=38]} {}
    \stopcombination
\stopplacefigure

\stopsectionlevel

\startsectionlevel[title=Examples]

\start \em

    The \CONTEXT\ distribution comes with a few test setups: \typ
    {spac-imp-tests.mkxl}. Once we have found a suitable set of values and sample
    texts we might discuss them here.

    Currently we provide the following predefined passes that you can enable with
    \typ {\setupalignpass}: \type {decent}, \type {quality}, \type {test1}, \type
    {test2}, \type {test3}, \type {test4}, \type {test5}. We hope that users are
    willing to test these.

\stop

\stopsectionlevel

\startsectionlevel[title=Pages]

While the par builder does multiple passes, the page builder is a single pass
progressive routine. Every time something gets added to the (so called) main
vertical list the page state gets updated and when the page overflows what has
been collected gets passed to the output routine. It is to a large extend driven
by glue (with stretch and shrink) and penalties and when content (boxes) is added
the process is somewhat complicated by inserts as these needs to be taken into
account too.

You can get pages that run from top to bottom by adding stretch between lines but
by default in \CONTEXT\ we prefer to fill up the bottom with white space.

It can be hard to make decisions at the \TEX\ end around a potential page break
because in order to get an idea how much space is left, one needs to trigger the
page builder which can have side effects.

Penalties play an important role and because these are used to control for
instance widows and clubs high values can lead to underfull pages so if we want
to influence that we need to cheat. For this we have three experimental
mechanisms:

\startitemize[packed]
\startitem tweaking the page goal: \type {\pageextragoal} \stopitem
\startitem initializing the state quantities: \type {\initialpageskip} \stopitem
\startitem adapting the state quantities as we go: \type {\additionalpageskip} \stopitem
\stopitemize

The first tweak is for me to play with, and when a widow or club is seen the
extra amount can kick in. This feature is likely to be replaced by a more
configurable one.

The second tweak lets the empty page start out with some given height, stretch
and shrink. This variable is persistent over pages. This is not true for the
third tweak: it kicks in when the page gets initialized {\em or} as we go, but
after it has been applied the value is reset. That makes it a feature like \type
{\looseness}. We could combine these into one (because one can set up a
persistent one in the macro package at well defined spots) but having an initial
one also nicely can compensate the usual topskip glue hackery with a more natural
control option.

\startbuffer[pagelooseness-1]
\starttext
    \showframe[text]
    \setuplayout[width=middle,headerdistance=5mm]
    \setupalign[vertical,height]
    \dorecurse{10}{
        \samplefile{tufte}\par
        \setpagelooseness[lines=2]%
        \dorecurse{5}{
            {\red   \samplefile{knuth}}\par
            {\green \samplefile {ward}}\par
            {\blue  \samplefile{davis}}\par
        }
        \page
    }
\stoptext
\stopbuffer

\startbuffer[pagelooseness-2]
\starttext
    \showframe[text]
    \setuplayout[width=middle,headerdistance=5mm]
    \setupalign[vertical,height]
    \dorecurse{10}{
        \samplefile{tufte}\par
        \setpagelooseness[-3]%
        \dorecurse{5}{
            {\red   \samplefile{knuth}}\par
            {\green \samplefile {ward}}\par
            {\blue  \samplefile{davis}}\par
        }
        \page
    }
\stoptext
\stopbuffer

Adapting the layout (within the regular text area) is done with \typ
{\setpagelooseness} an demonstrated in \in {figure} [fig:pagelooseness-1] and \in
{figure} [fig:pagelooseness-2]. Possible parameters are \type {lines}, \type
{height}, \type {stretch} and \type {shrink}. You can also directly specify the
number of lines. The other two features are not (yet) interfaced.

\startplacefigure[location=here,reference=fig:pagelooseness-1,title={Cheating with page dimensions: \type {[lines=2]}.}]
    \startcombination[4*1]
        {\typesetbuffer[pagelooseness-1][width=\combinationwidth,page=1,frame=on]} {}
        {\typesetbuffer[pagelooseness-1][width=\combinationwidth,page=2,frame=on]} {}
        {\typesetbuffer[pagelooseness-1][width=\combinationwidth,page=3,frame=on]} {}
        {\typesetbuffer[pagelooseness-1][width=\combinationwidth,page=4,frame=on]} {}
    \stopcombination
\stopplacefigure

\startplacefigure[location=here,reference=fig:pagelooseness-2,title={Cheating with page dimensions: \type {[-3]}.}]
    \startcombination[4*1]
        {\typesetbuffer[pagelooseness-2][width=\combinationwidth,page=1,frame=on]} {}
        {\typesetbuffer[pagelooseness-2][width=\combinationwidth,page=2,frame=on]} {}
        {\typesetbuffer[pagelooseness-2][width=\combinationwidth,page=3,frame=on]} {}
        {\typesetbuffer[pagelooseness-2][width=\combinationwidth,page=4,frame=on]} {}
    \stopcombination
\stopplacefigure

It is not that trivial to fulfill the wide range of user demands but over time
the \typ {\setupalign} commands has gotten plenty of features. Getting for
instance windows and clubs right in the kind of mixed usage that is common in
\CONTEXT\ is not always easy. One can experiment with scenarios (also to get some
understanding of matters) but none is probably perfect (unless one does something
close to manual tweaking). There is also the butterfly effect: a change here
might trigger na issue there.

The examples in \in {figure} [fig:vz-1], \in [fig:vz-2] and \in [fig:vz-3] scale
vertically in order ti fill up the text area; the \type {vz} parameter is set
with \typ {setuplayout}. In the example the widow and club penalties are set to
$10000$. In these examples we have enabled the \typ {layout.vz} trackers that
shows a small black rule indicating the amount of stretch.

\startbuffer[vz-1]
\starttext
    \showframe[text]
    \enabletrackers[layout.vz]
    \setuplayout[width=middle,headerdistance=5mm,vz=no]
    \clubpenalty  10000
    \widowpenalty 10000
    \dostepwiserecurse{0}{30}{1}{
        \dorecurse{#1}{\strut dummy line ##1\par}
        \dorecurse{4}{\samplefile{tufte}\par}
    }
\stoptext
\stopbuffer

\startbuffer[vz-2]
\starttext
    \showframe[text]
    \enabletrackers[layout.vz]
    \setuplayout[width=middle,headerdistance=5mm,vz=yes]
    \clubpenalty  10000
    \widowpenalty 10000
    \dostepwiserecurse{0}{30}{1}{
        \dorecurse{#1}{\strut dummy line ##1\par}
        \dorecurse{4}{\samplefile{tufte}\par}
    }
\stoptext
\stopbuffer

\startbuffer[vz-3]
\starttext
    \showframe[text]
    \enabletrackers[layout.vz]
    \setuplayout[width=middle,headerdistance=5mm,vz=2]
    \clubpenalty  10000
    \widowpenalty 10000
    \dostepwiserecurse{0}{30}{1}{
        \dorecurse{#1}{\strut dummy line ##1\par}
        \dorecurse{4}{\samplefile{tufte}\par}
    }
\stoptext
\stopbuffer

\startplacefigure[location=here,reference=fig:vz-1,title={Cheating with vertical expansion: \type {[vz=no]}.}]
    \startcombination[4*1]
        {\typesetbuffer[vz-1][width=\combinationwidth,page=1,frame=on]} {}
        {\typesetbuffer[vz-1][width=\combinationwidth,page=2,frame=on]} {}
        {\typesetbuffer[vz-1][width=\combinationwidth,page=3,frame=on]} {}
        {\typesetbuffer[vz-1][width=\combinationwidth,page=4,frame=on]} {}
    \stopcombination
\stopplacefigure

\startplacefigure[location=here,reference=fig:vz-2,title={Cheating with vertical expansion: \type {[vz=yes]}.}]
    \startcombination[4*1]
        {\typesetbuffer[vz-2][width=\combinationwidth,page=1,frame=on]} {}
        {\typesetbuffer[vz-2][width=\combinationwidth,page=2,frame=on]} {}
        {\typesetbuffer[vz-2][width=\combinationwidth,page=3,frame=on]} {}
        {\typesetbuffer[vz-2][width=\combinationwidth,page=4,frame=on]} {}
    \stopcombination
\stopplacefigure

\startplacefigure[location=here,reference=fig:vz-3,title={Cheating with vertical expansion: \type {[vz=2]}.}]
    \startcombination[4*1]
        {\typesetbuffer[vz-3][width=\combinationwidth,page=1,frame=on]} {}
        {\typesetbuffer[vz-3][width=\combinationwidth,page=2,frame=on]} {}
        {\typesetbuffer[vz-3][width=\combinationwidth,page=3,frame=on]} {}
        {\typesetbuffer[vz-3][width=\combinationwidth,page=4,frame=on]} {}
    \stopcombination
\stopplacefigure

There are a few other tweaks but these one can wonder about these. We can add
stretch and shrink to the baseline skip, something that can also be triggered
with the \quote {spread} option to \typ {\setupalign}, assuming that also \typ
{height} is given). An alternative is to permit an extra line and accept a visual
overflow, assuming that the layout is set up to make sure that the footer line
doesn't overlap. None of this guarantees that a whole document with plenty of
graphics and special constructs will come out well, but for text only it might
work okay. \in {Figures} [fig:extra-1], \in [fig:extra-2] and \in [fig:extra-3]
show some of this.

\startbuffer[extra-1]
\starttext
    \showframe[text]
    \setuplayout[width=middle,headerdistance=15mm,vz=no]
    \setupalign[height]
    \clubpenalty  10000
    \widowpenalty 10000
    \dorecurse{10}{
        \samplefile{tufte}\par
        \samplefile{knuth}\par
        \samplefile{ward}\par
        \samplefile{davis}\par
    }
\stoptext
\stopbuffer

\startbuffer[extra-2]
\starttext
    \showframe[text]
    \setuplayout[width=middle,headerdistance=15mm,vz=no]
    \setupalign[height]
    \clubpenalty  10000
    \widowpenalty 10000
    \baselineskip 1\baselineskip plus 1pt minus .1pt
    \dorecurse{10}{
        \samplefile{tufte}\par
        \samplefile{knuth}\par
        \samplefile{ward}\par
        \samplefile{davis}\par
    }
\stoptext
\stopbuffer

\startbuffer[extra-3]
\starttext
    \showframe[text]
    \setuplayout[width=middle,headerdistance=15mm,vz=no]
    \setupalign[height]
    \clubpenalty  10000
    \widowpenalty 10000
    \pageextragoal1\lineheight
    \dorecurse{10}{
        \samplefile{tufte}\par
        \samplefile{knuth}\par
        \samplefile{ward}\par
        \samplefile{davis}\par
    }
\stoptext
\stopbuffer

\page

\startplacefigure[location=here,reference=fig:extra-1,title={Cheating: just high penalties.}]
    \startcombination[4*2]
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-1][width=\combinationwidth,page=1,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-1][width=\combinationwidth,page=2,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-1][width=\combinationwidth,page=3,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-1][width=\combinationwidth,page=4,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-1][width=\combinationwidth,page=5,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-1][width=\combinationwidth,page=6,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-1][width=\combinationwidth,page=7,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-1][width=\combinationwidth,page=8,frame=on]}} {}
    \stopcombination
\stopplacefigure

\startplacefigure[location=here,reference=fig:extra-2,title={Cheating: \typ {\baselineskip 1\baselineskip plus 1pt minus .1pt}.}]
    \startcombination[4*2]
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-2][width=\combinationwidth,page=1,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-2][width=\combinationwidth,page=2,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-2][width=\combinationwidth,page=3,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-2][width=\combinationwidth,page=4,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-2][width=\combinationwidth,page=5,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-2][width=\combinationwidth,page=6,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-2][width=\combinationwidth,page=7,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-2][width=\combinationwidth,page=8,frame=on]}} {}
    \stopcombination
\stopplacefigure

\startplacefigure[location=here,reference=fig:extra-3,title={Cheating: \typ {\pageextragoal \lineheight}.}]
    \startcombination[4*2]
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-3][width=\combinationwidth,page=1,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-3][width=\combinationwidth,page=2,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-3][width=\combinationwidth,page=3,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-3][width=\combinationwidth,page=4,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-3][width=\combinationwidth,page=5,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-3][width=\combinationwidth,page=6,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-3][width=\combinationwidth,page=7,frame=on]}} {}
        {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra-3][width=\combinationwidth,page=8,frame=on]}} {}
    \stopcombination
\stopplacefigure

\stopsectionlevel

\startsectionlevel[title=Profiles]

You can have a paragraph with lines that exceed the maximum height and/or depth
or where spaces end up in a way that create so called rivers. Rivers are more a
curiosity than an annoyance because any attempt to avoid them is likely to result
in a worse looking result. The unequal line distances can be annoying too but
these can be avoided when bringing lines closer together doesn't lead to clashes.
This can be done without reformatting the paragraph by passing the \type
{profile} option to \typ {\setupalign}. It comes at the cost of a little more
runtime and (as far as we observed) it kicks in seldom, for instance when inline
math is used that has super- or subscripts, radicals, fractions or other slightly
higher constructs.

\stopsectionlevel

\page % colofon after flushed page float

\stopdocument

% \showmakeup[glue]

% \startsetups align:pass:whatever
%     \pretolerance 75
%     \tolerance    150
%     \parpasses    3
%         threshold            0.025pt
%         classes              \indecentparpassclasses
%         tolerance            150
%     next
%         threshold            0.025pt
%         classes              \indecentparpassclasses
%         tolerance            200
%         emergencystretch     .25\bodyfontsize
%     next
%         threshold            0.025pt
%         classes              \indecentparpassclasses
%         tolerance            200
%         optional             1
%         emergencystretch     .5\bodyfontsize
%     \relax
%     \linebreakpasses\plusone
% \stopsetups


% \dostepwiserecurse{80}{120}{2} {
%     \start
%         \hsize#1mm \getbuffer \getbuffer \blank
%         \hsize#1mm \setupalignpass[whatever] \getbuffer \getbuffer \page
%     \stop
% }

% discuss the disc options options pre, post orphaned, penalties (or maybe in a new
% lowlevel-discretionaries)

% \starttext
%     \showmakeup[line]
% %     \discretionaryoptions\zerocount
% %     \discretionaryoptions\prefernobreakdiscoptioncode
%     \hsize\widthofstring{sciencefiction}
%     science\discretionary{\red fict-}{\green ion}{\blue fiction}\par
%     science\discretionary{\red f\kern0ptiction}{}{\blue fiction}\par
%     science\-fiction\par
%     science\discretionary{-}{}{\blue fiction}\par
%     \hyphenation{science-fiction}
%     sciencefiction\par
% \stoptext

% optional content example (todo: show break and nobreak keywords):

% \start
%     \tttf
%     \hsize\widthofstring{short}
%     --:\par
%     \discretionaryoptions\zerocount
%     \discretionary{before}{after}{short}\par
%     \discretionary{before}{}{short}\blank
%     nb:\par
%     \discretionaryoptions\prefernobreakdiscoptioncode
%     \discretionary{before}{after}{short}\par
%     \discretionary{before}{}{short}\blank
%     br:\par
%     \discretionaryoptions\preferbreakdiscoptioncode
%     \discretionary{before}{after}{short}\par
%     \discretionary{before}{}{short}\blank
% \stop


