% language=us runpath=texruns:manuals/lowlevel

\environment lowlevel-style

\startdocument
  [title=inserts,
   color=middlecyan]

\startsectionlevel[title=Introduction]

This document is a mixed bag. We do discuss inserts but also touch elements of
the page builder because inserts and regular page content are handled there.
Examples of mechanisms that use inserts are footnotes. These have an anchor in
the running text and some content that ends up (normally) at the bottom of the
page. When considering a page break the engine tries to make sure that the anchor
(reference) and the content end up on the same page. When there is too much, it
will distribute (split) the content over pages.

We can discuss page breaks in a (pseudo) scientific way and explore how to
optimize this process, taking into accounts also inserts that contain images but
it doesn't make much sense to do that because in practice we can encounter all
kind of interferences. Theory and practice are too different because a document
can contain a wild mix of text, figures, formulas, notes, have backgrounds and
location dependent processing. It get seven more complex when we are dealing with
columns because \TEX\ doesn't really know that concept.

I will therefore stick to some practical aspects and the main reason for this
document is that I sort of document engine features and at the same time give an
impression of what we deal with. I will do that in the perspective of
\LUAMETATEX, which has a few more options and tracing than other engines.

{\em Currently this document is mostly for myself to keep track of the state of
inserts and the page builder in \LUAMETATEX\ and \CONTEXT\ \LMTX. The text is not
yet corrected and can have errors.}

\stopsectionlevel

\startsectionlevel[title=The page builder]

When your document is processed content eventually gets added to the so called
main vertical list (mvl). Content first get appended to the list of contributions
and at specific moments it will be handed over to the mvl. This process is called
page building. There we can encounter the following elements (nodes):

\starttabulate
\NC \type {glue}     \NC a vertical skip \NC \NR
\NC \type {penalty}  \NC a vertical penalty \NC \NR
\NC \type {kern}     \NC a vertical kern \NC \NR
\NC \type {vlist}    \NC a a vertical box \NC \NR
\NC \type {hlist}    \NC a horizontal box (often a line) \NC \NR
\NC \type {rule}     \NC a horizontal rule \NC \NR
\NC \type {boundary} \NC a boundary node \NC \NR
\NC \type {whatsit}  \NC a node that is used by user code (often some extension) \NC \NR
\NC \type {mark}     \NC a token list (as used for running headers) \NC \NR
\NC \type {insert}   \NC a node list (as used for notes) \NC \NR
\stoptabulate

The engine itself will not insert anything other than this but \LUA\ code can
mess up the contribution list and the mvl and that can trigger an error. Handing
over the contributions is done by the page builder and that one kicks in in
several places:

\startitemize[packed]
\startitem
    When a penalty gets inserted it is part of evaluating if the output routine
    should be triggered. This triggering can be enforced by values equal or below
    10.000 that then can be checked in the set routine.
\stopitem
\startitem
    The builder is {\em not} exercised when a glue or kern is injected so there can
    be multiple of them before another element triggers the builder.
\stopitem
\startitem
    Adding a box triggers the builder as does the result of an alignment which can
    be a list of boxes.
\stopitem
\startitem
    When the output routine is finished the builder is executed because the
    routine can have pushed back content.
\stopitem
\startitem
    When a new paragraph is triggered by the \type {\par} command the builder kicks in
    but only when the engine was able to enter vertical mode.
\stopitem
\startitem
    When the job is finished the builder will make sure that pending content is handled.
\stopitem
\startitem
    An insert and vadjust {\em can} trigger the builder but only at the nesting level zero
    which normally is not the case (I need an example).
\stopitem
\startitem
    At the beginning of a paragraph (like text), before display math is entered,
    and when display math ends the builder is also activated.
\stopitem
\stopitemize

At the \TEX\ the builder is triggered automatically in the mentioned cases but at
the \LUA\ end you can use \type {tex.triggerbuildpage()} to flush the pending
contributions.

The properties that relate to the page look like counter and dimension registers
but they are not. These variables are global and managed differently.

\starttabulate
\NC \type {\pagegoal}         \NC the available space \NC \NR
\NC \type {\pagetotal}        \NC the accumulated space \NC \NR
\NC \type {\pagestretch}      \NC the possible zero order stretch \NC \NR
\NC \type {\pagefilstretch}   \NC the possible one order stretch \NC \NR
\NC \type {\pagefillstretch}  \NC the possible second order stretch \NC \NR
\NC \type {\pagefilllstretch} \NC the possible third order stretch \NC \NR
\NC \type {\pageshrink}       \NC the possible shrink \NC \NR
\NC \type {\pagedepth}        \NC the current page depth \NC \NR
\NC \type {\pagevsize}        \NC the initial page goal \NC \NR
\stoptabulate

When the first content is added to an empty page the \type {\pagegoal} gets the
value of \type {\vsize} and gets frozen but the value is diminished by the space
needed by left over inserts. These inserts are managed via a separate list so
they don't interfere with the page that itself of course can have additional
inserts. The \type {\pagevsize} is just a (\LUAMETATEX) status variable that hold
the initial \type {\pagegoal} but it might play a role in future extensions.

Another variable is \type {\deadcycles} that registers the number of times the
output routine is called without returning result.

\stopsectionlevel

\startsectionlevel[title=Inserts]

We now come to inserts. In traditional \TEX\ an insert is a data structure that
runs on top of registers: a box, count, dimension and skip. An insert is accessed
by a number so for instance insert 123 will use the four registers of that
number. Because \TEX\ only offers a command alias mechanism for registers (like
\type {\countdef}) a macro package will implement some allocator management
subsystem (like \type {\newcount}). A \type {\newinsert} has to be defined in a
way that the four registers are not clashing with other allocators. When you
start with \TEX\ seeing code that deals with in (in plain \TEX) can be puzzling
but it follows from the way \TEX\ is set up. But inserts are probably not what
you start exploring right away away.

In \LUAMETATEX\ you can set \type {\insertmode} to 1 and that is what we do in
\CONTEXT. In that mode inserts are taken from a pool instead of registers. A side
effect is that like the page properties the insert properties are global too but
that is normally no problem and can be managed well by a macro package (that
probably would assign register the values globally too). The insert pool will
grow dynamically on demand so one can just start at 1; in \CONTEXT\ \MKIV\ we use
the range 127 upto 255 in order to avoid a clash with registers. In \LMTX\ we start
at 1 because there are no clashes.

A consequence of this approach is that we use dedicated commands to set the insert
properties:

\starttabulate[|l|l|p|]
\NC \type {\insertdistance}   \NC glue  \NC the space before the first instance (on a page) \NC \NR
\NC \type {\insertmultiplier} \NC count \NC a factor that is used to calculate the height used \NC \NR
\NC \type {\insertlimit}      \NC dimen \NC the maximum amount of space on a page to be taken \NC \NR
\NC \type {\insertpenalty}    \NC count \NC the floating penalty (used when set) \NC \NR
\NC \type {\insertmaxdepth}   \NC dimen \NC the maximum split depth (used when set) \NC \NR
\NC \type {\insertstorage}    \NC count \NC signals that the insert has to be stored for later \NC \NR
\NC \type {\insertheight}     \NC dimen \NC the accumulated height of the inserts so far \NC \NR
\NC \type {\insertdepth}      \NC dimen \NC the current depth of the inserts so far \NC \NR
\NC \type {\insertwidth}      \NC dimen \NC the width of the inserts \NC \NR
\stoptabulate

These commands take a number and an integer, dimension or glue specification.
They can be set and queried but setting the dimensions can have side effects. The
accumulated height of the inserts is available in \type {\insertheights} (which
can be set too). The \type {\floatingpenalty} variable determines the penalty
applied when a split is needed.

In the output routine the original \TEX\ variable \type {\insertpenalties} is a
counter that keeps the number of insertions that didn't fit on the page while
otherwise if has the accumulated penalties of the split insertions. When \type
{\holdinginserts} is non zero the inserts in the list are not collected for
output, which permits the list to be fed back for reprocessing.

The \LUAMETATEX\ specific storage mode \type {\insertstoring} variable is
explained in the next section.

\stopsectionlevel

\startsectionlevel[title=Storing]

This feature is kind of special and still experimental. When \type
{\insertstoring} is set 1, all inserts that have their storage flag set will be
saved. Think of a multi column setup where inserts have to end up in the last
column. If there are three columns, the first two will store inserts. Then when
the last column is dealt with \type {\insertstoring} can be set to 2 and that
will signal the builder that we will inject the inserts. In both cases, the value
of this register will be set to zero so that it doesn't influence further
processing.

\stopsectionlevel

\startsectionlevel[title=Synchronizing]

The page builder can triggered by (for instance) a penalty but you can also use
\type {\pageboundary}. This will trigger the page builder but not leave anything
behind. (This is experimental.)

\stopsectionlevel

\startsectionlevel[title=Migration]

{\em Todo, nothing new there, so no hurry.}

\stopsectionlevel

\startsectionlevel[title=Callbacks]

{\em Todo, nothing new there, so no hurry.}

\stopsectionlevel

\stopdocument
