% language=us

\usemodule[present-boring,abbreviations-logos]

% This was a talk about a preliminary mechanism so it has been adapted to run in
% the current version. I need to check other 2020 files and examples and will do
% that when I update (especially the low level) manuals. Let me know when there
% are issues.

\startdocument
  [title={CONCEPTS},
   banner={experiments turned features},
   location={context\enspace {\bf 2020}\enspace meeting}]

% see datatypes, hyphenation etc

\starttitle[title=Experiments]

There have been quite some experiments. Some results were rejected, some kept.
Here are a few (that come to mind). This talk is a mix of summary, discussion and
some demos.

\stoptitle

\starttitle[title=Math]

There are a couple of additional features in the math engine. Most concern a bit
more control over hard coded behavior, but some are sort of new:

\startbuffer
test $a = b \discretionary class 3 {$<$}{$>$}{$\neq$}  c$ test
\stopbuffer

\typebuffer

When there is enough room this will give

\getbuffer

When \type {\hsize} is limited we get:

{\hsize 4mm \getbuffer}

\page

\enabletrackers[math.toysymbol]

\definemathtoysymbol[Plus] [bin][+]
\definemathtoysymbol[Minus][bin][-]
\definemathtoysymbol[Equal][rel][=]

\start

\let\darkblue\darkyellow %\showmakeup[glue]

$\dorecurse{199} {
    x_{#1} \ifodd#1\relax\Plus\else\Minus\fi
} x_{200} \Equal n$

\dorecurse{10} {
    test \discretionary
        {\darkred   hel  $\sqrt{y}$ lo}
        {\darkgreen good $\sqrt{z}$ bye}
        {\darkblue  wel  $\sqrt{x}$ come} test
}

\dorecurse{10} {test #1
    $x + 2x \discretionary class 2
        {$+ \, \cdots $}
        {$     \cdots \, +$}
        {$+ \, \cdots \, +$} nx$ test
}

\stop

\stoptitle

\starttitle[title=More math]

In traditional \TEX\ the last setting wins:

\startbuffer
\def\whatevera
  {\Umathordrelspacing \textstyle=50mu
   \Umathopenbinspacing\textstyle=50mu}

\def\whateverb
  {\Umathordrelspacing \textstyle=25mu
   \Umathopenbinspacing\textstyle=25mu}

$\whatevera a = (-2)$ \par
$\whateverb a = (-2)$ \par
$\whatevera a = (-2) \quad \whateverb a = (-2)$ \par
\stopbuffer

\typebuffer \startpacked \getbuffer \stoppacked

\page

In \LUAMETATEX\ we can freeze settings on the spot:

\startbuffer
\def\whatevera
  {\frozen\Umathordrelspacing \textstyle=50mu
   \frozen\Umathopenbinspacing\textstyle=50mu}

\def\whateverb
  {\frozen\Umathordrelspacing \textstyle=25mu
   \frozen\Umathopenbinspacing\textstyle=25mu}

$\whatevera a = (-2)$ \par
$\whateverb a = (-2)$ \par
$\whatevera a = (-2) \quad \whateverb a = (-2)$ \par
\stopbuffer

\typebuffer \startpacked \getbuffer \stoppacked

{\em We can now also enable and disable specific features in the engine that
control traditional or \OPENTYPE\ approaches. This is only there for experimental
and educational purposes.}

\stoptitle

\starttitle[title=Macros]

Not storing arguments:

\starttyping
\def\foo#1#0#3{....}

\foo{11}{22}{33}

\foo #1#0#3->....
#1<-11
#2<-
#3<-33
\stoptyping

Ignoring arguments:

\starttyping
\def\foo#1#-#2{#1#2}

\foo{1}{2}{3}

13
\stoptyping

\page

Normal behaviour:

\starttyping
\def\foo#1#2#3{#1#2#3}

\foo{1}{{2}}{3}

\foo #1#2#3->#1#2#3
#1<-1
#2<-{2}
#3<-3
\stoptyping

Special behaviour:

\starttyping
\def\foo#1#+#3{#1#2#3}

\foo #1#2#3->#1#2#3
#1<-1
#2<-{{2}}
#3<-3
\stoptyping

{\em There are more specifiers and I admit that they are hard to remember. But
they are mostly used in low level macros anyway.}

\page

Optional tokens (we also show some \TEX-expansion-horror here):

\starttyping
\edef\a!space{\expandtoken \ignorecatcode \spaceasciicode}

\normalexpanded {

    \protected \def \noexpand \doifelseinset#1#2%
      {\noexpand\ifhasxtoks{,\a!space#1,}{,#2,}%
         \noexpand\expandafter\noexpand\firstoftwoarguments
       \noexpand\else
         \noexpand\expandafter\noexpand\secondoftwoarguments
       \noexpand\fi}

}
\stoptyping

or as tokens (\type{\showluatokens\doifelseinset}) on the next page:

{\em There are some expansion related extensions that are discussed in the low
level expansion manual.}

\page

\starttyping
591504   13    1  argument
643771   13    2  argument
595596   14    0  end match
633535  120   48  if test       ifhasxtoks
643789    1  123  left brace
643793   12   44  other char
643741    9   32  ignore
185919    5    1  parameter
633495   12   44  other char
 57752    2  125  right brace
167619    1  123  left brace
643686   12   44  other char
228803    5    2  parameter
643434   12   44  other char
643792    2  125  right brace
643788  114    0  expand after  expandafter
643775  125    0  call          firstoftwoarguments
590609  120    3  if test       else
643628  114    0  expand after  expandafter
643754  125    0  call          secondoftwoarguments
643763  120    2  if test       fi
\stoptyping

\page

Cheating with arguments:

\startbuffer
\def\foo#1=#2,{(#1/#2)}

\foo 1=2,\ignorearguments
\foo 1=2\ignorearguments
\foo 1\ignorearguments
\foo \ignorearguments
\stopbuffer

\typebuffer

\getbuffer

As in:

\startbuffer
\def\foo#1=#2,{\ifarguments\or(#1)\or(#1/#2)\fi}

\foo 1=2,\ignorearguments
\foo 1=2\ignorearguments
\foo 1\ignorearguments
\foo \ignorearguments
\stopbuffer

\typebuffer

\getbuffer

\stoptitle

\starttitle[title=Hyphenation]

Hyphenation at work:

\startlinecorrection[line] \small
\startcombination[distance=3cm,nx=6]
    {\vtop{\hsize 2mm \strut NEDERLANDS\vskip.5\lineheight}} {\type{NEDERLANDS}}
    {\vtop{\hsize 2mm \strut Nederlands\vskip.5\lineheight}} {\type{Nederlands}}
    {\vtop{\hsize 2mm \strut nederlands\vskip.5\lineheight}} {\type{nederlands}}
    {\vtop{\hsize 2mm \strut \CONTEXT  \vskip.5\lineheight}} {\type{\CONTEXT  }}
    {\vtop{\hsize 2mm \strut text\-test\vskip.5\lineheight}} {\type{text\-test}}
    {\vtop{\hsize 2mm \strut test-test \vskip.5\lineheight}} {\type{test-test }}
\stopcombination
\stoplinecorrection

Controlling hyphenation:

\startbuffer[1]
\nohyphens NEDERLANDS {\dohyphens Nederlands} nederlands
\stopbuffer

\startbuffer[2]
NEDERLANDS {\nohyphens Nederlands} nederlands
\stopbuffer

\typebuffer[1] and \typebuffer[2]

\startlinecorrection[line]
\startcombination[distance=4cm,nx=2]
    {\small \small \vtop{\hsize 2mm \strut \nl \getbuffer[1]}} {}
    {\small \small \vtop{\hsize 2mm \strut \nl \getbuffer[2]}} {}
\stopcombination
\stoplinecorrection

\page

There are several ways to implement this:

\startitemize
\startitem choose a language with no patterns:
    \startitemize[packed]
        \startitem it's quite efficient \stopitem
        \startitem we loose language specifics \stopitem
    \stopitemize
\stopitem
\startitem set the left and right hyphen min values high:
    \startitemize[packed]
        \startitem it works okay \stopitem
        \startitem it is a hack \stopitem
        \startitem we still enter the routine \stopitem
    \stopitemize
\stopitem
\startitem block the mechanism:
    \startitemize[packed]
        \startitem it provides detailed control \stopitem
        \startitem it is conceptually clean \stopitem
    \stopitemize
\stopitem
\stopitemize

The last method is what we use in \LMTX:

\type {\dohyphens} : {\tttf \meaning\dohyphens}

\type {\nohyphens} : {\tttf \meaning\nohyphens}

\page

For the moment we have this (it might evolve):

\starttyping[style=\tt\small\small]
\chardef \completehyphenationmodecode \numexpr
    \normalhyphenationmodecode            % \discretionary
  + \automatichyphenationmodecode         % -
  + \explicithyphenationmodecode          % \-
  + \syllablehyphenationmodecode          % pattern driven
  + \uppercasehyphenationmodecode         % replaces \uchyph
  + \compoundhyphenationmodecode          % replaces \compoundhyphenmode
  % \strictstarthyphenationmodecode       % replaces \hyphenationbounds (strict = original tex)
  % \strictendhyphenationmodecode         % replaces \hyphenationbounds (strict = original tex)
  + \automaticpenaltyhyphenationmodecode  % replaces \hyphenpenaltymode (otherwise use \exhyphenpenalty)
  + \explicitpenaltyhyphenationmodecode   % replaces \hyphenpenaltymode (otherwise use \exhyphenpenalty)
  + \permitgluehyphenationmodecode        % turn glue into kern in \discretionary
  + \permitallhyphenationmodecode         % okay, let's be even more tolerant
  + \permitmathreplacehyphenationmodecode % and again we're more permissive
\relax
\stoptyping

This replaces some \LUATEX\ mode variables and adds some more which is why we now use
a bitset instead of multiple parameters.

\page

In addition we have more detailed discretionary control:

\startbuffer
nederlands\discretionary           {!}{!}{!}nederlands
nederlands\discretionary options 1 {!}{!}{!}nederlands
nederlands\discretionary options 2 {!}{!}{!}nederlands
nederlands\discretionary options 3 {!}{!}{!}nederlands
\stopbuffer

\typebuffer

\startlinecorrection[line]
\startcombination[distance=4cm,nx=4]
    {\vtop{\hsize 1mm \strut \nl nederlands\discretionary           {!}{!}{!}nederlands}} {}
    {\vtop{\hsize 1mm \strut \nl nederlands\discretionary options 1 {!}{!}{!}nederlands}} {}
    {\vtop{\hsize 1mm \strut \nl nederlands\discretionary options 2 {!}{!}{!}nederlands}} {}
    {\vtop{\hsize 1mm \strut \nl nederlands\discretionary options 3 {!}{!}{!}nederlands}} {}
\stopcombination
\stoplinecorrection

At some point it will become \quote {frozen} functionality and that's when it gets
documented (first we need to integrate and play a bit more with it in \CONTEXT).

{\em There is now a plugin mechanism that provides more control over language
specific hyphenation, e.g.\ compound words combined with ligatures.}

\stoptitle

\starttitle[title=Local control]

In \LUATEX\ we have experimental (kind of ugly) immediate assignments that can be
used in expansions without blocking (resulting in tokens that is).

But now we now have local control:

\startbuffer
\newcount\foocounter

\def\foo
  {\advance\foocounter\plusone
   \the\foocounter}

\edef\oof{(\foo)(\foo)(\foo)(\foo)}

\meaning\oof
\stopbuffer

\typebuffer

{\tttf \nohyphens \veryraggedright \getbuffer}

\page

Immediate expansion:

\startbuffer
\def\foo
  {\beginlocalcontrol
     \advance\foocounter\plusone
   \endlocalcontrol
   \the\foocounter}

\edef\oof{(\foo)(\foo)(\foo)(\foo)}

\meaning\oof
\stopbuffer

\typebuffer

{\tttf \getbuffer}

Hidden assignments:

\startbuffer
\scratchcounterone \beginlocalcontrol
    \scratchcountertwo 100
    \multiply \scratchcountertwo by 4
\endlocalcontrol \scratchcountertwo
\the\scratchcounterone
\stopbuffer

\typebuffer

{\tttf \getbuffer}

% \the \beginlocalcontrol
%     \scratchcountertwo 100
%     \multiply \scratchcountertwo by 4
% \endlocalcontrol \scratchcountertwo

\page

Fancy expansion:

\startbuffer
\protected\def\foo
  {\beginlocalcontrol
     \advance\foocounter\plusone
   \endlocalcontrol
   \the\foocounter}

\edef\oof{(\foo)(\foo)(\foo)(\foo)}
\edef\ofo{(\expand\foo)(\expand\foo)(\expand\foo)(\expand\foo)}

\meaning\oof \par \meaning\ofo
\stopbuffer

\typebuffer

{\tttf \getbuffer}

And a teaser:

\starttyping
\protected\def\widthofcontent#1{\beginlocalcontrol
     \setbox\scratchbox\hbox{#1}\endlocalcontrol \wd\scratchbox}
\stoptyping

{\em These mechanisms can have surprising side effects due to input stacking.
There is some more info in the low level expansion manual.}

\stoptitle

\starttitle[title=Conditionals]

We can get nicer code that this:

\starttyping
\ifdim\scratchdimen=10pt
   \expandafter\one
\else\ifnum\scratchcounter=20
   \expandafter\expandafter\expandafter\two
\else
   \expandafter\expandafter\expandafter\three
\fi\fi
\stoptyping

This becomes:

\starttyping
\ifdim\scratchdimen=10pt
   \expandafter\one
\orelse\ifnum\scratchcounter=20
   \expandafter\two
\else
   \expandafter\three
\fi
\stoptyping

\page

There is a bunch of extra conditions like the generic:

\startnarrower \type {\ifcondition} \stopnarrower

some token testers like:

\startnarrower \type {\iftok} and \type {\ifhas(x)tok(s)} \stopnarrower

some specific for math:

\startnarrower \type {\ifmathstyle} and \type {\ifmathparameter} \stopnarrower

macro helpers:

\startnarrower \type {\ifarguments}, \type {\ifboolean} and \type {\ifempty} \stopnarrower

robust number and dimension interception:

\startnarrower \type {\ifchknum}, \type {\ifchkdim}, \type {\ifcmpnum}, \type {\ifcmpdim)}, \type {\ifnumval} and \type {\ifdimval} \stopnarrower

bonus checks:

\startnarrower \type {\iffrozen}, \type {\ifprotected} and \type {\ifusercmd} \stopnarrower

and the mentioned:

\startnarrower \type {\orelse} and \type {\orunless} \stopnarrower

\stoptitle

\starttitle[title=Migration]

\startbuffer
h: \setbox0\hbox{box    \footnote{h:     box}}\setbox2\hbox{\box    0}\box2\par
h: \setbox0\hbox{copy   \footnote{h:    copy}}\setbox2\hbox{\copy   0}\box2\par
h: \setbox0\hbox{unbox  \footnote{h:  unhbox}}\setbox2\hbox{\unhbox 0}\box2\par
h: \setbox0\hbox{uncopy \footnote{h: unhcopy}}\setbox2\hbox{\unhcopy0}\box2\par

v: \setbox0\hbox{box    \footnote{v:     box}}\setbox2\vbox{\box    0}\box2\par
v: \setbox0\hbox{copy   \footnote{v:    copy}}\setbox2\vbox{\copy   0}\box2\par
v: \setbox0\hbox{unbox  \footnote{v:  unhbox}}\setbox2\vbox{\unhbox 0}\box2\par
v: \setbox0\hbox{uncopy \footnote{v: unhcopy}}\setbox2\vbox{\unhcopy0}\box2\par

\starttabulate[||]
\NC tabulate \footnote{tabulate} \NC \NR
\stoptabulate
\stopbuffer

\typebuffer[style={\tt\small}]

\page

\startpacked \getbuffer \stoppacked

% \setbox0\hbox{test \footnote{test}} (\prelistbox0) (\postlistbox0)
% \setprelistbox0\hbox{BEFORE} \setpostlistbox0\hbox{AFTER}
% \box0

% \setbox0\hbox{test \footnote{test}} (\prelistcopy0) (\postlistcopy0)
% \setprelistbox0\hbox{BEFORE} \setpostlistbox0\hbox{AFTER}
% \box0

{\em Everything insert related will always have side effects. It's complicated
by the fact that the page flow interferes with expectations of where notes
break cq.\ end up.}

\stoptitle

\starttitle[title=Normalizing lines]

We can have predictable lines:

\startbuffer
    \hangindent3cm \hangafter 2 \leftskip1cm \rightskip1cm \input ward \par
\stopbuffer

\typebuffer

Standard (but already with left skips):

\start
    \bitwiseflip \normalizelinemode -\parindentskipnormalizecode
    \bitwiseflip \normalizelinemode -\normalizelinenormalizecode
    \showmakeup \relax \getbuffer
\stop

Normalized (enhanced, no shifts, indent skip):

\start
    \bitwiseflip \normalizelinemode \parindentskipnormalizecode
    \bitwiseflip \normalizelinemode \normalizelinenormalizecode
    \showmakeup \relax \getbuffer
\stop

\page

\startbuffer
    \parshape 2 1cm 10cm 2cm 15cm \leftskip1cm \rightskip1cm \input ward \par
\stopbuffer

\typebuffer

Standard:

\start
    \bitwiseflip \normalizelinemode -\parindentskipnormalizecode
    \bitwiseflip \normalizelinemode -\normalizelinenormalizecode
    \showmakeup \relax \getbuffer
\stop

Normalized:

\start
    \bitwiseflip \normalizelinemode \parindentskipnormalizecode
    \bitwiseflip \normalizelinemode \normalizelinenormalizecode
    \showmakeup \relax \getbuffer
\stop

% {\showmakeup \getbuffer}

{\em There might be some more normalization in the future in other subsystems of
the engine. One should be aware of this when manipulating node lists after they
come out such subsystems.}

\stoptitle

\starttitle[title=Freezing paragraph properties]

\startbuffer[sample]
\startplacefigure[location=left,number=no] \externalfigure[halslegacy.jpg][width=30pt] \stopplacefigure

{\bf David Stork:} \samplefile{stork}
\stopbuffer

\startbuffer[demo]
\forgetparagraphfreezing \getbuffer[sample]
\stopbuffer

\typebuffer[demo][style=\tt\small] \start \switchtobodyfont[6pt] \getbuffer[demo] \par \stop

\startbuffer[demo]
\setparagraphfreezing    \getbuffer[sample]
\stopbuffer

\typebuffer[demo][style=\tt\small] \start \switchtobodyfont[6pt] \getbuffer[demo] \par \stop

Sample: \typebuffer[sample][style=\tt\small\small]

{\em This feature will stepwise be applied to mechanism and might have side effects when
users have their own hacks around \TEX's limitations (and side effects).}

\stoptitle

\starttitle[title=Wrapping up paragraphs]

\setparagraphfreezing

We have \type {\wrapuppar} as new hook. An experimental mechanism has been build
around it so that Wolfgang and I can freak out on this.

\startbuffer
\def\TestA{\registerparwrapper
  {A}
  {[\ignorespaces}
  {\removeunwantedspaces]\showparwrapperstate{A}}}

\def\TestB#1{\registerparwrapper
  {B#1}
  {(\ignorespaces}
  {\removeunwantedspaces)\showparwrapperstate{B#1}}}

\def\TestC{\registerparwrapper
  {C}
  {<\ignorespaces}
  {\removeunwantedspaces>\showparwrapperstate{C}\forgetparwrapper}}

\def\TestR{\registerparwrapperreverse
  {R}
  {<\ignorespaces}
  {\removeunwantedspaces>\showparwrapperstate{R}}}
\stopbuffer

\typebuffer[style=\tt\small] \getbuffer

\page

Example 1:

\startbuffer
\TestA
\dorecurse{3}
    {1.#1 before \ruledvbox{\hsize2em\raggedcenter\TestB1 !\par} after\par}
\dorecurse{3}
    {2.#1 before \ruledvbox{\hsize3em\raggedcenter        !\par} after\par}
\dorecurse{3}
    {3.#1 before \ruledvbox{\hsize4em\raggedcenter\TestB2 !}     after\par}
\forgetparwrapper
\dorecurse{3}
    {4.#1 before \ruledvbox{\hsize5em\raggedcenter\TestB3 !}     after\par}
\TestC
\dorecurse{3}
    {5.#1 before \ruledvbox{\hsize2em\raggedcenter\TestA  !}     after\par}
\stopbuffer

\typebuffer[style=\tt\small]

\startcolumns  \startpacked \getbuffer \stoppacked \stopcolumns

\page

Example 2:

\startbuffer
\TestA
\dorecurse{3}{6.#1  before after\par} \blank
\TestB4
\dorecurse{3}{7.#1 before after\par} \blank
\TestB5
\TestR
\dorecurse{3}{8.#1 before after\par} \blank
\stopbuffer

\typebuffer[style=\tt\small]

\startcolumns  \startpacked \getbuffer \stoppacked \stopcolumns

{\em These are just weird examples, but you can expect more interesting features to
show up. Beware of stacking because order matters.}

\stoptitle

\stopdocument
