[XeTeX] Mysterious nonzero lastskip

[David M. Jones]

> Date: Tue, 1 Dec 2009 22:07:58 -0500
> From: dmj at ams.org

Earlier I wrote:

> I don't even begin to understand how OpenType fonts are integrated
> into XeTeX, so that's as far as I can go.

But on further reflection, why should I allow ignorance stop me from
speculating?

First, a plain xetex version of the test file:

    \showboxdepth=10000
    \showboxbreadth=10000

    \def\foo{%
        \ifdim\lastskip>0pt
            \message{!!! \the\lastskip\space isn't 0???}%
        \else
            \message{!!! normal lastskip}%
        \fi
    }

    \font\cmr=cmr10

    \cmr

    filler

    This is a\quad test\foo

    \font\lmr="[lmroman10-regular]"

    \lmr

    This is a\quad test\foo

    \bye

When using an external font engine, XeTeX presumably typsets words by
accumulating characters in a temporary list and expanding tokens
forward until it finds a non-character token, at which time it hands
the entire list to the external font engine and gets back a
native_word_node(?), which it then appends to the current list.

That would mean that when XeTeX encounters \foo, it expands it to see
if it contains any more characters.  It's not until it hits the
non-expandable, non-character token "\message" that XeTeX wraps up the
current word and hands it to the font engine, but by then the first
\lastskip has already been executed and reported the amount of space
after "a".  By the time the \lastskip inside the \message is
encountered, the node containing "test" has been added to the list, so
\lastskip now reports 0pt.

In other words, this is somewhat analogous to the problem with
plugging \if... into align preambles that Knuth discusses on page 240
or so of the TeXbook, and it means that when targeting XeTeX, macro
writers need to think carefully about whether they should write

    \def\foo{\if...}

or

    \def\foo{\relax\if...}

even when not beginning a macro with \ifmmode.

Cheers,
David.