[XeTeX] Confused (why are ignored characters not "removed from the input" as per Eijkhout's TeX by Topic ?)

[Zdenek Wagner]

ne 16. 4. 2023 v 18:44 odesílatel Philip Taylor (Hellenic Institute)
<P.Taylor at hellenic-institute.uk> napsal:
>
> On 16/04/2023 17:37, Jonathan Kew wrote:
>
> First note: I see the same result with plain TeX. So not a XeTeX issue.
>
> Agreed, but as I use only XeTeX this seemed the right place to ask.  I did test with PdfTeX (not having plain TeX in my TeXworks armoury) and PdfTeX did much the same but put a real <tab> in the transcript as opposed to ^^I.
>
> Anyway, this is expected behavior. The ^^I isn't part of the offending control sequence; it's just the preceding context on the line, which is what normally appears in a TeX error message.
>
> OK, so I should have realised that in the absence of a leading "\", ^^I~ could not possibly be a control sequence and therefore the "~" had to be an active character.  That fact had passed me by ...
>
> Ignored characters are not "removed from the input" (despite anything Eijkhout says); they're still present, just ignored.
>
> Having consulted Eijkhout first, I did then search the TeXbook to see if I could find a definitive statement concerning the treatment of ignored characters but failed to do so — perhaps I should search the PDF version rather than the printed ...
>
Just what I remember, TeX algorithms are separated to the "mouth" and
the "stomach". The mouth reads the input and assigns categories, thus
it sees ^^I as  the character with code 0x09, assigns category 9 to it
(ignored character) and sends this token to the stomach. The stomach
accepts a "character token" consisting from the character 0x09 with
category 9. The category says that the stomach should ignore it. What
causes an error message is and active character ~ (category 13) with
undefined definition. The error message contains the line (as given by
the mouth) up to the token which caused the error. This is the reason
why the ignored character appears in the error message, the character
is ignored by the stomach, not by the mouth.
> --
> Philip Taylor

Zdeněk Wagner
https://www.zdenek-wagner.eu/