[metapost] make_text related scanning improvements

[Hans Hagen]

On 5/18/2018 12:51 AM, Marcel Krueger wrote:
> Hi,
> 
> some time ago I found the undocumented but very useful `make_text` callback in mplib.
> Sadly the code for scanning the text between `btex/verbatimtex` and `etex`
> is not compatible with normal `btex...etex` scanning:
> Especially `etex` at the end of a line is not accepted because of an off-by-one error and only a single line of TeX material is allowed.
> 
> This is a problem when using this with `verbatimtex`, where `etex` often comes at the end of a line and sometimes multiple lines are used.
> 
> To fix these problems I implemented a new version, which also adds some error reporting and more consistent detection of the etex token.
> I think mplib would really benefit from merging this code into the official sources.
> The corresponding CWEB change file is
Thanks for the patch / suggestion. The space before etex / verbatim tex 
is deliberate (mandate) and that wil not change. Keep in mind that full 
compatibility is not our aim, definitely not with extensions (in mp it 
was kind of a kludge anyway) and the library has its own logic 
sometimes. The text gets passed to a lua function and people can
manipulate there.

Incidentely you patch can and will break other macro packages. We take 
context as reference here.

The newline issue you sketch can probably be dealt with but then we 
might also will look at strings in general and it might become an option 
controlled by some flag to honor spaces and newlines (basically any 
character then). We'll add it to our todo list to look at it in more detail.

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------