Update the Lexer to accept expected tokens from the parser

Created by: nthykier

The SVG spec allows paths to omit whitespace in some cases and expect parsers to gracefully handle this. In particularly, the parsers must greedly match as much of the token required as possible but stop as soon the token no longer matches. The latter bit is where the SVG standard gets interesting.

An Elliptical command (i.e. A or a) will accept among other: ..., number, flag, flag, number, ...

Where flag is defined as "0" or "1" (exactly one character). Given those tokens and the following input:

1 010

The spec requires scour to parse that as:

"1" (number), "0" (flag), "1" (flag) and "0" (number).

It might be tempting to just include "flag" in the default tokenization. Unfortunately this falls apart pretty quickly if you want to follow the spec. E.g. if you have 100 as input and the lexer has no hint about the next token then it can now parse it as:

• Three flags
• Two flags and a one-digit number (in that order)
• A flag and a two-digit number (in that order)
• A three-digit number

Therefore, to support this, the SVGPathParser must provide the Lexer with a hint about what it is expecting in some cases. This turns out to be trivially possible by exploiting the fact that "lex" is a generator function and can trivially be converted to a "coroutine" (by replacing "next(x)" with "x.send(value)").

Signed-off-by: Niels Thykier niels@thykier.net