Module dcanvas.dml.tokenizer
Functions
| Name | Description |
|---|---|
tokenize(code, _singleLineCommentPrefixes, skipSpace, skipEols, skipComments)
|
tokenize source into array of tokens (excluding EOF) |
trimSpaceTokens(tokens, trimBeginning, trimEnd)
|
exclude whitespace tokens at beginning and end of token sequence |
Classes
| Name | Description |
|---|---|
ParserException
|
|
Tokenizer
|
simple tokenizer for DCanvas ML |
Structs
| Name | Description |
|---|---|
Token
|
Enums
| Name | Description |
|---|---|
TokenType
|