Class Tokenizer

simple tokenizer for DCanvas ML

Constructors

Name	Description
this (source, filename, singleLineCommentPrefixes)

Fields

Name	Type	Description
_filename	string
_len	int
_line	ushort
_lines	LineStream
_lineText	dchar[]
_pos	ushort
_prevChar	dchar
_singleLineCommentPrefixes	string[]
_token	Token

Properties

Name	Type	Description
filename[get]	string
line[get]	int
pos[get]	int

Methods

Name	Description
emitError (msg)
emitError (msg, token)
getContextSource ()
isAlpha (ch)
isAlphaNum (ch)
isNum (ch)
nextToken ()	get next token
nextChar ()
parseEof ()
parseEol ()
parseError ()
parseFloating (n)
parseHex (prefixLen)
parseIdent ()
parseMultiLineComment ()
parseNumber ()
parseOp (op)
parseSingleLineComment ()
parseString ()
parseWhiteSpace ()
peekChar ()
peekNextChar ()
setTokenStart ()
skipChar ()

Aliases

Name	Description
~this