Token Models

Token models model leaf nodes in the AST, who may not have children.

Show code cell source Hide code cell source

from autobean_refactor import models, parser

p = parser.Parser()

# Most commonly, you'll want to parse a file
file = p.parse('2000-01-01 open Assets:Foo\n2000-01-02 close Assets:Foo', models.File)

# You can parse into other models as well
close = p.parse('2000-01-02 close Assets:Foo', models.Close)

# Token needs to be parsed with `parse_token` instead.
comment = p.parse_token('; comment', models.BlockComment)
string1 = p.parse_token(r'"foo\n"', models.EscapedString)
string2 = p.parse_token('"foo\n"', models.EscapedString)
number = p.parse_token('1,234.56', models.Number)

Common interfaces

raw_text

All token models have a string property raw_text which returns the raw text of the token.

value usually gives a more useful representation of the token, but raw_text allows more precise manipulation of the token.

(comment.raw_text, string1.raw_text, string2.raw_text, number.raw_text)

('; comment', '"foo\\n"', '"foo\n"', '1,234.56')

value

Many token models have a property value which returns the semantic value of the token.

(comment.value, string1.value, string2.value, number.value)

('comment', 'foo\n', 'foo\n', Decimal('1234.56'))

from_raw_text

All token models have a class method from_raw_text which constructs a new token model from the raw text.

This constructs a token model with the exact raw text, and therefore:

• ✅ type(token).from_raw_text(token.raw_text) == token

• ✅ type(token).from_value(token.value).value == token.value (if value is supported)

>>> comment == models.BlockComment.from_raw_text('; comment')
True
>>> comment == models.BlockComment.from_raw_text(';comment')
False

from_value

Many token models have a class method from_value which constructs a new token model from the semantic value.

Some default formatting will apply, and therefore:

• ❌ type(token).from_value(token.value) == token

• ✅ type(token).from_value(x).value == token.value

>>> models.BlockComment.from_value('comment').raw_text
'; comment'
>>> models.EscapedString.from_value('foo\n').raw_text
'foo\n'
>>> models.Number.from_value(decimal.Decimal('1234.56')).raw_text
'1234.56'

List of all token models

sorted([
    model.__name__ for model in models.__dict__.values()
    if isinstance(model, type) and issubclass(model, models.RawTokenModel)])

['Account',
 'AddOp',
 'Asterisk',
 'At',
 'AtAt',
 'BalanceLabel',
 'BlockComment',
 'Bool',
 'CloseLabel',
 'Comma',
 'CommodityLabel',
 'Currency',
 'CustomLabel',
 'Date',
 'DblLeftBrace',
 'DblRightBrace',
 'DedentMark',
 'DocumentLabel',
 'Eol',
 'EscapedString',
 'EventLabel',
 'Hash',
 'Ignored',
 'IncludeLabel',
 'Indent',
 'InlineComment',
 'LeftBrace',
 'LeftParen',
 'Link',
 'MetaKey',
 'MulOp',
 'Newline',
 'NoteLabel',
 'Null',
 'Number',
 'OpenLabel',
 'OptionLabel',
 'PadLabel',
 'PluginLabel',
 'PopmetaLabel',
 'PoptagLabel',
 'PostingFlag',
 'PriceLabel',
 'PushmetaLabel',
 'PushtagLabel',
 'QueryLabel',
 'RawTokenModel',
 'RightBrace',
 'RightParen',
 'Tag',
 'Tilde',
 'TransactionFlag',
 'UnaryOp',
 'Whitespace']