Package org.antlr.v4.runtime

Class CommonTokenFactory

java.lang.Object

org.antlr.v4.runtime.CommonTokenFactory

All Implemented Interfaces:

TokenFactory

public class CommonTokenFactory
extends Object
implements TokenFactory

This default implementation of TokenFactory creates CommonToken objects.

Field Summary

Fields

Modifier and Type	Field	Description
protected final boolean	copyText	Indicates whether CommonToken.setText(java.lang.String) should be called after constructing tokens to explicitly set the text.
static final TokenFactory	DEFAULT	The default CommonTokenFactory instance.

Constructor Summary

Constructors

Constructor	Description
CommonTokenFactory()	Constructs a CommonTokenFactory with copyText set to false.
CommonTokenFactory(boolean copyText)	Constructs a CommonTokenFactory with the specified value for copyText.

Method Summary

Modifier and Type	Method	Description
CommonToken	create(int type, String text)	Generically useful
CommonToken	create(Tuple2<? extends TokenSource,CharStream> source, int type, String text, int channel, int start, int stop, int line, int charPositionInLine)	This is the method used to create tokens in the lexer and in the error handling strategy.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT

public static final TokenFactory DEFAULT

The default CommonTokenFactory instance.

This token factory does not explicitly copy token text when constructing tokens.

copyText

protected final boolean copyText

Indicates whether CommonToken.setText(java.lang.String) should be called after constructing tokens to explicitly set the text. This is useful for cases where the input stream might not be able to provide arbitrary substrings of text from the input after the lexer creates a token (e.g. the implementation of CharStream.getText(org.antlr.v4.runtime.misc.Interval) in UnbufferedCharStream throws an UnsupportedOperationException). Explicitly setting the token text allows Token.getText() to be called at any time regardless of the input stream implementation.

The default value is false to avoid the performance and memory overhead of copying text for every token unless explicitly requested.

Constructor Details

CommonTokenFactory

public CommonTokenFactory(boolean copyText)

Constructs a CommonTokenFactory with the specified value for copyText.

When copyText is false, the DEFAULT instance should be used instead of constructing a new instance.

Parameters:

copyText - The value for copyText.

CommonTokenFactory

public CommonTokenFactory()

Constructs a CommonTokenFactory with copyText set to false.

The DEFAULT instance should be used instead of calling this directly.

Method Details

create

public CommonToken create(Tuple2<? extends TokenSource,CharStream> source,
 int type,
 String text,
 int channel,
 int start,
 int stop,
 int line,
 int charPositionInLine)

Description copied from interface: TokenFactory

This is the method used to create tokens in the lexer and in the error handling strategy. If text!=null, than the start and stop positions are wiped to -1 in the text override is set in the CommonToken.

Specified by:

create in interface TokenFactory

create

public CommonToken create(int type,
 String text)

Description copied from interface: TokenFactory

Generically useful

Specified by:

create in interface TokenFactory