Package org.antlr.v4.runtime.atn

Class LexerATNSimulator

java.lang.Object

org.antlr.v4.runtime.atn.ATNSimulator

org.antlr.v4.runtime.atn.LexerATNSimulator

public class LexerATNSimulator
extends ATNSimulator

ATN interpreter for lexers. Builds and consults a per-mode DFA cache while falling back to ATN simulation for unseen edges.

PERF: Reach config sets are retained on the simulator and cleared between edge computations so the common DFA-fill path avoids per-character OrderedATNConfigSet allocation. Clear of the retained pool is occupancy-aware (see ATNConfigSet.clear() / RetainedConfigSet): empty clear is O(1) for the merge map (no bulk Arrays.fill); non-empty clear nulls only the used list prefix and, when few configs remain in a grown open-addressed table, removes known merge keys (O(n)) instead of filling the full table capacity. Obtain/release under computeTargetState(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.dfa.DFAState, int) therefore does not pay repeated full-capacity zeroing. A single lexer (and its simulator) is not used concurrently for matching, so reuse is safe. Stored DFA states always receive a readonly ATNConfigSet.clone(boolean) of the reach set, never the reusable buffer itself.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	LexerATNSimulator.SimState	When we hit an accept state in either the DFA or the ATN, we have to notify the character stream to start buffering characters via IntStream.mark() and record the current state.

Field Summary

Fields

Modifier and Type	Field	Description
protected int	charPositionInLine	The index of the character relative to the beginning of the line 0..n-1
private static final int	CLOSURE_BUSY_DEPTH	Closure frames before closureBusy is consulted.
private final Map<ATNConfig,Boolean>	closureBusy	Closure busy set for the current outermost closure(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.atn.ATNConfig, org.antlr.v4.runtime.atn.ATNConfigSet, boolean, boolean, boolean) call.
private int	closureDepth	Nesting depth of closure(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.atn.ATNConfig, org.antlr.v4.runtime.atn.ATNConfigSet, boolean, boolean, boolean).
static final boolean	debug
static final boolean	dfa_debug
protected int	line	line number 1..n within the input
static int	match_calls	Deprecated. This field is no longer used.
static final int	MAX_DFA_EDGE
static final int	MIN_DFA_EDGE
protected int	mode
boolean	optimize_tail_calls
protected final LexerATNSimulator.SimState	prevAccept	Used during DFA/ATN exec to record the most recent accept configuration info
protected final Lexer	recog
private final RetainedConfigSet	retainedReach	Retained reach set for computeTargetState(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.dfa.DFAState, int).
protected int	startIndex	The current token's starting index into the character stream.

Fields inherited from class org.antlr.v4.runtime.atn.ATNSimulator

atn, ERROR, RULE_LF_VARIANT_MARKER, RULE_NOLF_VARIANT_MARKER, RULE_VARIANT_DELIMITER, SERIALIZED_UUID, SERIALIZED_VERSION

Constructor Summary

Constructors

Constructor	Description
LexerATNSimulator(ATN atn)
LexerATNSimulator(Lexer recog, ATN atn)

Method Summary

Modifier and Type	Method	Description
protected void	accept(CharStream input, LexerActionExecutor lexerActionExecutor, int startIndex, int index, int line, int charPos)
protected DFAState	addDFAEdge(DFAState from, int t, ATNConfigSet q)
protected void	addDFAEdge(DFAState p, int t, DFAState q)
protected DFAState	addDFAState(ATNConfigSet configs)	Add a new DFA state if there isn't one with this set of configurations already.
protected void	captureSimState(LexerATNSimulator.SimState settings, CharStream input, DFAState dfaState)
protected boolean	closure(CharStream input, ATNConfig config, ATNConfigSet configs, boolean currentAltReachedAcceptState, boolean speculative, boolean treatEofAsEpsilon)	Since the alternatives within any lexer decision are ordered by preference, this method stops pursuing the closure as soon as an accept state is reached.
protected ATNConfigSet	computeStartState(CharStream input, ATNState p)
protected DFAState	computeTargetState(CharStream input, DFAState s, int t)	Compute a target state for an edge in the DFA, and attempt to add the computed state and corresponding edge to the DFA.
void	consume(CharStream input)
(package private) final void	consume(CharStream input, int curChar)	Consume curChar (already input.LA(1)) and update line/column.
void	copyState(LexerATNSimulator simulator)
protected boolean	evaluatePredicate(CharStream input, int ruleIndex, int predIndex, boolean speculative)	Evaluate a predicate specified in the lexer.
protected int	execATN(CharStream input, DFAState ds0)
protected int	failOrAccept(LexerATNSimulator.SimState prevAccept, CharStream input, ATNConfigSet reach, int t)
int	getCharPositionInLine()
final DFA	getDFA(int mode)
protected ATNConfig	getEpsilonTarget(CharStream input, ATNConfig config, Transition t, ATNConfigSet configs, boolean speculative, boolean treatEofAsEpsilon)
protected DFAState	getExistingTargetState(DFAState s, int t)	Get an existing target state for an edge in the DFA.
int	getLine()
protected void	getReachableConfigSet(CharStream input, ATNConfigSet closure, ATNConfigSet reach, int t)	Given a starting configuration set, figure out all ATN configurations we can reach upon input t.
protected ATNState	getReachableTarget(Transition trans, int t)
String	getText(CharStream input)	Get the text matched so far for the current token.
String	getTokenName(int t)
int	match(CharStream input, int mode)
protected int	matchATN(CharStream input)
void	reset()
(package private) final ATNConfigSet	retainedReachBuffer()	Underlying retained reach buffer after first computeTargetState(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.dfa.DFAState, int), else null.
private boolean	reuseClosure(ATNConfig config, boolean currentAltReachedAcceptState)
void	setCharPositionInLine(int charPositionInLine)
void	setLine(int line)

Methods inherited from class org.antlr.v4.runtime.atn.ATNSimulator

checkCondition, checkCondition, clearDFA, deserialize, edgeFactory, stateFactory, toInt, toInt32, toLong, toUUID

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

debug

public static final boolean debug

See Also:

• Constant Field Values

dfa_debug

public static final boolean dfa_debug

See Also:

• Constant Field Values

MIN_DFA_EDGE

public static final int MIN_DFA_EDGE

See Also:

• Constant Field Values

MAX_DFA_EDGE

public static final int MAX_DFA_EDGE

See Also:

• Constant Field Values

optimize_tail_calls

public boolean optimize_tail_calls

recog

@Nullable
protected final Lexer recog

startIndex

protected int startIndex

The current token's starting index into the character stream. Shared across DFA to ATN simulation in case the ATN fails and the DFA did not have a previous accept state. In this case, we use the ATN-generated exception object.

line

protected int line

line number 1..n within the input

charPositionInLine

protected int charPositionInLine

The index of the character relative to the beginning of the line 0..n-1

mode

protected int mode

prevAccept

@NotNull
protected final LexerATNSimulator.SimState prevAccept

Used during DFA/ATN exec to record the most recent accept configuration info

retainedReach

private final RetainedConfigSet retainedReach

Retained reach set for computeTargetState(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.dfa.DFAState, int). RetainedConfigSet.obtain(int) always returns empty; RetainedConfigSet.release() runs in finally after each edge.

closureBusy

private final Map<ATNConfig,Boolean> closureBusy

Closure busy set for the current outermost closure(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.atn.ATNConfig, org.antlr.v4.runtime.atn.ATNConfigSet, boolean, boolean, boolean) call. The value is the currentAltReachedAcceptState flag of the most permissive visit (false means the config was added). A revisit is skipped unless a later path is more permissive, which stops EOF* and epsilon cycles without dropping a diamond that still has to add the config. Warm DFA edge walks do not enter closure(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.atn.ATNConfig, org.antlr.v4.runtime.atn.ATNConfigSet, boolean, boolean, boolean).

closureDepth

private int closureDepth

Nesting depth of closure(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.atn.ATNConfig, org.antlr.v4.runtime.atn.ATNConfigSet, boolean, boolean, boolean). Zero outside a closure root.

CLOSURE_BUSY_DEPTH

private static final int CLOSURE_BUSY_DEPTH

Closure frames before closureBusy is consulted. Shallow decisions pay no map probe. Deep enough that a same-config cycle stops well short of the thread stack, including when the ATN itself is large.

See Also:

• Constant Field Values

match_calls

@Deprecated
public static int match_calls

Deprecated.

This field is no longer used.

Constructor Details

LexerATNSimulator

public LexerATNSimulator(@NotNull
 ATN atn)

LexerATNSimulator

public LexerATNSimulator(@Nullable
 Lexer recog,
 @NotNull
 ATN atn)

Method Details

copyState

public void copyState(@NotNull
 LexerATNSimulator simulator)

match

public int match(@NotNull
 CharStream input,
 int mode)

reset

public void reset()

Specified by:

reset in class ATNSimulator

matchATN

protected int matchATN(@NotNull
 CharStream input)

execATN

protected int execATN(@NotNull
 CharStream input,
 @NotNull
 DFAState ds0)

getExistingTargetState

@Nullable
protected DFAState getExistingTargetState(@NotNull
 DFAState s,
 int t)

Get an existing target state for an edge in the DFA. If the target state for the edge has not yet been computed or is otherwise not available, this method returns null.

Parameters:

s - The current DFA state

t - The next input symbol

Returns:

The existing target DFA state for the given input symbol t, or null if the target state for this edge is not already cached

consume

final void consume(@NotNull
 CharStream input,
 int curChar)

Consume curChar (already input.LA(1)) and update line/column. Avoids a second IntStream.LA(int) on the per-character DFA walk.

Parameters:

input - character stream

curChar - the code point just matched; must not be EOF

computeTargetState

@NotNull
protected DFAState computeTargetState(@NotNull
 CharStream input,
 @NotNull
 DFAState s,
 int t)

Compute a target state for an edge in the DFA, and attempt to add the computed state and corresponding edge to the DFA.

Parameters:

input - The input stream

s - The current DFA state

t - The next input symbol

Returns:

The computed target DFA state for the given input symbol t. If t does not lead to a valid DFA state, this method returns ATNSimulator.ERROR.

retainedReachBuffer

@Nullable
final ATNConfigSet retainedReachBuffer()

Underlying retained reach buffer after first computeTargetState(org.antlr.v4.runtime.CharStream, org.antlr.v4.runtime.dfa.DFAState, int), else null. Same-package tests may observe identity and emptiness.

failOrAccept

protected int failOrAccept(LexerATNSimulator.SimState prevAccept,
 CharStream input,
 ATNConfigSet reach,
 int t)

getReachableConfigSet

protected void getReachableConfigSet(@NotNull
 CharStream input,
 @NotNull
 ATNConfigSet closure,
 @NotNull
 ATNConfigSet reach,
 int t)

Given a starting configuration set, figure out all ATN configurations we can reach upon input t. Parameter reach is a return parameter.

accept

protected void accept(@NotNull
 CharStream input,
 LexerActionExecutor lexerActionExecutor,
 int startIndex,
 int index,
 int line,
 int charPos)

getReachableTarget

@Nullable
protected ATNState getReachableTarget(Transition trans,
 int t)

computeStartState

@NotNull
protected ATNConfigSet computeStartState(@NotNull
 CharStream input,
 @NotNull
 ATNState p)

closure

protected boolean closure(@NotNull
 CharStream input,
 @NotNull
 ATNConfig config,
 @NotNull
 ATNConfigSet configs,
 boolean currentAltReachedAcceptState,
 boolean speculative,
 boolean treatEofAsEpsilon)

Since the alternatives within any lexer decision are ordered by preference, this method stops pursuing the closure as soon as an accept state is reached. After the first accept state is reached by depth-first search from config, all other (potentially reachable) states for this rule would have a lower priority.

Returns:

true if an accept state is reached, otherwise false.

reuseClosure

private boolean reuseClosure(@NotNull
 ATNConfig config,
 boolean currentAltReachedAcceptState)

Returns:

true when config was already closed under an equally or more permissive accept flag

getEpsilonTarget

@Nullable
protected ATNConfig getEpsilonTarget(@NotNull
 CharStream input,
 @NotNull
 ATNConfig config,
 @NotNull
 Transition t,
 @NotNull
 ATNConfigSet configs,
 boolean speculative,
 boolean treatEofAsEpsilon)

evaluatePredicate

protected boolean evaluatePredicate(@NotNull
 CharStream input,
 int ruleIndex,
 int predIndex,
 boolean speculative)

Evaluate a predicate specified in the lexer.

If speculative is true, this method was called before consume(org.antlr.v4.runtime.CharStream, int) for the matched character. This method should call consume(org.antlr.v4.runtime.CharStream, int) before evaluating the predicate to ensure position sensitive values, including Lexer.getText(), Lexer.getLine(), and Lexer.getCharPositionInLine(), properly reflect the current lexer state. This method should restore input and the simulator to the original state before returning (i.e. undo the actions made by the call to consume(org.antlr.v4.runtime.CharStream, int).

Parameters:

input - The input stream.

ruleIndex - The rule containing the predicate.

predIndex - The index of the predicate within the rule.

speculative - true if the current index in input is one character before the predicate's location.

Returns:

true if the specified predicate evaluates to true.

captureSimState

protected void captureSimState(@NotNull
 LexerATNSimulator.SimState settings,
 @NotNull
 CharStream input,
 @NotNull
 DFAState dfaState)

addDFAEdge

@NotNull
protected DFAState addDFAEdge(@NotNull
 DFAState from,
 int t,
 @NotNull
 ATNConfigSet q)

addDFAEdge

protected void addDFAEdge(@NotNull
 DFAState p,
 int t,
 @NotNull
 DFAState q)

addDFAState

@NotNull
protected DFAState addDFAState(@NotNull
 ATNConfigSet configs)

Add a new DFA state if there isn't one with this set of configurations already. This method also detects the first configuration containing an ATN rule stop state. Later, when traversing the DFA, we will know which rule to accept.

getDFA

@NotNull
public final DFA getDFA(int mode)

getText

@NotNull
public String getText(@NotNull
 CharStream input)

Get the text matched so far for the current token.

getLine

public int getLine()

setLine

public void setLine(int line)

getCharPositionInLine

public int getCharPositionInLine()

setCharPositionInLine

public void setCharPositionInLine(int charPositionInLine)

consume

public void consume(@NotNull
 CharStream input)

getTokenName

@NotNull
public String getTokenName(int t)