Class: Dhaka::LexerRun

### Home Classes Methods

### In Files

- lib/dhaka/lexer/lexer_run.rb

### Parent

Object

### Methods

- ::new
- #create_token
- #each

### Included Modules

- Enumerable

### Class/Module Index

Quicksearch

- Dhaka
- Dhaka::CompiledLexer
- Dhaka::CompiledParser
- Dhaka::Evaluator
- Dhaka::Grammar
- Dhaka::GrammarSymbol
- Dhaka::Lexeme
- Dhaka::Lexer
- Dhaka::LexerRun
- Dhaka::LexerSpecification
- Dhaka::LexerSupport
- Dhaka::LexerSupport::AcceptAction
- Dhaka::LexerSupport::CheckpointAction
- Dhaka::LexerSupport::DFARun
- Dhaka::LexerSupport::InvalidRegexException
- Dhaka::LexerSupport::LookaheadAcceptAction
- Dhaka::LexerSupport::RegexParser
- Dhaka::LexerSupport::RegexTokenizer
- Dhaka::LexerSupport::State
- Dhaka::LexerSupport::StateMachine
- Dhaka::NoStartProductionsError
- Dhaka::ParseErrorResult
- Dhaka::ParseSuccessResult
- Dhaka::ParseTreeCompositeNode
- Dhaka::ParseTreeLeafNode
- Dhaka::Parser
- Dhaka::ParserMethods
- Dhaka::PrecedenceBuilder
- Dhaka::Production
- Dhaka::ProductionBuilder
- Dhaka::Token
- Dhaka::Tokenizer
- Dhaka::TokenizerErrorResult
- Dhaka::TokenizerState
- Dhaka::TokenizerSuccessResult

No matching classes.

# Dhaka::LexerRun

Represents a run of a lexer on a given input string.

### Attributes

current_lexeme[R]

### Public Class Methods

new(lexer, input) click to toggle source

    # File lib/dhaka/lexer/lexer_run.rb, line 7
    def initialize lexer, input
      @lexer, @input          = lexer, input
      @input_position         = 0
      @not_yet_accepted_chars = []
      @last_saved_checkpoints = {}
    end

### Public Instance Methods

create_token(symbol_name, value = current_lexeme.characters.join) click to toggle source

Constructs a token of type `symbol_name` from the `current_lexeme`.

    # File lib/dhaka/lexer/lexer_run.rb, line 15
    def create_token(symbol_name, value = current_lexeme.characters.join)
      Token.new(symbol_name, value, current_lexeme.input_position)
    end

each() click to toggle source

Yields each token as it is recognized. Returns a TokenizerErrorResult if an error occurs during tokenization.

    # File lib/dhaka/lexer/lexer_run.rb, line 20
    def each
      reset_and_rewind
      loop do
        c = curr_char
        break if (c == "\00"" && @not_yet_accepted_chars.empty? && !@current_lexeme.accepted?)
        dest_state  = @curr_state.transitions[c]
        unless dest_state
          return TokenizerErrorResult.new(@input_position) unless @current_lexeme.accepted?
          token = get_token
          yield token if token
          reset_and_rewind
        else
          @curr_state = dest_state
          @not_yet_accepted_chars << c
          @curr_state.process(self)
          advance
        end
      end
      yield Token.new(END_SYMBOL_NAME, nil, nil)
    end

[Validate]

Generated with the Darkfish Rdoc Generator 2.