Class: CodeRay::Tokens

### Home Classes Methods

### In Files

- lib/coderay/tokens.rb

### Parent

Array

### Methods

- #begin_group
- #begin_line
- #count
- #encode
- #end_group
- #end_line
- #method_missing
- #split_into_parts
- #to_s

### Class/Module Index

Quicksearch

- CodeRay
- CodeRay::Duo
- CodeRay::Encoders
- CodeRay::Encoders::CommentFilter
- CodeRay::Encoders::Count
- CodeRay::Encoders::Debug
- CodeRay::Encoders::DebugLint
- CodeRay::Encoders::Div
- CodeRay::Encoders::Encoder
- CodeRay::Encoders::Filter
- CodeRay::Encoders::HTML
- CodeRay::Encoders::HTML::Output
- CodeRay::Encoders::JSON
- CodeRay::Encoders::LinesOfCode
- CodeRay::Encoders::Lint
- CodeRay::Encoders::Null
- CodeRay::Encoders::Page
- CodeRay::Encoders::Span
- CodeRay::Encoders::Statistic
- CodeRay::Encoders::Terminal
- CodeRay::Encoders::Text
- CodeRay::Encoders::TokenKindFilter
- CodeRay::Encoders::XML
- CodeRay::Encoders::YAML
- CodeRay::FileType
- CodeRay::ForRedCloth
- CodeRay::Plugin
- CodeRay::PluginHost
- CodeRay::Scanners
- CodeRay::Scanners::C
- CodeRay::Scanners::CPlusPlus
- CodeRay::Scanners::CSS
- CodeRay::Scanners::Clojure
- CodeRay::Scanners::Debug
- CodeRay::Scanners::Delphi
- CodeRay::Scanners::Diff
- CodeRay::Scanners::ERB
- CodeRay::Scanners::Go
- CodeRay::Scanners::Groovy
- CodeRay::Scanners::HAML
- CodeRay::Scanners::HTML
- CodeRay::Scanners::JSON
- CodeRay::Scanners::Java
- CodeRay::Scanners::JavaScript
- CodeRay::Scanners::Python
- CodeRay::Scanners::Raydebug
- CodeRay::Scanners::Ruby
- CodeRay::Scanners::SQL
- CodeRay::Scanners::Sass
- CodeRay::Scanners::Scanner
- CodeRay::Scanners::Taskpaper
- CodeRay::Scanners::Text
- CodeRay::Scanners::XML
- CodeRay::Scanners::YAML
- CodeRay::Styles
- CodeRay::Styles::Alpha
- CodeRay::Styles::Style
- CodeRay::Tokens
- CodeRay::TokensProxy
- CodeRay::WordList
- CodeRay::WordList::CaseIgnoring

No matching classes.

# CodeRay::Tokens

The Tokens class represents a list of tokens returned from a Scanner. It's actually just an Array with a few helper methods.

A token itself is not a special object, just two elements in an Array:

- the token text (the original source of the token in a String) or a token action (begin_group, end_group, begin_line, end_line)

- the token kind (a Symbol representing the type of the token)

It looks like this:

    ..., '# It looks like this', :comment, ...
    ..., '3.1415926', :float, ...
    ..., '$^', :error, ...

Some scanners also yield sub-tokens, represented by special token actions, for example :begin_group and :end_group.

The Ruby scanner, for example, splits "a string" into:

    [
     :begin_group, :string,
     '"',          :delimiter,
     'a string',   :content,
     '"',          :delimiter,
     :end_group,   :string
    ]

Tokens can be used to save the output of a Scanners in a simple Ruby object that can be send to an Encoder later:

    tokens = CodeRay.scan('price = 2.59', :ruby).tokens
    tokens.encode(:html)
    tokens.html
    CodeRay.encoder(:html).encode_tokens(tokens)

Tokens gives you the power to handle pre-scanned code very easily: You can serialize it to a JSON string and store it in a database, pass it around to encode it more than once, send it to other algorithms...

### Attributes

scanner[RW]

The Scanner instance that created the tokens.

### Public Instance Methods

begin_group(kind;) click to toggle source

    # File lib/coderay/tokens.rb, line 153
    def begin_group kind; push :begin_group, kind end

begin_line(kind;) click to toggle source

    # File lib/coderay/tokens.rb, line 155
    def begin_line kind; push :begin_line, kind end

count() click to toggle source

Return the actual number of tokens.

    # File lib/coderay/tokens.rb, line 148
    def count
      size / 2
    end

encode(encoder, options = {}) click to toggle source

Encode the tokens using encoder.

encoder can be

- a plugin name like :html oder 'statistic'

- an Encoder object

options are passed to the encoder.

    # File lib/coderay/tokens.rb, line 53
    def encode encoder, options = {}
      encoder = Encoders[encoder].new options if encoder.respond_to? :to_sym
      encoder.encode_tokens self, options
    end

end_group(kind;) click to toggle source

    # File lib/coderay/tokens.rb, line 154
    def end_group kind; push :end_group, kind end

end_line(kind;) click to toggle source

    # File lib/coderay/tokens.rb, line 156
    def end_line kind; push :end_line, kind end

method_missing(meth, options = {}) click to toggle source

Redirects unknown methods to encoder calls.

For example, if you call `tokens.html`, the HTML encoder is used to highlight the tokens.

    # File lib/coderay/tokens.rb, line 67
    def method_missing meth, options = {}
      encode meth, options
    rescue PluginHost::PluginNotFound
      super
    end

split_into_parts(*sizes) click to toggle source

Split the tokens into parts of the given `sizes`.

The result will be an Array of Tokens objects. The parts have the text size specified by the parameter. In addition, each part closes all opened tokens. This is useful to insert tokens betweem them.

This method is used by @Scanner#tokenize@ when called with an Array of source strings. The Diff encoder uses it for inline highlighting.

    # File lib/coderay/tokens.rb, line 82
    def split_into_parts *sizes
      return Array.new(sizes.size) { Tokens.new } if size == 2 && first == ''
      parts = []
      opened = []
      content = nil
      part = Tokens.new
      part_size = 0
      size = sizes.first
      i = 0
      for item in self
        case content
        when nil
          content = item
        when String
          if size && part_size + content.size > size  # token must be cut
            if part_size < size  # some part of the token goes into this part
              content = content.dup  # content may no be safe to change
              part << content.slice!(0, size - part_size) << item
            end
            # close all open groups and lines...
            closing = opened.reverse.flatten.map do |content_or_kind|
              case content_or_kind
              when :begin_group
                :end_group
              when :begin_line
                :end_line
              else
                content_or_kind
              end
            end
            part.concat closing
            begin
              parts << part
              part = Tokens.new
              size = sizes[i += 1]
            end until size.nil? || size > 0
            # ...and open them again.
            part.concat opened.flatten
            part_size = 0
            redo unless content.empty?
          else
            part << content << item
            part_size += content.size
          end
          content = nil
        when Symbol
          case content
          when :begin_group, :begin_line
            opened << [content, item]
          when :end_group, :end_line
            opened.pop
          else
            raise ArgumentError, 'Unknown token action: %p, kind = %p' % [content, item]
          end
          part << content << item
          content = nil
        else
          raise ArgumentError, 'Token input junk: %p, kind = %p' % [content, item]
        end
      end
      parts << part
      parts << Tokens.new while parts.size < sizes.size
      parts
    end

to_s() click to toggle source

Turn tokens into a string by concatenating them.

    # File lib/coderay/tokens.rb, line 59
    def to_s
      encode CodeRay::Encoders::Encoder.new
    end

[Validate]

Generated with the Darkfish Rdoc Generator 2.