Class: BufferedTokenizer

### Home Classes Methods

### In Files

- lib/em/buftok.rb

### Parent

Object

### Methods

- ::new
- #extract
- #flush

### Files

- ChangeLog
- DEFERRABLES
- EPOLL
- INSTALL
- KEYBOARD
- LEGAL
- LIGHTWEIGHT_CONCURRENCY
- PURE_RUBY
- RELEASE_NOTES
- SMTP
- SPAWNED_PROCESSES
- TODO

### Class/Module Index

Quicksearch

- EventMachine
- EventMachine
- EventMachine::Channel
- EventMachine::Completion
- EventMachine::Connection
- EventMachine::DNS
- EventMachine::DNS::Request
- EventMachine::DNS::RequestIdAlreadyUsed
- EventMachine::DNS::Resolver
- EventMachine::DNS::Socket
- EventMachine::DatagramObject
- EventMachine::DefaultDeferrable
- EventMachine::Deferrable
- EventMachine::DeferrableChildProcess
- EventMachine::Error
- EventMachine::EvmaKeyboard
- EventMachine::EvmaTCPClient
- EventMachine::EvmaTCPServer
- EventMachine::EvmaUDPSocket
- EventMachine::EvmaUNIXClient
- EventMachine::EvmaUNIXServer
- EventMachine::FileNotFoundException
- EventMachine::FileStreamer
- EventMachine::FileWatch
- EventMachine::Iterator
- EventMachine::LoopbreakReader
- EventMachine::PeriodicTimer
- EventMachine::Pool
- EventMachine::ProcessWatch
- EventMachine::Protocols
- EventMachine::Protocols
- EventMachine::Protocols::HeaderAndContentProtocol
- EventMachine::Protocols::HttpClient
- EventMachine::Protocols::HttpClient2
- EventMachine::Protocols::HttpClient2::Request
- EventMachine::Protocols::LineAndTextProtocol
- EventMachine::Protocols::LineProtocol
- EventMachine::Protocols::LineText2
- EventMachine::Protocols::Memcache
- EventMachine::Protocols::Memcache::ParserError
- EventMachine::Protocols::ObjectProtocol
- EventMachine::Protocols::Postgres3
- EventMachine::Protocols::SASLauth
- EventMachine::Protocols::SASLauthclient
- EventMachine::Protocols::SmtpClient
- EventMachine::Protocols::SmtpServer
- EventMachine::Protocols::Socks4
- EventMachine::Protocols::Stomp
- EventMachine::Protocols::Stomp::Message
- EventMachine::Protocols::TcpConnectTester
- EventMachine::Queue
- EventMachine::Reactor
- EventMachine::Selectable
- EventMachine::SpawnedProcess
- EventMachine::StreamObject
- EventMachine::SystemCmd
- EventMachine::ThreadedResource
- EventMachine::TickLoop
- EventMachine::Timer
- EventMachine::UuidGenerator
- EventMachine::YieldBlockFromSpawnedProcess
- Object
- Object::EM
- Object::P
- BufferedTokenizer
- IO
- StringIO
- TestConnection

No matching classes.

# BufferedTokenizer

BufferedTokenizer takes a delimiter upon instantiation, or acts line-based by default. It allows input to be spoon-fed from some outside source which receives arbitrary length datagrams which may-or-may-not contain the token by which entities are delimited. In this respect it's ideally paired with something like EventMachine (rubyeventmachine.com/).

### Public Class Methods

new(delimiter = $/) click to toggle source

New BufferedTokenizers will operate on lines delimited by a delimiter, which is by default the global input delimiter $/ ("n").

The input buffer is stored as an array. This is by far the most efficient approach given language constraints (in C a linked list would be a more appropriate data structure). Segments of input data are stored in a list which is only joined when a token is reached, substantially reducing the number of objects required for the operation.

    # File lib/em/buftok.rb, line 15
    def initialize(delimiter = $/)
      @delimiter = delimiter
      @input = []
      @tail = ''
      @trim = @delimiter.length - 1
    end

### Public Instance Methods

extract(data) click to toggle source

Extract takes an arbitrary string of input data and returns an array of tokenized entities, provided there were any available to extract. This makes for easy processing of datagrams using a pattern like:

    tokenizer.extract(data).map { |entity| Decode(entity) }.each do ...

Using -1 makes split to return "" if the token is at the end of the string, meaning the last element is the start of the next chunk.

    # File lib/em/buftok.rb, line 30
    def extract(data)
      if @trim > 0
        tail_end = @tail.slice!(-@trim, @trim) # returns nil if string is too short
        data = tail_end + data if tail_end
      end

      @input << @tail
      entities = data.split(@delimiter, -1)
      @tail = entities.shift

      unless entities.empty?
        @input << @tail
        entities.unshift @input.join
        @input.clear
        @tail = entities.pop
      end

      entities
    end

flush() click to toggle source

Flush the contents of the input buffer, i.e. return the input buffer even though a token has not yet been encountered

    # File lib/em/buftok.rb, line 52
    def flush
      @input << @tail
      buffer = @input.join
      @input.clear
      @tail = "" # @tail.clear is slightly faster, but not supported on 1.8.7
      buffer
    end

[Validate]

Generated with the Darkfish Rdoc Generator 2.