<?xml version="1.0" encoding="UTF-8"?>
<feed xml:lang="en-US" xmlns="http://www.w3.org/2005/Atom">
  <title>~:caboose</title>
  <id>tag:habtm.com,2005:Typo</id>
  <generator version="4.0" uri="http://www.typosphere.org">Typo</generator>
  <link href="http://habtm.com/xml/atom10/feed.xml" rel="self" type="application/atom+xml"/>
  <link href="http://habtm.com/" rel="alternate" type="text/html"/>
  <updated>2006-08-16T11:59:44-05:00</updated>
  <entry>
    <author>
      <name>technoweenie</name>
    </author>
    <id>urn:uuid:3d9fa115-1bb1-420e-9fca-478a8cb520b5</id>
    <published>2006-08-16T11:58:00-05:00</published>
    <updated>2006-08-16T11:59:44-05:00</updated>
    <title type="html">A forum on Rails</title>
    <link href="http://habtm.com/articles/2006/08/16/a-forum-on-rails" rel="alternate" type="text/html"/>
    <enclosure url="http://habtm.com/podcasts/2008-10-08.mp3" type="audio/mpeg"/>
    <category term="rails" scheme="http://habtm.com/articles/category/rails" label="rails"/>
    <category term="ruby" scheme="http://habtm.com/articles/category/ruby" label="ruby"/>
    <summary type="html">&lt;plaintext&gt;&lt;p&gt;Josh Goebel and I took an evening to bang out a little project: &lt;a href="http://beast.caboo.se/"&gt;Beast&lt;/a&gt;.  It&amp;#8217;s our minimal no-fluff Rails forum.  It&amp;#8217;s no beast of an application either, clocking in at 285 LOC and a 1:1.5 test ratio.  &lt;a href="http://svn.techno-weenie.net/projects/beast/"&gt;Check it out&lt;/a&gt;!&lt;/p&gt;</summary>
    <content type="html">&lt;plaintext&gt;&lt;p&gt;Josh Goebel and I took an evening to bang out a little project: &lt;a href="http://beast.caboo.se/"&gt;Beast&lt;/a&gt;.  It&amp;#8217;s our minimal no-fluff Rails forum.  It&amp;#8217;s no beast of an application either, clocking in at 285 LOC and a 1:1.5 test ratio.  &lt;a href="http://svn.techno-weenie.net/projects/beast/"&gt;Check it out&lt;/a&gt;!&lt;/p&gt;</content>
  </entry>
  <entry>
    <author>
      <name>courtenay</name>
    </author>
    <id>urn:uuid:32816298-7855-4d65-a832-5f6a92b1a4ad</id>
    <published>2006-08-11T12:11:13-05:00</published>
    <updated>2006-08-11T12:11:13-05:00</updated>
    <title type="html">Update on the documentation project</title>
    <link href="http://habtm.com/articles/2006/08/11/update-on-the-documentation-project" rel="alternate" type="text/html"/>
    <category term="rails" scheme="http://habtm.com/articles/category/rails" label="rails"/>
    <category term="ruby" scheme="http://habtm.com/articles/category/ruby" label="ruby"/>
    <summary type="html">&lt;p&gt;Please use the wiki - &lt;a href="http://caboose.stikipad.com/documentation"&gt;http://caboose.stikipad.com/documentation&lt;/a&gt; - to give us your ideas about how the documentation project can proceed. Thanks.&lt;/p&gt;</summary>
    <content type="html">&lt;p&gt;Please use the wiki - &lt;a href="http://caboose.stikipad.com/documentation"&gt;http://caboose.stikipad.com/documentation&lt;/a&gt; - to give us your ideas about how the documentation project can proceed. Thanks.&lt;/p&gt;</content>
  </entry>
  <entry>
    <author>
      <name>KirinDave</name>
    </author>
    <id>urn:uuid:6c028f36-f87a-4f53-b7e3-1f943d2341f0</id>
    <published>2006-08-10T01:09:02-05:00</published>
    <updated>2006-08-10T01:09:02-05:00</updated>
    <title type="html">Starfish - Easy Distribution of Site Maintenance</title>
    <link href="http://habtm.com/articles/2006/08/10/starfish-easy-distribution-of-site-maintenance" rel="alternate" type="text/html"/>
    <enclosure url="http://websrvr82il.audiovideoweb.com/ny60web16519/LTN/LawTechnologyNow/LawTechnologyNow_OCT08.mp3" type="audio/mpeg"/>
    <summary type="html">&lt;p&gt;Lots of Rails apps are very simple &lt;span class="caps"&gt;CRUD&lt;/span&gt; mappings, which is something Rails excels at. But, as more and more large, complex sites go live with Rails, there is a demand for more complex &amp;#8216;backend&amp;#8217; components in the system. Already we&amp;#8217;ve got &lt;a href="http://brainspl.at/articles/2006/05/15/backgoundrb-initial-release"&gt;BackgrounDRB&lt;/a&gt;, which is great, but enter Starfish, which is a tool for complex distributed tasks made easy.&lt;/p&gt;</summary>
    <content type="html">&lt;p&gt;Lots of Rails apps are very simple &lt;span class="caps"&gt;CRUD&lt;/span&gt; mappings, which is something Rails excels at. But, as more and more large, complex sites go live with Rails, there is a demand for more complex &amp;#8216;backend&amp;#8217; components in the system. Already we&amp;#8217;ve got &lt;a href="http://brainspl.at/articles/2006/05/15/backgoundrb-initial-release"&gt;BackgrounDRB&lt;/a&gt;, which is great, but enter Starfish, which is a tool for complex distributed tasks made easy.&lt;/p&gt;&lt;p&gt;Lucas Carlson (a.k.a. cardmagic) and I have been using Starfish for various distributed tasks at &lt;a href="http://mog.com"&gt;mog.com&lt;/a&gt; for awhile now. I&amp;#8217;d like to show everyone how it works, as an incentive for Lucas to gem up the project and share it with the world.&lt;/p&gt;


    &lt;p&gt;&lt;strong&gt;The Problem&lt;/strong&gt;: Due to a bug, many albums in our music database were assigned the wrong artist. We can fix this pretty easily by checking against other data, but there are a lot of albums to run this fix against and not a lot of downtime available each day.&lt;/p&gt;


&lt;strong&gt;The Solution&lt;/strong&gt;: A starfish file that looks like this:
&lt;pre&gt;
ENV['RAILS_ENV'] ||= 'production'
require File.dirname(__FILE__) + '/../config/boot'
require File.dirname(__FILE__) + '/../config/environment'
require 'user'
require 'artist'
require 'album'
require 'user_collection_cleaning'

class ScrubAlbumsTask
  def self.service
    @@service ||= ScrubAlbumsTask.new
  end

  def initialize
    @various_artists = Artist.find_by_name( "Various Artists" )
    @albums = Album.find(:all, :conditions =&amp;gt; ["artist_id = ?", @various_artists.id])
  end

  def next
    @albums.shift.id
  end

  def has_data?
    @albums.size &amp;gt; 0
  end
end

server do |object|
  object = ScrubAlbumsTask.service
end

client do |object|
  if object.has_data?
    album = Album.find(object.next)
    album.repair_artist_by_majority!
  end
end
&lt;/pre&gt;

    &lt;p&gt;&lt;strong&gt;The Explanation&lt;/strong&gt;: We create a simple singleton-like object to do our dirty work. All it does is get ahold of the albums we need to fix (in &lt;tt&gt;initialize&lt;/tt&gt;) and then provide a simple way to get ahold of the object id to repair.&lt;/p&gt;


    &lt;p&gt;The server object merely produces a drb proxy to our clients. The clients get an id from the queue, find the object in the database, then run the repair process on them.&lt;/p&gt;


    &lt;p&gt;You can notice a few things we &lt;i&gt;didn&amp;#8217;t&lt;/i&gt; do. We didn&amp;#8217;t write any networking or distribution or threading code. We didn&amp;#8217;t need to partition the table in any arbitrary fashion. We didn&amp;#8217;t need to poll the database over and over to fetch new work units, the server keeps the state. We didn&amp;#8217;t even need to worry about redoing work, the way we wrote our server makes sure that doesn&amp;#8217;t happen.&lt;/p&gt;


    &lt;p&gt;To invoke the system, we simply run &lt;tt&gt;starfish my_albums_task &amp;#38;&lt;/tt&gt;. The first time we do this, it will output a message that says &amp;#8220;Starting Server.&amp;#8221; Starfish is smart enough to find that there are no servers for a given task, and if that is the case, it forks one off. It then invokes a client process. If you invoke the task again, a new client will be added. We can run them on multiple machines, so long as they share a local network. No fancy &lt;span class="caps"&gt;CORBA&lt;/span&gt;-ish code needed. Starfish handles the details, and you can just kill the clients when they are done. Starfish protects its client block from signals and closes after a client finishes a task.&lt;/p&gt;


    &lt;p&gt;The key to the speed of this system is that it parallelizes database access. Most database setups have very high read speeds, so running the repair processes in parallel isn&amp;#8217;t a significant burden on the system. Starfish is most useful in situations where you need to go record by record through a table and perform some specific–possibly slow–task.&lt;/p&gt;


    &lt;p&gt;This is a very simple starfish task that I wrote in 20 minutes to handle this cleanup process. We use more sophsticated distributed systems for other aspects of our site, and starfish is scaling to work with them all. The most recent &lt;a href="http://rufy.com/starfish"&gt;darcs repo&lt;/a&gt; even has work to extend starfish to support MapReduce, of google fame.&lt;/p&gt;</content>
  </entry>
</feed>

