Vermonster's Blog

Survey says: require 'work' && require 'training'

2010-05-19T00:00:00-04:00

If you are from the Boston -area and are interested in one, some or all of the following:

    [ ] Professional Development
    [ ] Looking for Job Opportunities 
    [ ] Looking for Ruby Talent 
    [ ] Ruby Culture

Take about 5 minutes to fill out the real survey. Do it; it’s good for your karma.

NoSQL Workshop - Featuring Riak and CouchDB

2010-03-31T00:00:00-04:00

This past Tuesday we (Vermonster and Greg) hosted the fourth installment of Newer Yankee Workshop. This one was on the über-trendy topic of NoSQL data stores. We covered two - Basho’s Riak and Apache’s CouchDB. After this past NoSQL Live event, I personally became very exited about Riak, a relatively new player in NoSQL. CouchDB is a nice RESTful document data and has been around longer than Riak.

Rain, Rain, go Away!

The weather was crap, another day of solid rain. Although we filled our capacity on EventBrite, I wasn’t actually expecting many to trudge through a soggy walk to our office. But I was pleasantly surprised with a solid showing – we had about 17 people in attendance.

The Setup

Using a crude slide show as a guide, we had a super brief overview of NoSQL, Brewer’s CAP theorem, and an introduction to our two data stores. Most everyone in the crowd was armed with OS X, outside of a few running Ubuntu, so installing and starting Riak and CouchDB (for development) was a snap. I recommended people use the Riak binaries (although I think some built from source), and Greg instructed folks to use a simple one-click-installer for CouchDB. We got our RVM on, built a gemset for the workshop and loaded a few gems.

Modeling a Simple Business Requirement

After everyone was up and running with two shiny NoSQL databases, we presented a simple business problem. The context is a project we are working on here at Vermonster called OpenDocket. It is still very early on, and respecting brevity, I won’t go into too many details in this post. But this did introduce two concepts we modeled - 1) Docket Items and 2) Motions. A docket item could have several motions. And a motion belongs to a docket item. A motion also has a number of yea’s and nay’s (such that more yea’s then nay’s means the motion has passed).

We modeled things in both data stores; using Ripple as an ORM for Riak and Couch Potato for CouchDB. You can see the two versions of the code at github (see the riak and couch branches). Everyone cloned the repo, ran spec ./spec in each branch and saw green. Excellent!

Next we walked through the code describing some of the features of each database.

MapReduce

Both CouchDB and Riak feature MapReduce functionality. As the data is distributed in these kinds of data stores, collecting and searching (i.e. mapping) also needs to be distributed. Reduction is a phase after map and aggregates the results. Someone made a nice analogy (and I apologize for forgetting your name) with ruby, and Harold suggested a once for SQL:

For those too young to have missed (or too old to to remember) analogies on their SATs – these read “map is to reduce” as “enumerable is to inject in ruby” and “map is to reduce” as “where clause is to aggregate functions in SQL”. Examples of aggregate functions in SQL are things like COUNT and SUM.

Implementing MapReduce functions in both data stores can be written in Javascript. In fact, it is probably worth mentioning that both use a nice RESTful interface with JSON encoded data to do their business. So using Javascript is a natural choice. Also related, Riak can process MapReduce functions in Erlang.

Riak MapReduce

We used MapReduce to demonstrate the has many relationship between Docket Item and Motion. In Riak, using the Ripple ORM, this is expressed as:

ruby --

  motions = Riak::MapReduce.new(Ripple.client).
    add("motions").
    map(motions_map_function, :keep => true).run

To take a step back, Riak’s MapReduce specification comprises of two components, an input and query. The add method allows you to add bucket and tags as the input. The map method adds to the query portion (others include link and reduce).

It’s probably worth mentioning some naming conventions. In Riak there are buckets that contain objects, identified by keys. This is similar to other key-store databases, including Amazon S3 (which you may already be familiar with and using).

The motions_map_function is this method (which is essentially Javascript):

ruby --

  def motions_map_function
    <<-JAVASCRIPT
      function(v){
        var results = [];
        var motion = JSON.parse(v.values[0].data); motion["key"] = v.key;
        if(motion.docket_item_key == '#{self.key}') results.push(motion);
        return results;
      }
      JAVASCRIPT
  end

The map function in Riak can actually be expanded a bit (something we should have spent more time on in the workshop). There are three inputs to the function:

javascript --

  function(item, keydata, arg)

Where item is the value found at the current key. It has 4 attributes: bucket, key, vclock, and values. NOTE: values is an array because there could be siblings, like during a vclock conflict.
The second argument, keydata is the data passed in the input portion of the call.
And arg are any :arg => ... values passed in the actual map method.

Riak reduce (not shown for this has_many example) takes in two arguments:

javascript --

  function(valueList, arg)

valueList is the output of the map phase
arg is what’s passed into the reduce method

Both have an optional :keep parameter that tells Riak whether to save the results of the phase or not. For example, if you had both a map and reduce function, you would likely not keep the map results, but would keep the reduce.

Additional note, Ripple uses an attribute _type to track the klass of the object, so it knows what to instantiate when processing retrieved documents.

CouchDB MapReduce

With CouchDB and Couch Potato as our ORM, any time you would like to view results, you first create a view spec. View specs contain map and reduce methods. Couch Potato provides some syntactic sugar that automatically creates MapReduce methods for common views.

For instance, we have a view called by_id:

ruby --

  view :by_id,    :key => :_id

You might notice the key is _id, this is a convention in CouchDB as the identifier for the document. As Greg pointed out, this is kinda nice, where in ActiveRecord, there’s lots of effort making docket_item.id marry up to the primary id field in the database, all without colliding with Ruby’s built-in object#id. Additionally, Couch Potato will generate a MapReduce function with this key syntax. In this case, in CouchDB the map looks like this:

javascript --

  function(doc) {
    if(doc.ruby_class && doc.ruby_class == 'DocketItem') {
      emit(doc['_id'], 1);
    }
  }

And the reduce looks like this:

javascript --

  function(key, values) {
    return sum(values);
  }

You will notice in Ripple, _type was used to identify the ruby class type, Couch Potato uses the key ruby_class. To retrieve data, we pass the view spec as an argument to a #view method:

And we have docket items!

The docket_item.motions has-many relationship to motions was demonstrated by creating a view in the Motion class.

ruby --

  view :by_docket_item_id, :key => :docket_item_id

The auto-generated map function looks like this:

javascript --

  function(doc) {
    if(doc.ruby_class && doc.ruby_class == 'Motion') {
      emit(doc['docket_item_id'], 1);
    }
  }

(The reduce looks the same as above)

So then in the DocketItem, we have a method that calls this passing in the docket_item’s id as a key:

ruby --

  def motions
    if self._id
      view_spec = Motion.by_docket_item_id( :key => self._id )
      CouchPotato.database.view(view_spec)
    else
      []
    end
  end

Link Walking (Riak)

One Riak feature I am very excited about is link walking. This feature lets you create links from one object to another, while tagging. It is similiar to having named edges in graph databases.

Consider the relationship Docket Item has many Motions. Previously we showed this with a MapReduce method. But this could have just been implemented using links. This technique was shown by first creating links in an after_save in motion.rb.

ruby --

  after_save :create_links
  
  def create_links
    if self.key && self.docket_item_key
      self.robject.links << Riak::Link.new("/riak/docket_items/#{self.docket_item_key}", "docket_item")
      self.robject.store
 
      d = docket_item
      d.send(:robject).links << Riak::Link.new("/riak/motions/#{self.key}", "motions")
      d.send(:robject).store
    end
  end

One curve here is robject. This is the Riak object (in Ripple, there are two namespaces, Riak and Ripple, the former is a lower-level interface to the Riak server). This allows us access to the links getter/setter. Also worth noting is the robject saves with a store method.

So let’s examine what’s going on here. Basically, we create two links after each motion is saved.

And this sort of thing would happen for any number of motions for Docket Item 1.

Link walking is preformed by starting at a document object, then indicating which bucket to enter. Riak will follow any link that connects the object to any object in the specified bucket. To restrict which links to follow, a tag parameter can be supplied when walking. This basically means “follow any link to this bucket that are tagged that”. As Riak uses REST, you can actually test this in your web browser:

This would follow any links from docket item with the specified key to the motions bucket with any tag (the first _, the last _ is a keep flag). Using Ripple, we can preform a link-walk with a simple walk as seen in the motions_via_link method:

ruby --

  def motions_via_links
    motions = robject.walk(:bucket => "motions").flatten
 
    motions.map{|m| instantiate(m) }
  end

Where :bucket and :tag are parameters.

To build on this, suppose we created a link when a motion passes, with a passed_motions tag (and could also create one for failed_motions).

That would allow us to do things like

This allows for very interesting slicing and dicing of how data is related without writing methods in the application layer. AND link walking can be part of a MapReduce method, way cool indeed! Here’s an example getting the all the passing motion titles.

ruby --

  Riak::MapReduce.new(client).
    add("motions").
    link(:bucket => "motions", :tag => "passed_motions", :keep => false).
    map("function(v){ return [JSON.parse(v.values[0].data).title]; }", :keep => true).run

The Challenge

After all this, we issued a challenge, “Find the difference between yeas and nays”. Some used CouchDB and others used Riak and everyone had their hands on creating actual MapReduce functions.

Outcome

Victor was quick at the draw and came up with the following Riak solution:

ruby --

  def self.delta_yeas_nays
    motions = Riak::MapReduce.new(Ripple.client).
      add("motions").
      map(delta_map_function, :keep => false).
      reduce(delta_reduce_function, :keep => true).
      run
  end

  private

  # A map function to get the motions given a docket item key
  def self.delta_map_function
    <<-JAVASCRIPT
    function(v){
        var results = [];
        var motion = JSON.parse(v.values[0].data);
        results.push((motion.yeas || 0) - (motion.nays || 0));
        return results;
    }
    JAVASCRIPT
  end

  def self.delta_reduce_function
    <<-JAVASCRIPT
    function(results){
        var sum = 0;
        for(var i = 0; i < results.length; i++) { sum += results[i]; }
        return sum;
    }
    JAVASCRIPT
  end

And the following CouchDB solution:

ruby --

  view :delta_yeas_nays,
       :map            => "function(doc){ emit(doc._id, (doc.yeas || 0) - (doc.nays || 0)); }",
       :reduce         => "function(keys, values){ return sum(values) }",
       :type           => :raw,
       :include_docs   => false,
       :results_filter => lambda {|results| results['rows'].first['value'] }

Take-aways

As some attendees dove into TDD mode, it was quickly obvious that without database transactions, it is difficult to test without manually clearing the database. I think that a simple extension to database_cleaner to support Riak would go a long way.

Also, working on TDD for the reduce functions themselves using jspec would also be terrific.

This was a really fun workshop to put on, it really pressed our learning of NoSQL in preparation. And I’d like to thank everyone that made it out through the sogginess.

One thing not covered was embedded documents. Partially because we ran out of time in our preparation and partially because we ran out of time in the workshop.

Build-a-Gem Workshop Recap

2010-02-26T00:00:00-05:00

We had another Newer Yankee Workshop on Tuesday, following the success of our previous Cucumber session in our new office.

We started by reviewing gem basics – what it is, how it’s made, how it’d distributed, etc. Then we jumped into a simple hello-world project using the venerable Jeweler. Greg Sterndale then took the driver’s seat and walked us through “gemifying” an existing project (which happens to be a pretty useful library to talk to Nextbus, the provider of future MBTA bus data, among many other agencies).

We then paired up and started hacking our own gems. There were are few that used the Massachusetts RMV wait time feed (like Bouvier, by yours truly and Jay). I saw one team playing with Last.fm’s API and one that even ran python code (for some reason).

It was a good session and I’m looking a next one on a topic TBD.. Maybe using a NoSQL database, Rails 3, Rack-ifying a feature, or something else?

UPDATE: simple-twilio is another one by Pascal and Jeff

We Moved to 71 Broad St in Boston

2010-01-27T00:00:00-05:00

After 8ish years on Stuart St, we moved our office. We are still unpacking, setting things up and starting to settle in.

Our new office is an open floor space in what I believe is a building that once was used as a candle factory some 150+ years ago (yea, it’s that old). We will be posting pictures soon. In the meantime, update your address books:

Vermonster LLC
71 Broad St, 4th Floor
Boston, MA  02109

We have a new phone number too:

617-482-2020

Recommended Omnigraffle Guide for Rails ORM

2009-12-29T00:00:00-05:00

We recently developed a work-in-progress style guide for using Omnigraffle to describe Object Relationship Mappings (ORMs) in our Rails applications. We wanted to use simple, standard stencils while being able to exhaustively describe relationships. We also wanted to have them be fairly obvious to developers.

Has One / Belongs To

One of the more common relationships is belongs_to. We decided to show this as a line on the connector toward the belonging model. The sometimes-ambiguous sibling, has_one is shown as a block toward the belonging model. In both cases, we use the unfilled circle/square to show optional.

Has Many

The has_many relationship is shown as “crows feet” stretching toward the child model. We use the unfilled circle to show optional relationship. The fairly common combination of a belongs_to and has_many is shown as as a combination of the crows feet and line connector.

Through, HABTM and Polymorphic

The through relationship, or one that uses a joining table, is show shown as a block using the double -greater and -less than symbols <<through>>. Then we use an arrow to point at the joining model.

The has and belongs to many relationship is shows as double crows-feet with a perpendicular line on each end.

The polymorphic relationship is shown as a rectangle, like a model class, but the name of the relationship is shown in <brackets>. This lets us know the object is not an actual model.

And finally, a combination of all the relationships discussed here.

UPDATE: Jan 8, 2010 -- Changed the way through relationships are described. Swapped the belongs_to and has_many symbols to make room for the has_and_belongs_to_many (habtm) relationship.

Cucumber Workshop Recap

2009-12-16T00:00:00-05:00

Thanks to Greg Sterndale for running the show and Plectix for footing the food bill, we had a fantastic hands-on workshop last evening on Cucumber. The turnout was much better than expected – I think nearly 20 people attended!

Greg did a great job taking us through the Cucumber basics – walking through an example demonstrating pending and unimplemented features. Here are some take-aways and interesting conversation points.

Gem Bundler

We attempted to use modern tools and techniques for this project. The workshop project, called Code Stache, uses Rails 2.3.5 and the soon-to-be-de-facto gem bundler for gem specifying and inventorying (um, but it doesn’t work in ruby1.9). Gem bundler is fairly new, so it was not surprising that there was some discussion on this topic. The folks who use gem bundler seem to agree it is awesome but there are some missing features, including more usage instructions and visibility to what is actually going on during the bundle process.

After rake bundle, we were good to go. One important aspect (and feature) of gem bundler is any “binaries” that gems create (like cucumber or rspec) will end up in RAILS_ROOT/bin. These should be used instead of the system gems to ensure the correct version is being executed. So from RAILS_ROOT, instead of issuing:

cucumber

You should run:

bin/cucumber

Machinist vs. Factory Girl (vs. Fixtures)

Fairly quickly our of the gate, we had a discussion on using object factories instead of fixtures. Examples of object factories include factory_girl and Machinist - both work really well with Cucumber. In our example, we used factory_girl.

Some of the Machinist fans found the integration of Sham and Faker for completely random data, along with simple association declarations makes it a solid choice. It was also brought up that factory_girl could use the Machinist API (at least I think this was mentioned).

But without getting into a religious war, using an object factory pattern is clearly a win.

Features - Consumable by Non-Programmers?

The discussion of what the actual value of human-readable features is and if it’s worth the effort go through this DSL. Some developers out there believe it is not worth the effort, but the consensus at the workshop was cucumber human-readable features are worth it. It also seemed that developers are primarily responsible for authoring feature files; then QA or stakeholders review and provide acceptance.

Using IVARs

Along with non-programmer interaction with feature files, the use of instance variables is a common topic when discussing cucumber. Using instance vars does make individual steps less modular – you can’t cherry-pick a step if it relies on something being set, for one. But setting up a state, like an actor or current user, can simplify features with intentional and calculated deviation from pure “no-ivar” philosophy.

In the end, it seemed that being pragmatic and trying to minimize the use of instance variables was the consensus.

Pickle - Making Life less “Cucumbersum”

Pickle is a nice way to interact with your object factories in cucumber. Steve created a pickle branch in codestache where the manual factory step was replaced with Pickle steps.

features/share_code.feature
@@ -17,6 +17,6 @@ Scenario: Create a code snippet
     And I should be on the "sudo rm -rf /" snippet page

 Scenario: View a code snippet
-  Given a snippet exists with a body of "sudo rm -rf /"
+  Given a snippet exists with body: "sudo rm -rf /"

features/step_definitions/share_code_steps.rb
@@ -1,3 +0,0 @@
-Given /^a snippet exists with an? (.*) of "([^"]*)"$/ do |attr, value|
-  Factory(:snippet, attr.gsub(' ', '_') => value)
-end

Later today in IRC (#boston.rb), Dan Pickett pointed out that factory_girl actually has some undocumented pickle-like step definitions.

XPath Selectors (instead of CSS Selectors)

In the original codestache title branch, we had a feature like this to help us specify there ought to be two fields, body and title, using some webrat CSS selectors:

Scenario: Visit the home page
	Given I am on the home page
	 Then I should see "New snippet"	
	  And I should see "Title" within "label[for=snippet_title]"
	  And I should see "Body" within "label[for=snippet_body]"

Arguably, this exposes internal view implementation and not business requirements. Fortunately, ajvargo was in attendance and provided this step definition that uses XPath to marry up semantic form labels to their respective fields. It is still work-in-progress, and won’t work with the non-for, <label>Label<input id="foo" name="foo"/></label> style, but removes the CSS selector from the feature (and I believe he also mentioned this works well with Selenium).

    # Uses xpath to check for a form element based on the label,
    # instead of using a css selector.
    Then /^I should see a "([^\"]*)" for "([^\"]*)"$/ do |type, name|

      case type
        when 'Button'
          response.should have_xpath("//input[@type='submit' and @value='#{name}']")
        when 'Text Field'
          response.should have_xpath("//input[@type='text' and @id=..//label[normalize-space(text())=\"#{name}\"]//@for]")
        when 'Text Area'
          response.should have_xpath("//textarea[@id=..//label[normalize-space(text())=\"#{name}\"]//@for]")
        when 'Radio Button'
          pending
        when 'Select Field'
          response.should have_xpath("//select[@id=..//label[normalize-space(text())=\"#{name}\"]//@for]")
        else pending
      end

    end

So with this step definition, our feature could now like this:

Scenario: Visit the home page
  Given I am on the home page
   Then I should see "New snippet"	
    And I should see a "Text Field" for "Title"
    And I should see a "Text Area" for "Body"

Feedback

One area we didn’t get to much into was the reuse of scenarios in step definitions. For instance, some login feature could be cuke’d with ample of detail, then consolidated and referenced as a ‘When I successfully login’ step for subsequent new features.

Another item we missed talking about was cucumber tagging. As cucumber test suites can grow in size and run-time, using tagging can be a way to isolate tests locally during development (continuous integration should continue to run the entire suite). Also, we didn’t really cover the special @wip tag, which marks features as works-in-progress.

Other feedback included better instructions getting people setup before-hand. Our internet pipe was laboring as everyone was cloning the project at the same time.

The balance of roughly 50% discussion and 50% hands-on coding, seems to resonate well.

All-in-all, it was a very successful maiden “Newer Yankee Workshop” – we are looking forward (and looking for topic suggestions) for future ones!

Mysql::Error: Lock wait timeout exceeded; try restarting transaction

2009-12-11T00:00:00-05:00

This post pertains to MySQL with InnoDB.

With heavy concurrency it is common to see the “Lock wait timeout” error on some database transactions. This is a temporary error that can often be made to go away just by trying the transaction again. Here is a little helper method you can put in a Rails initializer, for instance, to give your model a concise way to retry a few times.

class ActiveRecord::Base

  def self.retry_mysql_error(tries=3, &b)
    count = 0
    begin
      count += 1
      return b.call
    rescue Mysql::Error
      if count < tries
        sleep(0.1 * count)
        retry
      end
      raise
    end
  end

end

Just pass it a block (and an optional number of tries) and it should help to reduce the number of timeouts you see. Be careful not to carry over inadvertently any assumptions across retries.

class Download < ActiveRecord::Base

  def Download.increment_download_count!(filename)
    Download.retry_mysql_error(5) do
      record = Download.find_by_filename(filename)
      record.download_count += 1
      record.save!
    end
  end

end

It also plays nice with RSpec.

describe Download do

  describe "increment_download_count!" do

    it "retries mysql error five times" do
      Download.should_receive(:find_by_filename).exactly(5).times.and_raise(Mysql::Error)
      lambda { Download.increment_download_count!(true) }.should raise_error(Mysql::Error)
    end

  end

end

Of course, this technique should be balanced against other considerations. If it is unable to complete successfully after all the retries, it will re-raise the Mysql::Error. If this happens often, you might need to make deeper changes to your application.

Fun with Tomatoi.st; the Pomodoro Technique

2009-11-11T00:00:00-05:00

I did a lightening talk at this past Boston.rb meeting about the Pomodoro Technique, including Stephen Caudill’s recently launched Tomatoi.st.

The Pomodoro Technique (PT) uses behavioral rules to help focus time and increase overall productivity. The rules attempt to build a nice mental rhythm while adding a feeling of time-importance to you and your workmates. But the main goal is to alleviating anxiety due to the perception of having not enough time via a bit of a mind hack – changing the way we use time.

Francesco Cirillo, creator of the technique, recommends that a modest measure of time, like 20-40 minutes, followed by a shorter break, like 5 minutes for a few cycles, then a longer break. He apparently did research to come to this time pattern.

The quickstart is:

Set a timer to 25 minutes
Take a 5 minute break
Repeat for 4 cycles
Take a 25 minute break

Tomatoi.st is a simple web-based timer to help you out. It is ideal as a Fluid application if you are using OSX (it even integrates with growl). Here is an icon you could use.

Full Version

This is a bit over-simplified, as the PT also involves a planning and progress worksheet, along with estimation techniques – I strongly recommend you read the PDF book. From my limited use and reading on the subject, the technique is something you can pick up fairly quickly, but to master and benefit from it, plan to spend at least a few months.

Amazon ELB - Root Domain Oversight!

2009-11-05T00:00:00-05:00

Amazon Elastic Load Balancer, or ELB, is a service from Amazon’s webservice group that allows you to setup a load balancer in the cloud without having to run a dedicated instance (or instances if you want failover) – which saves money and maintenance (it costs under $20 per month, plus bandwidth to use ELB, the smallest instance is close to $70).

So we wanted to us ELB and set it up in a staging environment, where it worked like a charm. Our staging environment had the subdomain prefix staging. But when we switched to production, we ran into a bit of a snafu.

From the ELB documentation:

You will still need a CNAME, or equivalent, in order to map 
the *generated* DNS name to a name that is meaningful to you 
and your customers (e.g. www.mywebsite.com).

So the DNS name for the loadbalancer ought to be used when mapping the actual domain of your application. This is likely because it is fairly straightforward to failover using a DNS name (verses an IP address). But the oversight is for root domains.

Root domains require an A record to point at an IP address. So if your application is intended to run from a root doamin (i.e http://myapp.com), using ELB out of the box is not straightforward. For now, we ended up making the application run on a www subdomain and used a designated instance with an elastic IP to forward any the root traffic to it.

There are a few other workarounds, including setting up DNS / named servers to use name-based failover or round-robin name resolution - but none seem that clean. This problem is apparently fairly known, and Amazon has indicated they are working on a solution, hopefully soon.

So I guess the moral of the story is if you want your application to run on a root domain, you may want to reconsider ELB, at least for the time being.

Vermonster.com, now with Jekyll

2009-10-24T00:00:00-04:00

It’s been a few years, so we decided to redesign our site again. Our original site was running Wordpress, which is a fine blogging/CMS package. As we’ve been simplifying our infrastructure, we decided to try Jekyll this time, a static site generator. We did this for a few reasons:

It is super simple, static site is simple HTML - any ole’ web server will do
The page sources are in markdown
Everything from templates to post content fits nicely into git, so we have fantastic version control
It’s written in Ruby

A few things did make us think:

Comments and contact form need 3rd party (WuFoo and Disqus respectively)
Embedded comments and contact form needs JS (but we do have degraded links)

But the yeas won, so we decided to go with it.

We did implement a few tricks, via a nice Rakefile. First, we are using rsync to publish the site (which is common for Jekyll -driven sites).

desc "Build then rsync the site to the public server"
task :publish do
  sh %{ rm -rf _site }
  sh %{ jekyll }
  sh %{ rsync -av --delete -e "ssh -i ./.ssh/dsa_key_file" ./_site/ user@server:/var/path/to/webroot }
end

If you use this code above, your key and destination will vary when rsync’ing.

We also have a task that uses a template to create a new post, prompting for a title and filling in the author.

def dasherize(title)
  title.gsub(/\s/, '-').gsub(/[\|\?\+\#\!\,\.\'\"\:\;\&]/,'')
end

desc "Create a new post using today's date, filled with instructions"
task :post do
  filename = File.dirname(__FILE__) + "/_posts/" + \
    Time.now.strftime("%Y-%m-%d")

  title = ENV['TITLE']
  if title.nil?
    print "\n>> What's the title? (skip by set env. var TITLE) : "
    title = STDIN.gets.strip
  end

  filename << "-#{dasherize(title)}".downcase + ".md"

  author = `git config --global user.name`.strip
  if author.nil?
    print "\nQUESTION: What's your name? "
    author = STDIN.gets.strip
  end

  content = File.read(File.dirname(__FILE__) + "/_posts/template")
  content.gsub!(/.*author:.*/, "author: \"#{author}\"")
  content.gsub!(/.*title:.*/, "title: \"#{title}\"")
  
  File.open(filename, 'w') {|f| f.write(content) }
  
  puts "\nMade new post by #{author}:"
  puts "  #{filename.gsub( File.dirname(__FILE__) + "/",'./')}"
  puts "\nCommit changes to git. Publish with 'rake publish'"
end

## ./_posts/template
__END__
---
layout: post
published: false
title:
#author: 
#tags:
#- "ruby"
---

Last, we implemented a fun changing theme, where the background images switch from day to night. We did thing with a simple crontab entry on the server that switches around a supplemental CSS style sheet.

All in all, we are very happy with the new site and hopeful the simplicity of Jekyll will promote more posts from our team (because we have lots to talk about!). We haven’t converted all our existing blog entries over yet, but plan to do so over the next week or so.

Shopify_API 0.0.1 Released

2009-05-29T00:00:00-04:00

You can see Shopify_API for a small library for driving the Shopify.com API. It is adapted from the official Shopify Rails plugin, but decoupled from the controller for use outside of web requests. It depends on ActiveResource, enjoy!

svn:mergeinfo - Addressing a Painfully Missing Subversion Feature

2009-02-12T00:00:00-05:00

I’ll preface this by saying article is about subversion and not git. Since release 1.5, subversion now includes a property, svn:mergeinfo, that is auto-applied when preforming merges. Before this, a merge was pretty much an ad-hoc diff between two files or lines of code. The best practice was to include the merge command used in the SVN comment, which was extremely unreliable. For clients and projects not yet on the git-train, svn:mergeinfo is a welcome and painfully overlooked feature.

We, like many other teams, typically follow the TRUNK = “bleeding edge”, branch-releases pattern to SCM. So at the end of each iteration, we cut a release branch from TRUNK. This release is deployed and we turn our efforts toward the next sprint. However, is is atypical to not have to patch the release. This is usually done in the release branch, tested on a staging environment, the ultimately deployed to production. Afterward, the patch is merged back to the mainline development trunk. This pattern works nice, but sometimes we do a series of patches in the release and there is definitely the opportunity to miss a commit when merging. Since subversion previously didn’t have a way to see merges (merges were not directly tracked), the process involved a session of reviewing commit logs and manual comparison. Enter git, which is built to merge, and the mass movement to git is inevitable.

However, with subversion 1.5, a new property is introduced svn:merginfo. This feature add merge tracking to svn. So when committing the results of a merge, this piece of metadata is recorded. This opens up many new features, like looking at the svn:merginfo property (you shouldn’t edit this property, only get it).
And one of my favorite features is the --reintegrate option. So we can now tell, with a single command, what merges in the release haven’t made their way back to trunk — no more missing merges!

There are a few points, echoed here (parting thoughts section at bottom), worth mentioning:

The subversion server and all developer clients ought to be at least running svn 1.5+
Do all the merges from a standard location, like the project root (RAILS_ROOT for RoR)
Inform developers of this property. After doing a merge, the svn:mergeinfo is put on the place the merge was preformed (preferably project root). So developers should know this and make sure to commit this property change.

Minimizing Software Entropy in the Agile Environment

2009-02-06T00:00:00-05:00

As part of the agile process, each development sprint has a planning prerequisite. During this planning process, user stories are scored using a consensus complexity and risk estimations. It is common that some stories are deemed “to risky” and either scaled-back, re-evaluated or hit the chopping block. Minimizing risk and defects is something desirable to all - management, developers and stakeholders. This mix of risk and defects is sometimes referred to as “software entropy”. From my thermodynamics class, I vaguely remember that entropy has something to do randomness, or was it heat - or is that enthalpy?

After a quick google + wiki session, I found that entropy is the amount of disorder in a system. And there is a principal that says in a closed system, one can’t decrease the amount of entropy, it either increases or could possible stay the same. So that doesn’t look good if we apply this to software. This basically says that each iteration the very best we can do is have the same unpredictability, but most likely our software will continuously get worse! Not good news for the software industry and not good news for agile practitioners.

However, we know software doesn’t always become more unpredictable, software can become more stable. So what’s the factor and how can being agile, which embraces change and uncertainty, be used to minimize software entropy?

The factor is actually refactor, as in software refactoring. So we follow the laws of thermodynamics in this house, our system is not a closed one (it all depends on how you draw your boundaries). Whenever we add or removed lines of code, we are doing so as an external influence. This can actually reduce software entropy. For instance, if we have a defect rate of say 3 defects per 1000 lines of code, and we remove 500 lines of code in an iteration, we’d have 1.5 less defects, right? Well that was pretty theoretical, but the concept does work in a very tangible, and common occurrence - bug fixing.

We consider any bug fix that turns out to be a software defect (i.e. one that is not misimplementation of some business requirement or feature) a refactoring effort. When a defect is addressed, the existing code is altered as we add or remove bits - and we are external influence. Fixing defects can have 3 possible outcomes: 1) the system is more stable, software entropy reduced (ah!, the fix works!), 2) the system is not any more or less stable, software entropy unchanged (the fix didn’t work, but at least you didn’t break anything), or 3) the system is less stable, software entropy increases (regression! whack a mole! uh-oh!). So if we are good bug fixers, we can reduce entropy.

Now for the agile part. As we work through our iterations, defects land on our plate. And properly fixing them does reduce entropy. But another way to more drastically reduce entropy is some good ole’, heads-down refactoring effort. Pure refactoring specifically targets minimization of software entropy. When we refactor, we are DRYing up methods, eliminating lines of code, cleaning and making our software more stable and manageable. We strongly recommend that each iteration ought to have some overhead earmarked for refactoring.

It is sometimes difficult to justify refactoring to stakeholders and non-developers, as the business value is difficult to measure. However, if a new feature is deemed “too risky”, a trade-off could be to add a pure refactoring effort to the iteration targeting the potential risk of the feature. Then build the feature.

So the moral is to minimize entropy and uncertainity in software, try to have refactoring a part of each development iteration.

Vermonster's Blog

Survey says: require 'work' && require 'training'

NoSQL Workshop - Featuring Riak and CouchDB

Rain, Rain, go Away!

The Setup

Modeling a Simple Business Requirement

MapReduce

Riak MapReduce

CouchDB MapReduce

Link Walking (Riak)

The Challenge

Outcome

Take-aways

Build-a-Gem Workshop Recap

We Moved to 71 Broad St in Boston

Recommended Omnigraffle Guide for Rails ORM

Has One / Belongs To

Has Many

Through, HABTM and Polymorphic

Links

Cucumber Workshop Recap

Gem Bundler

Machinist vs. Factory Girl (vs. Fixtures)

Features - Consumable by Non-Programmers?

Using IVARs

Pickle - Making Life less “Cucumbersum”

XPath Selectors (instead of CSS Selectors)

Feedback

Related Links

Mysql::Error: Lock wait timeout exceeded; try restarting transaction

Fun with Tomatoi.st; the Pomodoro Technique

Amazon ELB - Root Domain Oversight!

Vermonster.com, now with Jekyll

Shopify_API 0.0.1 Released

svn:mergeinfo - Addressing a Painfully Missing Subversion Feature

Minimizing Software Entropy in the Agile Environment