Sometimes in order to succeed, you have to give up

We went camping at Wallowa Lake, OR this summer.  What a beautiful place!  Of course, it took forever to get there from Bend, but it was well worth the trip.

As is typical when we go camping, Beth and I slept in our sleeping bag atop an air mattress.  This particular air mattress had been replaced several years ago because its predecessor started to leak right around the base of the circular baffles that join the top and the bottom of the air mattress together.

Well, apparently it was time for this air mattress to suffer the same structural defect.  I awoke one morning to find my hip touching the ground and realized I would need to fix this situation if I wanted a good night’s sleep going forward.

In the past, I had been able to super-inflate the mattress and use soapy water to spot the leak.  However, this leak was not cooperating — I would have to get more serious if I wanted to sleep well for the rest of the trip.  So, I took the air mattress to the lake, figuring that if I submerged the mattress, it would yield its leak(s) in the form of air bubbles.  After twenty minutes of pushing, prodding, and otherwise cajoling the mattress into giving up the secret location of the escaping air, I was defeated.  I knew it was there, but I couldn’t find the leak.

I was resigned to several more nights of subpar sleep.  I gave up.

What do you do when you are in a beautiful lake in the afternoon with an air mattress in front of you with the sun heading toward the horizon, much like your summer is sunsetting into autumn?   You jump on your air mattress and soak up some of summer’s final rays — that’s what you do!

It was a split-second after I landed on the air mattress that the telltale WHOOSH from the leak exposed its location.  I mapped the breach immediately by counting the x and y baffle “coordinates,” took the mattress back to camp, let it dry, and finally fixed the leak.

I slept just fine that night.

Occam versus House

When I’m troubleshooting a problem, the first tool I reach for is Occam’s Razor. 99% of the time, I find a simple solution to a problem, possibly write a unit test (assuming we’re not talking about a bad ground in my stupid old car), and move on.

Some days, though, I endure my own personal episode of House.

So it was, after trying to track down a particularly unsettling problem where my model observer — which tested out just fine over and over again — simply failed to fire in a staging environment AFTER it worked a few times first. After slashing at the problem with Occam’s Razor to the point where there was nothing left but very tiny bits of flesh (ew), I put on House’s cap (or cane, as it were).

What if the code was, literally, disappearing?

Well, it turned out that was exactly what was happening. For other reasons that remain unresolved, I was instantiating my observer class using ObserverClass.instance as opposed to adding the class to the config.active_record.observers in environment.rb. This, coupled with the unfortunate configuration mistake that kept the site running in “development” mode, and this reloading-all-of-Creation before every new request caused the observer simply to fall off into oblivion as if it had never existed.

And why did it work a couple of times before failing? nginx –> 3 mongrel servers. Succeeded once for each server before the observer went completely bye-bye.

Although the true solution lies in reworking my application-environment-specific configuration, the short term solution was simply to add the observer to the config.active_record.observers after I had instantiated it.

Finding :last

Rails makes it easy to find the first row of a query:

Order.find(:first)

But what if you want the last one? It’d be great to be able to go:

Order.find(:last)

… especially if you could pass in conditions, etc.

Well, this isn’t general purpose, but it tends to get you the last one that was created. Can be useful in testing circumstances:

last_order = Order.find(:first, :conditions => 'id = (select max(id) from orders)')

created_at and updated_at in fixtures

I’ve been working on some code that gets a list of “old” orders based on the created_at value for an order. Of course, I wrote a test using a fixture that gets the list of old orders and makes assertions about it. I was not as rigorous as I should have been and simply checked to make sure the correct count of orders was returned and went along my merry way.

Today the test broke when I made some other changes. During my investigation, I printed out the contents of the orders and discovered the created_at and updated_at values were being set to all zeros. OK, that would explain why they’re considered “old” if they’re from “the beginning of time.” But I still wanted to be able to put values in for created_at that would cause an order to be too new to be included in the query. So, I figured I’d do something like this:

created_at: <%= Time.now %>

No workie; still zeros. Well, thanks to this blog entry, I was able to get it right:

created_at: <%= Time.now.to_s(:db) %>

Rails REST testing using XML

One of the things that has been bugging me about my REST interfaces is that, although they are thoroughly tested in functional tests with all the GETs and POSTs and PUTs (and occasionally DELETEs), it just isn’t quite the same as literally POSTing the XML.

So, this morning I took the time to figure out a way to do this. It turns out that with an integration test, it’s quite easy. It’s also probably the Right Place™ to do this.

Witness:

class RestXmlTest < ActionController::IntegrationTest
  fixtures :model_fixture

  # Test creating a new resource by actually POSTing the XML.
  def test_create_resource
    post "/path_to_resource.xml",
      "<resource><attribute_1>attribute value</attribute_1>...</resource>",
      {:content_type => "application/xml"}

    assert_response 201
  end
end

Verifying a Verisign Certificate Using nginx and an Intermediate Certificate

I had an SSL certificate in place in an https server, but whenever I connected to the site, I would see a certificate error. I tried verifying the certificate and got the following error:

$ openssl verify cert.pemcert.pem: [details removed]
error 20 at 0 depth lookup:unable to get local issuer certificate

I stumbled across a reference to Verisign requiring an “intermediate certificate” in order to verify an issued certificate. Consequently, I downloaded the following intermediate certificate from Verisign, based on the type of certificate I was working with. Storing this certificate into a file I called “verisign_intermediate.crt,” I was now able to verify my certificate:

$ openssl verify -CAfile verisign_intermediate.crt cert.pemcert.pem: OK

The final hurdle was to get the https server (nginx) to play ball. Per nginx documentation here, I simply appended the intermediate certificate to my main certificate file, reloaded the nginx configuration, et voilà! No more certificate error.

Verisign Intermediate CA Installation Instructions

Constant crisis

In my Rails application, I have a file in

RAILS_ROOT/lib/pcm/web_services/constants.rb

On my development machine (OS X), I am able to reference items in this file/module using the prefix “PCM::WebServices::Constants” without having to require the file. Once deployed on Linux, however, this approach broke and I had to explicitly

require 'pcm/web_services/constants'

(or any other file I needed from the lib/pcm directory) for things to work. I suspected it had something to do with case sensitivity on Linux as opposed to not-so-case-sensitivity on OS X. I was wrong.

Upon further investigation, I find I can easily reproduce the problem, although I do not yet have an explanation.  The solution to the problems are (so far):

1) Explicitly require ‘pcm/web_services/constant’ wherever I need it.  No biggie, but I prefer the “less is more” philosophy where Rails automagically loads the constant without the explicit require.

More possible solutions coming soon…

—>8— cut here —>8—

1) Create a new Rails app, e. g. $ rails constant

2) Edit the environment.rb file to add your own module with
configuration constant default values with the idea you could
“override” these by specifying them in the specific environment file:

At the end of config/environment.rb, add:

module AppConstants
  IMPORTANT_VALUE = "x" unless defined?(AppConstants::IMPORTANT_VALUE)
end

3) Create a Ruby module in the file lib/app_constants/list.rb, noting
the “top level” name of the module matches the one specified in the
environment file:

module AppConstants
  module List
    LIST_CONSTANT = "list_constant"
  end
end

3) Fire up the console and probe the constant values:

Loading development environment.

>> AppConstants::List::LIST_CONSTANT
=> "list_constant"
>> AppConstants::IMPORTANT_VALUE
=> "x"

4) Quit from the console and add development-specific value for the
constant defined in the environment file:

At the end of config/environments/development.rb, add:

module AppConstants
  IMPORTANT_VALUE = "definitely_not_x"
end

5) Fire up the console again and probe the constant values:

Loading development environment.

>> AppConstants::List::LIST_CONSTANT
NameError: uninitialized constant
Rails::Initializer::AppConstants::List
from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/
active_support/dependencies.rb:263:in `load_missing_constant'
from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/
active_support/dependencies.rb:452:in `const_missing'
from (irb):1
>> AppConstants::IMPORTANT_VALUE
=> "definitely_not_x"
>> AppConstants::List::LIST_CONSTANT

NameError: uninitialized constant
Rails::Initializer::AppConstants::List
from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/
active_support/dependencies.rb:263:in `load_missing_constant'
from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/
active_support/dependencies.rb:452:in `const_missing'
from (irb):3

If it’s broke, fix it!

As I reach the end of the day at the end of the week, I run my tests one more time before checking them in, and … BANG! Failure!

Nuts. I wanted to check stuff in and call it a day. What do I do: check it in anyway, or take the time to fix it before checking it in?

Of course, if you’re even here, you know the answer: take the time to fix it. Here’s two reasons why:

1) If I leave it unfixed, it is more likely than not that something will prevent me from taking the time on Monday to fix it. Then Monday will pass by, and so will Tuesday… and by the end of the week a simple testing error becomes a burden to fix. Consequently, I no longer get the warm fuzzies and confidence of a clean test run at will. So, maybe I’ll just let the whole test suite decay since it’s obviously now broken. Not good — I’ve just given up the value of my entire test suite because I wouldn’t take the time to track down and fix an error in a test case that used to pass. And trust me — the longer you wait to fix something like this, the harder it becomes to get yourself psyched up to deal with it.

2) It took me so long to write #1 that I’ve forgotten #2. No matter — I did go ahead and fix the test case and I ended up with a better test anyway.

nginx + rails + PUT + curl = 411 Length Required

One of my RESTful resources supports a PUT in order to get the next item in the queue and dequeue it. It is a PUT as opposed to a GET because it changes the state of the queue, and we all know that GETs should be idempotent.

In order to test this, I was using the following sort of curl command:

$ curl -i -X PUT 'http://localhost:3000/resource_name.xml;dequeue'

Worked fine on localhost. However, when I setup this up in staging going through nginx and over https, I ran the same command and got an unexpected result:

411 Length Required
nginx/0.5.31

So, I knew it was nginx that was giving me grief as opposed to mongrel and/or Rails suddenly changing its mind. I found the following note on the subject and changed the command to include setting the Content-Length header to zero, resulting in the following successful form of the curl command:

$ curl -i -X PUT -H 'Content-Length: 0' 'https://staging/resource_name.xml;dequeue'

Do Not Forsake Me, Oh My localhost

I recently setup an nginx proxy to a mongrel cluster using only https. No big whoop, right? Although I could get a file from the public directory, whenever I tried to invoke something in Rails via the mongrel, it’d hang. The error in the log file was:

2007/09/21 18:17:52 [crit] 4104#0: *33 SSL_do_handshake() failed (SSL: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca) while reading client request line, client: 127.0.0.1, server: staging
2007/09/21 18:17:57 [info] 4104#0: *35 client 127.0.0.1 closed keepalive connection

Sure, I didn’t have a “proper” certificate for SSL, but what gives with the whole hanging thing? Why didn’t it just come screaming back with some professional handwringing(tm) and get on with things?

Well, somehow when this machine was configured it was done without a loopback interface. Consequently, when I tried something like “curl https://localhost/…” from the command line, it’d sit there for forever. So, I created the loopback interface by adding the following lines to /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

Restarted the machine, fired everything back up, and it works great — no more hanging trying to access a https://localhost/… URL. Go figure.