Empty Thoughts

Eric Holscher just posted a very nice article titled Automating tests in Django. The post goes through how to create integration tests for your Django applications in an automated way through the use of a Middleware that logs the test creation output to a file. It’s a creative approach and certainly very interesting. One additional benefit is that Eric created a screencast to go along with the post that is excellently done.

There is one thing about this approach to testing that doesn’t quite sit right with me and that’s that it seems like the testing process is backwards. If you’re creating tests based on what you have how are you possibly going to cover what’s specified but not implemented properly? It’s the same reason I’m not a fan of doctests. I think they encourage the wrong behavior, especially when often the output your matching to is so complex that the tendency is to just copy and paste from live results. I recognize that a lot of people don’t feel the same way, and perhaps I just need to give the idea more time to sink in.

I really appreciate all of the screencasts that are starting to show up within the Django community. I think it’s a vehicle that a lot of people enjoy and learn well from. I know that I’m certainly looking foward to more screencasts from Eric.

Here’s a syntax that I used quite frequently when doing unit tests in Ruby. A similar syntax in Python works quite well once you get past the death star syntax:

def create_category(self, **options):
    return Category.objects.create(**dict({'name': 'Python', 'description': 'Python rocks, mmmkay'}, **options))

By setting up the above method / function in your tests you can then use it as a default by just calling the function. But if you want to override a particular aspect of the create, for instance to set a required field to None, you can just pass that in easily with:

create_category(name=None):

It works quite well and makes it easy to provide defaults around which you modify to test certain aspects of your code.

I’ve been working on fleshing out the tests for Django-SQLAlchemy and right away I discovered a problem. Incidentally, this is why I love unit testing. The problems was that a very simple test of the query contains filter syntax was failing. I was expecting four items but was getting five back. After digging into it I discovered the problem was with SQLite. SQLite doesn’t seem to respect case when using the LIKE syntax.

Let us say we have the following setup:

from apps.blog.models import Category

class TestContains(object):
    def setup(self):
        Category.__table__.insert().execute({'name': 'Python'}, 
            {'name': 'PHP'}, {'name': 'Ruby'}, {'name': 'Smalltalk'}, 
            {'name': 'CSharp'}, {'name': 'Modula'}, {'name': 'Algol'},
            {'name': 'Forth'}, {'name': 'Pascal'})

a query with contains in Django on SQLite, like the following, will return five results, instead of the expected four results:

>>> Category.objects.filter(name__contains='a').count()
2008-03-28 20:41:43,228 INFO sqlalchemy.engine.base.Engine.0x..f0 BEGIN
2008-03-28 20:41:43,229 INFO sqlalchemy.engine.base.Engine.0x..f0 SELECT count(foo_category.id) AS count_1 
FROM foo_category 
WHERE foo_category.name LIKE ?
2008-03-28 20:41:43,229 INFO sqlalchemy.engine.base.Engine.0x..f0 ['%a%']
5

(Incidentally that’s actually going through the Django-SQLAlchemy backend and not Django’s ORM.)

So I kind of hemmed and hawed for a bit trying to figure out how I could make this work under different databases, at least from the testing standpoint. It finally occurred to me that SQLAlchemy has to be able to test their ORM against a lot of different backends so they must have a nice solution to this.

Decorators to the Rescue

It turns out that SQLAlchemy has implemented an elegant set of decorators for just this problem. They were also written in such a way that it was quite easy for me to extract them and modify them slightly to work with Django-SQLAlchemy tests. So what’s in this package?

• fails_if(callable_) – Mark a test as expected to fail if callable_ returns True.
• future – Mark a test as expected to unconditionally fail.
• fails_on(dbs) – Mark a test as expected to fail on one or more database implementations.
• fails_on_everything_except(dbs) – Mark a test as expected to fail on most database implementations.
• unsupported(dbs) – Mark a test as unsupported by one or more database implementations.
• exclude(db, op, spec) – Mark a test as unsupported by specific database server versions. This decorator allows an impressive list of options, for example @exclude('mydb', '<', (1,0))

There’s a lot more than that, but I will not detail them all here. If you want to dig through it all check out test/testlib/testing.py module.

The Implementation

So once I was able to extract and modify these decorators I ended up with very elegant syntax for my tests. Here is a sample:

class TestContains(object):
    def setup(self):
        Category.__table__.insert().execute({'name': 'Python'}, 
            {'name': 'PHP'}, {'name': 'Ruby'}, {'name': 'Smalltalk'}, 
            {'name': 'CSharp'}, {'name': 'Modula'}, {'name': 'Algol'},
            {'name': 'Forth'}, {'name': 'Pascal'})

    @fails_on('sqlite')
    def test_should_contain_string_in_name(self):
        assert 4 == Category.objects.filter(name__contains='a').count()
        assert 1 == Category.objects.filter(name__contains='A').count()

    @fails_on_everything_except('sqlite')
    def test_should_contain_string_in_name_on_sqlite(self):
        assert 5 == Category.objects.filter(name__contains='a').count()
        assert 5 == Category.objects.filter(name__contains='A').count()

    def test_should_contain_string_in_name_regardless_of_case(self):
        assert 5 == Category.objects.filter(name__icontains='a').count()
        assert 5 == Category.objects.filter(name__icontains='A').count()

Special thanks goes to Mike Bayer and the rest of the contributors to SQLAlchemy for providing such a great solution. I am constantly amazed by their code.

I was trying to spec out a few of my controllers that had actions on them requiring authentication. After jumping through many mind hoops to figure out how to stub them out properly I asked the RSpec-Users list and received this solution from Graeme Nelson:

def mock_user_authentication(allow_user_to_pass=true)       
  controller.stub!(:login_required).and_return(allow_user_to_pass)
end

It’s elegantly simple and works well. I’m still fumbling around with this RSpec stuff and did not realize I could stub out a method directly on the controller. This has cleared up a big missing piece in my thinking.

I just spent about an hour trying to figure out why none of my RESTful routes were working properly within my RSpec controller specs. In my controller I had some boilerplate code like this:

# POST /categories
# POST /categories.xml
def create
  @category = Category.new(params[:category])

  respond_to do |format|
    if @category.save
      flash[:notice] = 'Category was successfully created.'
      format.html { redirect_to category_url(@category) }
      format.xml  { head :created, :location => category_url(@category) }
    else
      format.html { render :action => "new" }
      format.xml  { render :xml => @category.errors.to_xml }
    end
  end
end

The line causing the problem was:

redirect_to category_url(@category)

I kept receiving an error on the eval of category_url with an error description of “can’t convert Fixnum into String”.

I tried replacing @category with @category.id to see if I would get different results. The error went away but the test failed indicating that the id returned from the @category instance was not the same as I was expecting. This led me to determine that I needed to stub out the id property on my class. So I added the following to my setup:

@category.stub!(:id).and_return(1)

Everything worked. Problem solved. But wait, that’s ugly and smells of something wrong. I should be able to just pass the object to the category_url and have it return the correct value. What I did next was go down a rat hole trying to figure out what the named route was sending to the object to get the id. I had assumed id, but in fact it’s to_param, which I had already stubbed out as follows:

@category = mock_model(Category, :to_param => 1)

So what’s the problem? It turns out that to_param must return a string. Makes sense. I changed it to the following and everything worked perfectly:

@category = mock_model(Category, :to_param => "1")

It’s little things like this that make learning so much fun. This issue is really indicative of a much bigger problem—my lack of understanding mocks and stubs. But, I’ll have more to write about this later.

I recently started using Topfunky’s Power Tools Plugin to clean up some of my testing routines. It’s a nice little package that brings together lots of different asserts that have been out there in the wild in one form or another.

I really enjoy the assert_required_fields method. I used to implement my model checking like so:

def test_should_require_login
  assert_no_difference User, :count do
    u = create_user(:login => nil)
    assert u.errors.on(:login)
  end
end

...and now with the Topfunky Power Tools Plugin I end up with the following:

def test_should_require_login_password
  assert_required_fields :create_user, :login, :password
end

Notice how it allows you to check multiple fields at once. There’s a lot more available in the plugin, but sometimes all it takes is one or two little things to make your day.