Empty Thoughts

Django-SQLAlchemy Update

It has been a couple of weeks since I announced the Django-SQLAlchemy project. I thought it would be a good idea to provide somewhat regular updates for those that are interested, but not following the project at a code level.

Summary

At the overall level we’ve seen tremendous progress on the project. In the last couple of week there were 77 commits affecting about 4500 lines of code. We implemented the testing framework into the project and as a result we have added about 45 tests. We are just getting started in the testing area and have a ways to go before we have adequately tested the existing code. But it is moving along nicely.

Additionally, we have a new regular committer with Samuel Hsiung coming on board and submitting patches to implement the inspectdb management command as well as all of the introspection stuff related to that. We really appreciate his help.

Details

As I mentioned in the previous post, we see the integration of Django and SQLAlchemy breaking down into three primary areas of development. I’ll outline how we’re progressing in each of these areas below:

1. Management Commands

As I mentioned above, Sam Hsiung implemented the inspectdb command. In addition to that we have implemented flush, reset, sql, sqlreset, and syncdb. Not all of these are what we would consider 100% complete, but they’re there and working for our current use case.

We are kind of stalled on the loaddata and dumpdata management commands until we implement extra. But there are lots of others that can be implemented. This is a great area for someone new to the project to get involved.

2. Declarative Mapper

This one is more difficult to gauge. Currently we have the declarative stuff working for our current test cases. Which includes straight models, Foreign Keys, and Many To Many. We’ve have not got into model inheritance nor have we really tested out the M2M relationships very extensively at this point.

All field mappings are done, but we continue to tweak them as we run into issues. There are currently very few tests related to the declarative area of things. This is an area that I plan to focus on over the next week.

Related fields are mapped although currently we’re letting Django handle the RelatedManager for us. Although this works it does so at the expense of some functionality that we’d like to provide down the road. We’ve modified the declarative mapper to support overriding this behavior but I’d like to provide a default implementation that takes advantage of SQLAlchemy’s attribute instrumentation to allow SQLAlchemy to manage this relationship but in a way that mirrors how Django manages it. Similar to how we’re managing QuerySet.

3. QuerySet Mapping

This area has been a lot of fun and also the area where we’ve provided the most tests so far. As I stated previously we are providing our own DjangoSQLAlchemyQuerySet in place of the default QuerySet. In doing so it’s necessary to map all of the methods that are available to Django. We’ve broken this into three smaller subsection of code:

a. QuerySet Methods – We have mapped a lot of the methods on the QuerySet. So instead of listing them all, I’ll list what is left. The QuerySet methods that still need some love are: dates, complex_filter, select_related, and extra (ugh).

Additionally there are a couple that need some work or further testing. order_by is implemented but we have not tested it for all forms of it. Remember order_by used to use one format and now uses a format similar to filters. We have a potential problem with delete regarding invalidating session information. I have not yet tested it so I can not say for sure if it is an issue.

We have added in an options method for passing extended options to SQLAlchemy. We have also added in a first method, just because it is convenient. Items like this are just in there for now. The end result of where Django stops and SQLAlchemy begins is to be determined.

b. Field Lookups – We have created a little mapper that handles the field lookups, and does the right thing in creating the join conditions needed. Currently we provide support for everything but range, year, month, day, search, regex, and iregex.

In addition to the above we have added two new constructs, just because we can, like and ilike. These two provide raw like ability for doing more complex LIKE matching in your field lookups.

Join conditions are working and tested for Foreign Key situations. They’ve yet to be tested for Many to Many relations.

c. Subclasses – This includes subclasses of QuerySet that are implemented within Django. Technically these probably do not need to be implemented as such, but the design makes sense on its own and it makes sense for what we are attempting to do.

So far we have implemented a DjangoSQLAlchemyValuesQuerySet and a DjangoSQLAlchemyValuesListQuerySet. Both of these function for the common cases although we have not tested them with Many to One values at this point.

We still need to implement the DateQuerySet and the EmptyQuerySet. So there’s some low hanging fruit there.

Wrap Up

Up to now a lot of the mapping that we’ve been doing has been just a lot of grunt work. Turning the crank and making sure things work as expected. We needed some early wins to know that we were heading in the right direction and we have that now. The next step is really hitting more of the edge cases, the more difficult areas of implementation. Before we get into that though I want to be sure we have good overall test coverage for what is implemented up to this point. That will be my focus over the next week or two.

It’s amazing to me how similar many of the approaches are between Django’s ORM and SQLAlchemy’s ORM, for the subset of functionality provided by Django. That makes the job a lot easier. Although there are also interesting differences.

For instance, Django’s order_by is not generative (they can not be chained), whereas with SQLAlchemy they are generative. Django’s get method respects current filters, whereas SQLAlchemy’s get method ignores any pre-existing filtering. (Note this is what I’m experiencing so if I got it wrong I’m sure I’ll be set straight.)

I must admit that the more I learn about SQLAlchemy the more I’m impressed with the flexibility and power of the framework. It has some amazing code, and the potential for Django that it provides through this project is exciting.

Once again I would like to thank Mike Bayer and Jason Kirtland for providing a ton of assistance. I know that I’ve been annoying at times, but they’ve been great in nudging us in the right direction when we lose our way.

---

Discussion

---