Tesla 0.2 changes

A few things have happened in the Tesla project over the last week:

  • Tesla is now on Cheeseshop ! This means you can install Tesla by doing easy_install (-U) Tesla.
  • Migrations. Yes, Tesla can do Rails-style migrations, thanks to the migrate library for SQLAlchemy. Read more about it here. Migrate commands can now be called from paster and a lot of manual steps are taken care of.

Plus, a bunch of unit tests and minor fixes.

Overall, I’ve tried to keep Tesla as lightweight as possible – adding some Elixir/SQLAlchemy bindings to a basic Pylons framework. The big news in Python this week has been that TurboGears 2.0 is going to be rewritten on top of Pylons, which is a great step forward for both communities and Python in general.

However, TG 2.0 will appeal to those who like the TurboGears way of doing things, while many “core” Pylons developers may prefer a more lightweight framework that stays out of their way but takes care of some lower-level details. That’s pretty much what I’m trying to achieve here. The challenge is to keep Tesla from bloating – so for example I won’t be choosing a particular template engine, widget library etc. That’s up to you.

Some things that are in the pipeline:

  • Integrating Mike Orr’s SAContext, which should provide more configuration options.
  • An AuthKit template based on Tesla, with some default identity classes and helpers.
  • The amazing thing about writing Tesla was how little work was involved – the heavy lifting is done by Paste and Pylons. You just write your glue code and templates and voila – a new framework.

    Now, when will we see Django running on Pylons ?

Advertisements

Tesla, a Pylons/Elixir framework

David Bowie as Nicola Tesla in the Prestige

Tesla is a framework built on top of Pylons and Elixir/SQLAlchemy. It’s being actively developed by myself and Ben Bangert (of Pylons fame). Like Pylons, Tesla is a Paste template, so you just download it, install it and run paster create -t pylons_elixir myapp to get started. A tutorial can be found here.

Pylons and Elixir/SQLAlchemy give you great power and flexibility; however, getting SQLAlchemy and Pylons to work together is a non-trivial task, as the mailing lists and IRC chats will attest. Tesla takes care of that for you by providing the easiest possible setup. It handles unit tests, setup code, and shell integration, and adds some extra commands for creating models and managing your database schema.

There will be more to come. Ben is working on a template based on Tesla with AuthKit integration and basic identity models, and we’ll probably be adding more commands. Integration with the SQLAlchemy migrate library is a likely possibility at some point.

At the moment Tesla is only available on SVN from Google Code. It’s really pre-alpha, so it’s not quite stable yet. Once it’s a bit more tested and stable I’ll probably add it to the Cheeseshop. Any feedback, suggestions and criticism are welcome (please post to the Pylons mailing list or IRC chat).

The name comes from Nicola Tesla, one of the greatest (and most unrecognized) scientists and inventors of the last century.

Turtles, all the way down : WSGI middleware

WSGI (Web Standard Gateway Interface) provides a standard for connecting a Python web application or framework to a web server. The framework provides a function, or callable, that is called by code on the server side, for example a CGI or FastCGI script. Here’s an example, from Wikipedia:

def app(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    return ['Hello Worldn']

As you can see, it’s pretty basic and low-level. However, it is a complete WSGI application. As an aside, Paste, which is used by Pylons, is a WSGI implementation, with building blocks for building and deploying your own WSGI frameworks.

The nice thing about it is that you can add middleware between the server and your application. Middleware is just functions (or callables) like the one shown above, but you can implement both application and server-side code. This is a powerful,yet beautifully simple concept. Here is an example:

class PrintTimeMiddleware(object):
       def __init__(self, app):
            self.app = app
       def __call__(self, environ, start_response):
            print str(datetime.now()
            return self.app(environ,start_response)

We then add this into our server script as so:

app = PrintTimeMiddleware(app)

The “app” in question is the instance of any previous middleware, so it will be executed in that order. Each middleware is called in that order as app(environ, start_response). In this case we simply print the current time to STDOUT. Middleware can be pretty much anything you imagine…for example, it can do XSLT transformations, user authentication, database connection pooling, browser checking…

The Pylons framework itself is middleware. If you look at a Pylons application, under config/middleware.py, you see this line:

app = pylons.wsgiapp.PylonsApp(config,helpers=myapp.lib.helpers,g=app_globals.Globals)

Now for something actually useful. In a Pylons application, or any other WSGI-based framework, you can of course add your own middleware. In a previous article, “Using Elixir with Pylons”, I mentioned that you need to create a SQLAlchemy session with each new HTTP request thread, to ensure your SQLAlchemy or Elixir objects can connect to the engine in that thread. In addition, when starting the application, we need to create a new engine for our database which is then cached for later use. Now, we could do it in our code (for example in BaseController) but it is a lot more elegant to add it to our middleware. Let’s create a little middleware class for this:

myapp/models/__init__.py
from elixir import metadata, objectstore
from pylons.database import create_engine, make_session

session_context = objectstore.context
engine = None

# call this with every thread
def resync():
    session_context.current = make_session()

# connects engine to metadata
def connect():
    global engine
    if not engine:
        engine = create_engine()
        metadata.connect(engine)

def flush_all():
    objectstore.flush()

myapp/models/middleware.py

from myapp import models as model

class ModelMiddleware(object):
    def __init__(self, app):
        self.app = app
        model.connect()
    def __call__(self, environ, start_response):
        model.resync()
        return self.app(environ,start_response)

Then, in config/middeware.py, under the # YOUR MIDDLEWARE # comment, we add the middleware instance:

app = ModelMiddleware(app)

When the middleware is initialized, it creates the engine once in model.connect(). When it is called, the model.resync() line ensures a session instance is created and passed to the Elixir session context. This will now happen with each HTTP request.

We might want to modify it slightly, so that all changes are committed to the database at the end of the request. Personally, I don’t like to do this, as I prefer a more fine-grained approach, but others might want to do so.Let’s change __call__ slightly:

def __call__(self, environ, start_response):
    model.resync()
    app = self.app(environ,start_response)
    model.flush_all()
    return app

As you can see, you can hook into any part of the request lifecycle. One thing to remember is that this will happen to all requests; if I wanted more fine-grained control over individual requests, then the place for that would be the application code itself. For example, in Pylons, I can implement __before__ and __after__ methods in my controllers.

Using Elixir with Pylons

There seems to be a lot of discussion about best practices on how to create connections to SQLAlchemy or Elixir engines from within Pylons. The best solution I have found is here:

myapp.models.__init__.py

from elixir import metadata, objectstore
from pylons.database import make_session

def connect():

    session = make_session()
    objectstore.context.current = session
    metadata.connect(session.bind_to)

Bearing in mind that Elixir provides its own Session Context through objectstore.

Then, in my BaseController:

myapp.lib.base.py

class BaseController(WSGIController):

    def __call__(self, environ, start_response):
        model.connect()
        return WSGIController.__call__(self, environ, start_response)

This seems to work fine. The bonus is that I can call model.connect() from anywhere in my Pylons app: in websetup.py, paster shell, even unit tests, and it automagically picks up the sqlalchemy.dburi setting in the correct configuration file (development.ini or test.ini, for example).

I gather that the above code should also take care of connection pooling, although I don’t really know for sure. If any SQLAlchemy experts have a take on this, please let me know.

Update: looking at the Pylons code, it seems that you don’t need to connect the engine to the metadata with each request. Pylons caches engine instances, so the engine is created once per application.  We just need to ensure that a new session is started with each new request thread. Here is the full recipe.

Exploring Mako

I’d like to take a tour of Mako, Mike Bayer’s excellent Python template engine. First, a few comments on my column on adding event behaviours to Elixir:

1. The pastebin code has a few mistakes (I’m sure those interested will have spotted them); I noticed them of course just after I posted the code, but haven’t found any way to correct it (unlike in djangosnippets, for example, there does not seem any way to edit your pasted code).

Updated: here is the improved code.

2. Comments on the Elixir google group have been quite positive. As pointed out, the code could be definitely improved, it’s a bit repetitive for starters.

3. In the same vein, someone has provided an excellent example of adding validation DSLs to Elixir.

Now, on to Mako. Mako is loosely based on Myghty, which itself is based on Mason, a Perl template engine of some vintage that has been used for example by Amazon. Myghty was pretty cool, I had only used it in “hello world” stuff but it seemed a lot easier than for example Cheetah. Cheetah had a lot of hassles such as having to precompile master templates which made it a bit of a PITA at times (that might have been fixed by now; it’s been a while since I have used it).

Mako is a text-based template engine, unlike Kid or Genshi; you can therefore use it in producing non-SGML output such as email contents or JavaScript and CSS. It has a number of tags for doing stuff like inheritance, control flow syntax (same as Python, but with closing tags like endif for getting round the significant whitespace issue) and variable inclusion. The syntax and tags are very simple and a Python coder (or even web designer) should get up to speed with Mako very quickly.

The really nifty thing about Mako, however, is that it combines a lot of the best ideas from other template engines into one package. For example, inheritance is borrowed from Myghty, and also Django/Jinja.

Some template designers don’t like too much logic in their code – Django for example. In Django, using their standard template engine makes it hard to do anything more than if/else conditionals and for loops. This is by intention; templates are for designers only, and it enforces MVC – you have to move code into the controller (or Django “view”) or special tags.

The way I see it, there is no problem with logic in the view / template layer – as long as that code concerns itself only with presentation logic. Often however that logic can get convoluted and repetitive, and you want to cordon it off into a function. With Mako you can do this with namespaces. A namespace is essentially just a library of template code.

For example, say you have a Mako file, formutils.mako, which looks something like this:

<%def name="label(field, content)">
<label for="${field}">${content}</label>
</%def>

Now in another file, you include the code in formutils.mako like so:

<%namespace name="form" file="/formutils.mako" />

Now you can call the functions defined in formutils.mako just like any other Python function:

${form.label('name', 'Your name:')}

For people who know Rails, this kind of like partials, but with simpler syntax. If you are using Pylons you can use helpers, but Mako namespaces are better if the code is more HTML than Python.

Another neat idea is template inheritance. In large web apps this is important because your site is often broken down into different modules or sections, each with 90% the same layout but maybe an extra stylesheet, menu, title etc. Some template engines don’t support inheritance or do it badly: Rails for example has layouts, but layouts cannot be (as far as I know) chained in any way, so you end up cutting and pasting code or putting in lots of conditionals; you can only replace the main body of the template, not selected pieces. They are also dependent on the controller layer, which kind of breaks MVC. In Mako you have a similar directive to namespaces:

<%inherit file='/base.mako' />

Now you can define functions in the same way in your base.mako file, for example:

<%def name='title()'>Hi de ho!</%def>

and call them in your template HTML code:

<title>${self.title()}</title>

(note the use of self, which is important if you want inheritance to work).

Any template which inherits base.mako should output:

<title>Hi de ho!</title>

However, if you want to change the title you just need to override the title() function in your template:

<%def name='title()'>Ho de hi!</%def>

Then you get:

<title>Ho de hi!</title>

When you are building a large, complex web site this is really, really useful. I’m working on a large Rails project right now and this simple feature would be so handy.

One final word : Mako performance rocks, see the Mako home page for a comparative study. Mako out-performs even Cheetah and Django.

Captain America ! I command you to –

From YesButNoButYes.

The Side Project

Side projects, if you have the time, are great way to keep your coding fresh. Most places I have worked in my IT career have been single-product businesses of varying size (tiny startups to medium sized companies). That is, you work on a single product or product family, using one language or platform, year after year. Now, depending the nature of your work and the nature of you, that can be interesting or deadly boring, or both; but often you just want to do something different, and the boss won’t let you. You may limp along all day in VB.NET, but at night fly in Perl; in office hours, you write accounting software macros; in your basement, Linux DVD drivers. At the water cooler, you are just a low-paid, invisible geek; on the OSS mailing lists, you are a guru, nay, a God.

So, the side project, be it open source or not, is one way to keep your coding skills fresh and your coding ego boosted (assuming, of course, you are good at what you do). And, who knows, it may one day bring you fame and fortune.

As an aside, I did work for a boss who was dead against developers doing side projects; he thought they should be 100% focused on our product, presumably saving their spare-time brain cycles for company use. To which I say: what an ass. Side projects not only are good for you, but they may be good for whomever you work for, as you get new ideas and perspectives that may be of use to the Man.

So, I have had an idea for some time about working on the perfect business web application. To be more exact, something which allows a company to manage its projects and sales, but without complexity and bloat. I’m thinking of an app that has blogs and wikis, so people and teams can build something as much from the ground up as well as from the management down. For example, you can keep up with your project as much with a blog as with Gantt charts and milestones, tag your meetings and contacts, add YouTube videos to your product lists.

These are all rather vague, Web 2.0-ish ideas, and at the moment half-baked and in need of some originality. And yes, I know 37Signals have Basecamp and Highrise, but they have yet to combine their individually decent products under a single roof and, to be honest, their focus on “simplicity” at all costs leaves their apps a nose short of greatness.

I don’t really plan on making money out of this; I’ll probably just post it on Google Code. I’m doing it because:

a) I do Ruby all day for bread and board, and I prefer to code in Python

b) I want to do stuff in Pylons, Elixir, Mako and other cool tools I’ve been writing about

c) I think there’s a kickass framework that can be built on these tools, but I want to harvest it from something that really works (more on this later)

d) I really think that businesses and organizations need something like this ( I may be completely wrong, of course)

e) You never know, it might make me rich and/or famous.

However, I have limited time: a full time job, and family commitments. Progress will be slow and sporadic. That’s OK , if I can at least get the ball rolling, and let others with more skill and time on their hands take over, then I’ll be happy, and I’ll go and start another side project, even if that side project is just mowing the lawn.