djangozen

This is a paginator that does not do a count and it all started with this blog post. Essentially Arecibo was getting hammered by the number of the requests that were being sent to it. It wasn’t the requests that were the problem, is the listing of them.

I was doing all the usual stuff, vacuuming and making indexes. However those counts were still slow and the indexes were not always being used. So mmalone had a good idea and an excellent gist.

The problem is that count can be hideously expensive. At one point, it was about 15 minutes.

By doing a query for one more than you need, it figures out if there’s a next. The difference is the select vs the count.

What do you lose? You don’t know how many records there are, you just know if there is a next and previous (and you can figure out how many came before). But if you are using postgresql, beware of how expensive those counts can be.

The default assumes there’s going to be 1000 pages, but we don’t really know how many there. There’s a max_safe_pages variable that gets updated as information is provided. For example if you set it to 3 pages… when try and access 4, it fail, thinking that there was no data.

That can be a bit confusing, perhaps in the future it should check and if not then try to get it… improvements welcome.

The power of Q

Q objects are lumbered with a name that’s very hard to Google. Down there with F objects and Plone’s event. You’ll probably come across Q objects when you need to complicated lookups such as not, or and so on. But these objects are so useful.

Complex lookups

But now we can at least do an or quite easily (archived or have a priority of 1):

List of lookups

One thing I do quite often is build up the query dynamically. You can do that easily in Python by making a dictionary and passing that to filter. You can also do it by building up a list of Q objects. For example:

We’ve now got a list of Q objects and we can do a filter. The use of reduce and operator.or_, applies the | to all the elements in the list:

Pickling lookups

If you would like to store your lookups you can pickle the list of objects. This means you can save the Q objects and use them again later. For example:

Pickling is a useful way of saving the queries for use again later. In some situations it can be benefical to pickle the queries as opposed to pickling the results. Django documentation notes that “If you pickle a QuerySet, this will force all the results to be loaded into memory prior to pickling.” (docs).

Javascript inline forms outside of contrib.admin

This recipe follows on from the last two and shows how to do the addition of inline fields to a model form, outside of contrib.admin.

In the last two recipes, we’ve seen how to add inlines through the admin interface. We’ll repeat this process in this last recipe of the series but without the admin interface. This uses some different API’s and therefore can be slightly different. Again, we’ll show a quick snippet of the end result:

The models have remain unchanged since recipe 70. The forms for this setup will also be similar to recipe 70. We’ll just need to add in one new form - the Recipe form. This is a basic Django ModelForm:

The add template is pretty straightforward. We are going to assume that the view is passing a form (the Recipe form) and a formset (the Ingredient formset) to the template. Notice in the template that I’ve used the same classes from the admin in order to make the Javascript a little bit easier. Each formset is also rendered in its own table. This makes for a slightly ugly markup but it works.

When rendering a formset manually in this manner, we also need to include the management form (line 1). This form is the value that gets incremented every time Add another row is pressed [1]. The JavaScript is very similar to the one from recipe 72 with a few minor alterations. Since we now have control over the template, the button is rendered in the template not the JavaScript.

Those are all the elements. Now we need to take a look at the main part of this recipe, the views. I covered how to use the formfield_callback and curry to process the field correctly in recipe 70 . We’ll repeat this process in our view. The admin won’t be constructing the formset for us this time. We’ll have to construct it using an inlineformset_factory [2]:

Once we’ve got that formset built, most of the rest of the form is straightforward and follows the standard form population, validation and save process. One slight wrinkle is before the formset can be saved we’ll need a recipe to link to it. For this reason, the view populates the formset, validates it, then repopulates it with the saved recipe:

There’s little difference for the edit page and most of that should be familiar by now. Since we ended up using the formfield_callback, it’s easy to have these models editable in both the admin and outside the admin.

Adding Javascript to contrib.admin inline fields

This recipe follows on from the last by adding Javascript to the inline fields of a model in the contrib.admin.

By default the inlines in contrib.admin are a fixed number. In each case Django adds in three extra ingredient inlines for you by default. However, a user may add in more. What we really need is an Javascript solution to add in more rows so the user can keep adding extra rows in one page.

There are two ways to do this. One way is to use the template parameter of the admin.TabularInline. You could add in a new template to display the entire inline field set using this. You’d want to go grab the inline template and then place it in your project to do this. Then you could customize it to add a button and some JavaScript. You can see this in the templates directory of this recipe.

I didn’t choose that way because it involves the copy and paste of large amounts of Django template code. Remember template code can change and you’ll need to maintain that template code across upgrades. The only real advantage of doing that copy and paste is to include a snippet of JavaScript and a bit of HTML. The JavaScript shouldn’t be in the template and should be in an external file anyway. So this is really a less desirable solution.

You’ll see what I chose to do instead in the media folder of this recipe. Take a look at the add_tabular_inline.js file. This uses jQuery to add an add button and then clone the last row of the inlines.

Let’s setup the JavaScript. You can add custom JavaScript using the Media class inside RecipeAdmin. This particular one pulls in jQuery from Google’s servers and then the JavaScript from our recipe [1].

The JavaScript creates a button and then binds a clone event to it. This works with the TabularInline. A StackedInline would require slightly different code. This isn’t much longer at all compared to the embedding in the template version:

To facilitate the addition and removal of inlines, Django provides an input field that calculates the total number of forms in the formset. When you insert or remove forms, you need to alter that number so that Django knows to process the data. This is the id_recipe-TOTAL_FORMS input. Line 24 of the above code increments that value.

Setting up inline forms in contrib.admin

This recipe covers how to set up inline forms in contrib admin and how to show foreign keys in those inline forms.

Note: this is excerpt a recipe from cleardjango. You can download the source from SVN. Once you’ve checked it out, do: python tools/configure.py -s 70 to install and run this recipe.

Two different models are often used, one related to another. These are normally shown to users in the Django interface as two separate pages. You could however set up one form that contains the forms for the other model so they can be edited inline. It’s probably simpler to show a quick graphic to demonstrate this:

In this case we’ve got a recipe, a “Cucumber-Rosemary Gin and Tonic” [1] and some ingredients. Let’s take a look at the two models that setup this relationship:

from django.db import models

class Recipe(models.Model):
    name = models.CharField(max_length=255)
    description = models.TextField()

    def __unicode__(self):
        return self.name

class Ingredient(models.Model):
    name = models.CharField(max_length=255)
    amount = models.CharField(max_length=255)
    
    recipe = models.ForeignKey(Recipe, related_name="recipe")
    other_recipe = models.ForeignKey(Recipe, related_name="other_recipe", blank=True, null=True)

The Ingredient model is slightly unusual because it has two keys pointing back to the recipe. The recipe field points to the Recipe model. However, some drinks may contain ingredients which are another recipe. There’s no point in covering how to make the recipe every time. Instead this is pointed to once and a reference made. Here it’s just a normal Gin and Tonic with some additions. This pointer to another recipe is the other_recipe field and is optional.

So next we’ll need to register these models in the Django admin. We’ll do this in the usual way. For example, in admins.py we’ll have:

admin.site.register(Recipe, RecipeAdmin)
admin.site.register(Ingredient, IngredientAdmin)

The admin registration for a Recipe is a little different because we need to have this show the ingredient inline. Here we’ll use the contrib.admin inlines functionality [2]. This requires modifying the admins.py and registering the Recipe slightly differently. Next we’ll use the inlines attribute on Recipes. This tells the admin interface what inline classes to use. Multiple inlines can be defined but in our case we only need one.

class RecipeAdmin(admin.ModelAdmin):
    inlines = [
        IngredientInline
    ]
    list_display = ("name",)

The IngredientInline class referred to is a class used to define what will be shown. This looks like:

class IngredientInline(admin.TabularInline):
    model = Ingredient
    fk_name = "recipe"

Now the code should be ready to run and play with. We’ve specified a fk_name on line 3 of the IngredientInline. That’s the foreign key that contains the relationship back to the recipe. There’s no reason for this to be set in a form since Django will automatically set this to the recipe you are adding or editing. The explicit definition of fk_name is only necessary in cases where there are more than one foreign key relationships back to the parent model [3].

I’d like to make one additional optimization to this form. We have a recipe “Cucumber-Rosemary Gin and Tonic” that contains a reference to another recipe, “Gin and Tonic”. But it could never contain a reference to itself. An ingredient of “Cucumber-Rosemary Gin and Tonic” cannot ever be itself. Recursion is cool for acronyms of open source projects but less cool in the world of cocktails.

If we know the current object, we can use the queryset api to return all recipes excluding all the current object. If the current object is instance that would look like this:

 Recipe.objects.exclude(pk=instance.pk)

The harder part is altering the form to do this. There a few different approaches and one is to create a custom form [4]. I chose a slightly different way because this allows us to work the same way with inlines outside of the admin in later recipes. When the admin creates a formset, it calls a method called get_formset We can pass a number of parameters into that. One of them is a formfield_callback.

When each form field is rendered, this method will be called so that we can alter each field in the form. Here’s how that callback would look given an instance. It’s called for each and every field so we need to check the field name before altering:

from models import Recipe

def ingredient_form_callback(instance, field, *args, **kw):
    if field.name == "other_recipe":
        if instance:
            return field.formfield(queryset=Recipe.objects.exclude(pk=instance.pk), **kw)
    return field.formfield(**kw)

We’re almost there but there’s one last thing. The instance isn’t normally passed to the field. We need to add the instance. This can be done quickly and easily by using the django curry method [5]. The entire admin.py will look like this:

from django.contrib import admin
from django.utils.functional import curry

from models import Recipe, Ingredient
from forms import ingredient_form_callback
    
class IngredientInline(admin.TabularInline):
    model = Ingredient
    fk_name = "recipe"

    def get_formset(self, request, obj=None, **kwargs):
        kwargs["formfield_callback"] = curry(ingredient_form_callback, obj)
        return super(IngredientInline, self).get_formset(request, obj, **kwargs)
    
class IngredientAdmin(admin.ModelAdmin):
    pass
        
class RecipeAdmin(admin.ModelAdmin):
    inlines = [
        IngredientInline
    ]
    list_display = ("name",)

admin.site.register(Recipe, RecipeAdmin)
admin.site.register(Ingredient, IngredientAdmin)

The end result is pretty cool. We have a Recipe form with inlines below it and a custom form on those inline forms to limit the valid data. As for the virtues of these drinks, feel free to take part in some exhaustive research.

Sponsored Recipe: This recipe was sponsored by Blue Fountain Systems Ltd. Please contact Clearwind if you’d like to sponsor a recipe.

[1] http://www.drinksmixer.com/drinkf164e99.html [2] http://docs.djangoproject.com/en/dev/ref/contrib/admin/#inlinemodeladmin-objects [3] http://docs.djangoproject.com/en/dev/ref/contrib/admin/#fk-name [4] http://www.artfulcode.net/articles/runtime-choicefield-filtering-in-djangos-admin/ [5] http://code.djangoproject.com/browser/django/trunk/django/utils/functional.py#L53

Tuning the count method on a queryset

The other week I faced a real slow down on Arecibo due to a high number of errors being generated by a few other sites. At the time I started to find out how bad Postgres is compared to MySQL in terms of performance when doing a SQL count. MySQL does more work to make that faster.

In Arecibo, we present a list of all the errors that have recently occured. This page is paginated using the Django pagination class. To figure out how big the result set is and hence how many pages you have, the default pagination class will call the method count on the queryset.

So how can we make this faster, the first obvious strategy, change count on your query set. To do this write a custom object manager and then create your own count. For example:

Hmm that actually works pretty well on some larger Arecibo error pages, because we don’t really need to do a count if you’ve got a lot of results. If you do this, then the pagination class just thinks the list is big and the user sees lots of “next” links.

Hmm again, ok. The problem is that this is fast on the second hit. The first hit is slow.

But you’ve got the general idea. The problem with these is that they change the ObjectManager and hence the change all count calls for that model (unless I start having multiple ObjectManagers). Actually for me that’s a problem in my application. I still want the django-admin and other parts of the site to work and this change broke quite a lot of unit tests as I quickly looked for a solution.

You could also overwrite count on a per query set basis. Since there was on key view in Arecibo that was causing the problem, I focused on that and wrote the following:

In the end my CountProxy class got quite complicated as it did work to cache and optimise the query based on the filtering. The nice thing is that you have an opportunity to make count as complex and as custom you’d like at this point. Add in some Postgres index tuning and things turned out a lot better. The end result was that I got page that was taking sometimes around 8 seconds (it was a bad day), down to about 40ms.

The lack of magic in Django

There’s been a bit of a discussion recently about magic from James Bennett and others. Also one of my friends recently had a play with Ruby and came to some conclusions. I did a bit of work in Rails a while back and it never did much for me I found large applications confusing and hard to debug.

If you ask me why I don’t like Rails, there’s a simple answer: it has too much magic.

We all know the quote about technology being indistinguishable from magic. In that law Arthur C. Clarke is describing how a lack of some basic underlying principles means that observers are unable to determine how things work and they seem magical. The same thing can happen in software, something happens when the following peice of code happens. But something magic has happened to make it do so and it’s not obvious to the person developing the software because they are lacking some underlying peice of knowledge.

Now we can say simply that I didn’t understand all of Rails and hence I was missing knowledge and hence it was all magic. I think that would be wrong, whilst I never got completely to grips with all of Rails, I knew the key parts well enough to be able to produce projects that worked. The test for this is reading someone else’s code and applications and trying to figure out what’s happenening and that’s where Rails (and to a small extent Ruby) do magic to solve a problem.

Let’s start with an example. Django has explicit models that declare the types of fields a model has. For example here’s a Django model:

Django (optionally) creates the SQL from your model and populates the database with it. Here’s a model from Rails.

In Rails you write the migration, the schema is created in the database and Rails infers from the database your model (don’t forget Rails will rename your table and try to pluralize it for, thanks Rails).

Now let’s take a scenario where you are working on a project that has over 200 migrations. To tell what’s in your model, you either a) wade through all the migrations that are relevant to your model or b) keep examining your database. It’s not that point b) is bad, it’s important to know what’s going on in your database.

In Django I’m explicitly saying what fields I’m expecting on a model, what they are and what they manage. I’m also declaring the methods and attributes on an object - or a namespace. I know what at a glance Poll.question (Django) is, figuring out what Post.name (Rails) isn’t as obvious. Indeed there’s nothing stopping me adding a method to the Post class called name. Rails is doing a looking up for you on the object to calculate the variable.

What does Sample.wtf give you? Does the method foo exist on the object (and presumably in the inheritance tree), does the method foo exist somewhere else in the namespace, does anything else import foo into the global namespace, does foo exist on the database for that model?

To me this is not explicit and hence somewhat magical. That is my definition of magical, something is going to happen that is not obvious or clear. There’s some implicit assumption that you know a whole bunch more information about the code and the context it’s being run in.

As a developer reading this you have to do all that work to figure out what foo is, or do runtime analysis. To me this flies in the face of another rule:

This philosophy goes on through Rails in many different areas, here’s some more examples.

Let’s look at Zope 2 for a minute and Acquisition and DTML. Alright I’m picking at pieces that either no longer exist or are now explicit, but that’s because the developers found way too many issues with the said technologies. Learning from past mistakes and not repeating them is an important thing to be able to do.

In Zope 2 there was a templating language called DTML that you could write on objects. When it was rendered it would act on the object in the database and then use Acquisition to figure out what to do next. Acquisition can be pretty mind bending, it doesn’t mean it was wrong, but to many developers it was just far too magical.

Code that worked in one place, wouldn’t work in another because the context it was being executed in wasn’t right. Not in itself wrong, but very hard to figure out and debug so to many it became magical. Never mind the confusing syntax which differentiated between:

I have to confess I don’t have any concrete examples of this since all that code has long since vanished into a distant past, but I wanted to point out that it’s not just Rails. Training developers in this was great fun, the principle was all well and good, but all too often I saw people resort to random guessing of code to get it working.

Technology is thought of as magical because something is happening that we don’t understand. Frameworks that make decisions for you at runtime based on things you haven’t explicitly declared can be seen as performing some magic. The level of magic has inverse relationship to the level of explicitness. Django is more explicit and much less magical. Rails is less explicit and rather magical. Zope 2 was very magical.

BTW: Yes, I’m quoting the Zen of Python, by Tim Peters. To read the whole thing start your Python and type “import this”.

Forms, forms, forms

The exact same way you do on a normal Form. If the name of the field matches a field that already exists on the model the form is using, it will override the default.

2. If I add fields to a ModelForm, the data in those fields does not get saved.

No, they won’t, because they aren’t attached to the model, so you would have to save them manually.

So for example, if you make a User Profile, you might have one HTML page to edit user details like email address (in the User model) and Facebook account (in the User Profile model). To do this:

Although the names are similar, a HTML Form and a Django Form are quite different.

Features of a good Django plugin

A while back I wanted to allow openid logins on my site. A simple problem which a few plugins released to the Django community make easier. However there’s quite a few of them (we list 4, there’s more) and none of them quite work the way I want.

So what’s wrong with them? Most of them define front end templates or particular models or a particular way of doing it. I don’t want that, I just want to make my application do openid authentication. So in frustration I have something which is a complete rip off of django-simple-openid from Benoît Chesneau.

If I’m going to create all these steps I want a plugin that fits into this pattern. A lot of them end up being like contrib.comments, something you have to fight to get customised. As an example, here’s what I would like to see in an openid plugin:

Thinking about it this mirrors the general Django philosophy in many ways. Perhaps a list for what a plugin should have:

The primary job of someone using your plugin is to integrate it with their application. Let’s make that a bit easier.

Disclaimer: the new openid implementation from Simon Willison looks great. Although it does look worryingly complex, it seems to (at my quick glances over the code) accomplish the above.

Other disclaimer: it’s easy to say this standing up on my ivory tower since I haven’t actually released my openid library, because its a hacked up mess. Chances are as soon as I write about this, someone will come along and point out their Django openid plugin which does it just the way I want. I hope so.

Read only fields in models

Supposing you have a field in your model and you don’t want it to be changed, ever.

How would you do that? Well actually you can’t from Django, which we’ll see in a minute. Here’s a script that should work for most people. Basically, set up a signal (or a custom save method) so that when your object is saved it compares the old to new and then figures out if it can be changed.

On a save, we look up the read_only attribute and see if any of those fields have changed and raise an error if they do. In this scenario, you can change the amount of money in a bank account, but the number of the account will not change once set.

This is awfully close to model validation. Something that Django doesn’t have yet. And it there’s two areas it can break down.

This is discussed here and here. I wonder if it’s worth going any further on this until model validation comes along. Perhaps not.