djangozen

Faster or lazier pagination

Sat, 17 Dec 2011 03:15:30 -0500

Or how to avoid slow counts in Postgres.

This is a paginator that does not do a count and it all started with this blog post. Essentially Arecibo was getting hammered by the number of the requests that were being sent to it. It wasn’t the requests that were the problem, is the listing of them.

I was doing all the usual stuff, vacuuming and making indexes. However those counts were still slow and the indexes were not always being used. So mmalone had a good idea and an excellent gist.

Taking this, I updated it for the latest Django and we got the LazyPaginator.

Example from a normal Paginator:

>>> from django.db import connection
>>> from listener.models.error import Error
>>> queryset = Error.objects.filter(account=1, archived=1)

The old way:

>>> from django.core.paginator import Paginator 
>>> p = Paginator(queryset, 10)
>>> p.page(1)

Causes:

>>> connection.queries
[ {'time': '21.953', 'sql': 'SELECT COUNT(*) FROM "listener_error" 
   WHERE ("listener_error"."account_id" = 1  
   AND "listener_error"."archived" = true )'}]

That’s one query. To get the object list it does another.

>>> p.page(1).object_list
[{'time': '0.000', 'sql': 'SELECT "listener_error"."id", ... FROM "listener_error"
    WHERE ("listener_error"."account_id" = 1  
    AND "listener_error"."archived" = true ) LIMIT 10'}]

The problem is that count can be hideously expensive. At one point, it was about 15 minutes.

The new way:

>>> from lazy_paginator.paginator import LazyPaginator 
>>> p = LazyPaginator(queryset, 10)
>>> p.page(1)

>>> connection.queries     
[{'time': '0.000', 'sql': 'SELECT "listener_error"."id",... FROM "listener_error" 
   WHERE ("listener_error"."account_id" = 1  
   AND "listener_error"."archived" = true ) LIMIT 11'}]

>>> p.page(1).object_list
[{'time': '0.000', 'sql': 'SELECT "listener_error"."id",... FROM "listener_error"
    WHERE ("listener_error"."account_id" = 1  
    AND "listener_error"."archived" = true ) LIMIT 10'}]

By doing a query for one more than you need, it figures out if there’s a next. The difference is the select vs the count.

What do you lose? You don’t know how many records there are, you just know if there is a next and previous (and you can figure out how many came before). But if you are using postgresql, beware of how expensive those counts can be.

The default assumes there’s going to be 1000 pages, but we don’t really know how many there. There’s a max_safe_pages variable that gets updated as information is provided. For example if you set it to 3 pages… when try and access 4, it fail, thinking that there was no data.

>>> p = LazyPaginator(queryset, 10, max_safe_pages=3)
>>> p.has_next(1)
True
>>> p.has_next(2)
True
>>> p.has_next(3)
False
>>> p.has_next(4)
False
>>> p.page(4)    
Traceback (most recent call last):
  File "", line 1, in 
  File "/var/arecibo/lazy_paginator/paginator.py", line 27, in page
    number = self.validate_number(number)
  File "/var/arecibo/lazy_paginator/paginator.py", line 20, in validate_number
    return super(LazyPaginator, self).validate_number(number)
  File "/usr/lib/python2.5/site-packages/django/core/paginator.py", line 32, in validate_number
    raise EmptyPage('That page contains no results')
EmptyPage: That page contains no results

Now if you start at the beginning:

>>> p.page(2)

>>> p.page(3)

>>> p.page(4)

That can be a bit confusing, perhaps in the future it should check and if not then try to get it… improvements welcome.

Install: Lazy paginator can be installed from PyPi or on github.

The power of Q

Sat, 17 Dec 2011 03:14:36 -0500

Q objects are a great tool in Django.

Q objects are lumbered with a name that’s very hard to Google. Down there with F objects and Plone’s event. You’ll probably come across Q objects when you need to complicated lookups such as not, or and so on. But these objects are so useful.

Complex lookups

You can make Q objects for the lookups you’d like. For example:

>>> from django.db.models import Q
>>> from listener.models.error import Error # our example model

To find all objects that archived and have a priority of 1, we could do:

>>> Error.objects.filter(archived=False).filter(priority=1).count()
39592

The equivalent in Q objects is:

>>> Error.objects.filter(Q(archived=False) & Q(priority=1)).count()
39592

But now we can at least do an or quite easily (archived or have a priority of 1):

>>> Error.objects.filter(Q(archived=False) | Q(priority=1)).count()
195871

And how about archived and a priority of not 1:

>>> Error.objects.filter(Q(archived=False) & ~Q(priority=1)).count()
41545

List of lookups

One thing I do quite often is build up the query dynamically. You can do that easily in Python by making a dictionary and passing that to filter. You can also do it by building up a list of Q objects. For example:

>>> qs = [Q(archived=False),]
>>> qs.append(Q(priority=1))
>>> qs
[, ]

We’ve now got a list of Q objects and we can do a filter. The use of reduce and operator.or_, applies the | to all the elements in the list:

>>> import operator
>>> Error.objects.filter(reduce(operator.or_, qs)).count()
195871

Similarly for an and:

>>> Error.objects.filter(reduce(operator.and_, qs)).count()
39592

Pickling lookups

If you would like to store your lookups you can pickle the list of objects. This means you can save the Q objects and use them again later. For example:

>>> import pickle
>>> saved = pickle.dumps(qs)
>>> Error.objects.filter(reduce(operator.and_, pickle.loads(saved))).count()
39592

Pickling is a useful way of saving the queries for use again later. In some situations it can be benefical to pickle the queries as opposed to pickling the results. Django documentation notes that “If you pickle a QuerySet, this will force all the results to be loaded into memory prior to pickling.” (docs).

Javascript inline forms outside of contrib.admin

Sat, 17 Dec 2011 03:13:00 -0500

This recipe follows on from the last two and shows how to do the addition of inline fields to a model form, outside of contrib.admin.

Note: this is excerpt a recipe from cleardjango. You can download the source from SVN. Once you’ve checked it out, do: python tools/configure.py -s 72 to install and run this recipe.

In the last two recipes, we’ve seen how to add inlines through the admin interface. We’ll repeat this process in this last recipe of the series but without the admin interface. This uses some different API’s and therefore can be slightly different. Again, we’ll show a quick snippet of the end result:

The models have remain unchanged since recipe 70. The forms for this setup will also be similar to recipe 70. We’ll just need to add in one new form - the Recipe form. This is a basic Django ModelForm:

from django import forms

from recipe_72.models import Recipe

class RecipeForm(forms.ModelForm):
    class Meta:
        model = Recipe

def ingredient_form_callback(instance, field, *args, **kw):
    if field.name == "other_recipe":
        if instance:
            return field.formfield(queryset=Recipe.objects.exclude(pk=instance.pk), **kw)
    return field.formfield(**kw)

We’ll set up two URLs for this recipe, one to add and one to add:

class RecipeForm(forms.ModelForm):
    class Meta:
        model = Recipe

The add template is pretty straightforward. We are going to assume that the view is passing a form (the Recipe form) and a formset (the Ingredient formset) to the template. Notice in the template that I’ve used the same classes from the admin in order to make the Javascript a little bit easier. Each formset is also rendered in its own table. This makes for a slightly ugly markup but it works.

                  {{ formsets.management_form }}
                  {% for formset in formsets.forms %}
                  
                      {{ formset }}
                  
                  {% endfor %}

When rendering a formset manually in this manner, we also need to include the management form (line 1). This form is the value that gets incremented every time Add another row is pressed [1]. The JavaScript is very similar to the one from recipe 72 with a few minor alterations. Since we now have control over the template, the button is rendered in the template not the JavaScript.

Those are all the elements. Now we need to take a look at the main part of this recipe, the views. I covered how to use the formfield_callback and curry to process the field correctly in recipe 70 . We’ll repeat this process in our view. The admin won’t be constructing the formset for us this time. We’ll have to construct it using an inlineformset_factory [2]:

    IngredientFormSet = inlineformset_factory(Recipe, Ingredient, 
                fk_name="recipe", 
                formfield_callback=curry(ingredient_form_callback, None))

Once we’ve got that formset built, most of the rest of the form is straightforward and follows the standard form population, validation and save process. One slight wrinkle is before the formset can be saved we’ll need a recipe to link to it. For this reason, the view populates the formset, validates it, then repopulates it with the saved recipe:

        form = RecipeForm(request.POST)
        formset = IngredientFormSet(request.POST)
        if form.is_valid() and formset.is_valid():
            recipe = form.save()
            formset = IngredientFormSet(request.POST, instance=recipe)

There’s little difference for the edit page and most of that should be familiar by now. Since we ended up using the formfield_callback, it’s easy to have these models editable in both the admin and outside the admin.

Sponsored Recipe: This recipe was sponsored by Blue Fountain Systems Ltd. Please contact Clearwind if you’d like to sponsor a recipe.

[1] http://docs.djangoproject.com/en/dev/topics/forms/formsets/#using-a-formset-in-views-and-templates [2] http://docs.djangoproject.com/en/dev/topics/forms/modelforms/#inline-formsets

Adding Javascript to contrib.admin inline fields

Sat, 17 Dec 2011 03:12:00 -0500

This recipe follows on from the last by adding Javascript to the inline fields of a model in the contrib.admin.

Note: this is excerpt a recipe from cleardjango. You can download the source from SVN. Once you’ve checked it out, do: python tools/configure.py -s 71 to install and run this recipe.

By default the inlines in contrib.admin are a fixed number. In each case Django adds in three extra ingredient inlines for you by default. However, a user may add in more. What we really need is an Javascript solution to add in more rows so the user can keep adding extra rows in one page.

There are two ways to do this. One way is to use the template parameter of the admin.TabularInline. You could add in a new template to display the entire inline field set using this. You’d want to go grab the inline template and then place it in your project to do this. Then you could customize it to add a button and some JavaScript. You can see this in the templates directory of this recipe.

I didn’t choose that way because it involves the copy and paste of large amounts of Django template code. Remember template code can change and you’ll need to maintain that template code across upgrades. The only real advantage of doing that copy and paste is to include a snippet of JavaScript and a bit of HTML. The JavaScript shouldn’t be in the template and should be in an external file anyway. So this is really a less desirable solution.

You’ll see what I chose to do instead in the media folder of this recipe. Take a look at the add_tabular_inline.js file. This uses jQuery to add an add button and then clone the last row of the inlines.

Let’s setup the JavaScript. You can add custom JavaScript using the Media class inside RecipeAdmin. This particular one pulls in jQuery from Google’s servers and then the JavaScript from our recipe [1].

    class Media:
        js = (
            "http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js",
            "/recipe_media/add_tabular_inline.js",
            )

The JavaScript creates a button and then binds a clone event to it. This works with the TabularInline. A StackedInline would require slightly different code. This isn’t much longer at all compared to the embedding in the template version:

function increment_form_ids(el, to) {
    var from = to-1;
    $(':input', $(el)).each(function(i,e){
          var old_name = $(e).attr('name')
          var old_id = $(e).attr('id')
          $(e).attr('name', old_name.replace(from, to))
          $(e).attr('id', old_id.replace(from, to))
          $(e).val('')
    })
}
 
function add_inline_button() {
    $(".inline-group p.tools").bind("click", function(e) {
        var rows = $(this).parents("div.inline-group").find("tr");
        var last = $(rows[rows.length-1]);
        var copy = last.clone(true);
        if (last.hasClass("row1")) {
            copy.attr("class", "row2");
        } else {
            copy.attr("class", "row1");
        }
        last.after(copy);
        $($(this).parents("div.inline-group").find("input")[0]).val(rows.length);
        increment_form_ids(copy, rows.length-1);
        return false;
    });
}

$(document).ready(add_inline_button);

To facilitate the addition and removal of inlines, Django provides an input field that calculates the total number of forms in the formset. When you insert or remove forms, you need to alter that number so that Django knows to process the data. This is the id_recipe-TOTAL_FORMS input. Line 24 of the above code increments that value.

This is based on the work in this blog. [2]

Sponsored Recipe: This recipe was sponsored by Blue Fountain Systems Ltd. Please contact Clearwind if you’d like to sponsor a recipe.

[1] See notes in MEDIA_URL and serving static content, something this recipe does on start up. [2] http://www.arnebrodowski.de/blog/507-Add-and-remove-Django-Admin-Inlines-with-JavaScript.html

Setting up inline forms in contrib.admin

Sat, 17 Dec 2011 03:11:00 -0500

This recipe covers how to set up inline forms in contrib admin and how to show foreign keys in those inline forms.

Note: this is excerpt a recipe from cleardjango. You can download the source from SVN. Once you’ve checked it out, do: python tools/configure.py -s 70 to install and run this recipe.

Two different models are often used, one related to another. These are normally shown to users in the Django interface as two separate pages. You could however set up one form that contains the forms for the other model so they can be edited inline. It’s probably simpler to show a quick graphic to demonstrate this:

In this case we’ve got a recipe, a “Cucumber-Rosemary Gin and Tonic” [1] and some ingredients. Let’s take a look at the two models that setup this relationship:

from django.db import models

class Recipe(models.Model):
    name = models.CharField(max_length=255)
    description = models.TextField()

    def __unicode__(self):
        return self.name

class Ingredient(models.Model):
    name = models.CharField(max_length=255)
    amount = models.CharField(max_length=255)
    
    recipe = models.ForeignKey(Recipe, related_name="recipe")
    other_recipe = models.ForeignKey(Recipe, related_name="other_recipe", blank=True, null=True)

The Ingredient model is slightly unusual because it has two keys pointing back to the recipe. The recipe field points to the Recipe model. However, some drinks may contain ingredients which are another recipe. There’s no point in covering how to make the recipe every time. Instead this is pointed to once and a reference made. Here it’s just a normal Gin and Tonic with some additions. This pointer to another recipe is the other_recipe field and is optional.

So next we’ll need to register these models in the Django admin. We’ll do this in the usual way. For example, in admins.py we’ll have:

admin.site.register(Recipe, RecipeAdmin)
admin.site.register(Ingredient, IngredientAdmin)

The admin registration for a Recipe is a little different because we need to have this show the ingredient inline. Here we’ll use the contrib.admin inlines functionality [2]. This requires modifying the admins.py and registering the Recipe slightly differently. Next we’ll use the inlines attribute on Recipes. This tells the admin interface what inline classes to use. Multiple inlines can be defined but in our case we only need one.

class RecipeAdmin(admin.ModelAdmin):
    inlines = [
        IngredientInline
    ]
    list_display = ("name",)

The IngredientInline class referred to is a class used to define what will be shown. This looks like:

class IngredientInline(admin.TabularInline):
    model = Ingredient
    fk_name = "recipe"

Now the code should be ready to run and play with. We’ve specified a fk_name on line 3 of the IngredientInline. That’s the foreign key that contains the relationship back to the recipe. There’s no reason for this to be set in a form since Django will automatically set this to the recipe you are adding or editing. The explicit definition of fk_name is only necessary in cases where there are more than one foreign key relationships back to the parent model [3].

I’d like to make one additional optimization to this form. We have a recipe “Cucumber-Rosemary Gin and Tonic” that contains a reference to another recipe, “Gin and Tonic”. But it could never contain a reference to itself. An ingredient of “Cucumber-Rosemary Gin and Tonic” cannot ever be itself. Recursion is cool for acronyms of open source projects but less cool in the world of cocktails.

If we know the current object, we can use the queryset api to return all recipes excluding all the current object. If the current object is instance that would look like this:

 Recipe.objects.exclude(pk=instance.pk)

The harder part is altering the form to do this. There a few different approaches and one is to create a custom form [4]. I chose a slightly different way because this allows us to work the same way with inlines outside of the admin in later recipes. When the admin creates a formset, it calls a method called get_formset We can pass a number of parameters into that. One of them is a formfield_callback.

When each form field is rendered, this method will be called so that we can alter each field in the form. Here’s how that callback would look given an instance. It’s called for each and every field so we need to check the field name before altering:

from models import Recipe

def ingredient_form_callback(instance, field, *args, **kw):
    if field.name == "other_recipe":
        if instance:
            return field.formfield(queryset=Recipe.objects.exclude(pk=instance.pk), **kw)
    return field.formfield(**kw)

We’re almost there but there’s one last thing. The instance isn’t normally passed to the field. We need to add the instance. This can be done quickly and easily by using the django curry method [5]. The entire admin.py will look like this:

from django.contrib import admin
from django.utils.functional import curry

from models import Recipe, Ingredient
from forms import ingredient_form_callback
    
class IngredientInline(admin.TabularInline):
    model = Ingredient
    fk_name = "recipe"

    def get_formset(self, request, obj=None, **kwargs):
        kwargs["formfield_callback"] = curry(ingredient_form_callback, obj)
        return super(IngredientInline, self).get_formset(request, obj, **kwargs)
    
class IngredientAdmin(admin.ModelAdmin):
    pass
        
class RecipeAdmin(admin.ModelAdmin):
    inlines = [
        IngredientInline
    ]
    list_display = ("name",)

admin.site.register(Recipe, RecipeAdmin)
admin.site.register(Ingredient, IngredientAdmin)

The end result is pretty cool. We have a Recipe form with inlines below it and a custom form on those inline forms to limit the valid data. As for the virtues of these drinks, feel free to take part in some exhaustive research.

Sponsored Recipe: This recipe was sponsored by Blue Fountain Systems Ltd. Please contact Clearwind if you’d like to sponsor a recipe.

[1] http://www.drinksmixer.com/drinkf164e99.html [2] http://docs.djangoproject.com/en/dev/ref/contrib/admin/#inlinemodeladmin-objects [3] http://docs.djangoproject.com/en/dev/ref/contrib/admin/#fk-name [4] http://www.artfulcode.net/articles/runtime-choicefield-filtering-in-djangos-admin/ [5] http://code.djangoproject.com/browser/django/trunk/django/utils/functional.py#L53

Tuning the count method on a queryset

Sat, 17 Dec 2011 03:10:23 -0500

Some alternate ways to manipulate the pagination of Django query sets.

The other week I faced a real slow down on Arecibo due to a high number of errors being generated by a few other sites. At the time I started to find out how bad Postgres is compared to MySQL in terms of performance when doing a SQL count. MySQL does more work to make that faster.

In Arecibo, we present a list of all the errors that have recently occured. This page is paginated using the Django pagination class. To figure out how big the result set is and hence how many pages you have, the default pagination class will call the method count on the queryset.

So how can we make this faster, the first obvious strategy, change count on your query set. To do this write a custom object manager and then create your own count. For example:

def count(self):
    # rather stupid count
    return 999999

Hmm that actually works pretty well on some larger Arecibo error pages, because we don’t really need to do a count if you’ve got a lot of results. If you do this, then the pagination class just thinks the list is big and the user sees lots of “next” links.

How about caching it? That’s straightforward too.

def count(self):
    # not fully tested
    if cache.get("count", None):
        return cache["count"]
    count = self.get_query_set().count()
    cache.set("count", count)
    return count

Hmm again, ok. The problem is that this is fast on the second hit. The first hit is slow.

But you’ve got the general idea. The problem with these is that they change the ObjectManager and hence the change all count calls for that model (unless I start having multiple ObjectManagers). Actually for me that’s a problem in my application. I still want the django-admin and other parts of the site to work and this change broke quite a lot of unit tests as I quickly looked for a solution.

You could also overwrite count on a per query set basis. Since there was on key view in Arecibo that was causing the problem, I focused on that and wrote the following:

class CountProxy:
    def __call__(self):
        # do something clever here, cache, guess etc
        # to create a count
        return count

Next in your view, assign the CountProxy to the object.

queryset = Error.objects.filter(...)
queryset.count = CountProxy()
Paginator(queryset, ...)

In the end my CountProxy class got quite complicated as it did work to cache and optimise the query based on the filtering. The nice thing is that you have an opportunity to make count as complex and as custom you’d like at this point. Add in some Postgres index tuning and things turned out a lot better. The end result was that I got page that was taking sometimes around 8 seconds (it was a bad day), down to about 40ms.

The ability to do fine grained control on that count can be quite useful.

The lack of magic in Django

Sat, 17 Dec 2011 03:08:50 -0500

…or why Rails and Zope 2 were too magical.

There’s been a bit of a discussion recently about magic from James Bennett and others. Also one of my friends recently had a play with Ruby and came to some conclusions. I did a bit of work in Rails a while back and it never did much for me I found large applications confusing and hard to debug.

If you ask me why I don’t like Rails, there’s a simple answer: it has too much magic.

We all know the quote about technology being indistinguishable from magic. In that law Arthur C. Clarke is describing how a lack of some basic underlying principles means that observers are unable to determine how things work and they seem magical. The same thing can happen in software, something happens when the following peice of code happens. But something magic has happened to make it do so and it’s not obvious to the person developing the software because they are lacking some underlying peice of knowledge.

Now we can say simply that I didn’t understand all of Rails and hence I was missing knowledge and hence it was all magic. I think that would be wrong, whilst I never got completely to grips with all of Rails, I knew the key parts well enough to be able to produce projects that worked. The test for this is reading someone else’s code and applications and trying to figure out what’s happenening and that’s where Rails (and to a small extent Ruby) do magic to solve a problem.

Let’s start with an example. Django has explicit models that declare the types of fields a model has. For example here’s a Django model:

class Poll(models.Model):
    question = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

Django (optionally) creates the SQL from your model and populates the database with it. Here’s a model from Rails.

class Post < ActiveRecord::Base
end

The schema is declared in a migration.

class CreatePosts < ActiveRecord::Migration
  def self.up
    create_table :posts do |t|
      t.string :name
      t.string :title
      [...snipped]

In Rails you write the migration, the schema is created in the database and Rails infers from the database your model (don’t forget Rails will rename your table and try to pluralize it for, thanks Rails).

Now let’s take a scenario where you are working on a project that has over 200 migrations. To tell what’s in your model, you either a) wade through all the migrations that are relevant to your model or b) keep examining your database. It’s not that point b) is bad, it’s important to know what’s going on in your database.

Explicit is better than implicit

In Django I’m explicitly saying what fields I’m expecting on a model, what they are and what they manage. I’m also declaring the methods and attributes on an object - or a namespace. I know what at a glance Poll.question (Django) is, figuring out what Post.name (Rails) isn’t as obvious. Indeed there’s nothing stopping me adding a method to the Post class called name. Rails is doing a looking up for you on the object to calculate the variable.

But it’s doing more than that. For example:

class Sample < ActiveRecord::Base
    def wtf
        foo
    end
end

What does Sample.wtf give you? Does the method foo exist on the object (and presumably in the inheritance tree), does the method foo exist somewhere else in the namespace, does anything else import foo into the global namespace, does foo exist on the database for that model?

To me this is not explicit and hence somewhat magical. That is my definition of magical, something is going to happen that is not obvious or clear. There’s some implicit assumption that you know a whole bunch more information about the code and the context it’s being run in.

As a developer reading this you have to do all that work to figure out what foo is, or do runtime analysis. To me this flies in the face of another rule:

In the face of ambiguity, refuse the temptation to guess.

Rails developers may call not defining the namespace DRY, but I call it laziness and ambiguity.

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

This philosophy goes on through Rails in many different areas, here’s some more examples.

Let’s look at Zope 2 for a minute and Acquisition and DTML. Alright I’m picking at pieces that either no longer exist or are now explicit, but that’s because the developers found way too many issues with the said technologies. Learning from past mistakes and not repeating them is an important thing to be able to do.

In Zope 2 there was a templating language called DTML that you could write on objects. When it was rendered it would act on the object in the database and then use Acquisition to figure out what to do next. Acquisition can be pretty mind bending, it doesn’t mean it was wrong, but to many developers it was just far too magical.

Code that worked in one place, wouldn’t work in another because the context it was being executed in wasn’t right. Not in itself wrong, but very hard to figure out and debug so to many it became magical. Never mind the confusing syntax which differentiated between:

, and

I have to confess I don’t have any concrete examples of this since all that code has long since vanished into a distant past, but I wanted to point out that it’s not just Rails. Training developers in this was great fun, the principle was all well and good, but all too often I saw people resort to random guessing of code to get it working.

Technology is thought of as magical because something is happening that we don’t understand. Frameworks that make decisions for you at runtime based on things you haven’t explicitly declared can be seen as performing some magic. The level of magic has inverse relationship to the level of explicitness. Django is more explicit and much less magical. Rails is less explicit and rather magical. Zope 2 was very magical.

BTW: Yes, I’m quoting the Zen of Python, by Tim Peters. To read the whole thing start your Python and type “import this”.

Forms, forms, forms

Sat, 17 Dec 2011 03:07:13 -0500

Some questions answered about Django forms.

…again based on questions that keep cropping up in the IRC channels.

1. How do I add or change fields on a ModelForm?

The exact same way you do on a normal Form. If the name of the field matches a field that already exists on the model the form is using, it will override the default.

class Cool(forms.ModelForm):
    new = forms.CharField()
    class Meta:
        model = User

2. If I add fields to a ModelForm, the data in those fields does not get saved.

No, they won’t, because they aren’t attached to the model, so you would have to save them manually.

3. Can one model form have more than one model?

No.

4. But my form is quite complicated….

Remember the following:

Django forms are about data validation
… they just happen to also render themselves nicely as some HTML
a ModelForm is a form for one and only one model
a Form is not bound to a model and can be any number of fields
a HTML Form can contain any number of Form or ModelForm instance

So for example, if you make a User Profile, you might have one HTML page to edit user details like email address (in the User model) and Facebook account (in the User Profile model). To do this:

Make two ModelForms, one for each model.
Pass both through to the view, have both render on the same template.
The template will look something like this:


{{ form_one }}
{{ form_two }}

When the form is submitted, all you have to do is check the two forms are valid, something like:

if form_one.is_valid() and form_two.is_valid():
    form_one.save()
    form_two.save()

Although the names are similar, a HTML Form and a Django Form are quite different.

Update: Sep 2009, improving some of the grammar and erm words.

Features of a good Django plugin

Sat, 17 Dec 2011 03:06:20 -0500

Well, my opinion on them anyway.

A while back I wanted to allow openid logins on my site. A simple problem which a few plugins released to the Django community make easier. However there’s quite a few of them (we list 4, there’s more) and none of them quite work the way I want.

So what’s wrong with them? Most of them define front end templates or particular models or a particular way of doing it. I don’t want that, I just want to make my application do openid authentication. So in frustration I have something which is a complete rip off of django-simple-openid from Benoît Chesneau.

When I’m building a website I do the following:

Create a user profile that contains custom user information. In this model I’d like to add in my openid information for the user.
Create an authentication backend that looks up the user profile defined before and logs the user in.
Create custom login, join, signout pages.
Create some middleware.

If I’m going to create all these steps I want a plugin that fits into this pattern. A lot of them end up being like contrib.comments, something you have to fight to get customised. As an example, here’s what I would like to see in an openid plugin:

Let me define my openid field on a model and make that a profile (standard Django practice)
Let me write my custom authentication backend (standard Django practice)
Document that I need to do the following views: login, login failed and signout and tell me the API to call
Document the data I need in forms to pass to the views
Provide an example set of models, views and template that I can choose to ignore
Let me write my own views and template (standard Django practice)

Thinking about it this mirrors the general Django philosophy in many ways. Perhaps a list for what a plugin should have:

No front end user interface
No standard models, just example models
No standard views, just example views
Unit tests
Documentation
An example implementation

The primary job of someone using your plugin is to integrate it with their application. Let’s make that a bit easier.

Disclaimer: the new openid implementation from Simon Willison looks great. Although it does look worryingly complex, it seems to (at my quick glances over the code) accomplish the above.

Other disclaimer: it’s easy to say this standing up on my ivory tower since I haven’t actually released my openid library, because its a hacked up mess. Chances are as soon as I write about this, someone will come along and point out their Django openid plugin which does it just the way I want. I hope so.

Read only fields in models

Sat, 17 Dec 2011 03:04:38 -0500

Supposing you have a field in your model and you don’t want it to be changed, ever.

How would you do that? Well actually you can’t from Django, which we’ll see in a minute. Here’s a script that should work for most people. Basically, set up a signal (or a custom save method) so that when your object is saved it compares the old to new and then figures out if it can be changed.

from django.db import models

class BankAccount(models.Model):
    amount = models.DecimalField(max_digits=20, decimal_places=2)
    number = models.CharField(max_length=10)
    
    read_only = ("number",)
        
def read_only(sender, instance, **kw):
    if instance.id:
        old = sender.objects.get(id=instance.id)
        for field in sender.read_only:
            if (getattr(old, field) != getattr(instance, field)):
                raise ValueError, "The field %s cannot be changed, once set, ever." % field

models.signals.pre_save.connect(read_only, sender=BankAccount)

On a save, we look up the read_only attribute and see if any of those fields have changed and raise an error if they do. In this scenario, you can change the amount of money in a bank account, but the number of the account will not change once set.

Here’s a test to check it works:

from django.test import TestCase                
from models import BankAccount
from decimal import Decimal

class tests(TestCase):
    def testReadOnly(self):
        bank = BankAccount()
        bank.number = "1234567890"
        bank.amount = Decimal("100.00")
        bank.save()
        
        id = bank.id
        
        bank.amount = Decimal("200.00")
        bank.save()
        
        bank.number = "9999999999"
        self.assertRaises(ValueError, bank.save)

This is awfully close to model validation. Something that Django doesn’t have yet. And it there’s two areas it can break down.

When someone writes raw SQL.
The update method on a QuerySet does not call the signal or the custom save method. For example:

        # Warning does not work with update
        BankAccount.objects.all().update(number="999999999")
        # eeek!
        assert BankAccount.objects.get(id=id).number == "999999999"

This is discussed here and here. I wonder if it’s worth going any further on this until model validation comes along. Perhaps not.

Class based views

Sat, 17 Dec 2011 03:01:54 -0500

We discussed these at a recent Vancouver Django (May 2009) meetup and it was also a pattern I used on a recent project.

Traditionally in Django a URL points to a function. So something like this is probably what you familar with:

from django.conf.urls.defaults import patterns

urlpatterns = patterns('',
    (r'^hello-function/$', 'recipe_9.views.hello_function')
)

And a view that looks like this:

from django.http import HttpResponse
        
def hello_function(request):
    return HttpResponse("Hello world!")

As it turns out, Django only needs something that is callable, so this can be a class. Let’s just show the difference. The URLs:

from django.conf.urls.defaults import patterns
from views import hello_class

urlpatterns = patterns('',
    (r'^hello-class/$', hello_class()),
)

The class:

from django.http import HttpResponse

class hello_class:
    def __call__(self, request):
        return HttpResponse("Hello world!")

Oh and let’s not forget some tests to prove this works:

from django.test import TestCase                
from django.test.client import Client

class tests(TestCase):
    def testFunction(self):
        clt = Client()
        res = clt.get("/hello-function/")
        assert res.status_code == 200
        assert res.content == 'Hello world!'
        
    def testClass(self):
        clt = Client()
        res = clt.get("/hello-class/")
        assert res.status_code == 200
        assert res.content == 'Hello world!'

What’s the advantage of this? Well since it’s a class you get all the advantages of being a class, doing things on the __init__, subclassing, overriding the __call__ and so on. Let’s take an example. An extension of the Kenyan project for me recently was a similar project in another country. Similar, but of course, not the same. There were differences in the text and how certain situation are handled. So I made all the views point to classes (at this point I will add I also altered our URLResolver)… anyway all the requests in these projects do that same thing:

parse the user input into a form
validate the user input
process it
return an response

This logic of processing the request in this order has been pushed into a class. All my views inherit from it. So now the code for country X only deals with the specific parts for that country. When I add country Z to the mix, I just need to change the specific parts for that country.

Whilst this may sound a bit specialised, many applications can use this. For example one of the first applications I have takes every request and then checks the account information on that request (it doesn’t use session for other reasons). So the first line of every view is a call to go and get that information. Making a part of template request processor sort of works, but you have to do work to dig that back out.

In the end making it a class based view adds only about 3 lines of code but makes you app much more reusable and adaptable in the future.

References:

Simon Willison in his talk at EuroDjangoCon 2009

Useful Django API's

Sat, 17 Dec 2011 03:00:31 -0500

Two useful Django API’s that I’ve found today.

Model to Dict

A useful little fella hiding away in the forms module. Ever want a dictionary of your model and it’s fields? Then this one will give you that. Useful for mapping a model into a big string. For example in a recent project we are sending a message about the facility.. pass in a facility into a string and you’ll get:

>>> "%(id)s" % facility
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'Facility' object is unsubscriptable

But with add in a model_to_dict call and you’ll get back a dictionary suitable for mapping:

from django.forms.models import model_to_dict
>>> "%(id)s" % model_to_dict(facility)
'1'

Finding models

This one requires a few assumptions. Supposing you want to find an model called Log in your project. You don’t care what the model is, what application it is in, just that there is something called Log. We would presume that Log had all the appropriate API and made sense in the project.

In my scenario, I need to find the table name of log, so that I can run some custom, raw sql on it. I did not want to hardcode the table name. So I needed to find log, to access the db_table. The class from django.db.models.loading import AppCache provides some very useful stuff.

from django.db.models.loading import AppCache

class _models:
    def __init__(self):
        app = AppCache()
        for m in app.get_models():
            setattr(self, m.__name__, m)

models = _models()

This creates a instance models that contains a pointer to the model definitions. I called this resolve.py, sounded like a reasonable name. So now I can do:>

>>> from resolve import models
>>> models.Log

And I’ve got the Log, regardless of where it came from.

Caching and signals

Sat, 17 Dec 2011 02:58:24 -0500

Adding in cached objects on signals.

Was reading this great talk on Django and performance from EuroDjangoCon. There are some good points in there, one I wanted to play with quickly was the caching framework and signals.

I’m a big fan of signals and the idea of setting and deleting the cache as objects change was just something I had to quickly play with. Nothing too revolutionary here, but it simply sets and gets the cache when the object changes.

from django.db import models
from django.core.cache import cache
from django.db.models.signals import post_save, post_delete

def _key(model, id):
    return "%s.%s" % (model.__name__, id)

class Car(models.Model):
    make = models.CharField(max_length=255)
    
    def key(self):
        return _key(self.__class__, self.pk)

def create_cache(sender, **kw):
    cache.set(kw["instance"].key(), kw["instance"])

def delete_cache(sender, **kw):
    cache.delete(kw["instance"].key())

post_save.connect(create_cache, sender=Car)
post_delete.connect(delete_cache, sender=Car)

So each time the object is created or deleted, the cache gets updated (and thanks to being signals, simple to reuse for any class). And here’s a test case.

from django.test import TestCase
from django.core.cache import cache
from models import Car

class CacheTest(TestCase):
    def setUp(self):
        car = Car()
        car.make = "Avensis"
        car.save()
        
        self.car = car
        self.id = car.pk
        
    def testCar(self):
        key = self.car.key()
         
        assert cache.has_key(key)
        assert cache.get(key).make == "Avensis"
        
        self.car.make = "Auris"
        assert cache.get(key).make == "Avensis"
        self.car.save()
        assert cache.get(key).make == "Auris"
        
        self.car.delete()
        assert not cache.has_key(key)

As an improvement (and indeed the next recipe in my book) it does this through an ObjectManager so that you can just call:

Car.objects.cache(id)

And get the cached object back, setting the object in the cache if it’s not already there. There are some problems, including possible race conditions, as slide 30 of the presentation does point out. But it’s an interesting start and definitely something I need to play with a bit more.

References

Django, Malnutrition, SMS and Kenya

Sat, 17 Dec 2011 02:57:00 -0500

How I ended up doing one of most interesting and important projects I’ve been involved with.

A while back I got home from dinner to find a quick Skype chat from a friend, Nate Aune that basically went “you free to go to Kenya to do some Django - oh and you have to leave on Sunday” (that was in about 4 days time). The response was “is this for real?”. But as it turns out it was and about 6 days later I was leaving on a plane to Kenya, to work on a Django project.

The person behind this project is Matt Berg, who works at Columbia University on the Millennium Villages project. This particular project was already in the works and was based on excellent work from Schuyler Erle, I was to provide support on site and add in some new features.

The goal of this particular Django project is to help health care workers resolve cases of malnutrition (and as it turned out malaria and other diseases) faster and help more children get treatment faster. To do this, the project uses cell phones.

So when a health care worker using the project finds a child with malnutrition, they enter the details into a their cell phone and send a text message to a phone number. For all you twitterers out there, you will realize we hit the downside, those details entered ideally need to be under 140 chars.

Set up is a modem listening to that number, it feeds into a Ruby daemon called spomsky which is part of the RapidSMS project. That pushes a request over to a RapidSMS project written in Django and that’s where it get’s relevant to this blog.

RapidSMS is a cool project that “is UNICEF’s open source platform for data collection, logistics coordination and communication allowing any mobile phone to interact with the web”. You can get the source here: http://rapidsms.org/.

Once the message comes in, it hits a Django process, unlike most Django web applications there are two processes running. The first is a RapidSMS process called route and this is a process that connects to spomsky and listens for text’s coming in. The second is the standard Django runserver which provides a web interface for managers to see what is going on.

So the message is evaluated, for example: does this child one of the levels of malnutrition that warrant attention. If it does then we send a message back to the health care worked asking for the child to be sent to the clinic. It will also route messages to the clinic and to the local health care co-ordinator.

This routing is done using a very clever decorator that allows you to assign what text message goes to what method, eg:

 @keyword(r'n(?:ote)? \+(\d+) (.+)')
    @authenticated
    def note_case (self, message, ref_id, note):         
        [...]

Sending out a message is a simple in RapidSMS as doing:

    message.respond("Thanks")

In this example, you can send in a text message that reads (for example):

note +12 has malnutrition and needs care now

Where +12 is the child’s ID. The + was chosen to signify an ID, because on the particular phones used, the + is easy to get to without hitting too many buttons. Real messages would be more complicated and involve use of a MUAC measurement for example:

muac +26 7.5 2150 1.4 n

That rather cryptic text would give you back (new lines added for readability):

MUAC> SAM Patient requires OTP care. 
  +26 MADISON, M, F/4 (Sally). 
  MUAC 75 mm, 2.1 kg, 140 cm

These responses would go to health care facilitator’s or co-ordinator’s or to the clinics as relevant. Not only then does this give people the notification to follow up and make sure that the children get the treatment, but allows recording of the data.

On the back end in the office, there’s that second Django process which serves out a front end, that allows them to search, find and get data on the users. As that UI evolves it will let the users look up that data by clinic, district and so on.

This might sound simple, but for all those incoming messages, we wrote a lot of tests. I’ve done some mission critical code before, but knowing that I could miss a child with a serious medical condition, made me even more paranoid than normal. RapidSMS includes the really nice ability to write tests in a simple way.

For example here’s an example of an incoming malaria test of a child from phone number 7654321 (signified by the >). It is then sent to back to the reporter and also to a supervisor at 7654322 (signified by the <)

7654321 > mrdt +26 y n f
7654321 < MRDT> Child +26, MADISON, Molly, F/4 has MALARIA. 
    Child is 4. Please provide 2 tabs of Coartem (ACT) 
    twice a day for 3 days
7654322 < MRDT> Child +26, MADISON, Molly, F/4 (None) has MALARIA. 
    CHW: @jdoe 7654321

And that’s it. For about 6 days I worked with Matt adding in new features, new forms and new tools for the health care workers. On one day I got the chance to go out into the field and watch the health care workers get trained. It was great to see the health care workers pick this system up quickly and get used to using the system. The first few times sending messages took a little bit of time, but after a while they picked up quickly and really got the hang of the system and really seemed to appreciate it.

A brief aside here, the use of cell phones for this is absolutely brilliant. In Kenya there are lots and lots of cell phones. I won’t go as far to say everyone has one, but lots do. One image that struck me was a farmer clearing a field and taking a call on his cell phone at the time. Africa skipped the whole “stringing copper wires from poles” step that Western societies took. In Canada on every street corner there’s a Starbucks. In Nairobi it’s a cell phone store or pre-paid top up.

And that’s it, a few days later I was on a plane back over to Canada (well 4 planes). Leaving some people with the task of driving adoption and training users in the field how to use the project.

Of all the projects I’ve done, it’s got the potential for being the most important project I’ve done - and that’s really cool. And the real potential of RapidSMS spreads way beyond just this project. The ability to set up a sophisticated distributed system quickly using SMS has so many possible applications. It could great for inventory management, disaster management, anything. With RapidSMS and Django it becomes a real possibility to set this system up quickly. Of course these sorts of systems have existed in the past, but I doubt they have been set up as quickly and easily as RapidSMS allows.

A big thanks by the way to Matt and Jess and other Millenium Villages people for looking after this muzungu while I was in Africa. Also thanks to Schuyler for doing lots of great work and helping out on RapidSMS, spomsky and git.

Note: The personal details of getting to Kenya and back and can be found over on my personal blog, the focus here is on the technical side. Further if you looking for details on the efficiency of malnutrition programs or this program in general, I don’t have them either, I don’t know anything about that, Django stuff only.

Turning off django signals

Sat, 17 Dec 2011 02:55:49 -0500

While I’m on the subject, how to turn off signals that you don’t want on.

The first and most obvious signal to turn off is the one asking you for a username and password on syncdb, gosh that’s annoying. You can do this by calling:

python manage.py syncdb --noinput

But if you want to do this in Python you can:

from django.contrib.auth import models as auth_app
from django.contrib.auth.management import create_superuser
from django.db.models.signals import post_syncdb
post_syncdb.disconnect(create_superuser, 
       sender=auth_app, 
       dispatch_uid = "django.contrib.auth.management.create_superuser")

The part that was annoying was figuring out just how to get it to disconnect, to find that dispatch_uid I had to go and read the source, not onerous at all. My previous post was about signals vs save. To all that I’ll add an annoyance with signals: if you load in fixtures the, signals get called. This is annoying because I wanted to do the following:

load in a set of users from fixtures
load in a set of profiles from fixtures

Unfortunately all the profiles failed, because the profiles are all created when the users are imported. To solve this on my test or reset scripts I need to turn off the creating a profile signal:

from django.db.models.signals import post_save
from django.contrib.auth.models import User
from users.models import create_profile # this is where my signal is registered
post_save.disconnect(create_profile, sender=User)

I will chalk that up as definite disadvantage for signals, perhaps there’s a more cunning way around that.

Signal vs overriding save

Sat, 17 Dec 2011 02:53:47 -0500

I’m a big fan of signals and wonder why people are so adverse to using them, preferring overriding save instead.

Probably every day on the Django IRC channel, someone asks about how to set a user on a model or about setting the time automatically. Fortunately thanks to this excellent article, we are all familiar with the former. The latter is easy to do as well, without having to resort to using the now deprecated (maybe) auto_now and auto_now_add.

Here’s how to do it overriding the save:

from django.db import models
from datetime import datetime

class Todo(models.Model):
    text = models.CharField(max_length=255)
    completed = models.BooleanField(default=False)
    timestamp = models.DateTimeField()
    
    def save(self, *args, **kw):
        if not self.id:
            self.timestamp = datetime.now()
        super(Todo, self).save(*args, **kw)

And here’s how to do it with a signal:

from django.db import models
from datetime import datetime

class Todo(models.Model):
    text = models.CharField(max_length=255)
    completed = models.BooleanField(default=False)
    timestamp = models.DateTimeField()
 
def add_date(sender, instance, **kw):
    instance.timestamp = datetime.now()

models.signals.pre_save.connect(add_date, sender=Todo)

Setting up a signal is surprisingly easy and one line less of code. Is it more expensive from a CPU point of view? Not really. Here’s a script that ran the two models:

from chapter_2.recipe_3.models import Todo
from time import time

start = time()
for x in range(0, 10000):
    todo = Todo()
    todo.text = "Get some milk %s" % x
    todo.save()
    
diff = time() - start
print "Time: %s" % diff

Running this with signals:

Time: 6.97824287415

Running this with save:

Time: 6.53473687172

I re-ran it a few times and never got a significant difference between the two. Sometimes the signals were faster. I think I can be reasonably sure that signals (in this simple scenario) do not add significant performance cost. So what’s the problem with signals?

The advantages:

The signal can be applied to any number of models, without having to repeat override save all over the place. Admittedly in the above example, if you made a base class and override save there, you wouldn’t have the problem.
Can be applied to models you don’t necessarily have access to or want to modify.
Overriding methods make me uncomfortable. The save is pretty obvious, but it’s easy to end up with a model that has a bunch of magic methods. Renaming one of them will cause grief as you are no longer overriding the correct one. And once I forgot to call super in the save, but that’s just my own incompetence.

The disadvantage:

You can end up with a lot of signals and forget what signal occurs when (easier to lose track than lots of save’s in the model inheritance)

Since creating a new signal and assigning to just one specific model is very easy, the argument “use save when its specific to the model” seems moot. It’s just as easy to make a signal as the save.

Where to put signals?

If it’s a signal related solely to one model, I recommend on the model
If it’s a signal that is used on multiple places, I recommend a signals.py module (just make sure to add it to __init__.py so that’s imported)

Content mirror and this site

Sat, 17 Dec 2011 02:52:35 -0500

How this site is configured and some more details on content mirror.

I’ve been asked for this a couple of times, so here’s some details on how Content Mirror works on this site.

This site’s main editing interface for editors (myself) is done in Plone. Plone is mirroring it’s content into a Postgres database, which is being synced to a Django site, you are then reading it in something served by Django.

Server configuration

My Plone site is running in a Virtual Machine on my Mac. It is in fact an out of the box Plone 3.x, that’s easy to install and set up. To that installation I added in Content Mirror and psycopg. For this I’m using Postgres, it would work with any database, but I much prefer Postgres.

Next I created a Postgres database locally. I did this locally because I wanted to develop the site and wanted to install and play with everything locally. Developing locally is fast and simple.

So I’ve got Plone and Postgres installed locally with Content Mirror pushing content to Postgres. With that up and running I tried it out by adding in a few documents, publishing them and checking that everything worked.

So here’s how the “About page” of Djangozen, you’ll notice it is vanilla out of the box Plone:

Building a front end

Once I’ve got everything going into the database, I can start to add in a front end. For this I chose Plango and then started modifying the heck out of it. Plango is here and is a front end in Django. It has a simple URL resolver that simply grabs the request, then looks that up in the database. So /about becomes a lookup in the database for a piece of content with no parent and an id of about.

There’s certainly some opportunity for optimization in those URL look ups, but we’ll about that later. The nice thing about having that Django front end is that it’s a snap to add in comments, simple openid login and so on without doing any Plone development at all.

The plugin’s section is added to and edited by end users, so that’s a Django app (that’s embarrasingly simple).

Deployment

Deployment of this site was actually very easy. I have a Webfaction account I use for various non-critical things. It’s cheap and easy to set up - I’m running 4 sites on it at the moment for under $10 a month.

The Django code went into SVN. And then I made Django app on my server, copying down the code I’d developed locally. Then I added in Postgres database.

To move data up is pretty simple, I stopped the local postgres and made an ssh tunnel. Restart Plone and it’s now writing to the remote postgres over the ssh tunnel. My script looks something like this:

sudo /etc/init.d/postgresql-8.3 stop
ssh -L 5432:server:5432 andymckay@server
sudo /etc/init.d/postgresql-8.3 start

The first time the connection is made, just run the bulk deployment script and the content is uploaded. After that all content will be incrementally deployed.

Gotchas

There’s currently a bug in Content Mirror that means the ATEvent content type has none of the extra fields (such as startdate, enddate, location…). Only I seem to get this issue at the moment however.
Changing the Postgres that the Plone site is talking to a lot can cause Plone to give errors, so try not to flip the Postgres database too often via the ssh tunnel.
If you have a lot of content that is being added to the site on the front end, you loose a lot of the Plone advantage. For example I want people to be able to add events to Django zen and have a review step… there isn’t a backwards step, I can’t review content in Plone.
Got to watch out for review states for things like navigation, RSS feeds, search and so on, if you want only published content to be viewed.
Things will be written to the database without Django knowing about. So, for example, Plango has a full text search. A cron job runs every 5 minutes, looks for things that have been updated and adds or removes them from the full text search, as needed. A database trigger works well, until I found Webfaction doesn’t allow them.
You need to fight Kupu to get it not to resolve uid’s in HTML so that if you embed an image in a document (such as this one) you can unresolve it.
Not everything in Plone is going to be translated, but that’s part of the cost.

Summary

For Djangozen it’s probably overkill. Although writing this blog post in Kupu, adding images etc has been quite a pleasure compared to the Django admin interface. But now most of the above gotchas have been resolved, it’s quite simple and quick to set this up and end up with a good site in no time.

Update: this doesn’t happen any more, it’s in tumblr now.

Keeping user profiles in sync

Sat, 17 Dec 2011 02:40:53 -0500

Django provides a user profile as a way to extend a users information. This isn’t created automatically for you however when you create a new User, but it is pretty simple to do this.

The easiest way to do this is to use signals. By listening to the post_save signal on users, you can create a user profile on a save.

When the user object is deleted, the profile will be deleted, because the delete cascades. To use this, don’t forget to register in the class UserProfile into AUTH_PROFILE_MODULE (see settings at the end):

from django.db import models
from django.db.models.signals import post_save
from django.contrib.auth.models import User
from django.core.exceptions import ObjectDoesNotExist

class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True)

def create_profile(sender, **kw):
    user = kw["instance"]
    if kw["created"]:
        up = UserProfile(user=user)
        up.save()

post_save.connect(create_profile, sender=User)

And here’s a test to demonstrate it working.

from django.test import TestCase
from models import UserProfile
from django.contrib.auth.models import User

class UserTest(TestCase):
    def testUser(self):
        u = User.objects.create(
             username='admin',
             email='andy@clearwind.ca')

        u.save()
        
        assert u.get_profile()
        assert User.objects.count() == 1
        assert UserProfile.objects.count() == 1
        
        u.delete()
        
        assert User.objects.count() == 0
        assert UserProfile.objects.count() == 0

Since this got asked on the django IRC channel I figured it had enough reason to justify a blog post.

References:

Django at PyCon

Sat, 17 Dec 2011 02:39:37 -0500

A wrap up of the happenings at PyCon.

Sadly I didn’t get to PyCon this year. In fact I haven’t ever been to PyCon, just what used to be the “Python Conference” which I think has been overtaken by PyCon.

Looking at the chat and IRC logs makes me realise how many people I know there and wish I’d been able to go. Next year. After being around the Zope, then Plone, then Django communities, it’s amazing how many people I knew and how many people those communities touched.

It also seemed this year that Django was a big component, perhaps due to the uptake in its popularity over the last year. Some links from what I’ve found:

Jacob’s State of Django talk (pdf)
James Tauber talks about Pinax (pdf)
Manfred Schwendinger talks about Google App Engine (pdf)
Jacob talks about Django’s design decisions (pdf)
Carlos de la Guardia talks about Content Mirror to serve Plone out through Django. Which is exactly what this site is doing. (pdf)
Django on Jython by Jim Baker and Lee Soto (odp)

Some of the blog posts I found:

James Bennett blogs on an ORM panel which included Django
Alex Gaynor’s PyCon wrap up
Jacob gave a talk on “Real World Django” (which on the live monitoring point I would like to add in Arecibo)

For the sheer humour and gall of standing up in front of an audience and letting anyone on IRC type you have to watch this talk. It’s not really about Django and is a bit off towards the end, but Ian Bicking is worth listening too. And the IRC chat is great.

The most important thing though was of course:

The Django pony going missing
The pony being held for ransom
The pony slaying the Pinax team and escaping
Phew!

Site launched

Sat, 17 Dec 2011 02:37:34 -0500

Djangozen is now live in some sort of state, hopefully working.

There’s nothing more boring than that first “oh this site is live” post. Usually because the normal developer person has gone into marketing mode and really thinks the other people care about the site. I know you care. How could you not with new Django sites going live every day.

Well here’s another.

It covers things I want to know about Django above and beyond what’s on the main website. There’s lots of other sites out there that cover this stuff, but I wanted to start off with two things:

Plugins. There’s a great site out there called djangoplugables, it has loads of products on it. However there’s a few features I wanted. You can’t add plugins or releases to it (i think it only indexes google code), there’s no categorisation and no RSS feed. Once I’ve got tagging working, those features are all here.
Clear Django. This is a book I’m starting to write that is a recipe style online book. It hasn’t got far as of the time of writing but it will all be online.

Of course all this is just empty words, with sites coming and going all the time, the real test is to come back in 6 months and see if I’m still here and doing this. Then you can really start to take notice.

In the meantime please bear with me as I finish off the features, clean up the typos and clean up the user interface.

djangozen

Faster or lazier pagination

The power of Q

Complex lookups

List of lookups

Pickling lookups

Javascript inline forms outside of contrib.admin

Adding Javascript to contrib.admin inline fields

Setting up inline forms in contrib.admin

Tuning the count method on a queryset

The lack of magic in Django

Forms, forms, forms

Features of a good Django plugin

Read only fields in models

Class based views

Useful Django API's

Finding models

Caching and signals

Django, Malnutrition, SMS and Kenya

Turning off django signals

Signal vs overriding save

Further reading

Content mirror and this site

Server configuration

Building a front end

Deployment

Gotchas

Summary

Keeping user profiles in sync

Django at PyCon

Site launched