This problem is current as of Django 1.3 and has no generic solution as far as I know.

A class_prepared signal is dispatched when the model class is instantiated for the first time; this happens inside class Model’s metaclass at django.db.models.base: 257:

        signals.class_prepared.send(sender=cls)

Therefore, we would expect that connecting to this signal from the __init__.py of an app which appears first in the INSTALLED_APPS list, would ensure that our listener receives the signal for all Models loaded in the remainder apps (following is the beginning of a typical such list):

INSTALLED_APPS = (
  'class_prepared_listener_app',
  'django.contrib.auth',
  'django.contrib.contenttypes',
  'django.contrib.sessions',
  'django.contrib.sites',
  'django.contrib.messages',
  'django.contrib.admin',
  ...
)

However, this is not (always) the case. Tracing, for example, shell execution, we find that django.core.management.commands.shell: 45:

from django.db.models.loading import get_models
loaded_models = get_models()

requests all models from the app cache which first calls _populate to initialize itself at django.db.models.loading: 167:

self._populate()

which loads all apps into the cache the first time it’s called and does nothing for subsequent calls at django.db.models.loading: 58:

            for app_name in settings.INSTALLED_APPS:
                if app_name in self.handled:
                    continue
                self.load_app(app_name, True)
            if not self.nesting_level:
                for app_name in self.postponed:
                    self.load_app(app_name)
                self.loaded = True

The apps are loaded in _populate in the order they’re listed in INSTALLED_APPS; some apps can be postponed though if they can’t be imported at the time of traversal at django.db.models.loading: 77:

        try:
            models = import_module('.models', app_name)
        except ImportError:
            self.nesting_level -= 1
            # If the app doesn't have a models module, we can just ignore the
            # ImportError and return no models for it.
            if not module_has_submodule(app_module, 'models'):
                return None
            # But if the app does have a models module, we need to figure out
            # whether to suppress or propagate the error. If can_postpone is
            # True then it may be that the package is still being imported by
            # Python and the models module isn't available yet. So we add the
            # app to the postponed list and we'll try it again after all the
            # recursion has finished (in populate). If can_postpone is False
            # then it's time to raise the ImportError.
            else:
                if can_postpone:
                    self.postponed.append(app_name)
                    return None
                else:
                    raise

A print statement in _populate indicates that no apps are postponed and that all apps are, in fact, loaded in sequence. Furthermore the signal gets connected before any other apps are loaded. The output of manage.py shell with several print statements:

Signal connected.
class_prepared_listener_app
django.contrib.auth
django.contrib.contenttypes
django.contrib.sessions
django.contrib.sites
django.contrib.messages
django.contrib.admin
...

Adding a print statement in the Model’s metaclass indicates that some classes are instantiated before their owner apps are loaded:

Preparing class 'django.contrib.contenttypes.models.ContentType'
Preparing class 'django.contrib.auth.models.Permission'
Preparing class 'django.contrib.auth.models.Group_permissions'
Preparing class 'django.contrib.auth.models.Group'
Preparing class 'django.contrib.auth.models.User_user_permissions'
Preparing class 'django.contrib.auth.models.User_groups'
Preparing class 'django.contrib.auth.models.User'
Preparing class 'django.contrib.auth.models.Message'
Preparing class 'django.contrib.sites.models.Site'
Signal connected.
Preparing class 'django.contrib.sessions.models.Session'
Preparing class 'django.contrib.admin.models.LogEntry'
...

Further snooping shows that these classes are instantiated by translation-related preparation code, before the shell command is properly handled, at django.core.management.base: 202:

        # Switch to English, because django-admin.py creates database content
        # like permissions, and those shouldn't contain any translations.
        # But only do this if we can assume we have a working settings file,
        # because django.utils.translation requires settings.
        if self.can_import_settings:
            try:
                from django.utils import translation
                translation.activate('en-us')

Models are instantiated inside the last call to activate() which eventually calls translation() at django.utils.translation.trans_real: 161:

        for appname in reversed(settings.INSTALLED_APPS):
            app = import_module(appname)
            apppath = os.path.join(os.path.dirname(app.__file__), 'locale')

            if os.path.isdir(apppath):
                res = _merge(apppath)

From the above, it is straightforward to see that by removing the admin and auth apps, or by placing our app after the admin and auth apps, our signal connects before any Models are loaded. Neither of these two “solutions” however appear robust or welcome.

It should be noted that, because the call to translation.activate() is being made inside the execute method of the base class from which all management commands derive, this behaviour is to be expected when Django starts up through commands other than shell.

Admittedly, the official Django documentation for this signal states “Django uses this signal internally; it’s not generally used in third-party applications.” however, I strongly feel that Django would benefit from a formal and deterministic load order for INSTALLED_APPS.

For instance, consider the following two Models:

class A(Model):
    pass

class B(Model):

    a = OneToOneField(A)

    value = IntegerField()

    def __unicode__(self):
        return str(self.value)

And the following sequence of statements:

>>> a = A.objects.create()
>>> b = B.objects.create(a = a, value = 69)
>>> b.value
69
>>> a.b.value
69
>>> b.value = 42
>>> b.save()
>>> b.value
42
>>> a.b.value
69

Clearly, the two instances of B are different objects, and updating one has no effect on the rest, as witnessed by the different values on lines 10 and 12.

It is possible to automatically propagate field changes to all aliases of a Model by (a) keeping track of aliases, and (b) propagating upon assignment. Assuming you have a list of all aliases of a Model instance (see below), then field changes from a reference alias to all the remainder target aliases can be achieved in the following straightforward way.

def propagate_field_update(reference, field_name, targets):
    value = getattr(reference, field_name)

    for target in targets:
        setattr(target, field_name, value)

Another option is to update all fields together (for instance, after a save signal, as done below); this can be achieved in the following way.

def propagate_field_updates(reference, targets):

    mapping = [(f.attname, getattr(reference, f.attname)) for \
        f in reference._meta.fields]

    for target in targets:
        for (f_attname, v) in mapping:
            setattr(target, f_attname, v)

Note, however, that updates to fields which are not handled through Django’s Model fields will not be captured using the above technique.

The following mini-framework uses post initialization and post save signal listeners to maintain weak references of aliases, as well as propagates field changes to all aliases using post save and post delete signals.

from django.db.models.signals import post_init, post_save, post_delete
from collections import defaultdict
from weakref import WeakValueDictionary

MODEL_INSTANCES_ALIASES = defaultdict(WeakValueDictionary)

def record_alias(sender, instance, **kwargs):
    """ Keeps a weak reference to all aliases of a Model instance, using the
        class and primary key as a key to a default dictionary of weak value
        dictionary entries, which in turn use the id of aliases mapped to weak
        references to the aliases themselves, in the following manner:

        { (Class, PK) : { id : instance, id : instance, ... }, ... }

        Recording will happen post initialization, but also post save in the
        event that a Model instance is created directly from a class (and not
        the create Manager method), and hence does not yet feature a primary
        key.
    """

    if instance.pk is not None:
        aliases = MODEL_INSTANCES_ALIASES[(sender, instance.pk)]
        if not aliases.has_key(id(instance)):
            aliases[id(instance)] = instance

post_init.connect(record_alias)
post_save.connect(record_alias)

def propagate_updates_to_aliases(sender, instance, **kwargs):
    """ Propagates all field values of a given Model instance to all its aliases
        post save and delete. Can also be invoked explicitly, or used in a
        decorator for methods which update fields.
    """

    mapping = [(f.attname, getattr(instance, f.attname)) for \
        f in instance._meta.fields]

    # Propagate field values to all aliases.
    for target in MODEL_INSTANCES_ALIASES[
        (instance.__class__, instance.pk)].itervalues():
        # Ignore the given instance.
        if not id(target) == id(instance):
            for (f_attname, v) in mapping:
                setattr(target, f_attname, v)

post_save.connect(propagate_updates_to_aliases)
post_delete.connect(propagate_updates_to_aliases)

Using the above technique, the example becomes as follows.

>>> a = A.objects.create()
>>> b = B.objects.create(a = a, value = 69)
>>> b.value
69
>>> a.b.value
69
>>> b.value = 42
>>> b.save()
>>> b.value
42
>>> a.b.value
42

Which demonstrates that the two aliases have consistent field values.

I’m currently experimenting with a mini-framework for propagating field updates as they happen. I am also planning on determining the kind of performance and memory impact this has on realistic Django production code; more updates in the future.

1. the instance object of the invoker (the subject),
2. the name of the method (the name), and
3. the invocation frame in the current stack (the frame).

This information is passed to the invoked method through a keyword argument, by default _invoker_info, which can be overridden through the name argument to the decorator.

import inspect

from functools import partial, wraps

def capture_invoking_method_info(name = '_invoker_info'):
    """ Decorator which captures invoking method information and places it in
        the keyword arguments of the decorated method, by default as the keyword
        argument '_invoker_info' which can be set via the 'name' argument. The
        information captured consists of the subject (who contains the method
        which is invoking), the invoking method name, and the stack frame of the
        invocation.
    """

    def _capture_controller(wrapped_method):

        def _capture_controlled(*m_args, **m_kwargs):

            frame = inspect.currentframe().f_back

            # If the invocation is from the shell, or the invoker frame's
            # locals are missing, the subject becomes None.
            if len(frame.f_code.co_varnames) > 0:

                subject = frame.f_locals[frame.f_code.co_varnames[0]]

            else:

                subject = None

            m_kwargs[name] = (subject, frame.f_code.co_name, frame)

            return wrapped_method(*m_args, **m_kwargs)

        return wraps(wrapped_method)(_capture_controlled)

    return partial(_capture_controller)

The following example demonstrates its use. Assume a class A which defines two methods, x and y, where x invokes y, and y is decorated with the above decorator.

class A(object):

  def x(self):
    return self.y()

  @capture_invoking_method_info()
  def y(self, _invoker_info):
    return _invoker_info

class B(A):
  pass

An instance object b of class B will return the following captured information tuple when its x method is invoked:

>>> b.x()
(<__main__.B object at 0x10162ee90>, 'x', <frame object at 0x101560090>)

And, since we are in the interactive python shell, the same object b will return the following captured information tuple when its y method is invoked:

>>> b.y()
(None, '<module>', <frame object at 0x101562750>)

Obviously, the subject here is None, as the invocation was performed from the shell and not another instance object.

This problem is current as of Django 1.3 and has no generic solution as far as I know.

Assuming the following Model:

class Category(models.Model):

    related = models.ManyToManyField('self', symmetrical = False, \
        through = 'CategoryRelation', null = True, blank = True)

Assuming the following intermediate ‘through‘ relation:

class CategoryRelation(models.Model):

    source = models.ForeignKey('Category', null = False, blank = False, \
        related_name = 'relation_source')

    target = models.ForeignKey('Category', null = False, blank = False, \
        related_name = 'relation_target')

    order = models.PositiveIntegerField(_('order'), default = 0, \
        null = False, blank = True)

    class Meta:
        ordering = ( 'order', )

How can we obtain the Category objects related to a given Category while preserving their ordering? The following code will produce the correct Category objects, not in the correct order, and include duplicate entries in the Query Set even when using .distinct():

relations = CategoryRelation.objects.filter(source = self)

Category.objects.filter(relation_target__in = relations).order_by('related')

The following works for ordering correctly, but does not leave out duplicate entries:

relations = CategoryRelation.objects.filter(source = self)

Category.objects.filter(relation_target__in = relations).order_by('relation_target')

Calling .distinct() will not make a difference, because .order_by(…) is applied by the ORM after the SQL SELECT DISTINCT clause is built. However, it is possible – in this case – to exploit the fact that the order is a positive integer field, and annotate each Category with the Min value of the order field of the relation_target field, and use this new annotation field for ordering:

return Category.objects.filter(relation_target__in = relations).annotate(
    relation_target_order = models.Min('relation_target__order')).order_by(
        'relation_target_order')

This is almost complete, but since the semantics of this query essentially make it unique, it would be wise to call .distinct() just to make sure the distinct flag is True so that later combinations with other distinct queries can take place:

return Category.objects.filter(relation_target__in = relations).annotate(
    relation_target_order =models.Min('relation_target__order')).order_by(
        'relation_target_order').distinct()

In this case .distinct() does not affect the query in the slightest, but ensures that db/models/sql/query.py method combine(self, rhs, connector) passes its assertion:

assert self.distinct == rhs.distinct, \ ...

This problem is current as of Django 1.3 and has no generic solution as far as I know.

This problem will manifest when two or more different Models instances are attached to instances of the Model with the Generic Foreign Key. In the following example, we have Model X which has a Generic Foreign Key, and two Models, Y and Z, which have Generic Relations to X and ordering based on these relations.

from django.db.models import CharField

from django.contrib.contenttypes.models import ContentType

from django.contrib.contenttypes.generic import GenericForeignKey, GenericRelation

class X(Model):
    """ This Model has a unique name and is attached to another Model through a
        Generic relation.
    """

    # The unique name of this X.
    name = CharField(u'name', unique = True, max_length = 128, null = False)

    # The Content Type of the generic relation to the attached content.
    content_type = models.ForeignKey(ContentType, null = True, blank = True,
        related_name = 'x_contents')

    # The ID of the attached content.
    content_id = models.PositiveIntegerField(null = True, blank = True)

    # The attached content.
    content = GenericForeignKey('content_type', 'content_id')

    def __unicode__(self):
        """ The textual representation of this Model is its name.
        """

        return self.name

    class Meta:

        app_label = 'cmf'

        # Order by X's name.
        ordering = ['name', ]

class Y(Model):
    """ This Model acts as content for an X, and has a convenient reverse
        Generic Relation to X.
    """

    # The unique name of this Y.
    name = CharField(u'name', unique = True, max_length = 128, null = False)

    # The X this Y is attached to.
    xs = GenericRelation(X, content_type_field = 'content_type',
        object_id_field = 'content_id')

    def __unicode__(self):
        """ The textual representation of this Model is its name.
        """

        return self.name

    class Meta:

        # Order by X's name.
        ordering = ['xs__name', ]

class Z(Model):
    """ This Model acts as content for an X, and has a convenient reverse
        Generic Relation to X.
    """

    # The unique name of this Z.
    name = CharField(u'name', unique = True, max_length = 128, null = False)

    # The X this Z is attached to.
    xs = GenericRelation(X, content_type_field = 'content_type',
        object_id_field = 'content_id')

    def __unicode__(self):
        """ The textual representation of this Model is its name.
        """

        return self.name

    class Meta:

        app_label = 'cmf'

        # Order by X's name.
        ordering = ['xs__name', ]

The following sequence of statements demonstrates this problem:

x1 = X.objects.create(name = 'first x')
x2 = X.objects.create(name = 'second x')
x3 = X.objects.create(name = 'third x')
x4 = X.objects.create(name = 'fourth x')
y1 = Y.objects.create(name = 'first y')
y2 = Y.objects.create(name = 'second y')
z1 = Z.objects.create(name = 'first z')
z2 = Z.objects.create(name = 'second z')
x1.content = y1
x1.save()
x2.content = z1
x2.save()
x3.content = y2
x3.save()
x4.content = z2
x4.save()
X.objects.all()
[<X: first x>, <X: fourth x>, <X: second x>, <X: third x>]
Y.objects.all()
[<Y: first y>, <Y: first y>, <Y: second y>]

Notice that ‘first y’ appears twice. The SQL COUNT statement will not have any ordering applied, and so returns the correct value:

Y.objects.count()
2

However, when ordering is applied, the duplicate instance is present:

len(Y.objects.all())
3