Chapter 17: The utility layer

Three ways to wait

django/utils/functional.py gives Django three ways to put off work, each with its own object, and they differ in when the work is done and whether its result is kept.

A lazy value is the proxy that a function made by lazy returns. lazy(func, str) returns a function that does not call func: it keeps func’s arguments in a new instance of a class that lazy built, a subclass of Promise, and returns that. The class has the methods of the result classes it was given, str here, and each of them calls func with the kept arguments and then acts on the result. The result is not kept, and the docstring of lazy says so: each use calls the function again. This is how django.utils.translation.gettext_lazy and reverse_lazy are made: a message translated in the language active when it is used, and a URL reversed when the URLconf can be read.

A lazy object stands in for an object that is made on its first use and then kept. A LazyObject holds the object in LazyObject._wrapped, which starts as a marker called empty; the first time an attribute is read, set or deleted on it, or one of the special methods it forwards is used, it calls LazyObject._setup, which a subclass writes to put the object there, and from then on it passes those straight to that object. SimpleLazyObject is a LazyObject made from a function that returns the object. django.conf.settings is a lazy object (The settings object), as are the default storage and, under the authentication middleware, the user that a request carries.

A cached property is computed once for each instance. cached_property is a descriptor that a class holds in place of a method. The first read of the attribute on an instance calls the method and writes the result into the instance’s own __dict__, under the attribute’s name; and since the descriptor defines neither __set__ nor __delete__, Python looks in the instance’s dictionary before it asks the descriptor, so every later read finds the value there and never reaches the method. Deleting the attribute from the instance makes the next read compute it again.

The harbour office’s notice board, a module of the example project, has one of each. FLEET says how many ships the harbour has, HOME is the home port’s row, and a Harbour has a list of the ships:

# harbour/office/board.py
from django.utils.functional import SimpleLazyObject, cached_property, lazy

from harbour.fleet.models import Port, Ship


def fleet_size():
    return f"{Ship.objects.count()} ships in harbour"


FLEET = lazy(fleet_size, str)()
HOME = SimpleLazyObject(lambda: Port.objects.get(name="Valletta"))


class Harbour:
    @cached_property
    def ships(self):
        return list(Ship.objects.order_by("name"))

A run of the example, with the database holding the ports Valletta and Piraeus and two ships, and the number of SQL statements each line sent beside it:

import harbour.office.board                             0 queries
str(FLEET)                   '2 ships in harbour'       1 query
str(FLEET)                   '2 ships in harbour'       1 query
(a third ship is saved)
str(FLEET)                   '3 ships in harbour'       1 query
HOME.name                    'Valletta'                 1 query
HOME.country                 'Malta'                    0 queries
isinstance(HOME, Port)       True                       0 queries
type(HOME).__name__          'SimpleLazyObject'         0 queries
harbour.ships                a list of 3 ships          1 query
harbour.ships                the same list              0 queries
"ships" in vars(harbour)     True                       0 queries
del harbour.ships
harbour.ships                a new list of 3 ships      1 query

Importing the module sent nothing: each of the three was made without its function being called. FLEET counted the ships every time it was made a string, and so saw the third ship once it was saved. HOME fetched its row on the first attribute read and kept it, and it passes for a Port: LazyObject answers __class__ from the wrapped object, so isinstance sees the row, while type, which does not ask the object, sees the proxy. The harbour’s list lived in the instance’s dictionary until it was deleted from there.

Lazy values, keep_lazy (which lets a function given a lazy argument return a lazy result) and the helpers built on lazy are in Lazy values: lazy, Promise and keep_lazy; the proxy’s machinery, its copying and its pickling in Lazy objects: LazyObject and SimpleLazyObject; and the descriptor, with classproperty and classonlymethod, in cached_property, classproperty and classonlymethod.

Code named by a string

A project’s settings name code by its dotted path, as text: each entry of MIDDLEWARE, the "BACKEND" of each cache, storage and task backend, the authentication backends, the password hashers. import_string turns such a path into the object. It first tries to import the whole path as a module; then it imports the module up to the path’s last dot and returns that module’s attribute of the last name, raising ImportError if there is none, so that a path naming a module returns the module, and a submodule wins over an attribute of the same name. Both are new in 6.2. The middleware chain is built by calling it on each entry of MIDDLEWARE (Building the middleware chain).

autodiscover_modules imports a module of a given name from every installed application, as the admin does for each application’s admin.py, and passes over an application that has no such module; an error raised inside a module that does exist is raised again. Django also asks a callable what it accepts before it calls it, through django/utils/inspect.py: in debug mode Signal.connect refuses a receiver that has no parameter for arbitrary keyword arguments, and Paginator.count calls an object list’s count only if it is a method, not a built-in one, that takes no arguments, and otherwise takes the list’s length. Both modules have their sections: Importing by name: import_string and autodiscover_modules and Asking a callable what it accepts: django/utils/inspect.py.

The shapes of data

A query string or a posted form may give one name several values, though most forms give each name one. A MultiValueDict is a dict whose values are lists: indexing it gives the last value under a key, and MultiValueDict.getlist gives them all. The request’s QueryDict is one (QueryDict). Beside it in django/utils/datastructures.py are CaseInsensitiveMapping, a mapping whose keys are looked up whatever their case, on which a request’s and a response’s headers are built; OrderedSet, which keeps distinct items in the order in which they were added; and ImmutableList, a tuple that answers most of a list’s methods for changing it with an AttributeError and a message of its maker’s choosing (Data structures: MultiValueDict and its neighbours). The choices of a model field and of a form field, which may be given as pairs, groups, a mapping, an enumeration or a callable, are made into one shape by normalize_choices (Choices: normalize_choices and the choice iterators).

Node, the tree of django/utils/tree.py, is what a Q and a query’s where clause are built from: a list of children, a connector that joins them, and a mark that negates the whole (filter, exclude and Q objects).

Text, values and paths

force_str makes anything a str, decoding bytes as UTF-8 unless told otherwise, and force_bytes makes anything bytes; smart_str and smart_bytes do the same but return a lazy value unchanged, so that a lazy translation stays lazy. The same module turns an internationalized resource identifier into a URI and back, and a file’s path into the path of a URL (Text and bytes: django/utils/encoding.py). django/utils/text.py has the helpers that the template filters and the model and storage code call for text: Truncator, slugify, get_valid_filename, capfirst and others (Text helpers: Truncator and django/utils/text.py).

Values arrive as text, from a database that stores dates as text, as SQLite does, or from a serialized fixture, and Django reads them with django/utils/dateparse.py. parse_datetime tries the standard library’s datetime.datetime.fromisoformat first, and Django’s own pattern only where that fails; for text in neither form it returns None, and for text that Django’s pattern reads but that names an impossible date it raises ValueError. parse_duration reads three forms of a duration (Dates, durations and addresses: parse_datetime, parse_duration and clean_ipv6_address). safe_join joins paths and refuses a result outside the first of them, which is how a storage and a template loader keep a name from reaching outside their directory (Paths and archives: safe_join, safe_makedirs and extract).

The exceptions

django/core/exceptions.py defines the exceptions its docstring calls global, many of them raised in one part of Django and caught in another. The exception classes that each model that is not abstract is given, raised when a lookup finds no row or more than one, derive from ObjectDoesNotExist and MultipleObjectsReturned (ModelBase: a class statement becomes a model). SuspiciousOperation and its subclasses, DisallowedHost and RequestDataTooBig among them, mark what may be an attack. Raised inside the middleware chain, one of them becomes a 400 and is logged under "django.security"; PermissionDenied becomes a 403 and BadRequest a 400 (How an exception becomes a response). ImproperlyConfigured says that a setting or the code is configured wrongly. ValidationError carries one error, a list of them, or a dictionary of lists by field name, each error with a message and a code (Validation: full_clean, cleaned_data and the errors). EmptyResultSet and FullResultSet are raised while a query is compiled, when a condition can match no row or every row (The where clause: build_filter, add_q and WhereNode). SynchronousOnlyOperation is raised by a function marked with async_unsafe, such as BaseDatabaseWrapper.connect, when it is called in a thread where an event loop is running, unless the environment variable "DJANGO_ALLOW_ASYNC_UNSAFE" is set. Each is in Django’s exceptions: django.core.exceptions.

Deprecations and versions

Django removes a feature in two steps. A release deprecates it: the feature still works, and its use raises a warning whose class names the release in which it will be removed. A later release removes it. At the pin there are two such classes, RemovedInDjango2028Warning, a DeprecationWarning, and RemovedInDjango2029Warning, a PendingDeprecationWarning. Python’s default filters hide a PendingDeprecationWarning wherever it is raised, and a DeprecationWarning unless it is attributed to the module run as the main program; otherwise a warning of either class is seen only where a filter, such as the interpreter’s -W option, shows it.

Two of the module’s helpers decide how a deprecated feature warns. warn_about_external_use warns only when the code that used the feature is outside Django: it walks up the stack to that caller, and stays silent when the caller is Django’s own code, so that one deprecated feature built on another does not set off a second warning. deprecate_posargs lets a function make arguments keyword-only while still accepting them by position for a time, with a warning (Deprecation: warnings named by year, warn_about_external_use and deprecate_posargs).

django.VERSION is a VersionTuple, at the pin (6, 2, 0, "alpha", 0), and django.__version__ the string get_version makes from it, here "6.2.dev20260926051415": for an alpha of iteration 0 whose django package is in a git repository, ".dev" and the time, in UTC, of the commit checked out there follow the number. From Django 2028 a version has no minor number, and VERSION four parts, (2028, 1, "final", 0) for 2028.1; the attributes VersionTuple.feature, VersionTuple.patch, VersionTuple.status and VersionTuple.iteration count from the end, where the two forms agree, and reading the tuple by index past its second part is deprecated (Django’s version: VersionTuple and get_version).

Where the other modules of django/utils/ are told