17.2 Lazy objects: `LazyObject` and `SimpleLazyObject`

A LazyObject stands in for an object that is made on its first use and then kept: it holds the object in LazyObject._wrapped, which starts as the marker empty, fills it by calling LazyObject._setup, and forwards attribute reads and a fixed set of special methods to it. SimpleLazyObject is the lazy object made from a function.

Django names some objects before it makes them: the settings and the default storage at the top of a module, a request’s user as each request comes in. A lazy object passes for such an object, makes it at its first use and keeps it, where a lazy value calls its function at every use (Lazy values: lazy, Promise and keep_lazy). Python gives a proxy two ways in: an attribute it lacks can be caught in general, by __getattr__, but a special method, the one behind == or len, is looked up on the type and never reaches __getattr__, so the proxy forwards only those its class names. The example is HOME of The utility layer, a SimpleLazyObject of a lambda that fetches the port Valletta.

The marker, `_setup` and assignment

empty is a bare object(). LazyObject.__init__ puts it in _wrapped; LazyObject._setup raises NotImplementedError, and a subclass writes it to assign the object to _wrapped. The comment in __init__ warns that a subclass overriding it will likely need its own copying methods: LazyObject copies an empty lazy object by calling type(self)(), with no arguments.

_wrapped is also a class attribute, None, so that tracing __init__ does not recurse (ticket 19456): a tracer may read an attribute, in the ticket __class__, before __init__ has set _wrapped, and with no _wrapped anywhere the read would fall to __getattr__, which reads _wrapped, and so on. With the class attribute deleted, a run raised RecursionError.

Assignment and deletion go through to the wrapped object. LazyObject.__setattr__ writes _wrapped into the proxy’s own __dict__ and, for any other name, sets the object up and sets the attribute on it; LazyObject.__delattr__ refuses _wrapped with TypeError and deletes anything else from the object. Assigning HOME a country sent the query and changed the Port.

Two ways in: `getattr` and the special methods

new_method_proxy makes every forwarding method of LazyObject but __setattr__ and __delattr__, as a function that takes the proxy, calls _setup if _wrapped is empty, calls the given function on the wrapped object, and is marked _mask_wrapped = False. LazyObject.__getattr__ is new_method_proxy(getattr), called when the ordinary lookup fails, as it does for any name the proxy does not have itself: HOME’s __module__ is the proxy’s, "django.utils.functional". The special methods are class attributes made the same way: __str__, __bytes__, __bool__, __dir__, __hash__, __eq__, __ne__, __lt__, __gt__, __getitem__, __setitem__, __delitem__, __iter__, __len__ and __contains__.

Figure 17.2 The two ways into a lazy object, an attribute through __getattr__ and a special method through the proxy’s class, each setting the object up while _wrapped is empty.

The special methods are also attributes of the proxy, so an ordinary read would find them, and hasattr would call a port iterable. LazyObject.__getattribute__, which every attribute read passes through, raises AttributeError for what the ordinary lookup finds marked _mask_wrapped = False, so that, as its comment says, __getattr__ answers from the wrapped object:

hasattr(type(HOME), "__iter__")  ->  True
hasattr(HOME, "__iter__")        ->  False   (1 query: HOME is set up)
iter(HOME)                       ->  raises TypeError: 'Port' object is not iterable
HOME.__eq__                      ->  <bound method Model.__eq__ of <Port: Valletta>>

What is not forwarded

A special method the class does not name falls to object’s, or to none. TONNAGE is a SimpleLazyObject of the fleet’s total tonnage, 601:

TONNAGE + 1       ->  602
1 + TONNAGE       ->  602
TONNAGE <= 700    ->  raises TypeError: '<=' not supported between instances of 'SimpleLazyObject' and 'int'
TONNAGE * 2       ->  raises TypeError: unsupported operand type(s) for *: 'SimpleLazyObject' and 'int'
f"{TONNAGE}"      ->  '601'
f"{TONNAGE:>5}"   ->  raises TypeError: unsupported format string passed to SimpleLazyObject.__format__
callable(SimpleLazyObject(lambda: len))  ->  False

Addition works because SimpleLazyObject adds __add__ and __radd__; __le__ and __ge__ are not forwarded; a format with no specification falls to str; callable asks the type, which defines no __call__.

__class__ is forwarded, as a property made by new_method_proxy, because, the comment says, objects that care about the class, in equality tests above all, must see the wrapped one. isinstance tries the type and then __class__, so isinstance(HOME, Port) is true while type(HOME) is SimpleLazyObject.

Copying and pickling

LazyObject.__reduce__ sets the object up and has pickle write unpickle_lazyobject with the wrapped object as its argument, a function that returns its argument. The comment above it, citing ticket 25389, gives the reason: the faked __class__ confuses pickle, and a proxy acting as the wrapped object outright would still have its own identity, a distinct object with the same __dict__. Pickling HOME sent its query and gave back a Port; the lambda, which pickle cannot write, was never pickled; and a list of HOME and its Port came back holding one object twice. The migration serializer reads a lazy object’s value from the same arguments (Serializing values: serializer_factory and deconstruct).

LazyObject.__copy__ and LazyObject.__deepcopy__ copy the wrapper while it is empty, by type(self) (not __class__, which the comment notes is proxied), and after setup the wrapped object, so that copy is not lazy. SimpleLazyObject overrides both, keeping the two cases; with HOME:

copy.copy(HOME), HOME empty   ->  a new SimpleLazyObject of the same lambda
  the copy and HOME used      ->  two queries, two Port objects
copy.copy(HOME), HOME set up  ->  <Port: Valletta>, a copy of the Port

`SimpleLazyObject`

SimpleLazyObject.__init__ writes the function straight into the instance’s __dict__ as "_setupfunc", where an assignment would have gone through __setattr__ to the wrapped object; SimpleLazyObject._setup calls it and stores the result. The docstring asks that the function be safe to run more than once and return the same value, since Django may copy the proxy, and each empty copy runs it.

SimpleLazyObject.__repr__ does not set the object up: it shows the function while the proxy is empty, and the object after.

repr(HOME), empty    ->  <SimpleLazyObject: <function <lambda> at 0x000001B67121E560>>
repr(HOME), set up   ->  <SimpleLazyObject: <Port: Valletta>>

A function that is a bound method of the proxy is shown only by its name, to avoid recursion, as the comment says: a bound method’s repr holds its object’s, and with the check removed a run raised RecursionError.

SimpleLazyObject.__copy__ and SimpleLazyObject.__deepcopy__, while the proxy is empty, make a SimpleLazyObject of the same function, not an instance of the subclass, whose __init__ may take other arguments. So an empty copy of an unused Truncator wraps its text, and its chars raises AttributeError: 'str' object has no attribute 'chars'.

Django’s lazy objects

Table 17.2 Django’s lazy objects: the first four subclass LazyObject, the rest are or subclass SimpleLazyObject.

object, and what makes it	its first use makes	told in
`django.conf.settings`, `LazySettings` django/conf/__init__.py	its first use makes a `Settings` from `"DJANGO_SETTINGS_MODULE"`, unless `LazySettings.configure` came first	told in The settings object
`default_storage`, `DefaultStorage` django/core/files/storage/__init__.py	its first use makes the `"default"` storage	told in File storage: `Storage`, `FileSystemStorage` and `storages`
`staticfiles_storage`, `ConfiguredStorage` django/contrib/staticfiles/storage.py	its first use makes the `"staticfiles"` storage	told in Collecting static files: the collectstatic command
the admin’s `site`, `DefaultAdminSite` django/contrib/admin/sites.py	its first use makes the class `SimpleAdminConfig.default_site` names	told in Registering models: `AdminSite.register` and `ModelAdmin`
a request’s nonce, `LazyNonce` django/utils/csp.py	its first use makes `generate_nonce()`	told in Content Security Policy: `ContentSecurityPolicyMiddleware` and the nonce
a `Truncator`, `Truncator` django/utils/text.py	its first use makes its text as a string	told in Text helpers: `Truncator` and `text.py` django/utils/
`HttpRequest.user`, `AuthenticationMiddleware` django/contrib/auth/middleware.py	its first use makes the user `get_user` finds	told in The user of a request: `login`, `logout` and `AuthenticationMiddleware`
the context’s `"csrf_token"`, `csrf` django/template/context_processors.py	its first use makes the token from `get_token`, or `"NOTPROVIDED"`	told in The context: `Context`, `RequestContext` and context processors
a test response’s resolver_match, `Client.request` django/test/client.py	its first use makes the path resolved	told in The test client: `Client` and `ClientHandler`

Putting empty back in _wrapped makes a lazy object lazy again, which Django does to both storages when a test overrides STORAGES, STATIC_ROOT or STATIC_URL.