Part IV · Templates and forms

Chapter 13: Templates

Django’s template system turns the name of a template and a dictionary of values into a string. It compiles a template’s text into a tree of nodes, by default once, and renders the tree against a fresh context each time it is asked, escaping for HTML as it goes. Around that language sits a thin layer of pluggable template engines, configured by TEMPLATES, through which a view asks for a template by name.

A view that answers with a page has a template’s name, "fleet/port_detail.html", and some values: a port and its ships. Something has to find the file, which may be in a directory of the project or in the templates directory of any installed application; make sense of its text, in which {{ port.name }} stands for a value and {% for ship in ships %} for a loop; and write out a string with the values put in, each escaped so that a ship named <Beagle> prints as text and not as markup.

Reading and parsing the file for every request would waste most of the work, since the text does not change between requests and the values do. So the system does the work in two stages. Compiling reads the text and builds a tree of objects from its pieces of text, its variables and its tags. With the default loaders it is done once for each template, and the result is kept. Rendering walks that tree with a context, the values for this one response, and is done every time. Nearly everything in the chapter belongs to one stage or the other.

In this chapter

13.1 Template engines: TEMPLATES and the backends django.template.engines, an EngineHandler, makes a template backend, a subclass of BaseEngine, for each entry of the TEMPLATES setting the first time it is asked for. get_template, select_template and render_to_string, of django.template.loader, ask the backends in turn and gather their failures into one TemplateDoesNotExist. Django ships three: DjangoTemplates, which holds an Engine, one for Jinja2, and TemplateStrings.
13.2 Finding a template: Engine and the loaders Engine holds the configuration of the Django template language, and Engine.get_template turns a name such as "fleet/port_detail.html" into a compiled template. The engine asks its loaders in order; a loader proposes the places the template might be, reads the first that exists and compiles it; and the cached loader, which by default wraps the others, keeps what they find.
13.3 From text to nodes: the lexer and the parser A Django template is compiled as its Template is made: Template.compile_nodelist cuts the source into tokens with a Lexer and builds the tree of nodes with a Parser, which hands a block tag to the compile function registered under its first word (Template.compile_nodelist). A compile function is given the parser and its tag’s token, may drive the parser on to its end tag, and returns a node.
13.4 Variables and filters: FilterExpression and Variable A variable tag such as {{ ship.name|upper }} is compiled into a FilterExpression: the value it starts from, a Variable or a constant, and the filters to apply. At render time Variable._resolve_lookup finds the value in the context, as a key, an attribute or an index, calling it if it can be called, and FilterExpression.resolve passes it through the filters, or decides what a missing variable becomes.
13.5 The context: Context, RequestContext and context processors A template renders against a context, a stack of dictionaries in which a name is looked up from the top down. BaseContext is the stack; Context adds the flags a render reads and the template it is bound to; RequestContext adds the output of the engine’s context processors, functions of the request, which make_context puts below the view’s own values.
13.6 Rendering: nodes and the render context Rendering walks a compiled template’s tree of nodes with a context and returns the output as one string. Each node renders its piece and NodeList.render joins them; VariableNode makes a value into text with render_value_in_context; and what a node must remember during the walk it keeps, ordinarily, in the RenderContext.
13.7 Autoescaping and safe strings Django’s template engine escapes the values it writes into a template’s output for HTML, unless a value, made a string, is safe: a SafeData, such as the SafeString that mark_safe, format_html or a filter returns. The mark is made in django/utils/safestring.py and the escaping in django/utils/html.py; the engine applies them in render_value_in_context, in its filters, and as the "autoescape" option and {% autoescape %} direct.
13.8 Inheritance and inclusion: {% extends %}, {% block %} and {% include %} {% extends %} and {% block %} let a template be written as changes to another, and {% include %} renders one template inside another. All three are tags of django.template.loader_tags, and both kinds of composition happen as the page renders: an ExtendsNode renders its parent’s tree with the most derived version of each block, taken from a BlockContext, and an IncludeNode renders a second template where the tag stands.
13.9 Template partials: {% partialdef %} and {% partial %} A template partial is a named fragment of a template. {% partialdef %} defines it while the template is compiled, {% partial %} renders it wherever the template asks, and a name such as "fleet/port_detail.html#ship_row", given to Engine.get_template or anything that loads through it, stands for the fragment alone, a PartialTemplate.
13.10 Libraries: Library, custom tags and {% load %} A library is a Library, a dictionary of tags and one of filters, kept in a module’s variable "register". The DjangoTemplates backend imports every such module of django.templatetags and of the installed applications when it is made, {% load %} copies a library into the parser of the one template being compiled, and simple_tag, simple_block_tag and inclusion_tag make a tag of an ordinary function.
13.11 The built-in tags The built-in tags of django.template.defaulttags are in every Django template without {% load %}: how {% for %} builds its loop variables, {% if %} parses a condition with django/template/smartif.py and {% url %} reverses a name, and how {% cycle %}, {% ifchanged %}, {% regroup %} and {% querystring %} work.
13.12 The built-in filters The built-in filters are the functions of django/template/defaultfilters.py, from |upper and |date to |default and |length, which every template has without {% load %}. Each is a plain function of a value and at most one argument, most of them registered with flags that tell FilterExpression.resolve how to treat its input and its output.
13.13 The libraries to load: static, i18n, l10n, tz and cache The package django.templatetags holds the template libraries of Django’s core that are not built in: a template has their tags after {% load static %}, {% load i18n %}, {% load l10n %}, {% load tz %} or {% load cache %}, or where an engine’s "builtins" option names one. Each is a thin layer of nodes over a service from elsewhere in Django: {% static %} over the static files storage or STATIC_URL, {% translate %} and {% blocktranslate %} over translation, {% localize %} and {% timezone %} over formats and time zones, and {% cache %} over the cache.
13.14 SimpleTemplateResponse and TemplateResponse SimpleTemplateResponse and TemplateResponse are responses whose content is a template not yet rendered. They keep the template and the context, render them when SimpleTemplateResponse.render is called, run their post-render callbacks after it, and raise ContentNotRenderedError if their content is read before it.
13.15 Template errors and debug information TemplateDoesNotExist and TemplateSyntaxError, the exceptions of django/template/exceptions.py, and the debug information the Django template language puts on an exception when the engine’s "debug" option is on: the template, the line and the failing tag, at compile time and at render time.

Two layers in one package

django.template holds two systems that its own docstring calls independent. The first is a set of template engines. Each entry of the TEMPLATES setting names a backend, a subclass of BaseEngine, and the object engines, an EngineHandler made when django.template is imported, holds one backend for each entry, made at its first use. Django ships DjangoTemplates, for its own language; Jinja2, for Jinja2; and TemplateStrings, which fills in plain text with the standard library’s string.Template. The functions of django/template/loader.py (get_template, select_template, render_to_string) ask each backend in turn for a template by name, and whatever template comes back is rendered by a method that takes a plain dictionary and, optionally, the request.

The second is the Django template language, Django’s own engine. Its centre is Engine, which holds the configuration: the directories, the loaders, the libraries of tags and filters, and options such as "debug" and "string_if_invalid". A DjangoTemplates backend makes one Engine from its entry of TEMPLATES, asks it for every template it is asked for, and wraps each compiled template it returns in a small object that adapts the language’s Template.render, which takes a Context, to the engine layer’s, which takes a dictionary.

The objects of the language, in the order a request meets them:

Table 13.1 The objects of the Django template language. Template here is Template; the backend’s wrapper of the same name is a different class.

object	what it is	made
`Engine`	what it is the language’s configuration and the entry to it	made with its backend
a loader	what it is finds a template’s source by name and returns it compiled	made at the engine’s first use of its loaders
`Template`	what it is a compiled template: its source, where it came from, and its tree of nodes	made once for each template, kept by the cached loader
the lexer and the parser	what it is cut the text into tokens and build the tree from them	made for each compile
`Library`	what it is a module’s registry of tags and filters, which the parser draws on	made when its module is first imported, at the latest as an engine that uses it is made
a node	what it is one piece of the tree: a run of text, a variable, or a tag	made when the template is compiled
`Context`	what it is the values a render can see, as a stack of dictionaries	made for each render
`RenderContext`	what it is a second stack, for the nodes’ own state	made with each context

Compiling: from text to a tree of nodes

A template is compiled when its Template is made: the constructor calls Template.compile_nodelist, which runs a lexer and then a parser over the source. Take this template:

{# fleet/port_detail.html #}
<h1>{{ port.name }}</h1>
{% if ships %}<ul>{% for ship in ships %}<li>{{ ship.name|upper }}</li>{% endfor %}</ul>{% endif %}

The lexer knows the delimiters and, of the tags, only {% verbatim %}, inside which it turns everything up to the matching end tag into text. It splits the text with one regular expression that matches {% … %}, {{ … }} and {# … #}, and makes a token of each piece: a text token for what lies between tags, a variable token for {{ … }}, a block token for {% … %}, and a comment token for {# … #}, each holding its contents with the delimiters stripped.

The parser turns the flat list into a tree. It takes the tokens in order. A text token becomes a TextNode. A variable token becomes a VariableNode, holding a FilterExpression that has already parsed ship.name|upper into a variable and a list of filters. A comment token becomes nothing. A block token’s first word names a tag, and the parser looks that word up in its table of compile functions and calls the function with itself and the token. The function returns a node, and on the way it may ask the parser to go on parsing until one of the end tags it expects: the function for if calls Parser.parse with "elif", "else" and "endif", and gets back the nodes in between. That recursion is how the tree gets its depth, and it is why an end tag leaves no node of its own. A tag the table does not hold is an error at compile time, with the template’s line.

Figure 13.1 The lexer cuts the text at every tag into a flat list of tokens. The parser builds the tree: the compile functions of if and for each parse the tokens up to their end tag and keep the nodes they get back, so the end tags leave no node. (The comment’s line and the file’s last newline are left out.)

The table of compile functions is filled from libraries. Every Engine has three built in, the modules django.template.defaulttags, django.template.defaultfilters and django.template.loader_tags, and any library it is given as a builtin; their tags and filters are in every template. Other libraries, such as those in the templatetags package of an application, are known to the engine by name and reach a template’s parser only through {% load %}.

Finding a template, once

Engine.get_template asks the engine’s loaders for the name, in order, and the first that finds it wins. Unless "OPTIONS" names loaders of its own, an engine has one: the cached loader, wrapped around the filesystem loader, which looks in the directories of "DIRS", and, when "APP_DIRS" is true, the application directories loader, which looks in each installed application’s templates directory. The source is read and the Template made, and so compiled, as soon as the name is found. The cached loader keeps the compiled template under a key made from its name, and the next request for that name gets the same object without reading or parsing anything. A compiled template is therefore shared by every request for it that the process serves, which is why a node must keep what it learns during a render out of itself, in the context or the render context.

Rendering: walking the tree with a context

The backend’s template is given a dictionary and the request. It makes a Context with the dictionary on top, or, when there is a request, a RequestContext, and calls the compiled template’s render.

A context is a stack of dictionaries. A name is looked up from the top down, and a tag that introduces names for its body, as for and with do, pushes a dictionary for them and pops it when its body is done, so that they vanish at its end tag. A RequestContext also runs the engine’s context processors, functions of the request that each return a dictionary of names (the CSRF token, the user, the request itself), when the template starts to render, and puts their names below the view’s.

Template.render then renders its NodeList: each node renders itself to a string, and the strings are joined. A TextNode returns its text. A VariableNode asks its FilterExpression for a value, which looks the variable up in the context (each part of its dotted name tried as a key, then as an attribute, then as a number used as an index, and what it finds called if it can be called, unless it is marked not to be) and passes it through the filters; it then turns the value into a string, localised, and escaped for HTML unless the context has autoescaping off or the value is already marked safe. A tag’s node does what the tag means: the IfNode renders the nodes of the first condition that holds, the ForNode renders its body once for each ship. With two ships named Endeavour and <Beagle>, "fleet/port_detail.html" renders as "\n<h1>Valletta</h1>\n<ul><li>ENDEAVOUR</li><li><BEAGLE></li></ul>\n": the comment leaves only its line’s newline, and the second name has gone through upper and then been escaped.

What the nodes need to remember while one template renders, such as the position of a {% cycle %} or the blocks of an inheritance chain, they ordinarily keep in the context’s render_context (Rendering: nodes and the render context), a second stack that Template.render pushes as it starts and pops as it ends, so that each template rendered through it starts with an empty level of its own. Inheritance itself is a matter of rendering: in a template that begins with {% extends %}, everything after the tag is parsed into one ExtendsNode, which at render time finds the parent template and renders the parent’s tree in its place, with the child’s blocks put where the parent’s stand (Inheritance and inclusion: {% extends %}, {% block %} and {% include %}).

Where templates meet the rest of Django

A view usually reaches templates through django.shortcuts.render, which calls render_to_string and puts the string in a response (The shortcuts), or by returning a TemplateResponse, which holds a template, or its name, and a context, and is rendered by the handler after the template-response middleware have seen it (The centre of the chain). The development server’s autoreloader watches the template directories and empties the loaders’ caches when a template changes (The autoreloader), and the system checks ask each engine for its own (The system checks). When the engine’s "debug" option is on, an error in a template carries the template’s lines to the debug page (The debug pages and ExceptionReporter). Forms and their widgets render through templates by way of a form renderer (Rendering a form: renderers and templates). The tags for translation, static files and the cache are libraries of django.templatetags, each loaded by name, and the machinery behind them belongs to Internationalization and time zones, Content types, static files and the other contrib apps and Caching, files, mail, signals and tasks.