20.8 Static file storages: ManifestStaticFilesStorage and hashed names

StaticFilesStorage

StaticFilesStorage is a FileSystemStorage whose location and base URL default to STATIC_ROOT and STATIC_URL (File storage: Storage, FileSystemStorage and storages). Its __init__ calls check_settings, which raises ImproperlyConfigured for an empty base URL, media settings equal to the static ones, and, while settings.DEBUG is on, a MEDIA_URL inside STATIC_URL. Without STATIC_ROOT a StaticFilesStorage is still made: since FileSystemStorage would fall back to MEDIA_ROOT, it puts the empty location back, writes URLs, and raises ImproperlyConfigured in StaticFilesStorage.path for whatever needs a file, making a ManifestStaticFilesStorage among them, since that reads its manifest.

Hashed names

HashedFilesMixin.file_hash gives the first 12 hex digits of the MD5 of the content. HashedFilesMixin.hashed_name puts the hash before the extension and keeps the name’s query string and fragment. Given no content, it opens the file of that name in the storage itself, and raises ValueError when there is none.

The patterns and the ignored blocks

HashedFilesMixin.patterns pairs a glob of file names with the expressions that find references: for "*.css", url(…), @import "…" and a source map’s comment; for "*.js", a source map’s comment. Each is given a template that writes the reference back in one form, so @import "base.css"; became @import url("base.932853cb9945.css"); in harbour.css, and the expression of the blocks to ignore: comments, escape sequences and quoted strings, and in JavaScript line comments and template literals too. Setting HashedFilesMixin.support_js_module_import_aggregation adds expressions for JavaScript modules’ imports and export … from.

HashedFilesMixin.get_ignored_blocks finds those blocks in a file, and a match that begins strictly inside one is left alone; one that begins where a block begins is rewritten, as a source map’s comment, itself a comment, needs.

Rewriting a reference: url_converter

HashedFilesMixin.url_converter makes the function each substitution calls with a match. It leaves the match alone when it begins in an ignored block; when the URL begins with a scheme in lower case, data: or https:, or with //; when it is an absolute path outside STATIC_URL; and when it has no path, as a bare fragment. Otherwise it finds the file meant: under STATIC_URL, by the rest of the path; a relative URL from the directory of the file being rewritten, so ../img/anchor.png in notes/harbour.css means img/anchor.png. HashedFilesMixin._url, with force=True so that settings.DEBUG makes no difference, gives its URL through HashedFilesMixin._stored_name: the name recorded so far in this collection, or else the hash of the file as it stands in the storage. Only the last segment is replaced, so a relative reference stays relative.

For a file that does not exist, the ValueError gets a note naming the file being rewritten, the text matched and its line. Handed a file notes/notice.css whose third line refers to ../img/buoy.png, which nothing provides, HashedFilesMixin.post_process yielded:

# post_process's ValueError for notes/notice.css, with its note
The file 'img/buoy.png' could not be found with <django.contrib.staticfiles.storage.ManifestStaticFilesStorage object at 0x00000229BE83FED0>.
'notes/notice.css' contains this reference 'url("../img/buoy.png")' on line 
3:     background: url("../img/buoy.png");

post_process and its passes

HashedFilesMixin.post_process is given every collected path with the storage and path a finder found it at, and does nothing on a dry run. A file that matches a glob of the patterns is adjustable. A pass is a call of HashedFilesMixin._post_process, which takes the files deepest directory first and opens each where the finder found it, not the copy, which, a comment says, might be far away, on S3 for instance. A file starts from the name recorded for it so far or, the first time, the name hashed from its original content. One that is not adjustable is saved under that name unless the name exists. An adjustable file is read as UTF-8, rewritten, and saved under a name hashed from the rewritten content; if HashedFilesMixin.keep_intermediate_files is true, also under the name it started from, an intermediate file.

The first pass goes over every file. Passes over the adjustable files follow, each rewriting the originals with the names recorded so far, until one changes no name, at most HashedFilesMixin.max_post_process_passes, 5. A reference to a stylesheet not yet processed gets the hash of its unprocessed copy, so a file can need another pass once the files it refers to have changed; files that refer to each other, the documentation adds, never converge. Names still changing after the last pass are yielded with a RuntimeError, “Max post-process passes exceeded.”; errors in decoding and rewriting a file are yielded too, not raised, and collectstatic stops at the first. Files that are not adjustable are yielded during the first pass, the others at the end with their final names.

In the run, the first pass saved base.css, which refers to nothing, as notes/base.932853cb9945.css and anchor.png as img/anchor.b357a19c8762.png; harbour.css, whose original content hashes to a10e57d9c6ea, was rewritten to refer to them and saved as notes/harbour.f60ecf1ef40a.css. The second pass got the same content for both stylesheets, so the same names, and was the last.

A file’s URL: stored_name

HashedFilesMixin.url answers through HashedFilesMixin._url, which hashes nothing while settings.DEBUG is true unless given force=True, and otherwise asks the storage’s stored_name.

HashedFilesMixin.stored_name, without a manifest, looks in the names this storage has recorded, by HashedFilesMixin.post_process or an earlier lookup, and otherwise works the name out from the files: it hashes the file, then the file of the hashed name, until a file’s name is its own hash. The intermediate files make that chain: after collecting with a storage of HashedFilesMixin over StaticFilesStorage, a fresh one went from notes/harbour.css to the intermediate notes/harbour.a10e57d9c6ea.css, holding the rewritten content, then to notes/harbour.f60ecf1ef40a.css, which hashes to itself.

The manifest: ManifestFilesMixin

ManifestFilesMixin keeps the names in a file instead, and sets keep_intermediate_files to false. Its ManifestFilesMixin.manifest_name is "staticfiles.json", kept in the storage itself unless manifest_storage= names another. __init__ reads it with ManifestFilesMixin.load_manifest: the "paths" and "hash" of a manifest of version "1.0" or "1.1", nothing without a file, ValueError for anything else. It is read once, when the storage is made, and storages makes one storage for an alias in a process: a server process that made its storage before collectstatic ran keeps the names it read.

ManifestFilesMixin.post_process empties the names, runs the passes, and, unless on a dry run, calls ManifestFilesMixin.save_manifest, which sorts the pairs, hashes their JSON with HashedFilesMixin.file_hash for the manifest’s own hash, and replaces the file:

# STATIC_ROOT/staticfiles.json, one line; the GIS and charts files' entries left out
{"paths": {..., "img/anchor.png": "img/anchor.b357a19c8762.png", "notes/base.css": "notes/base.932853cb9945.css", "notes/harbour.css": "notes/harbour.f60ecf1ef40a.css"}, "version": "1.1", "hash": "11564bbb3aff"}

ManifestFilesMixin.stored_name looks the name, without its query string and fragment, up in the manifest’s names. A missing name raises ValueError, “Missing staticfiles manifest entry for …”, while ManifestFilesMixin.manifest_strict is true, as by default; when it is false, the name is hashed from the file in the storage. The {% static %} tag asks the "staticfiles" storage for a URL when the static files application is installed (The libraries to load: static, i18n, l10n, tz and cache).