Chapter 16 · Caching, files, mail, signals and tasks
16.8 File storage: Storage, FileSystemStorage and storages
How Django saves a file’s content under a name and finds it again: the handler storages, which makes one storage for each alias of STORAGES; the base class Storage, which decides the name a file is saved under; and FileSystemStorage, which keeps files in a directory on the server.
Writing the bytes is the easy part of saving a file; the name is the hard part. It may come from a browser, with odd characters or a path that climbs out of the storage’s directory; another file may have it already; and it must fit the column that will keep it. Storage settles the name in methods its backends inherit; a backend stores the bytes under it, and answers for it.
STORAGES and the handler
STORAGES maps each alias to a dictionary that names a backend class under "BACKEND", with any "OPTIONS". Its default has two aliases, "default", a FileSystemStorage, and "staticfiles", the static files’ storage (Static file storages: ManifestStaticFilesStorage and hashed names).
The handler storages is a StorageHandler. It copies the setting into StorageHandler the first time a storage is asked for, and StorageHandler makes an alias’s storage with StorageHandler once and keeps it, so the process has one storage for each alias, shared by every thread. create_storage imports the class and calls it with the "OPTIONS" as keyword arguments, ignoring any other key. An alias the setting lacks, or a class that will not import, raises InvalidStorageError, an ImproperlyConfigured.
django is a DefaultStorage, a LazyObject that looks up the "default" alias at first use and stands for it from then on (Lazy objects: LazyObject and SimpleLazyObject); a django given no storage keeps it (File fields). A test’s override of STORAGES empties the handler and the lazy storages (Overriding settings).
Storage: the name in the base class, the bytes in the backend
Two public methods of Storage hand the work to private ones that the base class does not define: Storage.open calls _open, and Storage.save, once it has settled the name, calls _save. A comment above them says they should not be overridden unless absolutely necessary; a backend writes the private pair. The naming is the base class’s: Storage, Storage, and Storage with its two steps, Storage and Storage. Storage.delete, Storage.exists, Storage.listdir, Storage.size, Storage.url, the three times (Storage and its siblings) and Storage.path, a local path for Python’s open, raise NotImplementedError until a backend defines them. The documentation lets a backend leave some out, but the inherited is_name_available calls exists, so save needs it.
Storage.save
Storage.save takes a name, the content and, optionally, a maximum length, and returns the name it used. Given None for a name, it takes the content’s; content without a chunks method, such as a plain Python file, it wraps in a File (Files: File, ContentFile and their helpers). Then it checks the name before and after each of the two steps that may change it, get_available_name and _save, and returns what _save gave back.
Each check is validate_file_name, which raises SuspiciousFileOperation for a name whose last part is empty, "." or "..", or which, read with forward slashes, is absolute or has ".." among its parts. The comment in save says this duplicates a check inside get_available_name, for a subclass that overrides it and loses the validation.
The example’s Manifest.scan is a file field with upload_to="manifests/ over the default storage. Two new manifests were each given a scan named “scan.txt”; a run that printed the calls recorded the second save, the field’s own check of the name left out:
Storage.save checks the name around get_available_name, which found it taken and drew another, and _save.The first save had found “manifests/scan.txt” free:
Manifest.objects.values_list("serial", "scan") -> [('M-0001', 'manifests/scan.txt'), ('M-0002', 'manifests/scan_z7slDDE.txt')]
first.scan.url, second.scan.url -> ('/media/manifests/scan.txt', '/media/manifests/scan_z7slDDE.txt')
A free name: get_available_name
Storage refuses a directory with ".." among its parts and validates the file name. The extension is every suffix of the file name, so “scan.2026.txt” has the root “scan” and the extension “.2026.txt”. While is_name_available says no (exists finds the name, or it is over the maximum length), the method tries a name made by get_alternative_name: the root, an underscore, seven random letters and digits from get_random_string, and the extension. An alternative that is too long has its root cut by as much as it is over before another is drawn; when no root is left, a SuspiciousFileOperation asks for a larger maximum. Since length counts as taken, a free name that is too long gets the random characters as well as the cut. From the run (long is a free name of 91 characters):
save("manifests/scan.2026.txt", ...), the name taken -> 'manifests/scan_BP7bfxi.2026.txt'
save(long, ..., max_length=60) -> 'manifests/endorsement-of-the-harbour-master-for-_U1aH7as.txt' 60
A clean name: get_valid_name and generate_filename
Storage calls get_valid_filename, which strips the ends, turns spaces into underscores and drops every character but letters and digits of any script, dashes, underscores and dots; a result that is empty, "." or ".." raises SuspiciousFileOperation. Storage cleans the last part of a path so, refuses ".." in its directory, and normalizes it; a file field calls it on the name its upload_to gives (File fields). save cleans nothing, as a save to “scans”, a second alias of the example’s with "base_url": "/, shows:
get_valid_name("Harbour master's scan (2).txt") -> 'Harbour_masters_scan_2.txt'
storages["scans"].save("Endeavour 12 Oct.txt", ...) -> 'Endeavour 12 Oct.txt' ; url: /scans/Endeavour%2012%20Oct.txt
FileSystemStorage: settings read late
The constructor of FileSystemStorage stores its arguments and reads no setting. The storage uses cached properties, each the argument or, where that is None, a setting read at first use: FileSystemStorage, MEDIA_ROOT, made absolute either way; FileSystemStorage, MEDIA_URL (an argument gets a slash at its end); and the two modes, from FILE_UPLOAD_PERMISSIONS (0o644) and FILE_UPLOAD_DIRECTORY_PERMISSIONS (None). The constructor connects StorageSettingsMixin to setting_changed, which drops a property when its setting changes.
The class is deconstructible, as a storage must be for its field to go into a migration, and the decorator keeps only the arguments given: the example’s default storage deconstructs to its class’s path and no arguments, naming no directory (Serializing values: serializer_factory and deconstruct).
FileSystemStorage._save
FileSystemStorage finds the path with safe_join, which raises SuspiciousFileOperation for one outside the location (Paths and archives: safe_join, safe_makedirs and extract), and makes the missing directories, with safe_makedirs when a directory mode is set, since os.makedirs does not give the directories between its mode (the comment cites CPython issue 86533). Content with a temporary_file_path, an upload streamed to disk (Multipart parsing and upload handlers), it moves with file_move_safe, which raises FileExistsError if the destination exists. Other content it writes chunk by chunk, under an exclusive lock, into a file opened with os.O_CREAT and os.O_EXCL, so that the open fails if the file exists.
The comment explains the loop around the write: two threads may be given the same name, so on FileExistsError the method asks get_available_name again and retries. That call passes no maximum length, and a name found there is not cut to the column. Given a name the storage already had, _save saved under another:
_save("manifests/scan.txt", ...), the name taken -> 'manifests/scan_Tnirf4g.txt'
The file then gets the file mode, if one is set, and on POSIX the location’s group where it may; the name returned is relative to the location, with forward slashes even on Windows.
With allow_overwrite=True, FileSystemStorage considers only the length and FileSystemStorage returns the name unchanged, so a long name is only cut; _save opens with os.O_TRUNC in place of os.O_EXCL, and lets file_move_safe overwrite.
The rest of FileSystemStorage
Most other methods start from FileSystemStorage, the same safe_join. FileSystemStorage is os.path.lexists, so a broken symbolic link counts; FileSystemStorage ignores FileNotFoundError, which its comment puts down to a concurrent removal; FileSystemStorage joins the name, quoted by filepath_to_uri, to the base URL. The times are aware, in UTC, under USE_TZ, and naive local ones otherwise.
InMemoryStorage
InMemoryStorage keeps its files in the instance, in a tree of nodes whose files are each an InMemoryFileNode, a ContentFile that keeps its times. It inherits the naming and, given no location or base URL, takes them from MEDIA_ROOT and MEDIA_URL, but writes nothing to disk; the documentation offers it to speed up tests.