Chapter 11 · Database backends
11.16 Test databases: BaseDatabaseCreation
BaseDatabaseCreation is the class of a connection’s creation, the object that makes the database a test run works in and destroys it afterwards: BaseDatabaseCreation and BaseDatabaseCreation, the copies that parallel test processes work in, the rows kept for a TransactionTestCase, and BaseDatabaseCreation, which says whether two aliases are one database.
Tests write to the database, and must not write to the one the settings name. So for each database a project uses, a test run needs another: made empty, given the schema the models need, put where the settings point so that every connection to the alias reaches it, and taken away at the end. Several aliases may be one database; a run in several processes needs a copy for each; and a test that empties the tables after itself takes with it the rows that migrations wrote, which some tests need back. The runner decides which aliases need a database and in what order, and calls each one’s creation: setup_databases and teardown_databases belong to the test framework (Which databases a test uses: databases; setup_databases). The wrapper makes a creation for each connection from the backend’s BaseDatabaseWrapper, and the creation keeps the connection.
In the quotes below, recorded from a running Django on SQLite, a line at the margin is a line of Python, with -> and its value; the lines set in under it are the calls it set going that the passage is about, with -> what a call returned; and a line beginning # counts the statements a line sent.
Making a test database: create_test_db
BaseDatabaseCreation takes a verbosity, autoclobber=False and keepdb=False, and returns the name of the test database. Here it makes one for "default", whose "TEST" settings were given a name, a file beside the project’s own:
connection.settings_dict['TEST']['NAME'] = os.path.join(run_directory, 'test_harbour.sqlite3')
old_name = connection.settings_dict['NAME']
old_name -> 'harbour.sqlite3'
settings.DATABASES['default'] is connection.settings_dict -> True
test_name = connection.creation.create_test_db(verbosity=0, autoclobber=True)
BaseDatabaseCreation.create_test_db(verbosity=0, autoclobber=True)
DatabaseCreation._get_test_db_name -> 'test_harbour.sqlite3'
DatabaseCreation._create_test_db(verbosity=0, autoclobber=True)
DatabaseCreation._get_test_db_name -> 'test_harbour.sqlite3'
-> 'test_harbour.sqlite3'
DatabaseWrapper.close
BaseDatabaseWrapper.close
call_command('migrate', verbosity=0, interactive=False, database='default', run_syncdb=True)
BaseDatabaseWrapper.connect
call_command('createcachetable', database='default')
-> 'test_harbour.sqlite3'
# 29 statements were sent while this line ran: none is written
test_name -> 'test_harbour.sqlite3'
settings.DATABASES['default']['NAME'] -> 'test_harbour.sqlite3'
connection.settings_dict['NAME'] -> 'test_harbour.sqlite3'
os.path.exists(test_name) -> True
create_test_db as the base class takes them, with what NAME holds, in settings.DATABASES and in the connection’s settings_dict, and the state of the connection after each.The name. BaseDatabaseCreation is the "TEST" dictionary’s "NAME", or "test_" before the alias’s "NAME". Its docstring warns that it is only useful from create_test_db and BaseDatabaseCreation and when nothing else changes "NAME"; once the test database is made, "NAME" is the test database’s own. Oracle’s backend returns "NAME" itself, its docstring saying a database’s name as Django handles it has no counterpart in Oracle.
The database. BaseDatabaseCreation makes the database over a second connection, and keeps one that is already there, replaces it, or ends the run. It asks BaseDatabaseCreation for the options of the new database, and has BaseDatabaseCreation send "CREATE DATABASE" on a cursor of a connection made with no "NAME", which BaseDatabaseWrapper opens and closes again (Opening and closing a connection). Where the statement fails and keepdb=True was asked, the database is taken to be there and kept. Otherwise, unless autoclobber=True, the person running the tests is asked whether to delete the old test database; it is then dropped and made again, and a refusal, or an error in either statement, ends the process. The runner passes autoclobber=not interactive. PostgreSQL’s and MySQL’s _execute_create_test_db let only the error that the database exists through to that question, and end the process at once on any other. SQLite’s makes nothing: it removes an old file, asking as the base class does, and the new file appears when migrate first connects.
The settings. "NAME" is set in two places, in settings for the alias and in the connection’s settings_dict: the connection reads its own dictionary when it opens; other code reads the setting, as Oracle’s creation does when it makes its administrative connection. In this project the two are one dictionary, as the recording shows where it asks whether they are the same object. Both are changed in place and never replaced: a connection that another thread makes later is made from the connection handler’s settings (Connections by alias: DATABASES and ConnectionHandler), the same dictionaries, and so opens the test database as well.
The schema. create_test_db calls the migrate command for the alias with run_syncdb=True, which makes the tables of applications without migrations from their models. The project’s applications have no migrations, and their tables were made so; migrate opened the connection, on the test database. Where the "TEST" dictionary’s "MIGRATE" is False, create_test_db sets MIGRATION_MODULES to None for every application while migrate runs, so that every application is made from its models, and puts the setting back after, as this run with "MIGRATE" set to False shows:
settings.MIGRATION_MODULES -> {}
test_name = connection.creation.create_test_db(verbosity=0, autoclobber=True)
BaseDatabaseCreation.create_test_db(verbosity=0, autoclobber=True)
call_command('migrate', verbosity=0, interactive=False, database='default', run_syncdb=True) (settings.MIGRATION_MODULES is then {'fleet': None, 'crew': None, 'office': None})
...
settings.MIGRATION_MODULES -> {}
The rest. Then the createcachetable command makes the tables of the database caches that the router lets on this alias (The cache backends); BaseDatabaseWrapper opens the connection if nothing has; and where the environment variable "RUNNING_DJANGOS_TEST_SUITE" is "true", BaseDatabaseCreation marks the tests of Django’s own suite that this backend’s features list (What a database can do: BaseDatabaseFeatures).
Destroying it: destroy_test_db
BaseDatabaseCreation closes the connection, takes the test database’s name from "NAME", or from a clone’s settings where it is given a suffix, and has BaseDatabaseCreation remove it unless keepdb=True; then, where it is given the old name, it puts that back in both places. The runner destroys each clone by its suffix and then the test database, with the name it kept.
connection.creation.destroy_test_db(old_name, verbosity=0)
BaseDatabaseCreation.destroy_test_db(old_database_name='harbour.sqlite3', verbosity=0)
DatabaseWrapper.close
BaseDatabaseWrapper.close
DatabaseCreation._destroy_test_db(test_database_name='test_harbour.sqlite3', verbosity=0)
# 0 statements were sent while this line ran: none is written
os.path.exists(test_name) -> False
settings.DATABASES['default']['NAME'] -> 'harbour.sqlite3'
connection.settings_dict['NAME'] -> 'harbour.sqlite3'
The base _destroy_test_db sends "DROP DATABASE" on a cursor of a second connection, its comment saying a database cannot be dropped while connected to it. SQLite’s removes the file, and does nothing for a database in memory; PostgreSQL’s closes the connection’s pool first.
The "TEST" settings
ConnectionHandler gives each alias a "TEST" dictionary and fills in its defaults (Connections by alias: DATABASES and ConnectionHandler). The recording’s, before its name was set:
connection.settings_dict['TEST'] -> {'CHARSET': None, 'COLLATION': None, 'MIGRATE': True, 'MIRROR': None, 'NAME': None}
"TEST" dictionary read outside Oracle’s backend, which reads more, for its test user and tablespaces.| key | filled in | what reads it, and what it does |
|---|---|---|
"NAME" |
filled in None |
what reads it, and what it does Base: the test database’s name, in place of "test_" and "NAME" |
"MIGRATE" |
filled in True |
what reads it, and what it does Base: with False, the tables are made from the models and no migration runs |
"MIRROR" |
filled in None |
what reads it, and what it does get_: another alias whose test database this one uses |
"CHARSET" |
filled in None |
what reads it, and what it does PostgreSQL’s and MySQL’s sql_: PostgreSQL’s "ENCODING", MySQL’s "CHARACTER SET" |
"COLLATION" |
filled in None |
what reads it, and what it does the same: MySQL’s "COLLATE"; PostgreSQL’s raises Improperly, its message saying PostgreSQL does not support setting a collation when a database is created |
"DEPENDENCIES" |
filled in no | what reads it, and what it does get_: the aliases whose test databases are made before this one’s |
"TEMPLATE" |
filled in no | what reads it, and what it does PostgreSQL’s suffix: the template the database is made from |
The base class’s suffix is empty, and PostgreSQL’s and MySQL’s write their options from these keys. Without "DEPENDENCIES", an alias other than "default" whose database differs from "default"'s waits for "default"; and "default" is put first among the aliases of one database, a comment saying data migrations use the default alias.
Which databases to make: test_db_signature and mirrors
The runner groups the aliases by BaseDatabaseCreation, a tuple of the settings that say which database an alias reaches: the base class’s is the host, the port, the engine and the test database’s name. For each group the runner calls BaseDatabaseCreation on the first alias and BaseDatabaseCreation on the rest, which copies the first one’s "NAME"; then it does the same for each alias whose "TEST" names a "MIRROR", so that a replica in the settings reads the test database of the alias it replicates. Oracle’s set_as_test_mirror copies the user and the password instead, and Oracle’s signature adds the test user. SQLite’s signature is the real database’s name and the test database’s, or the alias in place of the latter where the test database is in memory, its docstring saying that in-memory databases are distinct even with one "TEST" name:
connection.creation.test_db_signature() -> ('harbour.sqlite3', 'default')
connection.settings_dict['TEST']['NAME'] = os.path.join(run_directory, 'test_harbour.sqlite3')
connection.creation.test_db_signature() -> ('harbour.sqlite3', 'test_harbour.sqlite3')
Rows kept for a TransactionTestCase: serialize_db_to_string
A TestCase undoes each test by rolling back a transaction where its databases have them; a TransactionTestCase does not, and empties the tables after each test with flush instead (Introspection: reading the schema back). Flushing takes with it the rows that migrations wrote, and a class that sets TransactionTestCase has them put back before each of its tests. A comment in setup_databases gives this as the way that tests on databases without transactions, or in a TransactionTestCase, still get a clean database for every test. Once all the test databases are made, its comment saying this is to account for mirrors and routing, setup_databases serializes each one whose alias is in its argument serialized_aliases, or every one where it is given none, and keeps the text on the connection; the runner names only the aliases that tests setting serialized_rollback use.
BaseDatabaseCreation writes as JSON every row of every model that can migrate on the connection and that the router allows there, in order of its primary key, for each application that has migrations and is not in TEST_NON_SERIALIZED_APPS; its docstring says it is meant for the test runner and will not handle large amounts of data.
BaseDatabaseCreation loads the rows back inside an atomic block, which a comment says handles forward references and cycles, and inside BaseDatabaseWrapper, which another comment says is because MySQL does not support deferred checks, and then hands the tables it wrote to the connection’s BaseDatabaseWrapper.
Clones for parallel test runs
When the runner runs the tests in several processes, setup_databases calls BaseDatabaseCreation once for each process, with the suffixes "1", "2" and on, right after it has made the test database. A clone’s settings are BaseDatabaseCreation: a copy of the connection’s, with the suffix after "NAME", which by then, its comment says, is the test database’s own. clone_test_db calls BaseDatabaseCreation, which in the base class raises NotImplementedError; Oracle’s backend has no other.
The other three copy the database each its own way. SQLite’s copies the file, to the name with the suffix before its extension, or writes a database in memory to a file where the way the workers are started needs one. PostgreSQL’s makes the clone with "CREATE DATABASE" from the test database as its template, closing the connection and its pool first, since, its comment says, the statement requires the connections to the template to be closed. MySQL’s makes an empty database and pipes mysqldump of the test database into the mysql client, both command lines built by its DatabaseClient. Asked once the test database is made, as a test run asks, SQLite’s clone settings give the test database’s file name with the suffix:
connection.settings_dict['NAME'] -> 'test_harbour.sqlite3'
connection.creation.get_test_db_clone_settings('2')['NAME'] -> 'test_harbour_2.sqlite3'
Each worker process, as it starts, calls BaseDatabaseCreation with its number for each alias the tests use, or for every alias where the suite names none. The base class’s updates the connection’s settings_dict with the clone’s settings and closes the connection; its comment says the update must be in place, because with a new dictionary assigned, new threads would connect to the default database instead of the clone. A worker started by spawn or forkserver is a new interpreter, and _init_worker first gives it the settings and the serialized rows of the process that made the databases.
SQLite: the in-memory test database and the start method
SQLite’s backend takes a missing "TEST" name as ":memory:", and makes it the name "file: for "default", with the alias in it. It is a URI, which the backend always asks the driver to open as one; its "mode=memory" is what DatabaseCreation looks for; the alias in it gives each alias a database of its own, as the signature does; and the testing documentation says the shared cache lets tests share the database between threads.
test_name = connection.creation.create_test_db(verbosity=0, autoclobber=True)
BaseDatabaseCreation.create_test_db(verbosity=0, autoclobber=True)
DatabaseCreation._get_test_db_name -> 'file:memorydb_default?mode=memory&cache=shared'
...
DatabaseWrapper.close
BaseDatabaseWrapper.close
call_command('migrate', verbosity=0, interactive=False, database='default', run_syncdb=True)
BaseDatabaseWrapper.connect
...
connection.settings_dict['NAME'] -> 'file:memorydb_default?mode=memory&cache=shared'
connection.is_in_memory_db() -> True
SQLite’s DatabaseWrapper does nothing for a database in memory, its comment saying that closing the connection destroys the database (DatabaseWrapper; SQLite). BaseDatabaseCreation closed the connection while "NAME" was still the real database’s, and the close went through. BaseDatabaseCreation calls close while "NAME" is the in-memory database’s, so that close does nothing; it removes no file and puts "NAME" back, and the connection keeps its driver’s connection, so the test database in memory is not destroyed, though the settings now name the file.
connection.creation.destroy_test_db(old_name, verbosity=0)
BaseDatabaseCreation.destroy_test_db(old_database_name='harbour.sqlite3', verbosity=0)
DatabaseWrapper.close
DatabaseCreation._destroy_test_db(test_database_name='file:memorydb_default?mode=memory&cache=shared', verbosity=0)
# 0 statements were sent while this line ran: none is written
connection.settings_dict['NAME'] -> 'harbour.sqlite3'
connection.connection is None -> False
Clones of a database in memory depend on how the worker processes are started, which multiprocessing says. A fork copies an in-memory database with the process, a comment says, so the clone’s settings are the test database’s own and nothing is copied. Under spawn or forkserver, DatabaseCreation writes the database with the driver’s sqlite3 to a file named for the alias and the suffix; any other start method raises NotSupportedError. Here the recording set "NAME" by hand to ":memory:", which DatabaseCreation takes to be in memory, like the test database’s URI:
multiprocessing.get_start_method() -> 'spawn'
connection.settings_dict['NAME'] = ':memory:'
connection.creation.get_test_db_clone_settings('2')['NAME'] -> 'default_2.sqlite3'
In the worker, SQLite’s DatabaseCreation, under spawn or forkserver, opens the clone’s file read-only, copies it with the same sqlite3 into a new database in memory, sets that as "NAME", and connects before it closes the copy. It does so whatever the test database was: under spawn, a worker on SQLite works in memory even where the test database is a file, as it is here:
connection.creation.setup_worker_connection('2') # as the second worker process calls it, here in this one
DatabaseCreation.setup_worker_connection('2')
DatabaseCreation.get_test_db_clone_settings('2') -> the connection's settings with 'NAME': 'test_harbour_2.sqlite3'
BaseDatabaseWrapper.connect
# 0 statements were sent while this line ran: none is written
connection.settings_dict['NAME'] -> 'file:memorydb_default_2?mode=memory&cache=shared'
full(sorted(connection.introspection.table_names())) -> ['crew_officer', 'crew_sailor', 'fleet_port', 'fleet_ship', 'fleet_voyage', 'fleet_voyage_calls', 'office_berth', 'office_licence', 'office_manifest']
Oracle: a test user and tablespaces
Oracle has no separate databases under one user, the docstring of DatabaseCreation says, so Oracle’s backend makes a test database a user of its own; Oracle tells how the user and its tablespaces are made.