11.2 The connection object: BaseDatabaseWrapper and what a backend provides

What a backend’s class declares

A backend’s DatabaseWrapper is mostly class attributes. In a recording from a running Django, each line under the title but the first, which closes the connection and so leaves the driver’s connection None, is a question put to the connection "default", SQLite’s, and its answer after ->, and full( is the recording’s own helper for writing a value whole:

What a connection carries: its class, the backend's names for itself, the driver, and the helper objects it made
    connection.close()
    type(connections['default'])  ->  django.db.backends.sqlite3.base.DatabaseWrapper
    connection.vendor  ->  'sqlite'
    connection.display_name  ->  'SQLite'
    connection.Database  ->  the module sqlite3.dbapi2
    connection.SchemaEditorClass  ->  django.db.backends.sqlite3.schema.DatabaseSchemaEditor
    type(connection.ops)  ->  django.db.backends.sqlite3.operations.DatabaseOperations
    type(connection.features)  ->  django.db.backends.sqlite3.features.DatabaseFeatures
    type(connection.introspection)  ->  django.db.backends.sqlite3.introspection.DatabaseIntrospection
    type(connection.creation)  ->  django.db.backends.sqlite3.creation.DatabaseCreation
    type(connection.client)  ->  django.db.backends.sqlite3.client.DatabaseClient
    type(connection.validation)  ->  django.db.backends.base.validation.BaseDatabaseValidation
    connection.ops.connection is connections['default']  ->  True
    connection.connection  ->  None
    full(list(connection.settings_dict))  ->  ['ENGINE', 'NAME', 'ATOMIC_REQUESTS', 'AUTOCOMMIT', 'CONN_MAX_AGE', 'CONN_HEALTH_CHECKS', 'OPTIONS', 'TIME_ZONE', 'USER', 'PASSWORD', 'HOST', 'PORT', 'TEST']
    connection.settings_dict['NAME']  ->  'harbour.sqlite3'

The base class leaves to the backend what only the backend can know, and raises NotImplementedError from each such method: BaseDatabaseWrapper.get_connection_params, BaseDatabaseWrapper.get_new_connection, BaseDatabaseWrapper.is_usable and BaseDatabaseWrapper.get_database_version, which Opening and closing a connection tells with each backend’s; BaseDatabaseWrapper.create_cursor, in Cursors: executing a statement, the query log and execute wrappers; and BaseDatabaseWrapper._set_autocommit, in Transactions: autocommit, atomic and savepoints. Other methods are hooks that do nothing until a backend fills them. BaseDatabaseWrapper.prepare_database is one: migrate calls it on its connection before it works out what to migrate, and PostGIS’s creates the extension "postgis" there where it is missing. The methods that turn foreign key checks off and on are others.

The helper objects

BaseDatabaseWrapper.__init__ ends by making the helpers, each by calling its class with the connection. Every connection object has its own: two threads’ connections for one alias share no helper, and a feature that is a cached property, worked out by asking the server, is worked out once for each connection object (What a database can do: BaseDatabaseFeatures). The schema editor is not among them; BaseDatabaseWrapper.schema_editor makes a new one each time it is called (The schema editor: BaseDatabaseSchemaEditor).

Each helper keeps the connection it was given as its attribute connection, as the recording’s line connection.ops.connection is connections['default'] shows, and reads there what it needs of it: the operations object the time zone and the features, the validation the display name.

A backend that has to adjust a helper does so after the base class’s __init__ has made it. Oracle’s DatabaseWrapper.__init__ sets two features, whether an insert can hand back columns and an update rows, from the option "use_returning_into".

The state of one connection

BaseDatabaseWrapper.__init__ takes the alias’s settings and the alias, opens nothing, and sets the rest of the connection’s state on the instance in groups, most of them under a comment of their own:

The settings dictionary is the connection handler’s own for the alias, and not a copy, so every thread’s connection for an alias holds the same one (Connections by alias: DATABASES and ConnectionHandler). BaseDatabaseWrapper.copy makes a second connection to the same database: a wrapper of the same class with a deep copy of the settings, under the same alias unless it is given another, registered nowhere: connections never hands it out. It is for tests that need two connections to one database, and nothing in Django calls it but Django’s own test suite.

The client: dbshell

The command dbshell runs the database’s own command-line program, connected as the alias’s settings say. It calls BaseDatabaseClient.runshell on the client of the connection its "--database" option names, with the arguments given after it on the command line. runshell asks the class method BaseDatabaseClient.settings_to_cmd_args_env for the program’s arguments and an environment, sets that environment over the process’s own where there is one, and runs the program with subprocess.run, raising if it exits with a failure. The base class’s class method raises NotImplementedError with a message that gives a backend its choice: provide the class method, or override runshell. The command turns two failures into a CommandError: FileNotFoundError, which it takes to mean that the program named by BaseDatabaseClient.executable_name is not installed, and the program’s exit with a failure, which it reports with the command line and passes on as its own status.

MySQL’s client gives its reason for the environment in a comment: MySQL’s documentation discourages the variable, since on old Unix systems the command ps can show it, but a password among the arguments is exposed on more systems, and the environment keeps it out of the subprocess.CalledProcessError raised when the program fails, whose text holds every argument. Oracle’s client puts the password among the arguments, and dbshell’s message for a program that failed is built from them.

Validation: check_field and the database checks

A connection’s validation object is how a backend adds its own warnings to the system checks, through two methods of BaseDatabaseValidation. BaseDatabaseValidation.check returns no messages in the base class, and is called for each alias by check_database_backends, a check registered under the tag for databases. A run of the checks that names neither aliases nor tags leaves such checks out, because, a comment says, they do more than read the code; they run under check given its "--database" option or that tag, under migrate for its alias, and under the test runner (The system checks).

BaseDatabaseValidation.check_field is called by the checks of fields, which are not under the tag for databases, so a run that names neither aliases nor tags calls it too, for each field and every alias, where the router allows the field’s model to be migrated there (The model checks). It does nothing unless the backend’s validation defines check_field_type, which it hands the field and its column type on this connection, passing over related fields and a few others. SQLite’s and PostgreSQL’s wrappers name no validation class, so theirs is the base class, as the recording shows for SQLite.

MySQL’s validation has both methods. Its check warns, as "mysql.W002", when neither of MySQL’s strict modes is among the server’s SQL modes, which it learns by asking the server; its check_field_type warns, as "mysql.W003", of a unique field whose column is a "varchar" longer than 255 characters or of no stated length, and, as "fields.W162", of an indexed field whose column is of a type MySQL cannot index in full, a text, a blob or JSON. Oracle’s has the last alone, for its large objects. Both take the types from the wrapper’s _limited_data_types; the warning’s hint says the index will not be created, and each backend’s schema editor reads the same attribute to leave it out.

Foreign key checks switched off: constraint_checks_disabled

Some writes put rows in an order the database’s foreign keys would refuse: a fixture whose rows refer forward to rows later in the file, or a cycle of references. BaseDatabaseWrapper.constraint_checks_disabled is a context manager for them. It calls BaseDatabaseWrapper.disable_constraint_checking, and on leaving calls BaseDatabaseWrapper.enable_constraint_checking only if the first returned true, which the base class’s docstring asks a backend to return only when the checks were turned off and need turning on again. Its callers then check what was written meanwhile with BaseDatabaseWrapper.check_constraints, which is to raise IntegrityError for a row that refers to nothing.

The backends differ because their foreign keys do. PostgreSQL’s and Oracle’s operations objects make each foreign key "DEFERRABLE INITIALLY DEFERRED", and SQLite’s schema editor writes its foreign keys so too, so that within a transaction a row may refer to one not yet written and the database checks when the transaction commits; setting the constraints to immediate, as PostgreSQL’s and Oracle’s check_constraints does before setting them back, makes the database check them at once. SQLite’s schema editor, some of whose alterations need the checks off, turns them off as it is entered, refusing where they are on inside a transaction, since the pragma cannot turn them off there, and as it is left checks every table and turns them on again (The schema editor: BaseDatabaseSchemaEditor). MySQL checks its foreign keys as each statement runs, which a comment gives as the reason for turning the checks off where the test databases are filled again. MySQL’s enable_constraint_checking clears needs_rollback while its statement runs, for constraint_checks_disabled nested inside atomic, its comment says; a cursor refuses every statement while needs_rollback is set (Transactions: autocommit, atomic and savepoints).

The callers pair the three, and both enter constraint_checks_disabled inside an atomic block: loaddata loads its fixtures within its own and then checks the tables it wrote to, and the test databases are filled again from their serialized rows the same way (Test databases: BaseDatabaseCreation). Inside a transaction only MySQL’s checks go off; SQLite’s disable_constraint_checking asks for the pragma off and reads it back still on, as this recording shows:

with transaction.atomic(), connection.constraint_checks_disabled(): Ship.objects.count()
  Atomic.__enter__
    SQL BEGIN
  BaseDatabaseWrapper.constraint_checks_disabled
    DatabaseWrapper.disable_constraint_checking
      SQL PRAGMA foreign_keys = OFF
      SQL PRAGMA foreign_keys
      -> False

On the other three the deferred foreign keys let the rows in, and check_constraints checks them. TestCase calls check_constraints as each test ends, before it rolls the test’s transaction back, on each database whose features say it can defer checks and whose connection is usable and not marked for rollback (A transaction around each test: TestCase).