12.17 The optimizer: MigrationOptimizer and reduce

reduce: one question, three kinds of answer

The optimizer calls Operation.reduce on one operation with another that stands later in the list, and with the label of the list’s application, which an operation does not hold. A list says: put these in place of the two of us. That is a fold: the list is empty where the two cancel, holds one operation where they merge, and of Django’s own only AlterField.reduce returns two. True says: no fold, but the two may change places. False says: no fold, and the later may not be moved in front of the earlier.

The base class (Operations: the base class and its two halves) answers False but for one case, Operation.elidable. An elidable operation answers any later one with a list of that one alone, and so drops out; an operation asked about a later one that is elidable answers with a list of itself. elidable is False on the class, and RunSQL and RunPython take it as an argument, by which a migration’s author marks an operation that is safe to leave out (RunPython, RunSQL and SeparateDatabaseAndState).

In the recordings here, of a running Django, only the lines a passage is about are written, the rest left out: a line with = is Python as it was run; -> is followed by what a call returned, and the calls it made are set in under it; python manage.py begins a command line and stdout | a line it printed, the path less the directory the recording ran in; an operation is written as its class, its description and, in parentheses, what the recording adds to tell it apart.

create = migrations.CreateModel('Crane', [('id', models.BigAutoField(primary_key=True))])
tonnes = migrations.AddField('crane', 'tonnes', models.PositiveSmallIntegerField(default=5))
...
no_crane = migrations.DeleteModel('Crane')
metres = migrations.AddField('dock', 'metres', models.PositiveSmallIntegerField(default=120))
...
sql = migrations.RunSQL('UPDATE yard_dock SET name = name')
passing = migrations.RunSQL('UPDATE yard_dock SET name = name', elidable=True)
create.reduce(tonnes, 'yard')  ->  [CreateModel: Create model Crane (fields id, tonnes)]
create.reduce(no_crane, 'yard')  ->  []
...
tonnes.reduce(metres, 'yard')  ->  True
...
sql.reduce(tonnes, 'yard')  ->  False
tonnes.reduce(sql, 'yard')  ->  False
passing.reduce(tonnes, 'yard')  ->  [AddField: Add field tonnes to crane (PositiveSmallIntegerField(default=5))]

Every reduce Django ships falls back on the base’s where it has no fold of its own, so an elidable operation drops out beside any operation of Django’s. RunSQL, RunPython and SeparateDatabaseAndState declare neither reduce nor references_model, the question a model operation puts to a later one before it lets the two change places: do you refer to my model? Operation.references_model says yes, True, whatever the model. So unless one of the two is elidable, such an operation answers False about every other, and every other answers False about it. It is a boundary: nothing changes places with it. So is an operation of a project’s own that overrides nothing.

MigrationOptimizer.optimize: one fold to a pass

MigrationOptimizer.optimize calls MigrationOptimizer.optimize_inner on the list, and again on what that returned, until a call returns a list equal to the one it was given; its docstring draws the consequence, that a fold must never return a list longer than what it replaces. Each call is a pass, and a pass makes at most one fold. Here the list is the CreateModel of Crane, the AddField of its field tonnes and an AlterField of that field.

MigrationOptimizer.optimize(<3 operations>, 'yard')
  MigrationOptimizer.optimize_inner(<3 operations>)  ->  [CreateModel: Create model Crane (fields id, tonnes), AlterField: Alter field tonnes on crane (PositiveIntegerField(default=5))]
  MigrationOptimizer.optimize_inner(<2 operations>)  ->  [CreateModel: Create model Crane (fields id, tonnes)]
  MigrationOptimizer.optimize_inner(<1 operations>)  ->  [CreateModel: Create model Crane (fields id, tonnes)]

In a pass, optimize_inner takes each operation in turn and asks it about every operation after it, nearest first. True lets the scan go on, and so does False, which is noted. The first list ends the scan, and the pass must decide where its operations are to stand, since the operations between the two must go to one side of them.

• To the right. If no answer before this one was False, the earlier operation could be moved across everything between: the operations between keep their places, and the list stands after them.
• To the left. Otherwise the pass asks each operation between about the later one. If every one answers True, the later operation could be moved across them all: the list stands where the earlier operation stood, and the operations between follow.
• If neither holds, the earlier operation stays and the pass goes on to the operation after it; the scan is not taken up again, and so misses any fold with an operation further on.

In the figure’s first list the field added to Crane is a foreign key to Hook, a model created between, and the merged operation goes right, after the model it now points at. In its second, Hook has a foreign key to Crane, and the merged operation goes left, before the model that points at it.

The model operations

ModelOperation.reduce answers with Operation.reduce’s list where there is one, and otherwise with ModelOperation.can_reduce_through: true exactly when the later operation’s references_model, asked about this operation’s model, says no. DeleteModel has no more than this. CreateModel.reduce has a fold for each of these operations on its own model.

RenameModel.reduce answers a later RenameModel from its new name with one rename; for anything else it passes over ModelOperation.reduce, which would test its old name, and tests the later operation against its new one. ModelOptionOperation.reduce, which the operations that alter a model’s options or its managers inherit, answers an operation of its own class on its model, or a DeleteModel of it, with a list of the later one alone. AlterTogetherOptionOperation.can_reduce_through adds that an AlterUniqueTogether and an AlterIndexTogether, the two subclasses of that class, let each other through on one model, so that two of one kind fold across one of the other.

The field operations

FieldOperation.reduce answers with Operation.reduce’s list where there is one, and otherwise with true exactly when the later operation’s references_field, asked about this operation’s field, says no. Its subclasses add folds, comparing names in lower case.

AddField.reduce answers a later operation on its field: an AlterField with an AddField of the altered field, a RemoveField with nothing, a RenameField with an AddField under the new name. RemoveField.reduce answers a DeleteModel of its model with the DeleteModel. AlterField.reduce answers a later AlterField or RemoveField of its field with the later operation, and a RenameField of it, where the altered field has no db_column, with the rename and then a copy of itself under the new name: a list no shorter than the two, but its AlterField can then fold with an AlterField or a RemoveField after the rename. RenameField.reduce answers a RenameField of the same model from its new name with one rename; for anything else it passes over FieldOperation.reduce and tests the later operation against both its names.

Indexes and constraints, and a cautious references_model

IndexOperation, the base of the operations on indexes and constraints, declares none of reduce, references_model and references_field. AddIndex.reduce answers a RemoveIndex of its index’s name with nothing and a RenameIndex from that name with an AddIndex of a renamed copy, comparing names and not models. RenameIndex.reduce answers a RenameIndex on its model, from its new name, with one rename. AddConstraint.reduce answers a RemoveConstraint of its model and name with nothing, and an AlterConstraint of them with an AddConstraint of the new constraint. AlterConstraint.reduce answers a later AlterConstraint or RemoveConstraint of its model and name with the later operation. To everything else they, like RemoveIndex and RemoveConstraint, give the base’s answers: but for these folds and CreateModel’s, each is a boundary.

Operation.references_model returns True by the rule in its docstring: if in doubt, return true, since a true too many makes the optimizer a little less efficient and a false too many may leave an optimized migration that cannot be used; Operation.references_field falls back on it. The model and field operations declare narrower ones (The field operations, and those on indexes and constraints); DeleteModel alone declares one that returns True for every model, since the model it deletes may point at any other.

The example project’s first makemigrations shows what the caution costs: fleet’s migration left the optimizer with Ship’s index and two constraints as operations of their own. Voyage’s CreateModel stood between and pointed at Ship, so the fold of Ship’s CreateModel with the AddIndex could not go right, and Voyage’s CreateModel answered False about the AddIndex, whose references_model is the base’s, so the fold could not go left.

The optimizemigration command

optimizemigration takes an application’s label and a migration’s name, or the beginning of one. It raises CommandError for a label no installed application has, for an unmigrated application, and for a name that no migration of the application, or more than one, begins with. Its loader has no connection, so no database is asked whether the migration is applied (Finding migrations: MigrationLoader).

It runs the optimizer on the migration’s operations and compares the two lists by length. Equal lengths end the command, even where the order changed, as with an AlterField and a RenameField. Otherwise it prints both lengths, unless --verbosity is 0, and with --check exits there with status 1, having written nothing; without, it writes over the migration’s file what a MigrationWriter makes of it with its new operations. Here the file is yard.0002_crane, written for the purpose with three operations that fold into one.

python manage.py optimizemigration yard 0002_crane --check
  stdout | Optimizing from 3 operations to 1 operations.
  exit status 1
python manage.py optimizemigration yard 0002_crane
  stdout | Optimizing from 3 operations to 1 operations.
  stdout | Optimized migration harbour/yard/migrations/0002_crane.py

Where the writer has set MigrationWriter.needs_manual_porting, for a function that must be copied by hand out of a migration file (Writing a migration file: MigrationWriter), the command leaves the file as it is: for a migration that itself has replaces it raises CommandError; for any other it makes a migration class with the original’s dependencies, the new operations and a replaces naming the original, calls it by the original’s name with "_optimized" after it, and writes that beside the original.