Record job failures with application context.

AT-AT needs to be able to track which user tasks failed and why. To
accomplish this we:

- Enabled Celery's results backend, which logs task results to a data
  store; a Postgres table, in our case.
  (https://docs.celeryproject.org/en/latest/userguide/tasks.html#result-backends)
- Created tables to track the relationships between the relevant models
  (Environment, EnvironmentRole) and their task failures.
- Added an `on_failure` hook that tasks can use. The hook will add
  records to the job failure tables.

Now a resource like an `Environment` has access to it task failures
through the corresponding failure table.

Notes:
- It might prove useful to use a real foreign key to the Celery results
  table eventually. I did not do it here because it requires that we
  explicitly store the Celery results table schema as a migration and
  add a model for it. In the current implementation, AT-AT can be
  agnostic about where the results live.
- We store the task results indefinitely, so it is important to specify
  tasks for which we do not care about the results (like `send_mail`)
  via the `ignore_result` kwarg.

alembic/versions/30ea1cb20807_add_job_failure_tables.py

@@ -0,0 +1,42 @@
+"""add job failure tables
+
+Revision ID: 30ea1cb20807
+Revises: 4a3122ffe898
+Create Date: 2019-09-06 06:56:25.685805
+
+"""
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+# revision identifiers, used by Alembic.
+revision = '30ea1cb20807' # pragma: allowlist secret
+down_revision = '4a3122ffe898' # pragma: allowlist secret
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('environment_job_failures',
+    sa.Column('id', sa.Integer(), nullable=False),
+    sa.Column('task_id', sa.String(), nullable=False),
+    sa.Column('environment_id', postgresql.UUID(as_uuid=True), nullable=False),
+    sa.ForeignKeyConstraint(['environment_id'], ['environments.id'], ),
+    sa.PrimaryKeyConstraint('id')
+    )
+    op.create_table('environment_role_job_failures',
+    sa.Column('id', sa.Integer(), nullable=False),
+    sa.Column('task_id', sa.String(), nullable=False),
+    sa.Column('environment_role_id', postgresql.UUID(as_uuid=True), nullable=False),
+    sa.ForeignKeyConstraint(['environment_role_id'], ['environment_roles.id'], ),
+    sa.PrimaryKeyConstraint('id')
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table('environment_role_job_failures')
+    op.drop_table('environment_job_failures')
+    # ### end Alembic commands ###