AT-AT needs to be able to track which user tasks failed and why. To
accomplish this we:
- Enabled Celery's results backend, which logs task results to a data
store; a Postgres table, in our case.
(https://docs.celeryproject.org/en/latest/userguide/tasks.html#result-backends)
- Created tables to track the relationships between the relevant models
(Environment, EnvironmentRole) and their task failures.
- Added an `on_failure` hook that tasks can use. The hook will add
records to the job failure tables.
Now a resource like an `Environment` has access to it task failures
through the corresponding failure table.
Notes:
- It might prove useful to use a real foreign key to the Celery results
table eventually. I did not do it here because it requires that we
explicitly store the Celery results table schema as a migration and
add a model for it. In the current implementation, AT-AT can be
agnostic about where the results live.
- We store the task results indefinitely, so it is important to specify
tasks for which we do not care about the results (like `send_mail`)
via the `ignore_result` kwarg.
