Record job failures with application context.

AT-AT needs to be able to track which user tasks failed and why. To
accomplish this we:

- Enabled Celery's results backend, which logs task results to a data
  store; a Postgres table, in our case.
  (https://docs.celeryproject.org/en/latest/userguide/tasks.html#result-backends)
- Created tables to track the relationships between the relevant models
  (Environment, EnvironmentRole) and their task failures.
- Added an `on_failure` hook that tasks can use. The hook will add
  records to the job failure tables.

Now a resource like an `Environment` has access to it task failures
through the corresponding failure table.

Notes:
- It might prove useful to use a real foreign key to the Celery results
  table eventually. I did not do it here because it requires that we
  explicitly store the Celery results table schema as a migration and
  add a model for it. In the current implementation, AT-AT can be
  agnostic about where the results live.
- We store the task results indefinitely, so it is important to specify
  tasks for which we do not care about the results (like `send_mail`)
  via the `ignore_result` kwarg.
This commit is contained in:
dandds
2019-09-05 11:07:51 -04:00
parent 1cbefb099b
commit 7010bdb09c
11 changed files with 158 additions and 2 deletions

View File

@@ -320,3 +320,12 @@ def notification_sender(app):
yield app.notification_sender
app.notification_sender = real_notification_sender
# This is the only effective means I could find to disable logging. Setting a
# `celery_enable_logging` fixture to return False should work according to the
# docs, but doesn't:
# https://docs.celeryproject.org/en/latest/userguide/testing.html#celery-enable-logging-override-to-enable-logging-in-embedded-workers
@pytest.fixture(scope="function")
def celery_worker_parameters():
return {"loglevel": "FATAL"}