7010bdb09c
AT-AT needs to be able to track which user tasks failed and why. To accomplish this we: - Enabled Celery's results backend, which logs task results to a data store; a Postgres table, in our case. (https://docs.celeryproject.org/en/latest/userguide/tasks.html#result-backends) - Created tables to track the relationships between the relevant models (Environment, EnvironmentRole) and their task failures. - Added an `on_failure` hook that tasks can use. The hook will add records to the job failure tables. Now a resource like an `Environment` has access to it task failures through the corresponding failure table. Notes: - It might prove useful to use a real foreign key to the Celery results table eventually. I did not do it here because it requires that we explicitly store the Celery results table schema as a migration and add a model for it. In the current implementation, AT-AT can be agnostic about where the results live. - We store the task results indefinitely, so it is important to specify tasks for which we do not care about the results (like `send_mail`) via the `ignore_result` kwarg.
41 lines
1.3 KiB
Python
41 lines
1.3 KiB
Python
from flask import current_app as app
|
|
|
|
from atst.queue import celery
|
|
from atst.database import db
|
|
from atst.models import EnvironmentJobFailure, EnvironmentRoleJobFailure
|
|
|
|
|
|
class RecordEnvironmentFailure(celery.Task):
|
|
def on_failure(self, exc, task_id, args, kwargs, einfo):
|
|
if "environment_id" in kwargs:
|
|
failure = EnvironmentJobFailure(
|
|
environment_id=kwargs["environment_id"], task_id=task_id
|
|
)
|
|
db.session.add(failure)
|
|
db.session.commit()
|
|
|
|
|
|
class RecordEnvironmentRoleFailure(celery.Task):
|
|
def on_failure(self, exc, task_id, args, kwargs, einfo):
|
|
if "environment_role_id" in kwargs:
|
|
failure = EnvironmentRoleJobFailure(
|
|
environment_role_id=kwargs["environment_role_id"], task_id=task_id
|
|
)
|
|
db.session.add(failure)
|
|
db.session.commit()
|
|
|
|
|
|
@celery.task(ignore_result=True)
|
|
def send_mail(recipients, subject, body):
|
|
app.mailer.send(recipients, subject, body)
|
|
|
|
|
|
@celery.task(ignore_result=True)
|
|
def send_notification_mail(recipients, subject, body):
|
|
app.logger.info(
|
|
"Sending a notification to these recipients: {}\n\nSubject: {}\n\n{}".format(
|
|
recipients, subject, body
|
|
)
|
|
)
|
|
app.mailer.send(recipients, subject, body)
|