5.4 Upgrade Guide¶

In this document we describe changes to be aware of when upgrading from orchestrator-core v5.3 to v5.4.

About 5.4¶

This release introduces 2 changes:

Tasks that fail on an AssertionError now get status failed
A new custom process step handler hook, which can be used to restore the previous behavior

Raising `AssertionError` now makes the task status `failed`¶

Short answer

If you do not rely on the inconsistent_data process status for tasks that fail with an AssertionError, no action is required.

Otherwise, either raise InconsistentDataError instead of using assert, or restore the old behavior with a custom process step handler.

Long answer

After each step the orchestrator stores the outcome of the step on the process, using a process step handler. In versions before 5.4, a failed task whose error class was AssertionError was stored with last_status=inconsistent_data and assigned to Assignee.NOC. The condition for InconsistentData in that handler never matched, because the error class has been named InconsistentDataError for years.

In 5.4 this SURF specific behavior has been removed (#1882). The default handler now maps the error class of a failed task as follows:

Error class	`last_status` before 5.4	`last_status` since 5.4
`AssertionError`	`inconsistent_data` (assignee `NOC`)	`failed`
`InconsistentDataError`	`failed`	`inconsistent_data`
`MaxRetryError`	`api_unavailable`	`api_unavailable`
`ApiException`	`api_unavailable` for status code 502, 503 or 504, otherwise `failed`	`api_unavailable` for any status code
anything else	`failed`	`failed`

The default handler also no longer touches the assignee. Before 5.4 a failed task was assigned to Assignee.SYSTEM (or Assignee.NOC for inconsistent_data); now the assignee stays the one of the step that failed.

Note

This only applies to tasks (is_task=True). The last_status of failed workflows is unaffected.

This can affect you if you have automation, monitoring or tests that look at the inconsistent_data status or the assignee of failed tasks.

Raise `InconsistentDataError`¶

The preferred fix is to raise the exception that is meant for this purpose, instead of relying on assert:

from orchestrator.core.utils.errors import InconsistentDataError

# before
assert subscription.status == SubscriptionLifecycle.ACTIVE, "Subscription is not active"

# after
if subscription.status != SubscriptionLifecycle.ACTIVE:
    raise InconsistentDataError("Subscription is not active")

As an additional benefit, assert statements are stripped when Python runs with optimizations enabled (-O).

Restoring the previous behavior¶

If you cannot change the code that raises the AssertionError, you can override the process step handler to map it to inconsistent_data again. See Custom Process Step Handler for details.

your_orchestrator/__init__.py

from orchestrator.core.db import ProcessTable
from orchestrator.core.services import processes
from orchestrator.core.workflow import Process as WFProcess
from orchestrator.core.workflow import ProcessStatus


def legacy_process_step_handler(p: ProcessTable, process_state: WFProcess) -> ProcessTable:
    p = processes._default_process_func(p, process_state)
    if process_state.isfailed() and p.is_task and process_state.unwrap().get("class") == "AssertionError":
        p.last_status = ProcessStatus.INCONSISTENT_DATA
    return p


processes.PROCESS_STEP_HANDLER = legacy_process_step_handler

To also restore the old assignees, set p.assignee to Assignee.NOC in the if branch and to Assignee.SYSTEM for other failed tasks (Assignee is in orchestrator.core.config.assignee).