2.0 Migration Guide

In this document we'll help you migrate your orchestrator application from orchestrator-core 1.3 to 2.0.

About 2.0

In this release we migrate the orchestrator-core from Pydantic v1 to Pydantic v2. This major change took a lot of effort (PR) as Pydantic v2 is quite different.

There were quite a few breaking changes that we had to deal with. We have adapted orchestrator-core to mitigate the impact of these breaking changes. Your orchestrator application may need to be changed as well, we will guide you in this process. Several changes have been automated through a migration script - more about this later.

Steps

Make sure you have upgraded to the latest orchestrator-core 1.3.x release before starting.

Create a new branch in your own orchestrator
Create and activate a virtualenv with python 3.11 (recommended), 3.10 or 3.9 and install your dependencies
Run pip install -U orchestrator-core~=2.0.0 to get the latest 2.0 core version and upgrade all other dependencies
- Add this to your pinned dependencies
Run pip install bump-pydantic to install a migration tool made by the Pydantic maintainers
- No need to add this to your dependencies
Run bump-pydantic .
- It is a bit fragile and not very fast on large codebases. You can run it against specific subdirectories to work around problematic files or slow performance
- The tool will leave # TODO[pydantic] comments on pydantic 1.x code that it cannot automatically rewrite to pydantic 2.x
Run python -m orchestrator.devtools.scripts.migrate_20 <dir> and point <dir> to your orchestrator code and tests
- This migration script will perform a number of rewrites on your code for orchestrator-core 2.0

Then continue with the following sections.

Pydantic v2 Changes

First and foremost, the full Pydantic v2 migration guide is located here https://docs.pydantic.dev/2.5/migration.

We recommend to read through it quickly before continuing with this guide.

Next we'll give you some pointers of what to change based on our experience of migrating the SURF orchestrator.

Change renamed BaseModel functions

Replace these deprecated function calls on any Pydantic Models (i.e. Subscriptions, ProductBlocks).

Pydantic V1	Pydantic V2
__fields__	model_fields
__validators__	__pydantic_validator__
construct()	model_construct()
copy()	model_copy()
dict()	model_dump()
json_schema()	model_json_schema()
json()	model_dump_json()
parse_obj()	model_validate()
update_forward_refs()	model_rebuild()

Full overview: https://docs.pydantic.dev/2.5/migration/#changes-to-pydanticbasemodel

Replace constrained types to Python Annotated[ ]

Most constrained types have been removed from Pydantic v2. We will show constrained int as an example.

The symbols in the after example can be imported with:

from typing import Annotated
from annotated_types import Ge, Le, Len, MinLen, MaxLen, doc

conint

Before:

ipv4_prefixlen: conint(ge=30, le=31)

After:

ipv4_prefixlen: Annotated[int, Ge(30), Le(31)]

ConstrainedInt

Before:

class NumberOfPeerings(ConstrainedInt):
    """Number of peerings."""

    ge = 1
    le = MAX_NUMBER_OF_PEERINGS

After:

NumberOfPeerings = Annotated[
    int, Ge(1), Le(MAX_NUMBER_OF_PEERINGS), doc("Number of peerings.")
]

orchestrator-core 2.0 changes

The following breaking changes have been made:

Removed SubscriptionInstanceList -> see SubscriptionInstanceList removed (covered by migration-script)
Removed @serializable_property -> use @computed_field instead (covered by migration-script)
Removed DomainModel.get_properties -> use BaseModel.model_computed_fields() instead
Removed build_extendend_domain_model() -> use build_extended_domain_model() instead
(pydantic-forms) Removed class UniqueConstrainedList -> see UniqueConstrainedList
(pydantic-forms) Removed class ChoiceList -> use choice_list() instead
(pydantic-forms) Removed class ContactPersonList -> use contact_person_list() instead
(pydantic-forms) Moved ReadOnlyField import (covered by migration-script)
(pydantic-forms) Changed ReadOnlyField from a Field to an Annotated Literal -> change your code from field: int = ReadOnlyField(123) to field: ReadOnlyField(123)

SubscriptionInstanceList removed

There are 2 different usecases of SubscriptionInstanceList that can be changed as follows.

Note: this change is covered by the migration script.

Note: you can no longer instantiate an annotated list type.

I.e. before you could write mylist = ListMax2() but this is no longer valid. Instead, write mylist: ListMax2 = [].

1. Generic List

class ListMax2(SubscriptionInstanceList[SI]):
    max_items = 2

becomes

ListMax2 = Annotated[list[SI], Len(max_length=2)]

Which can then be used in a pydantic model, i.e. a product block, with a type subscription

class MainProductBlockInactive(ProductBlockModel):
    values: ListMax2[SubProductBlockInactive]

When inheriting from this model you can change the type of the list, however a type: ignore comment is needed to silence errors from type checkers like mypy.

class MainProductBlockProvisioning(MyModel):
    values: ListMax2[SubProductBlockProvisioning]  # type: ignore

2. Typed List

class ListMax2Numbers(SubscriptionInstanceList[int]):
    max_items = 2

Becomes:

ListMax2Numbers = Annotated[list[int], Len(max_length=2)]

Which can then be used in a pydantic model without a type subscription

class MyModel(BaseModel):
    values: ListMax2Numbers

UniqueConstrainedList

Class type has been removed. Replace with one of * unique_conlist(T) can be imported from pydantic-forms * Annotated[list[T], AfterValidator(validate_unique_list)] * Annotated[set, ...]

Before:

class ListOfTwo(UniqueConstrainedList[T]):
    min_items = 2
    max_items = 2

After:

ListOfTwo = Annotated[list[T], AfterValidator(validate_unique_list), Len(2, 2)]

Recommendations and examples

Set validators on the annotated type rather than on a FormPage/Model

For example, instead of:

AddedServicePorts = conlist(
    BgpServicePort, min_items=0, max_items=6 - len(current_service_ports)
)


class ModifySN8IPForm(FormPage):
    added_service_ports: AddedServicePorts

    _validate_single_vlan: classmethod = validator(
        "added_service_ports", allow_reuse=True
    )(validate_single_vlan)
    _validate_unique_vlans: classmethod = validator(
        "added_service_ports", allow_reuse=True
    )(validate_service_ports_unique_vlans)

Write:

AddedServicePorts = Annotated[
    list[BgpServicePort],
    Len(0, 6 - len(current_service_ports)),
    AfterValidator(validate_single_vlan),
    AfterValidator(validate_service_ports_unique_vlans),
]


class ModifySN8IPForm(FormPage):
    added_service_ports: AddedServicePorts

This encapsulates validation logic to the type, making it easier to reuse in other places.

This also makes it possible to extend types, for example we could write the following to perform an extra validation on top of the existing ones.

AddedServicePorts_Extra = Annotated[AddedServicePorts, AfterValidator(validate_extra)]

(mypy) Define a TypeAlias for Choice fields

A form using a Choice with dynamic values requires a # type: ignore on the FormPage field to prevent mypy errors.

Example in core v1:

from orchestrator.forms.validators import Choice
from orchestrator.forms import FormPage
from orchestrator.types import FormGenerator, SubscriptionLifecycle, UUIDstr


def node_selector(enum: str = "NodesEnum") -> Choice:
    node_subscriptions = subscriptions_by_product_type(
        "Node", [SubscriptionLifecycle.ACTIVE]
    )
    nodes = {
        str(subscription.subscription_id): subscription.description
        for subscription in sorted(
            node_subscriptions, key=lambda node: node.description
        )
    }
    return Choice(enum, zip(nodes.keys(), nodes.items()))  # type:ignore


def initial_input_form_generator(product: UUIDstr, product_name: str) -> FormGenerator:
    class SelectNode(FormPage):
        class Config:
            title = f"{product_name} - select node"

        node_subscription_id: node_selector()  # type:ignore # noqa: F821

    select_node = yield SelectNode

The same code migrated to core v2, and with a TypeAlias to prevent the # type: ignore on the FormPage field.

from pydantic import ConfigDict

from orchestrator.types import SubscriptionLifecycle, UUIDstr
from pydantic_forms.core import FormPage
from pydantic_forms.validators import Choice
from pydantic_forms.types import FormGenerator


def node_selector(enum: str = "NodesEnum") -> type[Choice]:
    node_subscriptions = subscriptions_by_product_type(
        "Node", [SubscriptionLifecycle.ACTIVE]
    )
    nodes = {
        str(subscription.subscription_id): subscription.description
        for subscription in sorted(
            node_subscriptions, key=lambda node: node.description
        )
    }
    return Choice(enum, zip(nodes.keys(), nodes.items()))  # type:ignore


def initial_input_form_generator(product: UUIDstr, product_name: str) -> FormGenerator:
    NodeChoice: TypeAlias = cast(type[Choice], node_selector())  # noqa: F821

    class SelectNode(FormPage):
        model_config = ConfigDict(title=f"{product_name} - select node")

        node_subscription_id: NodeChoice

    select_node = yield SelectNode

What now?

After following all of these tips your IDE may still show plenty of errors in your orchestrator.

The best thing to do next is try to start your orchestrator and see which errors are critical to actually running it.

After that, you can start running your testsuite. Pytest has (among many other things) the helpful option --stepwise that runs all your tests until the first failure. Rerunning the command will resume running tests from the last failed test. This will make it more manageable to start fixing the errors.

Once that is done, your pytest or orchestrator output may still contain many warnings to look into regarding deprecated changes.

If you are not sure how to proceed, don't hesitate to reach out through Slack or a Github Discussion.