Writing tests

API validation tests

All APIs are expected to have good API validation test coverage.

API validation tests are tests which ensure that any validations defined in the API and included in the CRD perform as expected. Add API validation tests for your controller in test/apivalidations.

E2E tests

All controllers are expected to have good test coverage. ORC uses kuttl for end-to-end testing.

General consideration when writing kuttl tests

Documentation

Kuttl tests can get very complicated very quickly and it can be difficult to follow what each of them does. We ask that each test contains a README.md file describing what it does. Don't hesitate to sprinkle your test files with comments as well.

Naming convention

Each test runs in a separate Kubernetes namespace, however because they all run in the same OpenStack tenant, we still need to be careful about name clashes when referencing resources, and filters for importing resources:

• a test must only reference resources it created. Resources created on OpenStack must have a unique name among the whole test suite. Typically, you would use the name of the test wherever possible.
• a test must ensure their filters match resources uniquely

These constraints ensure that tests can safely run concurrently.

Also, to make things easier when debugging, we require that all tests are named differently. In kuttl, the test name is the name of the directory containing the test files. For this reason, we prefix all the test names with the name of the resource being tested, to differentiate them.

Resource cleanup

At the end of each test, kuttl destroys the namespace. This deletes all leftover ORC objects, and respective OpenStack resources, your test has created.

However, if you have created OpenStack resources outside of ORC as part of the test, you MUST clean them to avoid leaking resources. Keep in mind that if the test fails before it had a chance to manually clean the resources, you would still have a leak. As a counter measures, only create OpenStack resources externally when it is an absolute necessity, and avoid writing tests that fail.

Testing patterns

This section describes the expected tests for controllers. Use all the ones that apply to your controller, to ensure the best coverage.

create-minimal

The create-minimal test consists of creating a resource, setting only the required fields and validating that the observed state corresponds to the spec.

All controllers should implement this test.

When a resource doesn't have any required field, use an empty resource spec:

apiVersion: openstack.k-orc.cloud/v1alpha1
kind: Network
metadata:
  name: network-create-minimal
spec:
  cloudCredentialsRef:
    cloudName: openstack
    secretName: openstack-clouds
  managementPolicy: managed
  resource: {}

Because this test creates a resource with the minimal dependencies, we overload it to validate the dependency on secrets. Every resource that interacts with OpenStack should have a dependency on the credentials secret, meaning that we can't delete the secret while the object exists.

Concretely, after trying to manually delete the secret with kubectl delete secret openstack-clouds --wait=false, we check that the secret was flagged for deletion but not deleted due to the presence of finalizers. We use a CEL assertion for this:

apiVersion: kuttl.dev/v1beta1
kind: TestAssert
resourceRefs:
    - apiVersion: v1
      kind: Secret
      name: openstack-clouds
      ref: secret
assertAll:
    - celExpr: "secret.metadata.deletionTimestamp != 0"
    - celExpr: "'openstack.k-orc.cloud/flavor' in secret.metadata.finalizers"

The network-create-minimal test is a good example of this test.

create-full

Very similar to the create-minimal test, except that it sets all available fields of the resource spec.

Sometimes you'll find options that are mutually exclusive. In that case you may need to write separate tests to exercise all the options. Try to come up with a meaningful name for the scenario, for example create-sriov.

All controllers should implement this test.

It can be seen with the network-create-full test for example.

dependency

Whenever your controller has dependencies on other resources, we want to ensure it is able to wait for the resource to exist and be ready, and optionally have a dependency guard on the resource to prevent its deletion while we depend on it.

To know if your controller needs to implement this test, check the API and see if it accepts any <Resource>Ref field in its resource spec. This is for example the case for a subnet, which has a dependency on a network, and possibly a router.

The testing pattern goes like this:

1. Create the resource with all dependencies satisfied except for one
   1. Check that ORC is waiting on the missing dependency to exist
2. Repeat with another dependency until you've checked all possible dependencies
3. Create the missing dependency
   1. Check that the resource is finally created
4. Delete all dependencies
   1. Validate that ORC prevented the deletion due to a finalizer. There might be exceptions (for example flavor for a server)
5. Delete the resource
   1. Verify that all dependencies are now deleted

See how the subnet-dependency test implements it.

import

The import test validates import filters, and adoption.

The testing pattern goes like this:

1. Create an unmanaged resource using all of the available filters for the resource
   1. Verify it is waiting on an external resource matching the filters to exist
2. Create a resource matching all of the import filter except for the name that is a superstring of the one specified in the import filter, e.g. if we are waiting on a resource called foo, create a resource called foobar.
   1. Verify that this resource is not being imported -- it validates that we don't perform regex-based name search (some OpenStack projects do it)
3. Create a resource marching all of the import filters, including the name.
   1. Verify that the imported resource is available and the observed status correspond to the one of the created resource
   2. Validate that the previously created resource wasn't imported as the new one, again, because of regex-based name matching

All controllers should implement this test.

See the flavor-import test for an example.

import-error

In the import-error test, we verify that an import filter matching multiple resources returns an error.

All controllers should implement this test.

See the flavor-import-error test for an example.

import-dependency

The import-dependency test is required for resources that allow a <Resource>Ref field in their import filters.

We want to verify that ORC waits for the dependency to be created and available when it creates the resource, but that it allows deleting the dependency without the resource being deleted first. This ensures that unmanaged resources don't add deletion guards on dependencies.

One such example is the port-import-dependency test.

The pattern for this test is:

1. Create two unmanaged resources: one for the resource being tested, another one for its dependency
   1. Verify that the resource under test is waiting for the dependency to be ready
2. Create a dummy resource matching the import filter except for the dependency
   1. Verify it is not being imported
3. Create resources matching the import filter for the resource being tested and for the dependency
   1. The resource under test must move to Available, and its observed status correspond to the created resource
4. Delete the dependency
   1. Verify it's gone
5. Finally delete the resource
   1. Verify it's gone

update

The update test is required for resources that implement mutability. It should test both setting and unsetting resources properties.

The testing pattern consists in:

1. Create a resource using only mandatory fields, similar to the minimal test
2. Update all fields that support mutability
   1. Verify that the changes have been reflected in the observed status
3. Revert the changes
   1. Verify that the resource status is similar to the one we had in the first step

As support for mutability is still being worked on, we don't have tests that implement this patter yet. The closest we have is the securitygroup-update test.

Kuttl tips

It is possible to shell out and run openstack command:

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
  - script: |
      cd $(dirname ${E2E_KUTTL_OSCLOUDS})
      export OS_CLOUD=openstack
      openstack port list

Similarly, you can run kubectl commands with:

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
  - command: kubectl get port
    namespaced: true

Running tests

We use environment variables to configure how the tests run.

For example, to run the import-dependency test from the subnet controller:

E2E_KUTTL_DIR=internal/controllers/subnet/tests E2E_KUTTL_TEST=import-dependency make test-e2e

Controller-specific tests

Tests other than the above, for example tests covering functionality specific to a single controller, should live in the controller's directory.