Upgrade a worker

This page documents how to upgrade an existing Arkindex worker (using base-worker < 0.5.1) to this new configuration system.

Requirements

  • The worker source code must be available on your computer.

  • The worker source code must contain a .arkindex.yml file describing one or several Arkindex workers.

  • The worker source code must have a valid CI pipeline with pre-commit.

Step 1: Upgrade base-worker to 0.5.0

At the end of upgrade process, we’ll need to upgrade your worker dependency towards base-worker 0.5.1 (or whatever more recent version is available at the upgrade time). This will be easier if you first have a worker based on base-worker 0.5.0.

Make sure the worker runs as expected after this upgrade.

Step 2: Automatically convert the configuration

We provide a CLI tool named worker-validator in this package, which you can install locally and then use to convert your .arkindex.yml file.

To convert the configuration, run the following command in your worker directory:

worker-validator convert

A few changes happen when the conversion has been successful:

  • A new folder named arkindex will contain one YAML file per worker.

  • The .arkindex.yml file has been removed.

To start the conversion process, create a git branch and commit all these changes. Make sure the .arkindex.yml is removed.

If the conversion process fails, errors will be displayed explaining which part triggered a failure. You can always ask on our support forum for help.

Step 3: Validate your configuration

Once each worker has its own YAML file in arkindex/, you should add a pre-commit hook to your local .pre-commit-config.yaml which will validate the syntax of all these configuration files.

To add the hook, append the following to .pre-commit-config.yaml in the repos section:

  - repo: https://gitlab.teklia.com/arkindex/workers/configuration
    rev: 0.2.0
    hooks:
      - id: validate-worker-configuration

Now commit that change, and run pre-commit run -a to validate all the files.

Step 4: Simplify description

As the new configuration system will describe automatically all your fields, we encourage you to remove any tables that describe options from your description. The description will be shorter and easire to maintain, letting Arkindex display all the configuration options neatly.

In some cases, the description may also have been wrapped in double-quotes ", contains escaped line breaks \n and is poorly readable. This is likely caused by the description containing lines that start with spaces, something which is not supported by YAML’s multiline string syntax.

To summarize, this step is optional if your worker description is already short and readable in the converted files.

Otherwise, you’ll need to manually edit the description, and make a new commit with these changes.

You should also make a commit removing the now outdated description Markdown files as they are no longer used by the configuration.

Step 5: Upgrade base-worker to 0.5.1

Before testing the new configuration, you need to update base-worker dependency to version 0.5.1 (at least, but use whatever more recent version is available).

To upgrade:

  1. edit dependencies section in pyproject.toml, and change the base-worker version

  2. update your local environment by doing pip install -e .

  3. commit your changes

Step 6: Use secrets

The new worker configuration supports a secret type which allows you to use Arkindex secrets from your worker as any other configuration value. The main benefit is for the worker user that can select its own secret through an easy-to-use interface on Arkindex.

Every secret listed in the old configuration under the secrets list must be converted to a new option with type: secret such as:

- key: openai
  display_name: OpenAI credentials to use Chat-GPT on your behalf
  type: secret

Secrets are still saved in self.secrets, you don’t need to update your worker’s code.

Then you should be able to test the worker with its new configuration on your usual Arkindex test instance.

It is important that you test the worker before making potential code changes in the next step.

Step 7: Group options

The new worker configuration supports a group type which allows you to group together meaningful options.

You should use that type when you repeat the same prefix over several options, and if it makes more sense to the worker’s users to group them.

You’ll need to edit both the worker configuration and worker source code, as it changes the configuration option’s key.

To create a new group, add a new option whose key is the shared prefix:

- key: my_prefix
  display_name: A group of options
  type: group
  children: []

Then move all the options you want under the children list, removing their prefix so their key becomes shorter.

For example, if you had my_prefix_option_A, this will become option_a under the group my_prefix:

- key: my_prefix
  display_name: A group of options
  type: group
  children:
    - key: option_a
      display_name: Some option in a group
      type: int
      default: 42

Rinse and repeat for all the options in the group.

Once you have edited your YAML file, you need to update the worker code as the options key changes from my_prefix_option_a to my_prefix.option_a. You need to find all occurrences of self.config for the old keys, and change for the new format.