Arkindex 1.7.3

We are happy to announce that a new Arkindex release is available. You can explore Arkindex and try out the newest features on our demo instance, demo.arkindex.org.

Budgets

As part of a broader plan to better control the execution costs of a project, we have started introducing a new concept of Budgets.

Along with the previously introduced worker costs, budgets will allow to keep track of the execution costs of a process as well as cancel a process if it exceeds the allowed budget. This feature will only be available in the Enterprise Edition.

In this release, only the foundation has been laid out. Budgets are only available through APIs and in the administration interface, and they are not automatically filled in from the costs of each process. They will be accessible in the frontend and will interact with processes in the next release.

This development was supported by the pictorIA consortium-HN.

Process management

In Arkindex 1.7.0, we had introduced a concept of time-to-live (TTL) to limit the execution time of each task on a process, either per project or on the whole Arkindex instance.

An undocumented worker run TTL was also available through the API, but not through the frontend. It was meant to allow finer control over each TTL in a process. This TTL has been removed in this release because it was not worth the additional complexity.

This removal may resolve some unexpected behaviors that you may have noticed with TTLs, particularly when retrying processes or applying templates.

Datasets

The Generic Training Dataset Extractor has been added as a new system worker to save some time when preparing a dataset.

This worker is used to generate the necessary dataset cache archives before training a model or running an inference worker on a dataset. This update makes it easily available on all Arkindex instances in the same way as the workers used for imports or exports.

Misc

  • All Arkindex projects now officially support Python versions from 3.10 to 3.12. Support for Python 3.10 will be dropped in future release 1.9.0. Developers can read the upgrade notes for more information on upgrading.

  • In Arkindex Enterprise Edition only, Ponos agents now use a simpler authentication method that makes setting up an agent quicker.

  • Fixed a bug that intermittently prevented selecting any project on the search page, especially when logged out.

  • The Restart button on tasks is now disabled when you do not have the necessary access rights to restart a task.

Upgrade notes

To upgrade a development instance, follow this documentation.

To upgrade a production instance, you need to:

  • Deploy this release’s Docker image: registry.gitlab.teklia.com/arkindex/backend:1.7.3

  • Run the database migrations: docker exec ark-backend arkindex migrate

  • Update the system workers: docker exec ark-backend arkindex update_system_workers

The main changes impacting developers and system administrators are detailed below.

Python 3.12 upgrade

Arkindex now officially supports both Python 3.10 and 3.12. The official Docker image uses Python 3.12. Support for Python 3.10 will be dropped in version 1.9.0.

Developers are encouraged to upgrade their local installations to Python 3.12 now by recreating their virtual environment and reinstalling all dependencies:

cd arkindex/backend
rmvirtualenv arkindex
mkvirtualenv -a $(pwd) -p /usr/bin/python3.12 arkindex
pip install -e '.[test]'

For more information on setting up the virtual environment from scratch, see the development setup documentation.

Simplified Ponos agent authentication

This change only affects Arkindex Enterprise Edition.

The authentication method used by Ponos agents has moved away from JSON Web Tokens and private keys to a simpler method similar to task token authentication. As a consequence:

  • All Ponos agents will need to upgraded at the same time as upgrading the Arkindex instance.

    Arkindex 1.7.3 requires Ponos agents with version 0.12.4 Ponos agents with version 0.12.3 and earlier will only support Arkindex 1.7.2 and earlier.

  • The jwt_signing_key setting has been removed, since it was only used by the previous authentication system. You can remove it from your backend YAML configuration.

  • The private_key setting has been removed from the Ponos agent configuration.

  • The private key files previously used by Ponos agents are no longer used and can be removed.