Arkindex 1.12.0
We are happy to announce that a new Arkindex release is available. You can explore Arkindex and try out the newest features on our demo instance, demo.arkindex.org.
Removals
Old configuration format
As announced in the previous release, support for importing workers based on the previous configuration format with .arkindex.yml has been removed. Workers that were previously imported may continue to work, but they will not be officially supported.
Worker developers must follow the migration guide to migrate workers to the new configuration format.
The configuration and modern_configuration attributes have been removed from worker versions. This also affects the APIs related to worker runs.
JSON elements listing
As announced in the previous release, support for the generation of elements.json files in Workers processes has been removed. All workers will now need to call StartWorkerActivity to request a new worker activity to process.
The Skip JSON elements listing option has been removed from the advanced settings of processes.
The associated skip_elements_json attribute has been removed from most APIs, with the exception of RetrieveWorkerRun, for backwards compatibility with workers that would assume the value to be false when it is missing.
Worker developers must ensure their workers use version 0.5.1 or later of Base Worker to support this change, and should use version 0.5.2b4 or later to be ready for the removal of skip_elements_json from RetrieveWorkerRun.
Worker cache support
The Cache optimization option has been removed from the advanced settings of processes. The associated use_cache attribute has been removed from the process APIs.
This feature was originally meant to help lower the amount of API requests made by workers. Over the years, other performance and stability improvements have outweighted the maintenance costs of this feature, and it has not been used in many processes. This option is also incompatible with the aforementioned removal of the JSON elements listing.
Workers
The list of worker versions for a worker has been improved. The Version, Tag and Branch columns have been merged into one to save space, and tags and branches use icons to be easier to distinguish. A new button allows to open the details page for a specific worker version, which includes the list of worker configuration fields that it defines.
Worker configurations
A limited subset of Markdown can now be used on the optional help text of worker configuration fields. This can be used for example to provide links to external documentation.
The RetrieveWorkerRunConfiguration API endpoint now ignores fields defined in the configurations of model versions that were not also defined as worker configuration fields. Model versions may still override fields that have been declared as not editable by users. This means that workers are now guaranteed to only receive values for fields that have been defined in their YAML configuration and nothing else.
User worker runs
User worker runs have been greatly simplified in this release.
They now only require to type a custom summary, allowing users to more explicitly define what this worker run is for:
User worker runs are automatically assigned to worker versions in a single system worker managed by the API. They no longer require creating single-use worker versions that cannot be used in real processes, or creating workers solely for the purpose of generating specific summaries.
API
Image management helpers
The latest release of the Arkindex API client introduces new functions to create, upload or manage images on Arkindex using the existing APIs. We recommend using those functions rather than calling CreateIIIFURL or CreateImage directly in scripts and workers. They can handle already existing images, re-upload images that were not created properly, and are ready for a future migration of those endpoints to the new error format.
The system workers have been updated to use those functions. This can make file imports and S3 bucket imports more reliable.
Misc
In Enterprise Edition, the CPU frequency is no longer stored for Ponos agents. It was not used by the task scheduling algorithm.
Upgrade notes
To upgrade a development instance, follow this documentation.
To upgrade a production instance, you need to:
-
Deploy this release’s Docker image:
registry.gitlab.teklia.com/arkindex/backend:1.12.0 -
Run the database migrations:
docker exec ark-backend arkindex migrate -
Update the system workers:
docker exec ark-backend arkindex update_system_workers
The main changes impacting developers and system administrators are detailed below.
Configuration format migration
This release includes a complex database migration to migrate old worker versions to the new configuration format, numbered process.0060.
This migration is not intended to keep those worker versions working after this release. It cannot change the workers themselves to make them support the RetrieveWorkerRunConfiguration API, or handle any other breaking changes introduced in the development of this new format. This migration only helps to keep older configurations and processes viewable, for example to troubleshoot unexpected outputs long after they were created.
Because the previous configuration format had little to no server-side validation, the contents of the configuration on a worker version might not be exactly what Arkindex used to officially support. The new configuration format also adds new constraints that may not be met by old configurations. To mitigate this, the database migration will ignore default values, empty values, or values that never had any effect when running processes.
Manual review
When the database migration still encounters something that it cannot understand, it will not be migrated, and will not removed from the configuration. The worker version will then be marked as being in an Error state, and the arkindex migrate command will show an error message with the number of worker versions that were not fully migrated.
When this occurs, you can connect to the database directly and inspect the partially migrated worker versions by filtering on modern_configuration. Only the worker versions that were not fully migrated will be left with modern_configuration set to False. This query will show the contents of the remaining configurations:
SELECT version.id, worker.name, version.configuration
FROM process_workerversion version
INNER JOIN process_worker worker ON worker.id = version.worker_id
WHERE NOT version.modern_configuration
ORDER BY worker.name;You can update the configurations and retry the migration to let it recreate modern configuration values from this, or delete the offending values from the configurations. For example, this query lets you delete the old configuration from all worker version of the same worker:
UPDATE process_workerversion
SET configuration = '{}'::jsonb
WHERE worker_id = 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa'::uuid;Skipping the migration
If you have reviewed all of the affected configurations and wish to delete them entirely, you can manually mark the database migration as completed then restart the migration command, which will remove the configuration attribute entirely:
docker exec ark-backend arkindex migrate --fake process 0060
docker exec ark-backend arkindex migratePonos agent registration
| This change only applies to Enterprise Edition. |
Ponos agents must specify a mode to register to Arkindex. This mode defines how tasks are assigned to the agent. Modes were previously named after the technologies used by the agents, such as docker and slurm. In this release, the modes are renamed to be more generic and to reflect their impact on task scheduling, which introduces a breaking change for agents.
Due to this breaking change, only versions 0.16.0 or later will be compatible with Arkindex 1.12.0.
Frontend dependency updates
Many dependencies of the Arkindex frontend have been updated, and some removed. This change has no impact on system administrators, but developers with a local frontend setup may wish to run the following to ensure it is up-to-date with no unnecessary dependencies installed:
npm install
npm prune