Arkindex 1.9.2

We are happy to announce that a new Arkindex release is available. You can explore Arkindex and try out the newest features on our demo instance, demo.arkindex.org.

Processes

The behavior of the Load children option on processes has changed. When running a process from a folder or from a selection, loading children elements will now cause only the children elements to be included. Previously, both the children and the folder or selected elements were included.

This breaking change has been made after we found that most common use cases for the use of Load children involve only wanting to use children elements, and the inclusion of the parent elements was causing confusion. Running a process on a whole project, or running one without using this option, behaves the same as in previous releases.

Model caching

The Enterprise Edition Ponos agent that runs all the Machine Learning tasks is now able to cache the models. So instead of downloading the same model multiple times, it will be shared across tasks running on the same server. This represent a great speed and disk space improvement, especially when using large models, like open source LLMs.

Datasets

It is now possible to populate a dataset from selected elements using the new Populate a dataset option of the Actions dropdown in the selection page. While the existing Add to a dataset set option adds all elements to a single set within a dataset, populating a dataset can spread the selected elements across multiple sets randomly.

The CSV export command of the Arkindex CLI also now supports a new option to export datasets for each element.

User management

In the user management page of the administration interface, a new Disable and clear access rights option under the Action dropdown allows to disable multiple users at once. This removes all of their access rights and permissions.

This makes it easier for instance administrators to remove users without losing any data on their projects, as the user account is still there and can for example still be assigned as the creator of an element.

Misc

  • The description is not anymore required to create a new Project.

  • In the selection page, a new Unselect option has been added to the Actions dropdown to unselect all elements from one project at once.

  • The details page for a model version now includes a button to download the model version’s archive when the model version is marked as available and you have contributor access to the model.

  • RetryProcess now returns the same output as RetrieveProcess, including the newly created tasks. This allows the frontend to display the new tasks immediately after retrying.

Upgrade notes

To upgrade a development instance, follow this documentation.

To upgrade a production instance, you need to:

  • Deploy this release’s Docker image: registry.gitlab.teklia.com/arkindex/backend:1.9.2

  • Run the database migrations: docker exec ark-backend arkindex migrate

  • Update the system workers: docker exec ark-backend arkindex update_system_workers

The main changes impacting developers and system administrators are detailed below.

Model cache on Ponos agents

This only affects the Enterprise Edition of Arkindex.

Ponos agents will now download model versions in a models directory under data_dir. This allows each model version to be re-used in multiple tasks, which reduces bandwidth and disk space usage. Tasks can start quicker, especially when running a process with multiple chunks.

A new ARKINDEX_MODEL_DIR environment variable is provided to each task to specify the path to the model version. The previous undocumented download path of $PONOS_DATA/extra_files/model is still available, but it may be removed in future releases. Arkindex workers should be updated to use ARKINDEX_MODEL_DIR instead.

There is no automatic cleanup of the models directory. Instance administrators may safely delete it when no tasks are currently running.

The integrity of downloaded model versions can only be verified once they are first downloaded. When the directory for one model version already exists under models, it will be re-used without re-downloading and without any integrity checks. To force a re-download, delete the parent directory entirely.