Arkindex 1.13.0

We are happy to announce that a new Arkindex release is available. You can explore Arkindex and try out the newest features on our demo instance, demo.arkindex.org.

Datasets

Following evolutions in the way Machine Learning models are being trained on Arkindex, we are simplifying dataset management: dataset states, dataset archives, and the dataset extraction step are being removed.

  • Datasets can no longer be closed or reopened. They now behave like they are always open.

  • Task artifacts cannot be associated with datasets anymore.

  • Any set of any dataset can be used in any training process, without requiring the use of the Generic Training Dataset Extractor worker;

  • The Generic Training Dataset Extractor worker is now archived.

UI changes

  • The buttons to edit and delete a dataset are now also available when viewing a single dataset.

  • The total number of elements across all sets is now visible both in the datasets list and when viewing a single dataset.

  • In the datasets list, it is now possible to filter datasets by name.

Name filter and elements count in the dataset list

Embeddings

This release introduces embeddings, a new type of worker results. Embeddings are vectors produced by models that can be used in clustering or searching by similarity.

Workers can publish and manage embeddings using the new embedding APIs. Embeddings are also available in the latest version of the database exports.

Arkindex currently only allows to store and retrieve embeddings through the Rest API and Project exports. Embeddings are intended to be used by other workers to produce other results that are more meaningful to humans (such as classifications), or by external tools to provide other services (similarity search by content).

Processes

Consistent task states

The definition of the Unscheduled task state has been changed and has been applied more consistently in both editions of Arkindex.

This state now only refers to a task that will not be executed now because it is waiting on something else, or because another part of the process has failed. Any task that may be executed soon enters the Pending state, where it will be prepared and started as soon as compute resources are available.

In Community Edition, this means that processes will no longer have all of their tasks be pending all at once. Tasks will progressively move from Unscheduled to Pending as their parent tasks complete.

In Enterprise Edition, this means that the first tasks of a process will become Pending as soon as the process is ready, instead of having tasks unexpectedly change state only when they are assigned to a Ponos agent.

Elements initialization

Inference processes no longer begin with an initialisation task.

This task was used to list all elements to process and output JSON files, but this was removed in Arkindex 1.12.0. This task was then only used to wait for Arkindex to finish initializing worker activities.

With the help of more consistent task states, inference processes will now only contain tasks for their configured worker runs, and all tasks will remain Unscheduled until worker activities are ready. Only then will Arkindex move the first tasks to Pending.

This helps to make inference processes start quicker, and avoid wasting compute resources on a task that only waited for worker activities to be ready and did nothing else.

API

Removals

  • The skip_elements_json attribute had been removed in Arkindex 1.12.0, but it was still available in RetrieveWorkerRun to help with the transition and permanently set to True. In this release, the attribute is fully removed.

  • In ListProcessElements, the with_image parameter has been removed. This parameter was only used by the initialisation task when listing elements.

New error format

The following API endpoints have been migrated to the new error format:

The documentation for each of those API endpoints contains a detailed description of the structure of this new format.

Misc

  • Pressing the G key to open the command bar no longer types G into the bar on Chromium-based browsers.

  • Cloning training processes now disables worker activities properly, so that starting a cloned training process no longer fails.

  • Cloning processes created from a selection now properly clones the selected elements.

  • When configuring a worker, the modal to select an existing database export now also includes a Dataset column to help distinguish between project and dataset exports.

  • Project guests can no longer change the status of uploaded files without having contributor access.

  • Attempting to delete a file that is currently being handled by a file import process now shows an explicit error message.

  • Creating, updating or deleting budget entries in the administration interface now updates the total of a budget immediately without requiring the use of the arkindex update_budgets command.

Upgrade notes

To upgrade a development instance, follow this documentation.

To upgrade a production instance, you need to:

  • Deploy this release’s Docker image: registry.gitlab.teklia.com/arkindex/backend:1.13.0

  • Run the database migrations: docker exec ark-backend arkindex migrate

  • Update the system workers: docker exec ark-backend arkindex update_system_workers

The main changes impacting developers and system administrators are detailed below.

Removed system workers

Two system workers have been removed in this release:

  • init_elements, because the initialisation taskss that used this worker have been removed from inference processes;

  • dataset_extractor, because the Generic Training Dataset Extractor is no longer used to prepare datasets for training.

Both features will be automatically unassigned from any existing worker version when running the database migrations.

System administrators may want to archive these workers to prevent them from being used in future processes.

Updated Gunicorn access logs

The access logs generated by the arkindex gunicorn command, used to start the Arkindex backend’s server, now only shows the IP provided through the X-Forwarded-For HTTP header. Running arkindex gunicorn without a reverse proxy, or behind a reverse proxy that does not expose the client IP through this header, could cause IP addresses to be missing from the logs.

Embeddings migration

Some projects have been using embeddings in Arkindex before they were implemented as a new worker result. A JSON representation of the vector, as an array of floating-point numbers, was stored as a transcription.

To allow those projects to migrate existing data to proper embeddings, this release introduces a new arkindex migrate_embeddings command. This command will migrate any transcription that contains a valid JSON array of floating-point numbers into an embedding.

As this command is only intended to help with migrating existing projects and not as a permanent feature, it will be removed in Arkindex 1.14.0.

Configurable export batch size

A new export.batch_size setting is available to configure how database exports transfer data from the Arkindex database to an SQLite export. Reducing the batch size can reduce memory usage while exporting, but can slow exports down. This setting can help on instances where exports are executed with significant memory constraints.