Arkindex 1.6.3

We are happy to announce that a new Arkindex release is available. You can explore Arkindex and try out the newest features on our demo instance, demo.arkindex.org.

New menus for projects and elements

The Actions menu, found when browsing a project or an element, has been split up into multiple menus to make it easier to find the feature you were looking for.

Figure 1. New menus for projects and elements

Permissions management

We have replaced our user scopes system with a simpler, more user-friendly permission system. This refers to permissions that are not related to projects, workers, or models, but that are instance-wide, such as uploading images or importing from an S3 bucket.

The profile page now includes a Permissions tab showing which permissions have been granted for the current user:

Figure 2. The new user permissions page

Instance administrators can also view and edit those permissions more easily through the administration interface:

Figure 3. Managing a user’s permissions in the administration interface

The worker and worker version creation and edition features now respect the new worker permissions, which can prevent abuse of computing and storage resources on instances with open registrations.

Repository removal

We have simplified the management of worker versions by removing repositories, revisions and Git references. Instead, some attributes are stored directly on workers and worker versions:

Workers may have repository URLs;
Worker versions may have a commit URL, a branch, and a tag.

As a consequence, we have simplified the workers listing to display more relevant information and made it consistent across Arkindex.

Figure 4. Simplified worker versions list

System workers

A new concept of system workers has been introduced to replace our tasks project, allowing workers to provide some Arkindex features. The init_elements task in Workers processes and the import task in S3 ingest processes are now provided by workers. File imports will be migrated in a later release.

We have made many upgrades to make these essential workers easier on system administrators (instead of managing arkindex tasks upgrades). Check out the technical release notes to learn more. This change only concern system administrators.

Processes

When a task ends, any worker activities that were Queued or Started are automatically marked as Failed, making it easier to try processing those again.

In Community Edition, Docker containers for tasks now have names and can be cleaned up automatically. Learn more in the technical release notes.

Many other bug fixes have been made:

Adding failed elements to your selection from a process is now possible even when a previous run of the process did not complete successfully.
Creating or applying a template now properly clones the selected GPU usage for each worker version.
It is no longer possible to apply a template on a process where a template was already applied.
Fixed a bug that hid the Show all models toggle when selecting a model version to use in a process.
Fixed a bug that prevented highlighting the currently selected model when selecting a model version to use in a process.
Multiple pagination errors were fixed with the Show all models toggle, as well as the Display all versions toggle found in worker version lists and worker details.

CLI

The worker publication command uses the updated API endpoints to handle the removal of Git repositories from Arkindex.
CSV exports now also include the rotation angle of each element.
Fixed missing classification confidences when running a CSV export with the --with-classes option.
Fixed a crash when running a DOCX export without the --merge option.
The arkindex elements ml-splits command now uses the random order feature of the Arkindex API to list the elements to split faster.

API updates

A significant breaking change has been made in the API in this version.

The classes attribute of elements returned in ListElements, ListElementChildren and ListElementParents has been renamed to classifications, as it contains classifications. This matches the attribute already returned in RetrieveElement.

Small improvements were made on other endpoints:

Calling CreateElement with image or polygon set to null now behaves as if those properties were missing from the request, instead of causing validation errors.
UpdateElement and PartialUpdateElement can now remove the image and polygon from an element when they are set to null.
Calling DestroyElements, DestroyElementChildren or DestroyElementParents without setting the delete_children parameter now sets it to True by default, to match the behavior of the frontend when using the Delete filtered elements action.
Updating a worker’s type with UpdateWorker or PartialUpdateWorker will create a worker type automatically if it does not exist.
Multiple errors have been fixed on the CreateElementChildren endpoint, and significant performance improvements have been made for large instances.

Misc

When there are no projects on the instance, or none accessible to the user, a message now invites the user to create a new one.
Reaching the Page not found message now properly shows whether you are currently logged in.
In the project information page, the search indexation options are no longer visible when the search feature is disabled on the instance.
[Enterprise Edition] In the administration interface, viewing a group that has many rights on many objects should now load faster.
The API client no longer depends on the unmaintained APIStar project.

Upgrade notes

A new Arkindex release is available.

To upgrade a development instance, follow this documentation.

To upgrade a production instance, you need to:

Deploy this release’s Docker image: registry.gitlab.teklia.com/arkindex/backend:1.6.3
Run the database migrations: docker exec ark-backend arkindex migrate
Update the system workers: docker exec ark-backend arkindex update_system_workers

The main changes impacting developers and system administrators are detailed below.

Simpler management of system workers

System workers, or workers that provide specific Arkindex features like Workers processes and S3 ingest processes, are no longer managed using the YAML configuration. This means the following keys can be removed from your configuration files:

docker.tasks_image
docker.init_elements_image
docker.ingest_image

Previously, these values had to be manually set to the correct tag of the correct Docker image, then a worker version needed to be created manually with the same tag and specific settings. Upgrading required restarting the backend once or twice to apply and validate the necessary changes.

The assignation of a Docker image as providing a specific feature is now done through the worker versions themselves. Worker versions now have a feature attribute, which lets them be assigned to specific Arkindex features:

init_elements: The elements initialisation worker used in Workers processes to list the elements to process and distribute them between chunks.
s3_ingest: The worker that executes S3 ingest processes. This is not used when the ingest feature flag is disabled.
file_import: Reserved. In a future version, this worker will execute file import processes.

The feature attribute is unique: only one worker version at a time can provide each of the features.

The recommended way to manage features is to use the new arkindex update_system_workers command, which will in most cases be able to automatically create or update all of the necessary worker versions and assign features, using a set of Docker images that our team recommends for a specific Arkindex release. We also recommend that this command be run on each Arkindex instance upgrade, so that future updates of those workers are still reflected in your database.

Features can also be manually configured through the administration interface.

The W014 and W015 system checks have been updated to mention this new feature system and recommend the use of arkindex update_system_workers.

Arkindex developers also need to be aware of the new E012 system check, which could fail when releasing a new version of Arkindex if the recommended Docker images that update_system_workers relies on have not been manually approved for this new release.

S3 ingest feature flag

To simplify deployments, the S3 ingest feature is now disabled by default, and can be toggled using the new features.ingest feature flag.

When this feature flag is disabled:

The S3 ingest options in the frontend are not visible.
The ListBuckets and CreateS3Import endpoints are not available.
Existing S3 ingest processes cannot be retried.
There will be no system check warnings if no worker version exists to execute S3 import processes.

Docker container management

When running tasks with Community Edition, the Docker containers now have names starting with ponos- and ending in the UUID of each task, instead of the default names provided by Docker.

In addition, a new ponos.auto_remove_container option has been introduced. When enabled, each container is automatically removed once the task finishes.