Arkindex 1.10.1

We are happy to announce that a new Arkindex release is available. You can explore Arkindex and try out the newest features on our demo instance, demo.arkindex.org.

Navigation

Command bar

The command bar is a newly introduced navigation tool available anywhere on the Arkindex web interface. You can access it by clicking on Go to… in the navbar or by pressing the G key.

The command bar
Figure 1. The command bar

The command bar provides two new ways to navigate on Arkindex:

  • A few shortcuts can be accessed by typing part of their keywords: projects, processes and documentation. This can both help in accessing commonly used sections more quickly for advanced users, and in discovering features for newcomers.

  • An element ID can be pasted in, then pressing Enter will open the details page for this element. This is particularly useful during worker development, or to view an element mentioned in the logs of a process.

The command bar will probably later be extended to support additional shortcuts.

Keyboard navigation of modal dialogs

More than 60 modal dialogs have also been updated to use the dialog features supported by modern browsers. This makes them more consistent, and comes with improvements for keyboard navigation:

  • The Esc key can now close all modals.

  • In most confirmation modals, the Enter key will confirm the action.

  • In most modals with forms, the first form control will be focused immediately when the modal opens. For example, renaming a process opens a modal where you can immediately type in the new name.

Projects

  • The Elements  Reorder elements by name action introduced in the previous release now sorts numbers by their mathematical value, rather than treating them as regular characters. This is also known as a natural sort order.

    For example, instead of 010, 1, 11, 2, the reordering will now result in 1, 2, 010, 11. This also works with elements that contain characters other than numbers, such as Page 1 or book1_chapter1.

  • A new DestroyTranscriptionEntity endpoint has been introduced to allow deleting transcription entities individually.

Imports and exports

  • File imports and S3 imports now have default names set to make them easier to distinguish in the process list.

  • Similarly to the View element button added in the previous release, the Download export button has been moved out the Actions dropdown so that exports can be more easily downloaded from export processes.

  • Document exports now output ZIP archives compressed with an older and less efficient algorithm to improve compatibility on Windows systems.

Workers

  • The worker versions list now features a new Import button with a simplified form to import worker versions using the modern configuration format.

    Worker version import modal
    Figure 2. Worker version import modal

    The form to create worker versions with the previous format is still available under a dropdown next to the new button.

    To learn more, see the documentation page on creating a worker version.

  • The Generic Training Dataset worker is no longer hidden from the workers list like other system workers. This makes it easier to prepare datasets for model training.

Worker configurations

  • Our migration to the modern configuration format continues. The previous format is now considered to be deprecated. No new features will be added, and most issues will not be resolved if they do not occur with the modern format.

    A deprecation notice is now shown on the form to create worker versions with the previous format:

    Deprecation warning in the worker version creation modal
    Figure 3. Deprecation warning in the worker version creation modal

    Worker developers are encouraged to upgrade their workers soon.

  • The modern configuration format now includes a new corpus_export field type that allows selecting a database export. This field is intended to be used by document export workers.

  • The specification for the modern format now includes character limits on worker names, slugs, types, and field keys and display names. This matches the existing limitations in the API. The command line tool has also been updated to enforce those limits.

  • The Docker parameters on both configuration formats have been moved from the configuration of worker versions to separate attributes. The configuration should now only contain options related to the previous configuration format, and will be removed once the migration is complete.

    In the worker version APIs, configuration.docker.command and configuration.docker.shm_size are thus moved to docker_command and docker_shm_size. These attributes are typically not used directly by workers, so this should not affect any worker. The most notable change is that those attributes are now more clearly visible in the worker version details page.

  • Fixed an issue on RetrieveWorkerRunConfiguration that prevented running processes with multiple chunks on Slurm Ponos agents.

  • Most system workers have been updated to use the modern configuration format. This has no impact on the execution of processes.

Misc

  • The RetrieveImage, PartialUpdateImage and UpdateImage API endpoints can now return an s3_put_url under the same conditions as for CreateImage. This allows custom scripts that upload images to recover from errors and reupload an image until it is marked as checked.

Upgrade notes

To upgrade a development instance, follow this documentation.

To upgrade a production instance, you need to:

  • Deploy this release’s Docker image: registry.gitlab.teklia.com/arkindex/backend:1.10.0

  • Run the database migrations: docker exec ark-backend arkindex migrate

  • Update the system workers: docker exec ark-backend arkindex update_system_workers

The main changes impacting developers and system administrators are detailed below.

Updated setting for local Docker images in Ponos agents

These changes only apply to Enterprise Edition.

The docker.check_local_cache setting of Ponos agents has been replaced with docker.cache_usage. This new setting can still provide the same behavior that check_local_cache did:

  • check_local_cache: true must be updated to cache_usage: cache_first;

  • check_local_cache: false must be updated to cache_usage: require_pull.

When upgrading to the latest release of the Ponos agent, the docker.check_local_cache will now be ignored. To ensure the agent continues to behave in the intended way, you must define the new setting. Otherwise, the agent will use the new remote_first default.

To learn more about the new behaviors provided by the docker.cache_usage setting, see the Docker agent configuration documentation.