Arkindex 1.10.0

We are happy to announce that a new Arkindex release is available. You can explore Arkindex and try out the newest features on our demo instance, demo.arkindex.org.

IIIF 3

The main focus of this release has been to add support for IIIF Image API 3.0 and IIIF Presentation API 3.0:

Image servers now have a concept of IIIF version to distinguish between IIIF 2 and 3.
This version is available in most API endpoints, as well as database exports, to help workers, API users and other tools to generate compatible image URLs.
The IIIF import worker now supports IIIF 3 manifests and collections, as well as IIIF 3 images embedded within IIIF 2 manifests.
When viewing an element, the IIIF version used for its image is now shown below the image.

This development was supported by the pictorIA consortium-HN.

Elements

A new Elements Reorder elements by name action allows to reorder the child elements of any element by their name. This can help in resolving common ordering issues after importing data into an existing folder or running multiple processes on the same page.
When viewing a folder and its children with filters applied, navigating to the previous or next folder now preserves those filters.
Fixed an issue that caused the Delete filtered elements action to be disabled when it should have been enabled and vice-versa.

Processes

Some workers print log messages that expect a dark background like in most terminals. To keep those log messages readable, task logs now use a dark background by default. The background can be toggled between dark and light with a new button on each task:

Figure 1. Togglable background color in task logs
When a file import is completed, a View element button is now shown next to the Actions dropdown to make it easier to browse to the newly imported data:

Figure 2. Button to view imported elements
Export processes will now output ZIP archives (.zip) instead of Zstandard-compressed TAR archives (.tar.zst), to make them easier to extract on all operating systems.
In Enterprise Edition, stopping a process will now mark Pending tasks without Ponos agents assigned as Stopped instead of Stopping. Processes should no longer be stuck in a Stopping state in this situation.

Workers

Worker version fields within worker configurations now also allow a No worker version option.

This explicitly selects the absence of a worker version, instead of selecting nothing at all. This can be used to select elements, transcriptions, etc. that have not been created by any worker version, thus have likely been created manually.
The CreateWorkerVersion API endpoint has been deprecated. This endpoint was used to create worker versions using the old worker configuration format (.arkindex.yml).

ImportWorkerVersion should be used instead, along with the new worker configuration format.
The arkindex worker publish command, which was used to create workers and worker versions using CreateWorkerVersion, has been deprecated.

arkindex worker import should be used instead, along with the new worker configuration format.
Fixed an issue in ImportWorkerVersion that would cause HTTP 500 errors when a configuration field has a default of null.

Search

The search page now uses a simpler query syntax and should handle syntax errors more gracefully. This new syntax, along with the rest of the search feature, is now documented.
Some performance optimizations have been made on the search indexation. We are aware of other existing performance issues that may be resolved in future releases.

Misc

In Enterprise Edition, the emails sent by the arkindex cleanup_unused_corpus command now include a link to the project.
In database exports, the worker run export step should now run quicker on large instances.

Upgrade notes

To upgrade a development instance, follow this documentation.

To upgrade a production instance, you need to:

Deploy this release’s Docker image: registry.gitlab.teklia.com/arkindex/backend:1.10.0
Run the database migrations: docker exec ark-backend arkindex migrate
Update the system workers: docker exec ark-backend arkindex update_system_workers

The main changes impacting developers and system administrators are detailed below.

Image server versions

The changes we made to support IIIF 3 include adding a concept of IIIF version to distinguish between IIIF 2 and 3 on Image servers. The database migrations will assign version 2 to all existing image servers. If you already had some IIIF 3 image servers in your database, you will need to update their version manually.

To verify whether an image server uses IIIF 2 or 3, you can ask Arkindex to perform a new image check on any image of this server using PartialUpdateImage with {"status": "checked"}. Any incompatibility between the server’s attributes and those found on the image will result in an error, including the IIIF version.

If you are using the Cantaloupe IIIF server and wish to use IIIF 3, you can change the image server’s URL to use iiif/3 instead of iiif/2. This is the case in the Arkindex development setup and with Teklia’s IIIF servers.

MinIO replaced with Garage

The development setup included a MinIO container to provide S3 storage. Due to MinIO not providing Docker images or documentation anymore to open-source users, we are switching to Garage to provide a local S3 compatible server. Developers will need to either start from scratch, losing all of their data on the local S3 buckets, or to migrate manually. In all cases, they will need to pay special attention to some changes they may have to make to their configurations.

To start from scratch

Ensure all Arkindex containers are stopped: docker compose -p arkindex down
Erase the previous MinIO data: docker volume rm arkindex_miniodata
Restart the backend’s third-party services.
If you haven’t done so already, you will need to setup local dependencies for the Arkindex backend.
Ensure your database is up to date, as this may prevent running other Arkindex commands: arkindex migrate
Re-run the development setup bootstrap command to configure Garage automatically: arkindex bootstrap

To migrate existing data

Migrating existing data requires running both MinIO and Garage at the same time, copying data between the two, then erasing the old MinIO data.

As a precaution, those instructions will also backup every bucket into a local directory so that they can be re-uploaded later manually to Garage if the migration went wrong.

Start the backend’s third-party services from this new release, including Garage.
If you haven’t done so already, you will need to setup local dependencies for the Arkindex backend.
Ensure your database is up to date, as this may prevent running other Arkindex commands: arkindex migrate
Run the development setup bootstrap command to configure Garage automatically: arkindex bootstrap

Run the following script to migrate data from MinIO:

# Start a temporary MinIO server for this migration
docker run \
    --rm \
    --name ark-minio-migration \
    --detach \
    --publish 9002:9002 \
    --env MINIO_ROOT_USER=minio1234 \
    --env MINIO_ROOT_PASSWORD=minio1234 \
    --volume arkindex_miniodata:/data \
    --volume "$(pwd)"/minio_backup:/backup \
    --network arkindex_default \
    minio/minio:RELEASE.2025-02-28T09-55-16Z \
    server /data --console-address :9002

# Configure the MinIO client to perform the migration
docker exec -it ark-minio-migration mc alias set minio http://localhost:9000/ minio1234 minio1234
docker exec -it ark-minio-migration mc alias set garage http://ark-garage:3900/ GKAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

for bucket in export iiif-cache ingest ponos-artifacts ponos-logs staging thumbnails training uploads; do
    mkdir -p "minio_backup/$bucket"
    # Backup the MinIO data locally
    docker exec -it ark-minio-migration mc mirror --preserve "minio/$bucket" "/minio_backup/$bucket"
    # Run the migration on each bucket
    docker exec -it ark-minio-migration mc mirror --preserve --skip-errors "minio/$bucket" "garage/$bucket"
done

Use the MinIO client built into the migration server, or use any other S3 client, to check that the data has been properly migrated to Garage:
```
# List all buckets
docker exec -it ark-minio-migration mc ls garage/
# List contents of a bucket
docker exec -it ark-minio-migration mc ls garage/uploads/
```
The list of buckets should include all of the buckets that the Arkindex dev setup uses:
- export
- iiif-cache
- ingest
- ponos-artifacts
- ponos-logs
- staging
- thumbnails
- training
- uploads
If you had any files on those buckets, listing the contents of those buckets should still show those files.
Once you are satisfied, run the following to clean up the remaining MinIO data:
```
docker stop ark-minio-migration
docker volume rm arkindex_miniodata
```

Notable changes

No matter how a development setup was upgraded, there may be some other changes to make on the local backend configuration or on the system. This section references some known significant changes that may impact development.

Regions

MinIO did not check the validity of the S3 region, and accepted any value. Garage requires a region and checks its validity. We have configured Garage to expect the local region. You may need to update the s3.region or ingest.region settings to this value.

Updated URLs

While MinIO was using minio.ark.localhost as its S3 API endpoint, Garage uses s3.ark.localhost. While MinIO offered a web console at minio-console.ark.localhost, Garage does not provide any such user interface. An administration API is instead available at garage.ark.localhost.

New credentials

The S3 access key ID and secret access key had to be changed because Garage requires those to have a specific format. We have updated the configuration used to run Arkindex in Docker with make stack, but you may need to also update any configuration used to run Arkindex locally, particularly arkindex/config.yml:

s3:
  access_key_id: GKAAAAAAAAAAAAAAAAAAAAAAAA
  secret_access_key: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
  endpoint: https://s3.ark.localhost
  region: local

You may also need to apply the same settings in the ingest section of the configuration for the S3 ingest feature.

Instead of updating an existing configuration, you can recreate it from scratch with our sample, like what is usually done when setting up for the first time:

cp config.yml.sample arkindex/config.yml

To test your updated configuration, you can try to upload files to any project.

New request style

Garage supports both path-style and virtual-hosted—style requests, while we only had set up MinIO for path-style requests. For example, a GetObject request can be made both at https://s3.ark.localhost/bucket/key or at https://bucket.s3.ark.localhost/key. Using virtual hosts may cause SSL issues as the local certificate was not meant for such subdomains.

If you encounter SSL issues with this new style, you can delete docker/ssl/ark-cert.pem and restart the third-party services. This will generate an updated SSL certificate with support for the new subdomains.