Arkindex 1.8.0

We are happy to announce that a new Arkindex release is available. You can explore Arkindex and try out the newest features on our demo instance, demo.arkindex.org.

Budgets

In Arkindex 1.7.3, we laid the groundwork for a new concept of budgets, to keep track of the execution cost of processes and (potentially) limit them. This feature is only available in the Enterprise Edition.

In this release, budgets can now be viewed from the frontend. A list of all budgets that users have access to can be found through the Budgets action under the user menu.

The details for each budget include a list of their budget entries, and their access rights can be controlled without using the administration interface.

Budget entries can now also be automatically created for all processes that use workers with costs. System administrators will need to take additional steps to enable these. They can both be listed per budget and per process, through a new Budget entries action in the process status page.

Finally, a first step towards adding limitations based on budgets to processes has been taken. System administrators may enable a new option to prevent running a process that uses a paid worker when there is no budget, or no funds on the budget.

This development was supported by the pictorIA consortium-HN.

Frontend upgrades

Some multi-year efforts towards reducing technical debt on our frontend are approaching their completion. The development tooling has been updated to modern recommendations, helping us find and fix existing bugs and avoid introducing new ones.

Bulma, the design framework the frontend is based on, has been updated to its latest version. While we have made the necessary updates to ensure the user interface remains the same, some cosmetic changes are still visible, particularly with modal dialogs and tooltips.

Bug fixes

Fixed a bug that prevented browsing to the previous or next pages when listing elements in an inference process.
Fixed a bug that prevented browsing to the previous or next pages when browsing search results when no sources were explicitly selected.
Integers are no longer treated as invalid default values on a worker version’s user configuration.
Editing a user or a group’s role in a membership now selects the existing role by default instead of falling back to Guest.
Fixed a bug that could cause the database export to immediately fail when running an export process with a new database export.

Upgrade notes

To upgrade a development instance, follow this documentation.

To upgrade a production instance, you need to:

Deploy this release’s Docker image: registry.gitlab.teklia.com/arkindex/backend:1.8.0
Run the database migrations: docker exec ark-backend arkindex migrate
Update the system workers: docker exec ark-backend arkindex update_system_workers

The main changes impacting developers and system administrators are detailed below.

Support for PostgreSQL 14 dropped

Arkindex now requires PostgreSQL 15 and later. Official support for PostgreSQL 14 is dropped.

When attempting to run Arkindex on any PostgreSQL database with a version below 15, a new critical system check issue will be reported.

This has no impact for developers, as the development setup has been running PostgreSQL 17 since Arkindex 1.7.2.

New budget update command

This feature is only available in Arkindex Enterprise Edition.

A new arkindex update_budgets command has been added to the backend. This command creates new budget entries for all processes that ran using workers with costs. To properly keep track of the current budget usage by processes, system administrators should configure their instances to run this command regularly, similarly to arkindex cleanup.

New setting for process budget enforcement

This feature is only available in Arkindex Enterprise Edition.

A new process_enforce_budgets setting has been introduced to restrict running any process or task when workers with costs are involved. It will not be possible to run a worker with costs unless a budget has funds available, and the user has the necessary access rights to use the budget.

This setting is disabled by default to preserve the current behavior, but may be enabled by default in a future release.

Updated email configuration

Some options in the backend configuration related to sending emails have been changed:

email.user is now optional and nullable. Omitting it, or setting it to null, will disable authentication to the SMTP server.
email.password is now optional and nullable. Omitting it, or setting it to null, will disable authentication to the SMTP server.
A new email.tls setting allows disabling TLS when connecting to the SMTP server. It defaults to true, as Arkindex was only ever using TLS before this option was introduced.

New frontend development tooling

Numerous upgrades have been done on the development and build tools on the Arkindex frontend:

We now use Vite instead of Vue CLI as our build tool.
Unit tests now use Vitest instead of Mocha.
The pre-commit hooks have been replaced with Husky.
The code is now automatically formatted using Prettier.

This removes many deprecated dependencies, and brings the project to the current recommendations for modern Vue.js development. Many issues with linting tools not running properly, or only running well when using Volar, have been resolved.

Developers should update their frontend repositories and run npm install to upgrade their development setup. No other changes are required.

This has no impact for system administrators.

MinIO upgrade

The development setup includes a MinIO container to provide S3 storage, which was pinned to a release from 2021. This older version provided its simple filesystem-based storage using "filesystem mode", a feature that has since been dropped. We have upgraded MinIO to a more recent release. Developers will need to either start from scratch, losing all of their data on the local S3 buckets, or to migrate manually.

This has no impact for system administrators.

To start from scratch

Ensure all Arkindex containers are stopped: docker compose -p arkindex down
Erase the previous MinIO data: docker volume rm arkindex_miniodata
Restart the backend’s third-party services. This will recreate the buckets that Arkindex needs.

To migrate existing data

This is a simplified version of the official migration guide adapted to the MinIO instance provided by the Arkindex dev setup. During the migration, you will have 2 instances on MinIO running in parallel on your computer. This is required to be able to copy all the data from the old instance towards the new one.

Start the old MinIO server in your previous development setup with make services. You can use the 1.7.3 arkindex release for that (git checkout 1.7.3)
Verify that your server is correctly running: the MinIO console should be available with your existing buckets.

Run the following script to perform the migration:

# Start a server for this migration with the new MinIO version
docker run \
    --rm \
    --name ark-minio-migration \
    --detach \
    --publish 9002:9002 \
    --env MINIO_ROOT_USER=minio1234 \
    --env MINIO_ROOT_PASSWORD=minio1234 \
    --volume arkindex_miniodata:/olddata \
    --volume arkindex_minio_migration:/newdata \
    --network arkindex_default \
    minio/minio:RELEASE.2025-02-28T09-55-16Z \
    server /newdata --console-address :9002

# Configure the MinIO client to perform the migration
docker exec -it ark-minio-migration mc alias set old http://ark-minio:9000/ minio1234 minio1234
docker exec -it ark-minio-migration mc alias set new http://localhost:9000/ minio1234 minio1234

# Run the migration on each bucket
for bucket in export iiif-cache ingest ponos-artifacts ponos-logs staging thumbnails training uploads; do
    docker exec -it ark-minio-migration mc mb "new/$bucket"
    docker exec -it ark-minio-migration mc mirror --preserve "old/$bucket" "new/$bucket"
done

Open the MinIO console for the temporary migration server and login with minio1234 as the username and password.
Check that the same buckets are listed in both consoles, with the correct amount of objects.

Run the following to complete the migration:

# Stop the old server
docker compose -p arkindex down

# Overwrite the old server's data with the migrated data
docker exec -it ark-minio-migration sh -c 'rm -rf /olddata/.minio.sys /olddata/* && cp -rf /newdata/.minio.sys /newdata/* /olddata/'

# Stop the migration server
docker stop ark-minio-migration

Update your backend repository to get the newer server in your Docker Compose configuration. You should be on the release-1.8.0 branch or the 1.8.0 tag once available.
Start the new services with make services.
Check that the MinIO console is now the console of the newer version and still has your buckets.
Delete the migration data: docker volume rm arkindex_minio_migration