Arkindex 1.11.1
We are happy to announce that a new Arkindex release is available. You can explore Arkindex and try out the newest features on our demo instance, demo.arkindex.org.
Kubernetes support
Arkindex now officially support Kubernetes as a deployment strategy, using our standard Docker images.
The following features have been implemented to support Kubernetes:
-
custom SSL certificate authority in both backend & ponos-agent,
-
customizable Docker registry in
update_system_workerscommand to allow Docker proxies usage, -
Kubernetes support in ponos-agent to run Machine Learning workflows,
-
Docker rootless images across the whole stack (including workers),
-
a new health check endpoint (
RetrieveHealthcheck)
File imports
The file import worker now better handles errors when uploading images. There are multiple retries when uploading an image and performing image checks, and existing images will be re-uploaded when retrying a file import that failed to upload an image.
Document exports
The DOCX document export no longer creates duplicate subdirectories for each folder.
User interface
Clicking on the Select button of a corpus export while configuring a process now no longer attempts to create a worker configuration immediately. Over one hundred other buttons in other parts of Arkindex have been updated to ensure this type of issue can no longer occur.
This has also allowed to improve other forms. The Enter key can now be used to submit them from any form field when:
-
creating element types;
-
adding a new folder using the action in a project or a folder;
-
creating a manual transcription on an element;
-
creating or updating elements in the annotation mode;
-
creating or updating entity types;
-
creating or updating classes;
-
creating or updating workers;
-
updating model versions;
-
creating or updating models;
-
renaming a process;
-
creating a process template;
-
editing sets within a dataset:
-
creating or updating groups in Enterprise Edition;
-
creating or updating secrets in Enterprise Edition;
-
using the Go to button to type a page number and access it immediately, on any list with pages.
Security
User permissions are now also checked when API requests are made using task tokens. This means some workers may start to show permission errors when some users run them if they were creating or updating images, workers, and worker versions, or using S3 imports, and the user did not have the necessary permissions.
API
ImportWorkerVersion now returns an HTTP 409 error when a worker version already exists with a revision URL. This error includes an additional existing_id attribute for the existing worker version. This allows the Arkindex CLI to better handle updates of existing worker versions when using arkindex worker import.
A possible stale read issue has been resolved for all multipart upload APIs. This could have caused instances that use a replica to return HTTP 404 errors when attempting to run a multipart upload on a newly created model version or file.
New error format
The following API endpoints have been migrated to the new error format:
-
Asynchronous jobs
The following Enterprise Edition-only endpoints have been migrated to the new error format:
The documentation for each of those API endpoints contains a detailed description of the structure of this new format.
A new version of the Arkindex API client has been released to add new exception types to help developers make use of the new error format.
Upgrade notes
To upgrade a development instance, follow this documentation.
To upgrade a production instance, you need to:
-
Deploy this release’s Docker image:
registry.gitlab.teklia.com/arkindex/backend:1.11.1 -
Run the database migrations:
docker exec ark-backend arkindex migrate -
Update the system workers:
docker exec ark-backend arkindex update_system_workers
The main changes impacting developers and system administrators are detailed below.
Healthcheck
A new RetrieveHealthcheck API endpoint has been added to help with deployments. This endpoint is designed to always return HTTP 200 with the backend’s version number whenever the backend started successfully, without making any connection to the database or to any external dependency.
The healthchecks in our Docker images have been removed, since they could not work when using the images to run anything other than the backend server, such as workers for asynchronous jobs, or for Ponos task execution in Community Edition.
You can configure your own deployment to use the new healthcheck endpoint instead. We have updated our setup documentation to include the healthcheck in our Docker Compose examples.
Ponos on Kubernetes
Ponos agents now include a first iteration on Kubernetes support. An agent can be executed within a Kubernetes pod, or outside of it, to create other Kubernetes pods to execute tasks.
S3 compatibility
Support for the AWS_CA_BUNDLE environment variable has been restored. This allows to use a custom CA certificate bundle for all requests made by the Arkindex backend to an S3 API.
Additionally, checksums are now only used for S3 API requests when they are required by the S3 specification, not when they are optionally supported specifically by Amazon S3. This improves compatibility with other storage providers, particularly when uploading task logs, artifacts or database exports.