Releases

The WorkerVersionMixin has been removed. It was deprecated since version 0.3.7.

The create_required_types helper has been removed. Workers should call the new create_element_type helper (calling the CreateElementType endpoint) instead.

The check_required_types helper has been updated:

This releases introduces compatibility with the 1.9.0 release of Arkindex

The trim_polygon image helper has been patched to prevent an odd behavior removing non-unique points from the provided polygon.

Image download and upload helpers now wait longer before retrying their requests. This has been done to avoid spamming servers.

LLM workers should add multiline: true to string type parameters like prompts. Refer to the documentation for more details.

The resized_images helper has been reworked to:

The Arkindex API Client library no longer depends on apistar. Some imports should be updated, most notably:

# Old way
from apistar.exceptions import ErrorResponse

# New way
from arkindex.exceptions import ErrorResponse

The method BaseWorker.request has been removed. Developers should rely on BaseWorker.api_client.request instead.

The create_iiif_url helper has been added to create an image from an existing IIIF image by URL, using the CreateIIIFURL endpoint.

The list_elements helper has been added to list elements in the current project, using the ListElements endpoint.

The create_element_children helper has been added to link multiple elements to a parent element at once, using the CreateElementChildren endpoint.

The list_corpus_types helper has been added to list element types in the current project and store them as instance attribute.

The download_export helper has been added to download a project SQLite export, using the DownloadExport endpoint.

The download_latest_export helper has been added to download the latest SQLite export of a project.

The list_process_elements helper has been added to list the elements of the current process, using the ListProcessElements endpoint.

ElementsWorker supports processing dataset sets.

ElementsWorker now supports Export processes, introduced in the latest Arkindex release.

Most bulk endpoints now publish their results in batches, to avoid too large queries at once on Arkindex. The default batch size is 50 but a larger value can be set through the batch_size argument of the helper.

Workers now rely on the value set in the mandatory field docker.command in the YAML configuration to know each worker’s command. The CMD statement in the Dockerfile is no longer needed and should be removed.

The section Run your worker locally was updated.

Workers and arkindex-base-worker now support Python 3.12.

A new pre-commit hook to report test files with too many lines is now added by default in new workers.

Pillow has an image size limit to avoid "decompression bombs". To still be able to process very large images, this limit can be increased through the ARKINDEX_MAX_IMAGE_PIXELS environment variable.

Some tools have an image disk size limit instead of a dimensions limitations. When the image is too large, a new function resized_images is able to generate downsized versions of an image that can be used until the image is small enough in terms of disk size.

A new helper is available to automatically pluralize some words. This is mostly helpful in the log messages a worker might send. Default behaviour consist in adding an 's' at the end but some exceptions are supported like "entity" and "child".

  # Old way
  logger.info(f"Published {transcriptions_count} transcription{'s' if len(transcriptions_count) > 1 else ''}")

  # New way
  from arkindex.utils import pluralize
  logger.info(f"Published {transcriptions_count} {pluralize('transcription', transcriptions_count)}")

The Teklia CA certificate is no longer needed in the Docker images of the worker. The Dockerfile can be updated accordingly.

 WORKDIR /src

- # Install curl
- ENV DEBIAN_FRONTEND=non-interactive
- RUN apt-get update -q -y && apt-get install -q -y --no-install-recommends curl

# Install worker as a package
...

- Add archi local CA
- RUN curl https://assets.teklia.com/teklia_dev_ca.pem > /usr/local/share/ca-certificates/arkindex-dev.crt && update-ca-certificates
- ENV REQUESTS_CA_BUNDLE /etc/ssl/certs/ca-certificates.crt

The CLI arguments --element and --elements-list were converting the element IDs to different types uuid.UUID versus str. They now both convert to str.

The dependency to peewee has been loosened to support any patch release of the 3.17 cycle. This has been done knowing that this library does not introduce massive breaking changes in its patch releases.

The bump to teklia-toolbox==0.1.4 broke support for

This release updates the internal behavior of DatasetWorker, meant to process dataset sets, to accommodate for the changes introduced by Arkindex 1.6.0.

The create_metadatas helper has been renamed to create_metadata_bulk. Make sure to update existing imports.

The model version configuration and the user configuration are now updated at the very end of ElementsWorker.configure and DatasetWorker.configure. This means that there is no need to do it in workers.

The migration started in 0.3.6 is now finished and all project dependencies are now stored in pyproject.toml for both arkindex-base-worker and new workers, through the template.

The create_classifications helper has been updated to use the right parameter of the CreateClassifications endpoint. Missing ML classes are now created automatically, as in create_classification.

The DatasetMixin has been updated following changes to Arkindex’s dataset processes.

The details of the loaded model is now always stored in the model_details attribute.

The TrainingMixin exposes a new property, is_finetuning, to know if the worker has a model version set. This is helpful for training workers, to know if they are fine-tuning an existing model.

Arkindex has deprecated the usage of worker_version in many endpoints. This change has been reflected in affected endpoints. Support for the equivalent worker_run argument has been added where it was missing.

The load_parents parameter is now exposed on the list_element_metadata helper.

There is an issue with the ValidateModelVersion endpoint in the latest Arkindex releases. This endpoint may return HTTP errors (codes 403 or 500) even though the model version has been successfully updated. To avoid raising false errors, a warning is logged when that happens and the worker’s processing will no longer stop at that exception.

The worker template has been updated:

A new section explaining how to publish a worker to an Arkindex instance has been added.

A summary message is now logged at the end of the run method, even if no error was encountered during processing.

A new helper was added to parse source arguments, mostly used for worker_version and worker_run arguments. To filter manual sources, the Arkindex API expects the False value. This helper maps "manual" to this value.

A new helper to upload a Pillow image has been added.

SSL verification is now skipped for Arkindex local development hosts. This only affects instance whose URL is matching the pattern *ark.localhost.

A warning is now logged when calling an helper that doesn’t support cache.

The arkindex_worker.git module was removed. It was not used locally by any workers, this module was only used to expose some workflows from python-gitlab. Please refer to their documentation if your worker needs to communicate with a Git instance.

Following Arkindex’s 1.5.3 release, the model_usage configuration parameter has been updated to a tri-enum. To migrate your workers:

PEP 621 encourages user to store most of the package’s metadata in the pyproject.toml. We followed this proposition both for the arkindex-worker package and the worker template.

The details of the model available to the worker is now stored under the model_details attribute.

The list_corpus_entities API helper now stores the entities in the entities attribute instead of returning them.

A reminder was added to prevent making changes to the Arkindex Cache schema without bumping the Version of said cache.

Each dataset’s archive is now properly deleted after processing.

The path to a Dataset's archive is now stored under the filepath property.

The new create_element_parent API helper allows to create a link between two elements.

The create_sub_element was updated to support creating children element without zones and under a parent without a zone.

A new user configuration type was introduced to be able to select Arkindex Models. Learn more about it in the documentation.

When the provided slug had more than one word, it was invalid for either:

A link to the documentation was added:

Some sections in the documentation were renamed to improve readability.

While we removed the black formatter from our CI workflow, we replaced it by Ruff’s which respects most of its rules.

Many linting rules supported by the Ruff formatter were added to improve the style of the codebase:

This project is now licensed under the MIT license.

The arkindex_worker.reporting module has been removed as the JSON report file was no longer needed.

The --model-dir CLI argument was renamed to --extras-dir as it was more suited to its use. This folder now stores dataset archives, hence the more generic name.

Following Arkindex 1.5.2 release,

Added a unicity check on the input of the create_transcription_entities helper.

The partial_update_element helper was updated to better match the endpoint.

Some modules were poorly displayed in the documentation. Class methods are now only listed under their class’s section.

A Makefile was added to the worker template to deploy new releases more easily. The default branch expects master, make sure to change it to main depending on your settings.

The base image used in the worker’s docker image was changed from python:3.11 to python:3.11-slim, in an effort to reduce their size.

During the configuration stage, a summary of the worker is now logged instead of the revision’s hash. This was changed to support workers not linked to any revision on Arkindex.

A retry mechanism on HTTP 50x errors was added. Additionally, when the requested size exceeds the maximum size allowed by the IIIF server, a new try is done with max instead of full as size parameter. More information about these parameters in the IIIF documentation.

When running the worker locally without the ARKINDEX_CORPUS_ID variable set in the environment, an explicit exception will be raised when trying to access the corpus_id attribute.

This release adds support for Python 3.12.