Deployment

The Newspaper processing Rest API is only available as a Docker image.

Requirements

  • Linux server, ideally with a GPU

  • Docker 28+, with Nvidia runtime for Docker (if you use a GPU)

  • Docker credentials provided by Teklia to fetch the image from registry.gitlab.teklia.com

  • Arkindex API credentials provided by Teklia to fetch the models from arkindex.teklia.com

Check Docker setup

A GPU is not strictly required to run the service, but will provide a huge speed increase.

To validate that your GPU is available through Docker, you can run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

If the GPU is available through Docker, you should see a standard nvidia-smi output listing your GPU & some hardware infos.

Architecture

The application is split in 3 major components:

  1. the Rest API service, using the registry.gitlab.teklia.com/finlam/api/web docker image, which is lightweight (300Mb+) and can run on small VPS or VMs. It’s the only part that needs to be publicly exposed (or behind a reverse proxy),

  2. the asynchronous worker, using the registry.gitlab.teklia.com/finlam/api/worker docker image, which is really heavy (11Gb+) and requires a GPU for ideal performance (processing one newspaper under a minute). This can run on dedicated hardware, and does not need to be publicly exposed,

  3. the databases (postgresql & redis). Redis only serves as a data broker for asynchronous task, no need to use backups for its data. Postgresql holds the tasks and their results.

Run the Rest API container

Here is a sample docker-compose.yml that will deploy 4 containers:

  1. a postgres container with a Docker volume to persist its data,

  2. a redis container without any persistence,

  3. a worker container that uses a GPU from the host and uses both postgres & redis to communicate with the API service and a volume to cache Machine Learning models,

  4. an api container that uses both postgres & redis to communicate with the worker, and needs to be publicly exposed (here with a simple port exposition).

---
services:

  api:
    container_name: newspaper-api
    image: registry.gitlab.teklia.com/finlam/api/web:0.1.0
    environment:
      DATABASE_URL: postgres://backend:teklia12345@postgres:5432/newspaper_api

      # Run in prod mode
      DEBUG: false

      # Allow public host
      ALLOWED_HOSTS: localhost
      CSRF_TRUSTED_ORIGINS: http://localhost:8000

      # Connect backend & worker to redis
      REDIS_URL: redis://redis:6379/0

    ports:
      - "127.0.0.1:8000:80"
    restart: unless-stopped
    depends_on:
      - postgres
      - redis

  worker:
    container_name: newspaper-worker
    image: registry.gitlab.teklia.com/finlam/api/worker:0.1.0
    environment:
      DATABASE_URL: postgres://backend:teklia12345@postgres:5432/newspaper_api

      # Run in prod mode
      DEBUG: false

      # Connect backend & worker to redis
      REDIS_URL: redis://redis:6379/0

      # Custom models dir
      MODELS_DIR: /models

      # Arkindex credentials to fetch models
      # Warning: you need to update that value using credentials provided by Teklia
      ARKINDEX_API_TOKEN: xxx

    volumes:
      - models:/models
    depends_on:
      - postgres
      - redis

    # Comment this whole block if you do not have a GPU
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  postgres:
    container_name: newspaper-database
    image: postgres:18-alpine
    environment:
      POSTGRES_USER: backend
      POSTGRES_PASSWORD: teklia12345
      POSTGRES_DB: newspaper_api
    volumes:
      - database:/var/lib/postgresql

  redis:
    container_name: newspaper-redis
    image: redis:8.8.0-alpine


volumes:
  database:
    driver: local
  models:
    driver: local

First run

On your initial deployment, you need to run the following commands.

API

You need to initialize the database structure, then create a super-user to manage the instance:

# Initialize database structure
docker exec newspaper-api django-admin migrate

# Create super-user
docker exec -it newspaper-api django-admin createsuperuser

Worker

You need to download the Machine Learning models using the credentials provided by Teklia:

You need to set the correct value as ARKINDEX_API_TOKEN, otherwise you’ll get some permission denied errors during model download. 
# Allow docker user to write models
docker exec --user=root -it newspaper-worker chown teklia /models

# Initialize database structure
docker exec newspaper-worker django-admin load_models

Next step

Once the first run actions have been done, you can use the API and administration interface.