Deploying Models

Info

Apple users: Most models won't run on Apple Silicon (ARM64) processors, more information below.
Check the list of available services to see which ones are compatible.

Deployment Options

There's different ways to deploy a service.

1. Deployment via container + compose.yaml
   Deploy your model locally or in the cloud, using Docker or Podman compose
   Recommended (when available)
2. Deployment via container
   Deploy your model locally or in the cloud, using Docker or Podman build
3. Local deployment using a Python virtual environment
   Install requirements and manage your own environment
4. Cloud deployment to Google Cloud Run
   Leverage cloud computing power, using a no-code deployment
   Easy setup
5. Cloud deployment to Red Hat OpenShift
   Leverage cloud computing power
6. Cloud deployment to SkyPilot on AWS
   Leverage cloud computing power

Deployment via Container + compose.yaml Recommended

When available, containerizing a model using Docker Compose / Podman Compose is the easiest (and our recommended) way to deploy a model service.

Support

To check if a service supports compose, check the service's details in our list available services, or look in the service's GitHub repo if a compose.yaml file is present.

If you're running on an Apple computer, make sure to check the Apple Silicon section below.

Preparation

Create the .openad_models folder in your home directory.

Bash

mkdir -p ~/.openad_models

Build and Start

Note

Chosing a port: Before you start, consider the port you want to run the service on. By default, 8080:8080 maps host port 8080 to container port 8080. If you will be running multiple service, you may want to change the host port in the compose.yaml, eg. 8081:8080

First build the container image:

Bash

[ docker/podman ] compose create

Next, start the container:

Bash

[ docker/podman ] compose start

Note

If your device does not have a descrete GPU (as is the case for Apple Silicon devices), the start command will fail with the following error:

Bash

Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

In this case, you can simply disable this part of the compose.yaml instructions:

Bash

# deploy:
#   resources:
#     reservations:
#       devices:
#         - capabilities: [gpu]

Then run the create and start commands again.

Once the service is running, continue to Cataloging a Containerized Model.

Deployment via Container

Prerequisites

Make sure you have Docker and the Docker Buildx plugin installed.

Then create the .openad_models folder in your home directory.

Bash

mkdir -p ~/.openad_models

Set up Container

Clone the model's GitHub repository:

Bash

git clone https://github.com/acceleratedscience/<repository_name>

Set the working directory to the downloaded repository and run the build:

Note: The <model_name> you can choose yourself.

Bash

cd <repository_name>

Bash

docker build -t <model_name> .

Note

Apple users: If you're running on Apple Silicon, you'l need to add --platform linux/amd64 to the build command, to force the AMD64 architecture using an emulator.

Bash

docker build --platform linux/amd64 -t <model_name> .

After the build is complete, run the container and make the server available on port 8080:

Bash

docker run -p 8080:8080 <model_name>

Once the service is running, continue to Cataloging a Containerized Model.

Cataloging a Containerized Model

Once the container is running, you can catalog the model in OpenAD:

First, lauch the OpenAD shell:

Bash

openad

Then catalog the service, and check the status.

Bash

catalog model service from remote 'http://127.0.0.1:8080' as <service-name>

Text Only

model service status

If all goes well, the status should say Connected

Text Only

Service         Status       Endpoint                Host    API expires
--------------  -----------  ----------------------  ------  -------------
<service-name>  Connected    http://127.0.0.1:8080/  remote  No Info

As a reminder, to see all available model commands, run:

Bash

model ?

Local deployment using a Python virtual environment

Info

If you are using an Apple Silicon device, deploy using Docker instead. See Apple Silicon for more info.

• Create a virtual environment running Python 3.11. We'll use pyenv.

  Note: Python 3.10.10+ or 3.11 are supported, Python 3.12 is not.

  Bash

  pyenv shell 3.11
  

  Bash

  python -m venv my-venv
  

  Bash

  source my-venv/bin/activate
  

• Install the model requirements as described in the model's repository (not to be confused with the OpenAD service wrapper). Models not listed below should deploy with Docker instead.

  Model Name	GitHub
  SMI-TED	github.com/IBM/materials
  BMFM-SM	github.com/BiomedSciAI/biomed-multi-view
  BMFM-PM	github.com/BiomedSciAI/biomed-multi-alignment
  REINVENT	github.com/MolecularAI/REINVENT4

• Install the OpenAD service utilities:

  Bash

  pip install git+https://github.com/acceleratedscience/openad_service_utils.git
  

  Note

  Downloading of the models will be prompted by your first request and may take some time.

  You can pre-load the models using AWS CLI.

  Bash

  mkdir -p ~/.openad_models/properties/molecules && aws s3 sync s3://ad-prod-biomed/molecules/mammal/ /tmp/.openad_models/properties/molecules/mammal --no-sign-request --exact-timestamps
  

• Clone the service repo:

  Bash

  git clone https://github.com/acceleratedscience/openad-service-<service-name>
  

• Move the working directory into the cloned service repo:

  Bash

  cd openad-service-<service-name>
  

• Start the service:

  The start command differs per model:

  Bash

  # SMI-TED / REINVENT
  python app.py
  

  Bash

  # BMFM-SM
  python ./bmfm_sm_api/sm_implementation.py
  

  Bash

  # BMFM-PM
  python ./implementation.py
  

• Open a new terminal session and launch OpenAD:

  Bash

  openad
  

• Within the OpenAD shell, catalog the service you just started:

  Bash

  catalog model service from remote 'http://127.0.0.1:8080' as <service-name>
  

• To see the available service commands, run:

  Bash

  <service-name> ?
  

Cloud deployment to Google Cloud Run

Deploying a model to Google Cloud Run is the easiest way to deploy a model to the cloud, as Google Cloud can spin up the service directly from the GitHub repository using only the Dockerfile.

Step 1: Preparation

• Install Google Cloud CLI
• Go to Available Models to verify if the model you want to deploy supports Google Cloud
• Fork the GitHub repo of the model you want to deploy.

Step 2: Google Cloud Run

• Go to console.cloud.google.com/run
• Create a project
• Start your free trial if you need to (credit card required)
• Click the "Create service" button in an empty project
  (or click "Deploy container" and select "Service")
• Select second option: "Continuously deploy from a repository"
  • Click "Setup with Cloud Build"
  • Connect GitHub & install Cloud Build for your fork
  • Under "Build Type" select "Dockerfile" (keep default location)
• Choose "Require authentication"
• Click "Container(s), Volumes, Networking, Security" at the bottom
  • Under Resources, set Memory & CPU to 32GB & 8CPU or higher
  • Under Requests, set the Request timeout to the maximum of 3600 sec
• Click "Create"

• Copy the service URL on top (we'll refer to it as <service_url>)
  It should look something like this:

  Text Only

  https://openad-service-reinvent4-012345678999.us-central1.run.app
  

  Note: Your service will have a green checkmark next to its name, but it won't be available until "Building and deploying from repository" is done and also displays the green checkmark.

Step 3: Terminal

• Login to Google Cloud

  Note: The application-default clause stores your credentials on disk and is required to auto-refresh your auth tokens.

  Text Only

  gcloud auth application-default login
  

• If you haven't done so yet, you may have to set your project.

  Note: To find your project's ID, go to the Google Cloud Console and click the button with your project's name (probably 'My FIrst Project') next to the Google Cloud logo.

  Text Only

  gcloud config set project <project_id>`
  

• Fetch your auth token

  Text Only

  gcloud auth print-identity-token
  

• Copy the token

Step 4: OpenAD

• Create auth group for Google Cloud

  Note: You can call this group anything, we'll call it gcloud

  Text Only

  model auth add group gcloud with '<auth_token>'
  

• Catalog the service

  Text Only

  catalog model service from remote '<service_url>' as <service_name>
  

• Service will be listed as "Not ready" until the build is done
  (~15-20 min depending on the model)

  Text Only

  model service status
  

Cloud deployment to Red Hat OpenShift

If the service you're trying to deploy has a /helm-chart folder, it's been prepared for deployment on OpenShift.

Note: The <service-name> and <build-name> you can choose yourself. We'll show an example for SMI-TED.

1. Install the helm chart

   Bash

   helm install <service-name> <path-to-helm-chart-dir>
   

   Bash

   # Example for SMI-TED
   helm install smi-ted ./helm-chart
   

2. Start a new build

   Bash

   oc start-build <build-name>
   

   Bash

   # Example for SMI-TED
   oc start-build smi-ted-build
   

3. Wait for the build to complete

   Bash

   LATEST_BUILD=$(oc get builds | grep '<build-name>-' | awk '{print $1}' | sort -V | tail -n 1)
   

   Bash

   # Example for SMI-TED
   LATEST_BUILD=$(oc get builds | grep 'smi-ted-build-' | awk '{print $1}' | sort -V | tail -n 1)
   

   Bash

   oc wait --for=condition=Complete build/$LATEST_BUILD --timeout=15m
   

4. Run a request test

   Bash

   curl "http://$(oc get route <service-name>-openad-model -o jsonpath='{.spec.host}')/health"
   

   Bash

   # Example for SMI-TED
   curl "http://$(oc get route smi-ted-openad-model -o jsonpath='{.spec.host}')/health"
   

Cloud deployment to SkyPilot on AWS

Setting up SkyPilot

1. AWS account

   • Head to aws.com
   • Click the [Create an AWS Account] button in the top right corner
   • Follow instructions, including setting up a root user

2. AWS user with correct permissions

   Starting from your AWS dashboard:

   • Search for "IAM" in the search bar
   • From your IAM dashboard, click "Users" in the lefthand sidebar
   • Click the [Create user] button in the top right hand corner
   • Leave the "Provide user access to the AWS Management Console" box unchecked
   • Up next on the "Set Permissions" screen, select the third option: "Attach policies directly"
   • In the box below, click the [Create policy] button
   • Create a new policy with minimal permissions for Skypilot, following thye Skypilot instructions
   • On the next screen, search for the policy you just created, which would be called minimal-skypilot-policy per the instructions
   • Finish the process to attach the policy to your user

3. AWS Access key

   Starting from the IAM dashboard:

   • Click "Users" in the lefthand sidebar
   • Click on the user you created in the previous step
   • Click "Create access key" on the right side of the summary on top
   • Select the first option, "Command Line Interface (CLI)" as use case
   • Finish the process to create the access key
   • Store the secret access key in your password manager, as you will not be able to access it after creation

4. AWS command line tool

   Starting from a terminal window:

   • Install awscli

     Bash

     python -m pip install awscli
     

     Note: For more nuanced instructions, please refer to pypi

   • Add the credentials for the AWS user you set up in step 3.

     Bash

     aws configure
     

     • Your user's access key can be found in your IAM dashboard > Users, however the secret access key should have been stored in your password manager or elsewhere.
     • The fields "Default region name" and "Default output format" can be left blank

5. SkyPilot

   SkyPilot is a framework for running AI and batch workloads on any infrastructure. We're using AWS.

   Starting from a terminal window:

   • If you are running OpenAD in a virtual environment, make sure your virtual environment is activated. If you followed the default installation instructions, you should be able to run:

     Bash

     source ~/ad-venv/bin/activate
     

   • Install Skypilot for AWS

     Bash

     pip install "skypilot[aws]"
     

   • After installation, verify if you have cloud access

     Bash

     sky check
     

Apple Silicon

Apple Silicon chips (aka M1, M2, M3 etc.) utilize the ARM64 instruction set architecture (ISA), which is incompatible with the standard x86-64 ISA the models are compiled for.

Some are able to run on Apple Silicon via emulator (with some impact on performance), however support is not consistent.

Also, because Apple M processors are a SoC without discrete GPU, there is no support for GPU deployment. When using Docker or Podman compose, make sure to disable this part in the compose.yaml file:

Text Only

# deploy:
#   resources:
#     reservations:
#       devices:
#         - capabilities: [gpu]

If you will be using our models in a production environment, it's recommended to deploy the container to the cloud.