Add a metric

This guide will show you how to create a new Viash component.

A metric is a quantitative measure used to evaluate the performance of the different methods in solving the specific task problem.

This guide will show you how to create a new Viash component. In the following we will show examples for both Python and R. Note that the Label Projection task is used throughout the guide, so make sure to replace any occurrences of "label_projection" with your task of interest.

Tip

Make sure you have followed the “Getting started” guide.

Step 1: Create a new component

Use the create_component component to start creating a new metric.

viash run src/common/create_component/config.vsh.yaml -- \
  --task label_projection \
  --type metric \
  --name my_metric_py \
  --language python
Check inputs
Check language
Check API file
Read API file
Create output dir
Create config
Create script
Done!

This will create a new folder at src/tasks/label_projection/metrics/my_metric_py containing a Viash config and a script.

src/tasks/label_projection/metric/my_metric_py
    ├── script.py                    Script for running the metric.
    ├── config.vsh.yaml              Config file for metric.
    └── ...                          Optional additional resources.
viash run src/common/create_component/config.vsh.yaml -- \
  --task label_projection \
  --type metric \
  --name my_metric_r \
  --language r
Check inputs
Check language
Check API file
Read API file
Create output dir
Create config
Create script
Done!

This will create a new folder at src/tasks/label_projection/metrics/my_metric_r containing a Viash config and a script.

src/tasks/label_projection/metrics/my_metric_r
    ├── script.R                     Script for running the metric.
    ├── config.vsh.yaml              Config file for metric.
    └── ...                          Optional additional resources.
Tip

Some tasks have multiple metric subtypes (e.g. batch_integration), which will require you to use a different value for --type corresponding to the desired metric subtype.

Change the --name to a unique name for your metric. It must match the regex [a-z][a-z0-9_]* (snakecase).

  • A config file contains metadata of the component and the dependencies required to run it. In steps 2 and 3 we will fill in the required information.
  • A script contains the code to run the metric. In step 4 we will edit the script.

Step 2: Fill in metadata

The Viash config contains metadata of your metric, which script is used to run it, and the required dependencies.

Generated config file

This is what the config.vsh.yaml generated by the create_component component looks like:

Contents of config.vsh.yaml 
# The API specifies which type of component this is.
# It contains specifications for:
#   - The input/output files
#   - Common parameters
#   - A unit test
__merge__: ../../api/comp_metric.yaml

functionality:
  # A unique identifier for your component (required).
  # Can contain only lowercase letters or underscores.
  name: my_metric_py

  # Metadata for your component
  info:
    metrics:
      # A unique identifier for your metric (required).
      # Can contain only lowercase letters or underscores.
      name: my_metric_py
      # A relatively short label, used when rendering visualisarions (required)
      label: My Metric Py
      # A one sentence summary of how this metric works (required). Used when 
      # rendering summary tables.
      summary: "FILL IN: A one sentence summary of this metric."
      # A multi-line description of how this component works (required). Used
      # when rendering reference documentation.
      description: |
        FILL IN: A (multi-line) description of how this metric works.
      # A reference key from the bibtex library at src/common/library.bib (required).
      reference: bibtex_reference_key
      # URL to the documentation for this metric (required).
      documentation_url: https://url.to/the/documentation
      # URL to the code repository for this metric (required).
      repository_url: https://github.com/organisation/repository
      # The minimum possible value for this metric (required)
      min: 0
      # The maximum possible value for this metric (required)
      max: 1
      # Whether a higher value represents a 'better' solution (required)
      maximize: true

  # Component-specific parameters (optional)
  # arguments:
  #   - name: "--n_neighbors"
  #     type: "integer"
  #     default: 5
  #     description: Number of neighbors to use.

  # Resources required to run the component
  resources:
    # The script of your component (required)
    - type: python_script
      path: script.py
    # Additional resources your script needs (optional)
    # - type: file
    #   path: weights.pt

platforms:
  # Specifications for the Docker image for this component.
  - type: docker
    image: ghcr.io/openproblems-bio/base_python:1.0.4
    # Add custom dependencies here (optional). For more information, see
    # https://viash.io/reference/config/platforms/docker/#setup .
    # setup:
    #   - type: python
    #     packages: scib==1.1.5

  # This platform allows running the component natively
  - type: native
  # Allows turning the component into a Nextflow module / pipeline.
  - type: nextflow
    directives:
      label: [midtime,midmem, midcpu]
Contents of config.vsh.yaml 
# The API specifies which type of component this is.
# It contains specifications for:
#   - The input/output files
#   - Common parameters
#   - A unit test
__merge__: ../../api/comp_metric.yaml

functionality:
  # A unique identifier for your component (required).
  # Can contain only lowercase letters or underscores.
  name: my_metric_r

  # Metadata for your component
  info:
    metrics:
      # A unique identifier for your metric (required).
      # Can contain only lowercase letters or underscores.
      name: my_metric_r
      # A relatively short label, used when rendering visualisarions (required)
      label: My Metric R
      # A one sentence summary of how this metric works (required). Used when 
      # rendering summary tables.
      summary: "FILL IN: A one sentence summary of this metric."
      # A multi-line description of how this component works (required). Used
      # when rendering reference documentation.
      description: |
        FILL IN: A (multi-line) description of how this metric works.
      # A reference key from the bibtex library at src/common/library.bib (required).
      reference: bibtex_reference_key
      # URL to the documentation for this metric (required).
      documentation_url: https://url.to/the/documentation
      # URL to the code repository for this metric (required).
      repository_url: https://github.com/organisation/repository
      # The minimum possible value for this metric (required)
      min: 0
      # The maximum possible value for this metric (required)
      max: 1
      # Whether a higher value represents a 'better' solution (required)
      maximize: true

  # Component-specific parameters (optional)
  # arguments:
  #   - name: "--n_neighbors"
  #     type: "integer"
  #     default: 5
  #     description: Number of neighbors to use.

  # Resources required to run the component
  resources:
    # The script of your component (required)
    - type: r_script
      path: script.R
    # Additional resources your script needs (optional)
    # - type: file
    #   path: weights.pt

platforms:
  # Specifications for the Docker image for this component.
  - type: docker
    image: ghcr.io/openproblems-bio/base_r:1.0.4
    # Add custom dependencies here (optional). For more information, see
    # https://viash.io/reference/config/platforms/docker/#setup .
    # setup:
    #   - type: r
    #     packages: tidyverse

  # This platform allows running the component natively
  - type: native
  # Allows turning the component into a Nextflow module / pipeline.
  - type: nextflow
    directives:
      label: [midtime,midmem, midcpu]

Required metadata fields

Please make sure that the following fields in the functionality and functionality.info sections in the config file are filled in. The metrics component can contain several metric values these are listed in the functionality.info.metrics.

  • .__merge__: The API specifies which type of component this is. It contains specifications for:

    • The input/output files
    • Common parameters
    • A unit test

  • .functionality.name: A unique identifier. Can only contain lowercase letters, numbers or underscores.

  • .functionality.info.metrics[].label: A unique, human-readable, short label. Used for creating summary tables and visualisations.

  • .functionality.info.metrics[].summary: A one sentence summary of purpose and methodology. Used for creating an overview tables.

  • .functionality.info.metrics[].description: A longer description (one or more paragraphs). Used for creating reference documentation and supplementary information.

  • .functionality.info.metrics[].reference: A bibtex reference key to the paper where the component is described.

  • .functionality.info.metrics[].min: The lowest possible value of the metric.

  • .functionality.info.metrics[].max: The highest possible value of the metric.

  • .functionality.info.metrics[].maximize: Whether a higher metric value is better.

Step 3: Add dependencies

Each component has it’s own set of dependencies, because different components might have conflicting dependencies.

base images

For your convenience we have created several base images that can be used for python or R scripts. These images can be found in the OpenProblems github repo base_images. Click on the packages to view the url you need to use. You are not required to use these images but install the required packages to make sure OpenProblems works properly.

  • ghcr.io/openproblems-bio/base_images/python Base image for python scripts.

  • ghcr.io/openproblems-bio/base_images/r Base image for R scripts.

  • ghcr.io/openproblems-bio/base_images/pytorch_nvidia Base image for scripts that use pytorch with nvidia gpu support.

  • ghcr.io/openproblems-bio/base_images/tensorflow_nvidia Base image for scripts that use tensorflow with nvidia gpu support.

custom image

Update the setup definition in the platforms section of the config file. This section describes the packages that need to be installed in the Docker image and are required for your method to run.

If you’re using a custom image use the following minimum setup:

platforms:
  - type: docker
    Image: your custom image
    setup:
      - type: apt
        packages:
          - procps
      - type: python
        packages:
          - anndata~=0.10.0
          - scanpy~=1.10.0
          - pyyaml
          - requests
          - jsonschema
platforms:
  - type: docker
    Image: your custom image
    setup:
      - type: apt
        packages:
          - procps
          - libhdf5-dev
          - libgeos-dev
          - python3
          - python3-pip
          - python3-dev
          - python-is-python3
      - type: python
        packages:
          - rpy2
          - anndata~=0.10.0
          - scanpy~=1.10.0
          - pyyaml
          - requests
          - jsonschema
      - type: r
        packages:
          - anndata
          - BiocManager
          - reticulate
          - bit64

Please check out this guide for more information on how to add extra package dependencies.

Note

Tip: After making changes to the components dependencies, you will need to rebuild the docker container as follows:

viash run src/tasks/label_projection/metrics/my_metric_py/config.vsh.yaml -- \
  ---setup cachedbuild
Output 
[notice] Building container 'ghcr.io/openproblems-bio/label_projection/metrics/my_metric_py:dev' with Dockerfile

Step 4: Edit script

A component’s script typically has five sections:

  1. Imports and libraries
  2. Argument values
  3. Read input data
  4. Generate results
  5. Write output data to file

Generated script

This is what the script generated by the create_component component looks like:

Contents of script.py 
import anndata as ad

## VIASH START
# Note: this section is auto-generated by viash at runtime. To edit it, make changes
# in config.vsh.yaml and then run `viash config inject config.vsh.yaml`.
par = {
  'input_solution': 'resources_test/label_projection/pancreas/solution.h5ad',
  'input_prediction': 'resources_test/label_projection/pancreas/prediction.h5ad',
  'output': 'output.h5ad'
}
meta = {
  'functionality_name': 'my_metric_py'
}
## VIASH END

print('Reading input files', flush=True)
input_solution = ad.read_h5ad(par['input_solution'])
input_prediction = ad.read_h5ad(par['input_prediction'])

print('Compute metrics', flush=True)
# metric_ids and metric_values can have length > 1
# but should be of equal length
uns_metric_ids = [ 'my_metric_py' ]
uns_metric_values = [ 0.5 ]

print("Write output AnnData to file", flush=True)
output = ad.AnnData(
  uns={
    'dataset_id': input_prediction.uns['dataset_id'],
    'normalization_id': input_prediction.uns['normalization_id'],
    'method_id': input_prediction.uns['method_id'],
    'metric_ids': uns_metric_ids,
    'metric_values': uns_metric_values
  }
)
output.write_h5ad(par['output'], compression='gzip')
Contents of script.R 
library(anndata)

## VIASH START
par <- list(
  input_solution = "resources_test/label_projection/pancreas/solution.h5ad",
  input_prediction = "resources_test/label_projection/pancreas/prediction.h5ad",
  output = "output.h5ad"
)
meta <- list(
  functionality_name = "my_metric_r"
)
## VIASH END

cat("Reading input files\n")
input_solution <- anndata::read_h5ad(par[["input_solution"]])
input_prediction <- anndata::read_h5ad(par[["input_prediction"]])

cat("Compute metrics\n")
# metric_ids and metric_values can have length > 1
# but should be of equal length
uns_metric_ids <- c("my_metric_r")
uns_metric_values <- c(0.5)

cat("Write output AnnData to file\n")
output <- anndata::AnnData(
  uns = list(
    dataset_id = input_prediction$uns[["dataset_id"]],
    normalization_id = input_prediction$uns[["normalization_id"]],
    method_id = input_prediction$uns[["method_id"]],
    metric_ids = uns_metric_ids,
    metric_values = uns_metric_values
  )
)
output$write_h5ad(par[["output"]], compression = "gzip")

Required sections

Imports and libraries

In the top section of the script you can define which packages/libraries the metric needs. If you add a new or different package add the dependency to config.vsh.yaml in the setup field (see above).

Argument block

The Viash code block is designed to facilitate prototyping, by enabling you to execute directly by running python script.py (or Rscript script.R for R users). Note that anything between “VIASH START” and “VIASH END” will be removed and replaced with a CLI argument parser when the components are being built by Viash.

Here, the par dictionary contains all the arguments defined in the config.vsh.yaml file (including those from the defined __merge__ file). When adding a argument in the par dict also add it to the config.vsh.yaml in the arguments section.

Read input data

This section reads any input AnnData files passed to the component.

Generate results

This is the most important section of your script, as it defines the core functionality provided by the component. It processes the input data to create results for the particular task at hand.

Write output data to file

The output stored in a AnnData object and then written to an .h5ad file. The format is specified by the API file specified in the __merge__ field in the config file.

Step 5: Try component

Your component’s API file contains the necessary unit tests to check whether your component works and the output is in the correct format.

You can test your component by using the following command:

viash test src/tasks/label_projection/metrics/my_metric_py/config.vsh.yaml
Output 
Running tests in temporary directory: '/tmp/viash_test_f112663350524841803070'
====================================================================
+/tmp/viash_test_f112663350524841803070/build_executable/f1 ---verbosity 6 ---setup cachedbuild
[notice] Building container 'ghcr.io/openproblems-bio/label_projection/metrics/f1:test' with Dockerfile
[info] Running 'docker build -t ghcr.io/openproblems-bio/label_projection/metrics/f1:test /tmp/viash_test_f112663350524841803070/build_executable -f /tmp/viash_test_f112663350524841803070/build_executable/tmp/dockerbuild-f1-jdcjkr/Dockerfile'
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 594B done
#1 DONE 0.0s

#2 [internal] load metadata for ghcr.io/openproblems-bio/base_python:1.0.4
#2 DONE 0.0s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/2] FROM ghcr.io/openproblems-bio/base_python:1.0.4@sha256:e907268bee0b400e0eb1555d9cf6727ac64d976fb097eb669debd686fceaa705
#4 CACHED

#5 [2/2] RUN pip install --upgrade pip &&   pip install --upgrade --no-cache-dir "scikit-learn"
#5 0.485 Requirement already satisfied: pip in /usr/local/lib/python3.11/site-packages (24.1)
#5 0.547 Collecting pip
#5 0.557   Downloading pip-24.2-py3-none-any.whl.metadata (3.6 kB)
#5 0.564 Downloading pip-24.2-py3-none-any.whl (1.8 MB)
#5 0.591    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 101.9 MB/s eta 0:00:00
#5 0.702 Installing collected packages: pip
#5 0.703   Attempting uninstall: pip
#5 0.716     Found existing installation: pip 24.1
#5 0.755     Uninstalling pip-24.1:
#5 0.765       Successfully uninstalled pip-24.1
#5 1.523 Successfully installed pip-24.2
#5 1.523 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
#5 1.958 Requirement already satisfied: scikit-learn in /usr/local/lib/python3.11/site-packages (1.5.0)
#5 2.080 Collecting scikit-learn
#5 2.133   Downloading scikit_learn-1.5.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
#5 2.143 Requirement already satisfied: numpy>=1.19.5 in /usr/local/lib/python3.11/site-packages (from scikit-learn) (2.0.0)
#5 2.143 Requirement already satisfied: scipy>=1.6.0 in /usr/local/lib/python3.11/site-packages (from scikit-learn) (1.13.1)
#5 2.144 Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.11/site-packages (from scikit-learn) (1.4.2)
#5 2.144 Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.11/site-packages (from scikit-learn) (3.5.0)
#5 2.161 Downloading scikit_learn-1.5.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.3 MB)
#5 2.493    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.3/13.3 MB 43.5 MB/s eta 0:00:00
#5 2.578 Installing collected packages: scikit-learn
#5 2.578   Attempting uninstall: scikit-learn
#5 2.581     Found existing installation: scikit-learn 1.5.0
#5 2.649     Uninstalling scikit-learn-1.5.0:
#5 2.660       Successfully uninstalled scikit-learn-1.5.0
#5 4.318 Successfully installed scikit-learn-1.5.1
#5 4.319 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
#5 DONE 4.6s

#6 exporting to image
#6 exporting layers
#6 exporting layers 2.2s done
#6 writing image sha256:9b940288ad2c8398db1dd71d4f6ec14869ce2b34ce19f4de864b3a90cc2ee31a done
#6 naming to ghcr.io/openproblems-bio/label_projection/metrics/f1:test done
#6 DONE 2.2s
====================================================================
+/tmp/viash_test_f112663350524841803070/test_check_metric_config/test_executable
Load config data
check general fields
Check info fields
Check platform fields
All checks succeeded!
====================================================================
+/tmp/viash_test_f112663350524841803070/test_run_and_check_adata/test_executable
>> Running test 'run'
>> Checking whether input files exist
>> Running script as test
Load data
Encode labels
Compute F1 score
Store metric value
Writing adata to file
>> Checking whether output file exists
>> Reading h5ad files and checking formats
Reading and checking input_solution
  AnnData object with n_obs × n_vars = 213 × 1500
    obs: 'label', 'batch'
    var: 'hvg', 'hvg_score'
    uns: 'dataset_description', 'dataset_id', 'dataset_name', 'dataset_organism', 'dataset_reference', 'dataset_summary', 'dataset_url', 'normalization_id'
    obsm: 'X_pca'
    layers: 'counts', 'normalized'
Reading and checking input_prediction
  AnnData object with n_obs × n_vars = 213 × 1500
    obs: 'batch', 'label_pred'
    var: 'hvg', 'hvg_score'
    uns: 'dataset_id', 'method_id', 'normalization_id'
    obsm: 'X_pca'
    layers: 'counts', 'normalized'
Reading and checking output
  AnnData object with n_obs × n_vars = 213 × 1500
    obs: 'batch', 'label_pred'
    var: 'hvg', 'hvg_score'
    uns: 'dataset_id', 'method_id', 'metric_ids', 'metric_values', 'normalization_id'
    obsm: 'X_pca'
    layers: 'counts', 'normalized'
All checks succeeded!
====================================================================
SUCCESS! All 2 out of 2 test scripts succeeded!
Cleaning up temporary directory

Visit “Run tests” for more information on running unit tests and how to interpret common error messages.

You can also run your component on local files using the viash run command. For example:

viash run src/tasks/label_projection/metrics/my_metric_py/config.vsh.yaml -- \
  --input_prediction resources_test/label_projection/pancreas/knn.h5ad \
  --input_solution resources_test/label_projection/pancreas/solution.h5ad \
  --output output.h5ad

Next steps

If your component works, please create a pull request.