Skip to content

Get started with cg_lims

Jump to bottom
mayabrandi edited this page Dec 21, 2021 · 28 revisions

To get familiar with our lims system and the cg_lims code base, you first need to have a user lims account. Make sure you have one!

Clarity Lims

Clarity lims consist of A postgress database which we never (more or less) touch directly. Instead we integrate with the database via the Rest API Rest API GUI - web interface where wi configure workflows and were the lab is doing their job.

Learning Task

CG Lims Code

All lims code at cg are depending on the genologics package, which is a python api that reflects the clarity Rest API.

Learning Task

At cg we have code to interact with the lims database. We can separate the code in to two categories:

  1. We fetch lots of lims information and save it in other databases, such as status-db and vogue. This is currently done by the related applications: cg and vogue.
  2. We read and save information back to the lims database via the clarity GUI. It can be that some numbers are being fetched from the database, some calculations are being done and the new values are being saved. This is done all the time by le lims users as they work. The code for this type of interactions with the database used to be stored in the clinical_EPPs repository.

Project plan cg-lims

We are in the process of moving all lims related code inte the “new” package cg_lims. The work is in progress but lots of work is still to be done. Coupled to the build of the new cg_lim package, I have made a limsmock package with the purpose of easily making tests for our lims code. We need to write lots of tests. Swedac requires it!

Write a simple script using the genologics package:

Create a lims conda environment with python verssion 3.8 and activate it:

conda create -n lims python=3.8
conda activate lims

Clone the genologics repo and install:

git clone [email protected]:SciLifeLab/genologics.git
cd genologics/
pip install -e . -r requirements.txt

Create the requiered config file:

vim ~/.genologicsrc

With the content:

[genologics]
BASEURI=https://clinical-lims-stage.scilifelab.se
HOST=clinical-lims-stage.scilifelab.se
USERNAME=apiuser
PASSWORD=<secret password>
[logging]
MAIN_LOG=/home

Create a file my_first_lims_code.py using some editor. (Visual Studio Code or PyCharm eg.)

Print the folowing content into the file:

# Login parameters for connecting to a LIMS instance.
from genologics.config import BASEURI, USERNAME, PASSWORD


from genologics.lims import Lims
from genologics.entities import Process, Artifact

# Create the Lims API instance
lims = Lims(BASEURI, USERNAME, PASSWORD)


# Get the process.
process = Process(lims, id='24-236607')


print(process, process.id, process.type, process.type.name)

# Process udfs
print(process.udf.items())

# all input artifacts
all_input_artifacts = process.all_inputs()
print(all_input_artifacts)


# the first input artifact of the list:
artifact = all_input_artifacts[0]

print(artifact.name, artifact.id)
print(artifact.udf.items())

# change the value of a artifact udf:
artifact.udf['Level of Multiplexing']=7
artifact.put()

print(artifact.udf.items())

Run the file:

python my_first_lims_code.py

You can pay around with this file.

Queue some samples to a step: lims-stage

and change the step id in the file to the step that you just started.

look in the genologics/enteties.py and genologics/lims.py and try to use the classes and functions that are there. At the same time follow what you are doing in the rest API

Write your first EPP in cg_lims:

Set up a local version of cg_lims

  • Clone the cg_lims repo and make a conda env for it.
  • Create the required config - talk to me! (not available on servers yet)

Preparations

Find an appropriate EPP in the clinical_EPPs repository that should be transferred to cg_lims. (Ask me for help)

Try to understand what the script is doing.

Identify in what steps the EPP is being actively used. You do that by running the following command in your cg_lims environment

lims -c <config file> scripts -l <log file> check-config --automation-string <script name>

You will get a log file with information. Ask me if you need help understanding.

Now that you know in what step(s) the script is being used you can prepare some samples to work with while scripting.

Log into lims-stage GUI and queue samples to the step where the script should be used! When you have samples in the step you can start to write code and test against this step. The step-id is always given as argument to a EPP.

Getting the step id

Getting the step id is a bit annoying.

The format is this: <prefix>-<url_step_sufix>

where prefix is either 24, 122, 151 depending on sort of step.

and url_step_sufix is the last digits in the url of the step:

The step ID for the step above is: 24-223881

You can look at the step via the REST-API: https://clinical-lims.scilifelab.se/api/v2/processes/24-223881

Start coding

  • Think about where the new EPP should be added. Is it making calcuations, copying information, creating files..? See the EPPs tree below.
  • Look at other EPPs to se how they are written and called by the CLI.
  • Don't just copy the script from clinical_EPPs. Think about how it can be better and how you could use existing code in cg_lims/get eg.
  • Test the script against the step in lims-stage
  • Write tests using the limsmock!

EPP tree

This is what the EPP tree look like today (14/12-2021)

EPPs/
├── __init__.py
├── arnold
│   ├── __init__.py
│   ├── base.py
│   ├── prep.py
│   └── sample.py
├── base.py
├── files
│   ├── __init__.py
│   ├── base.py
│   ├── csv_for_kapa_truble_shooting
│   │   ├── __init__.py
│   │   ├── csv_for_kapa_debug.py
│   │   └── models.py
│   ├── file_to_udf.py
│   ├── hamilton
│   │   ├── __init__.py
│   │   ├── base.py
│   │   ├── make_kapa_csv.py
│   │   ├── models.py
│   │   ├── normalization_file.py
│   │   └── sars_cov2_prep_file.py
│   ├── placement_map
│   │   ├── __init__.py
│   │   ├── hmtl_templates.py
│   │   ├── make_96well_placement_map.py
│   │   └── models.py
│   └── pooling_map
│       ├── __init__.py
│       ├── hmtl_templates.py
│       ├── make_pooling_map.py
│       └── models.py
├── move
│   ├── __init__.py
│   ├── base.py
│   ├── move_samples.py
│   ├── place_samples_in_seq_agg.py
│   └── rerun_samples.py
├── qc
│   ├── __init__.py
│   ├── base.py
│   └── set_qc_fail.py
└── udf
    ├── __init__.py
    ├── base.py
    ├── calculate
    │   ├── __init__.py
    │   ├── base.py
    │   ├── calculate_amount_ng.py
    │   ├── calculate_beads.py
    │   ├── calculate_resuspension_buffer_volumes.py
    │   ├── calculate_water.py
    │   ├── calculate_water_volume_rna.py
    │   ├── get_missing_reads.py
    │   ├── maf_calculate_volume.py
    │   ├── molar_concentration.py
    │   ├── sum_missing_reads_in_pool.py
    │   ├── twist_aliquot_amount.py
    │   ├── twist_aliquot_volume.py
    │   ├── twist_get_volumes_from_buffer.py
    │   ├── twist_pool.py
    │   └── twist_qc_amount.py
    ├── copy
    │   ├── __init__.py
    │   ├── artifact_to_sample.py
    │   ├── base.py
    │   ├── process_to_sample.py
    │   ├── reads_to_sequence.py
    │   └── sample_to_artifact.py
    └── set
        ├── __init__.py
        ├── base.py
        ├── set_sample_date.py
        └── set_samples_reads_missing.py

as you can see the EPPs are first divided in arnold, files, move, qc and udf.

arnold EPPs that load data into the arnold database.

files EPPs that create files.

move EPPs that queue samples to other steps

qc EPPs that evaluate data and set qc-flags

udf EPPs that write to udfs.

Examples

Ill add some examples of EPPs here that have been transfered from clinical_EPPs to cg_lims

Calculate twist amount

Script in clinical_EPPs

Transfered to this script in cg_lims.

Tests for the new script.

Clone this wiki locally