Merge pull request #13 from quiltdata/make-testable

Make testable

.github/workflows/mega-linter.yml

@@ -0,0 +1,56 @@
+# MegaLinter GitHub Action configuration file
+# More info at https://megalinter.github.io
+name: MegaLinter
+
+on:
+  # Trigger mega-linter at every push. Action will also be visible from Pull Requests to main
+  push: # Comment this line to trigger action only on pull-requests (not recommended if you don't pay for GH Actions)
+permissions: read-all
+
+env: # Comment env block if you do not want to apply fixes
+  # Apply linter fixes configuration
+  APPLY_FIXES: all # When active, APPLY_FIXES must also be defined as environment variable (in github/workflows/mega-linter.yml or other CI tool)
+  #APPLY_FIXES_EVENT: pull_request # Decide which event triggers application of fixes in a commit or a PR (pull_request, push, all)
+  #APPLY_FIXES_MODE: pull_request # If APPLY_FIXES is used, defines if the fixes are directly committed (commit) or posted in a PR (pull_request)
+  DISABLE_LINTERS: SPELL_CSPELL,COPYPASTE_JSCPD,PYTHON_BANDIT,PYTHON_MYPY,PYTHON_PYRIGHT,PYTHON_PYLINT,MARKDOWN_MARKDOWN_LINK_CHECK,REPOSITORY_TRIVY
+
+concurrency:
+  group: ${{ github.ref }}-${{ github.workflow }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    name: MegaLinter
+    runs-on: ubuntu-latest
+    steps:
+      # Git Checkout
+      - name: Checkout Code
+        uses: actions/checkout@v3
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          fetch-depth: 0 # If you use VALIDATE_ALL_CODEBASE = true, you can remove this line to improve performances
+
+      # MegaLinter
+      - name: MegaLinter
+        id: ml
+        # You can override MegaLinter flavor used to have faster performances
+        # More info at https://megalinter.github.io/flavors/
+        uses: oxsecurity/megalinter/flavors/[email protected]
+        env:
+          # All available variables are described in documentation
+          # https://megalinter.github.io/configuration/
+          VALIDATE_ALL_CODEBASE: true  
+          # VALIDATE_ALL_CODEBASE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} # Validates all source when push on main, else just the git diff with main. Override with true if you always want to lint all sources
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          # ADD YOUR CUSTOM ENV VARIABLES HERE OR DEFINE THEM IN A FILE .mega-linter.yml AT THE ROOT OF YOUR REPOSITORY
+          DISABLE: COPYPASTE,SPELL # Uncomment to disable copy-paste and spell checks
+
+      # Upload MegaLinter artifacts
+      - name: Archive production artifacts
+        if: ${{ success() }} || ${{ failure() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: MegaLinter reports
+          path: |
+            megalinter-reports
+            mega-linter.log

.github/workflows/python-package.yml

@@ -0,0 +1,48 @@
+# This workflow will install Python dependencies and run tests
+name: Python package
+
+on:
+  push:
+permissions: read-all
+
+jobs:
+  build:
+    permissions:
+      contents: read
+      id-token: write
+      issues: write
+      pull-requests: write
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9"]
+        os: [ubuntu-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Configure AWS Credentials
+      uses: aws-actions/configure-aws-credentials@v2
+      with:
+        role-to-assume: arn:aws:iam::712023778557:role/github/GitHub-Testing-BenchlingPackager
+        aws-region: us-east-1
+    - name: Test with pytest
+      run: |
+        make test TEST_OS=${{ matrix.os }}
+      env:
+          BENCHLING_TENANT: ${{ secrets.BENCHLING_TENANT }}
+          BENCHLING_CLIENT_ID: ${{ secrets.BENCHLING_CLIENT_ID }}
+          BENCHLING_CLIENT_SECRET_ARN: ${{ secrets.BENCHLING_CLIENT_SECRET_ARN }}
+          DST_BUCKET: ${{ secrets.DST_BUCKET }}
+          PKG_PREFIX: ${{ secrets.PKG_PREFIX }}
+          QUILT_CATALOG_DOMAIN: ${{ secrets.QUILT_CATALOG_DOMAIN }}
+    - name: Get Coverage Report
+      uses: orgoro/[email protected]
+      with:
+        coverageFile: coverage.xml
+        token: ${{ secrets.GITHUB_TOKEN }}
+        thresholdAll: 0.8
+      if: github.event_name == 'pull_request'

Install.md

@@ -124,7 +124,7 @@ since it was already created there.
 1. Under `Parameters`:
     1. Enter the name of the event bus created at step 1 as `BenchlingEventBusName`.
     1. Enter the client ID from settings of app created at step 2 as `BenchlingClientId`.
-    1. Enter the your Benchling tenant name (i.e. $BenchlingTenant in https://$BenchlingTenant.benchling.com) as `BenchlingTenant`.
+    1. Enter the your Benchling tenant name (i.e. $BenchlingTenant in <https://$BenchlingTenant.benchling.com>) as `BenchlingTenant`.
     1. Enter the name of the S3 bucket to use for storing packages as `DestinationBucket`.
     1. Optional: change the `PackageNamePrefix` used when creating new packages (default: `benchling/`).
     1. Specify the hostname of your Quilt Catalog as `QuiltWebHost`
@@ -144,12 +144,12 @@ click on its Physical ID.
 In order for the lambda to update Benchling with the package information,
 the notebook must have a schema containing exactly the following fields:
 
-| Name                  | Required  | Multi-select  | Definition    |
-| --------------------- | --------- | ------------- | ------------- |
-| Quilt+ URI            |           |               | Text          |
-| Quilt Revise URL      |           |               | Text          |
-| Quilt Catalog URL     |           |               | Text          |
-| Sentinel              |           |               | Integer       |
+| Name              | Required | Multi-select | Definition |
+|-------------------|----------|--------------|------------|
+| Quilt+ URI        |          |              | Text       |
+| Quilt Revise URL  |          |              | Text       |
+| Quilt Catalog URL |          |              | Text       |
+| Sentinel          |          |              | Integer    |
 
 You can either create a brand-new schema, or add these fields to an existing schema.
 Each new notebook will need to have this schema applied to it.

Makefile

@@ -2,18 +2,18 @@ sinclude .env
 TARGET = build/benchling_packager.yaml
 ACTIVATE = ./venv/bin/activate
 PKG_URL = "https://open.quiltdata.com/b/quilt-example/packages/examples/benchling-packager"
-.PHONY: all clean install template upload 
+.PHONY: all clean install install-dev template test upload 
 
 all: template upload
 
 clean:
-	rm -rf build
-	rm -rf venv
-	rm -f *requirements.txt
+	rm -rf build venv
+	rm -f *requirements.txt .pytest_cache .DS_Store
+	rm -f .coverage coverage.xml
 
 template: $(TARGET) 
 
-$(TARGET): build venv install make.py lambdas/lambda.py
+$(TARGET): build venv install make.py lambdas/main.py
 	. $(ACTIVATE) && python3 make.py > $(TARGET)
 
 upload:
@@ -38,3 +38,23 @@ install: venv/bin/pip-sync requirements.txt
 
 requirements.txt: venv/bin/pip-compile requirements.in
 	. $(ACTIVATE) && pip-compile requirements.in
+
+test: venv install-dev
+	. $(ACTIVATE) && python3 -m pytest  --cov --cov-report xml:coverage.xml
+
+test-partials: venv install-dev
+	SKIP_PARTIALS=False . $(ACTIVATE) && python3 -m pytest
+
+coverage: venv install-dev
+	printenv BENCHLING_ENTRY_ID
+	. $(ACTIVATE) && python3 -m pytest --cov --cov-report html:coverage.html
+	open coverage.html/index.html
+
+watch: venv install-dev
+	. $(ACTIVATE) && ptw . --now
+
+install-dev: venv/bin/pip-sync dev-requirements.txt
+	. $(ACTIVATE) && pip-sync dev-requirements.txt
+
+dev-requirements.txt: venv/bin/pip-sync dev-requirements.in 
+	. $(ACTIVATE) && pip-compile dev-requirements.in

README.md

@@ -5,15 +5,18 @@ This repository generates a CloudFormation template for processing
 (and link, if possible) a [Quilt](https://quiltdata.com/)
 package for every Benchling notebook.
 
-## Template generation and upload
+## Template generation
 
 Requires a recent version of Python 3.
 
-```bash
-make all
+```shell
+python3 -m venv venv
+. ./venv/bin/activate
+python3 -m pip install -r requirements.txt
+python3 make.py > build/benchling_packager.yaml
 ```
 
-This will:
+## Template upload
 
 - setup the Python environment
 - generate the template in the `build` directory
@@ -24,3 +27,21 @@ This will:
 
 To install and configure the template, see [Install.md](Install.md).
 Note: this is the file that's distributed as `README.md` in the package.
+
+## Testing for Developers
+
+If you want to modify the actual lambda function, you can run automated tests via:
+
+```shell
+make test
+```
+
+In order to run these tests, you'll need to set the following environment variables
+(usually in the `.env` file, which is auto-included by the Makefile):
+
+- `BENCHLING_TENANT`: the part before ".benchling.com" in your Benchling URL (e.g. "mycompany" for "mycompany.benchling.com")
+- `BENCHLING_CLIENT_ID`: the client ID for the Benchling API`
+- `BENCHLING_CLIENT_SECRET_ARN`: the ARN of the AWS Secrets Manager secret containing the client secret for the Benchling API
+- `DST_BUCKET`: the name of the S3 bucket (no prefix) where the generated packages should be stored
+- `PKG_PREFIX`: the prefix to use for the generated packages, with a trailing "/" (e.g. "benchling/" to store packages in the "benchling" directory)
+- `QUILT_CATALOG_DOMAIN`: the domain name of your Quilt catalog (if any) where the generated packages can be viewed

dev-requirements.in

@@ -0,0 +1,9 @@
+# https://suyojtamrakar.medium.com/managing-your-requirements-txt-with-pip-tools-in-python-8d07d9dfa464
+aws-lambda-powertools ~= 2.15
+benchling-sdk ~= 1.6
+botocore ~= 1.31
+jinja2 ~= 3.1
+quilt3 ~=5.3
+pytest
+pytest-coverage
+pytest-watcher