cpg0037-oasis #92
Replies: 13 comments 35 replies
-
|
Hi Shantanu, thanks for getting the ball rolling on this! Excited! Confirming the assay used is the standard Cell Painting v3.0 dye set to stain primary human hepatocytes (PHHs). There are 68 x 384 well plates in the dataset. We use the whole plate and do not exclude any edge wells. We image 15 sites (field of view) using the 40x objective per well. The field images are comprised of 5 fluorescent channels and 1 BF image. Each individual channel image is around 8 MB. Therefore it looks like the images will be around 20 TB, in sum. Images were acquired on an Operetta - we will also transfer the associated flat field correction (FFC) profiles and xml metadata for each plate. In addition to the images, we are also happy to share our in-house analysis QC pipeline results, including:
Leveraging the embeddings has proven particularly insightful. We have seen that ~70% of the compounds in this dataset produce image embeddings that are significantly different than the DMSO vehicle control embeddings in the same plate, and could be considered bioactive. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for all these details @ktitterton! Let's start with image data first. Do you anticipate any issues in structuring the data like this? As for the remaining components, I suspect they will all go into one of the existing folders in
And then finally, would it be possible for you to create metadata for each of these plates, as detailed in https://broadinstitute.github.io/cellpainting-gallery/data_structure.html#arrayed-metadata? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Shantanu, Currently our data are indeed under a The raw image data follows the standard PerkinElmer/Revvity format:
The associated metadata and features folder structure is:
Some misc. notes:
Yes, we are happy to create metadata following the specifications. Is this to be provided in addition to the data as it currently exists, described above? tagging my colleague Alex @cabreraalex who is our resident data wizard and much more expert in these matters than myself! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for all these details
Yes please (Please see my next comment) |
Beta Was this translation helpful? Give feedback.
-
|
The Axiom team just chatted a bit internally, and we were wondering if it would be most helpful to go ahead and copy over the data as it currently exists into an s3 bucket, and share it with y'all - that way you can let us know exactly what should be added, deleted, or rearranged. Let us know if this would be a good next step! Excited! |
Beta Was this translation helpful? Give feedback.
-
|
Great idea! I will ping @leoank via email to send you credentials |
Beta Was this translation helpful? Give feedback.
-
|
@ktitterton The images are structured exactly per specs, so all set there. The metadata.parquet looks good but will eventually need to be wrangled to follow the structure, but that can wait. Next, can you please provide the following for 1-2 plates? Please upload that to |
Beta Was this translation helpful? Give feedback.
-
|
Hi @shntnu and @ktitterton - are the images ready for @ErinWeisbart to start processing with CellProfiler, or are we waiting on some more checks/data? |
Beta Was this translation helpful? Give feedback.
-
|
The images look good to me but @ktitterton can you confirm these are ALL the images? |
Beta Was this translation helpful? Give feedback.
-
|
Hi ya'll! Confirmed all the images are in, and I transferred the rest of the metadata into the scratch workspace folder. LMK if it all looks alright or if anything looks off. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @cabreraalex - thanks for transferring everything, we've kicked off the image processing on our end. Do you, @ktitterton, or @shntnu know the plan for transferring the Dino features? In other projects, we've created additional folders to hold alternative features, for example in this case we have 'images' for the raw .tiff, 'workspace' for CellProfiler features, and 'workspace_dl' for some deep learning-derived features: 
I imagine that we'll want to do something like this as well. Has it already been discussed? |
Beta Was this translation helpful? Give feedback.
-
|
(I see the comment about with @shntnu's mock file structure, but don't see that reflected in the CellPainting gallery) |
Beta Was this translation helpful? Give feedback.
-
|
Hi Jessica! The embeddings should be there in a path like |
Beta Was this translation helpful? Give feedback.
-
|
Oh - it's possible that I'm getting ahead of things :) I thought that the data was finished syncing but Ank just told me that it'll take ~10 more hours. That's probably why I can't see it. |
Beta Was this translation helpful? Give feedback.
-
|
Ah interesting it should already be in s3 but maybe syncing locally? |
Beta Was this translation helpful? Give feedback.
-
|
We have a staging bucket where Ank/Shantanu/Erin would have reviewed, and then it gets synced to the "real" cpg that the rest of the lab can access. So I think the sync that Ank is referring to is between those two locations. |
Beta Was this translation helpful? Give feedback.
-
|
@ktitterton But instead were uploaded with image files directly in the plate folder: How might we go about getting the |
Beta Was this translation helpful? Give feedback.
-
|
Hi Erin! I can upload the xml, will update when done |
Beta Was this translation helpful? Give feedback.
-
|
Should be there now - prod 30 only has half the images (only half were OASIS), lmk if that causes an issue |
Beta Was this translation helpful? Give feedback.
-
|
@ktitterton @cabreraalex I also want to confirm that channel mapping I've parsed below is correct and is consistent across all plates and all batches? channels: channelid: |
Beta Was this translation helpful? Give feedback.
-
|
Upon closer inspection, there are only 2 .xml files that were generated with Harmony V7, though my questions still stand. |
Beta Was this translation helpful? Give feedback.
-
|
Queried version with: |
Beta Was this translation helpful? Give feedback.
-
|
Hi Erin, Yes, I can confirm there are two different Harmony versions and two different microscopes in the dataset. The vast majority of the data was acquired on O1, with a very small minority on O2. There was an unfortunate automation error in prod_25, so there are some technical artifacts in this batch, including the microscope switch for a couple of plates. In our unbiased image clustering, we have noticed some patterning by microscope in high-tox conditions. In general, the O2 microscope seems to generate images which have slightly more texture - but the raw image histograms are otherwise extremely similar. Yup, I can also confirm that channel mapping looks correct! Please do let us know if you have any other questions- no question is too small. We are all very much looking forward to seeing your results sometime soon! Thanks for checking in |
Beta Was this translation helpful? Give feedback.
-
|
@ktitterton There are several that are off by 6 files (34555 files) which would be a single field of view and I am not concerned about these, but I will note them here for the sake of completeness. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Erin! for prod 25 - 27 those missing plates are the reference plates, didn't upload the images. |
Beta Was this translation helpful? Give feedback.
-
|
Great, thanks for confirming @cabreraalex ! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @cabreraalex and @ktitterton - I'm not seeing any positive controls or negative controls in the Dino embeddings (and wondering if @ErinWeisbart has any in the images / resulting CellProfiler features). After compiling the metadata.parquet across all plates in locations like this, I don't see any compounds with the name "DMSO" or "dimethyl sufoxide", and no compounds with a concentration of 0. Can you let me know if you either A) didn't collect poscons/negcons on each plate or B) just didn't include them in the data that you uploaded? Poscons I can live without, but the DMSO samples are pretty crucial for the normalization and the concentration-response analysis. |
Beta Was this translation helpful? Give feedback.
-
|
Alright looks like the images for DMSO are uploaded but I excluded it from all the metadata. Could I reupload to the same bucket and it should work? |
Beta Was this translation helpful? Give feedback.
-
|
I'll tag @leoank here - I believe that you originally uploaded to a staging bucket which was synced with the real CPG after review. I assume uploading to the same place that you originally uploaded makes sense, but maybe Ank can confirm? I don't believe you'll be able to upload directly to the bucket that I linked. |
Beta Was this translation helpful? Give feedback.
-
|
Cool I just uploaded updated metadata and biochem to the staging bucket and am uploading the embeddings and other features now |
Beta Was this translation helpful? Give feedback.
-
|
Great, thanks so much! |
Beta Was this translation helpful? Give feedback.
-
|
@jessica-ewald I'm copying over updated metadata.parquet and biochem.parquet files. You'd need to reverify that everything looks good. #!/bin/bash
# Function to check if a file exists in S3
file_exists_in_s3() {
if aws s3 ls "$1" >/dev/null 2>&1; then
return 0
else
return 1
fi
}
SOURCE_DIR="s3://staging-cellpainting-gallery/cpg0037-oasis/axiom/workspace/scratch/"
SOURCE_BUCKET="s3://staging-cellpainting-gallery/"
DEST_BUCKET="s3://cellpainting-gallery/"
# List all biochem.parquet files in the source bucket
aws s3 ls --recursive "$SOURCE_DIR" | grep -E '(biochem|metadata)\.parquet' | sort | while read -r line; do
# Extract the file path
file_path=$(echo "$line" | awk '{print $4}')
# Construct the source and destination paths
source_path="$SOURCE_BUCKET$file_path"
dest_path="$DEST_BUCKET$file_path"
# Check if the source file exists
if file_exists_in_s3 "$source_path"; then
# Copy the file
aws s3 cp "$source_path" "$dest_path"
else
echo "Source file does not exist: $source_path"
fi
done
|
Beta Was this translation helpful? Give feedback.
-
|
Hi folks, our plan was to upload Axiom's processed data as-is and then re-organize later. Let's do that now We will follow the instructions here This is the current structure, under Proposed structure Under The Then under
Additionally, @jessica-ewald has new CNN-based from Ray Jones and Bram Gorissen; these would be structured like this: Does all this sound ok, @ktitterton @cabreraalex @jessica-ewald? We (Broad) should be able to help complete this task but might ping some of you for help |
Beta Was this translation helpful? Give feedback.
-
|
maybe '825c11' if I understand correctly? That's a hash for https://github.com/facebookresearch/dinov2/blob/main/dinov2/configs/train/vitl14.yaml which is the train config for the dino version we use. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the hash What's more appropriate: |
Beta Was this translation helpful? Give feedback.
-
|
let's keep fieldnorm since we do normalize by field and should make it explicit |
Beta Was this translation helpful? Give feedback.
-
|
we can add both if we want too: dinov2_b_vitl14_fieldnorm_825c11 go crazy |
Beta Was this translation helpful? Give feedback.
-
|
Great, sync is done! Sync code: import boto3
import logging
import argparse
from botocore.exceptions import ClientError
# Set up logging
logging.basicConfig(
level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
)
# Initialize S3 client
s3 = boto3.client("s3")
# Define S3 bucket and paths
BUCKET_NAME = "cellpainting-gallery"
SOURCE_PREFIX = "cpg0037-oasis/axiom/workspace/scratch/"
WORKSPACE_PREFIX = "cpg0037-oasis/axiom/workspace/"
WORKSPACE_DL_PREFIX = "cpg0037-oasis/axiom/workspace_dl/"
# List of production batches
PROD_BATCHES = ["prod_25", "prod_26", "prod_27", "prod_30"]
# Hardcoded model identifier
MODEL_IDENTIFIER = "dinov2_b_vitl14_fieldnorm_825c11"
def s3_copy_object(src_key, dst_key, dry_run=False):
"""Copy object within the same S3 bucket."""
if dry_run:
logging.info(f"[DRY RUN] Would copy {src_key} to {dst_key}")
else:
try:
s3.copy_object(
Bucket=BUCKET_NAME,
CopySource={"Bucket": BUCKET_NAME, "Key": src_key},
Key=dst_key,
)
logging.info(f"Copied {src_key} to {dst_key}")
except ClientError as e:
logging.error(f"Error copying {src_key} to {dst_key}: {e}")
def reorganize_workspace(dry_run=False):
"""Reorganize files in the workspace directory for all production batches."""
for prod_batch in PROD_BATCHES:
batch_prefix = f"{SOURCE_PREFIX}{prod_batch}/"
logging.info(f"Processing batch: {prod_batch}")
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=BUCKET_NAME, Prefix=batch_prefix):
for obj in page.get("Contents", []):
src_key = obj["Key"]
filename = src_key.split("/")[-1]
plate = src_key.split("/")[-2]
if filename == "segmentation.parquet":
dst_key = f"{WORKSPACE_PREFIX}segmentation/{prod_batch}/{plate}/{filename}"
elif filename == "image_metrics.parquet":
dst_key = f"{WORKSPACE_PREFIX}qc/{prod_batch}/{plate}/{filename}"
elif filename == "metadata.parquet" or filename == "biochem.parquet":
dst_key = (
f"{WORKSPACE_PREFIX}metadata/{prod_batch}/{plate}/{filename}"
)
else:
continue
s3_copy_object(src_key, dst_key, dry_run)
def reorganize_workspace_dl(dry_run=False):
"""Reorganize files in the workspace_dl directory for all production batches."""
for prod_batch in PROD_BATCHES:
batch_prefix = f"{SOURCE_PREFIX}{prod_batch}/"
logging.info(f"Processing batch: {prod_batch}")
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=BUCKET_NAME, Prefix=batch_prefix):
for obj in page.get("Contents", []):
src_key = obj["Key"]
filename = src_key.split("/")[-1]
plate = src_key.split("/")[-2]
if filename == "dinov2_b_fieldnorm.parquet":
dst_key = f"{WORKSPACE_DL_PREFIX}profiles/{MODEL_IDENTIFIER}/{prod_batch}/{plate}/{plate}.parquet"
s3_copy_object(src_key, dst_key, dry_run)
def main():
parser = argparse.ArgumentParser(description="Reorganize S3 files")
parser.add_argument(
"--dry-run",
action="store_true",
help="Perform a dry run without making actual changes",
)
args = parser.parse_args()
if args.dry_run:
logging.info("Performing a dry run. No actual changes will be made.")
else:
logging.info("Starting S3 file reorganization")
reorganize_workspace(args.dry_run)
reorganize_workspace_dl(args.dry_run)
if args.dry_run:
logging.info("Dry run completed. No actual changes were made.")
else:
logging.info("S3 file reorganization completed")
if __name__ == "__main__":
main()Validation code: import boto3
import logging
from collections import defaultdict
# Set up logging
logging.basicConfig(
level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
)
# Initialize S3 client
s3 = boto3.client("s3")
# Define S3 bucket and paths
BUCKET_NAME = "cellpainting-gallery"
WORKSPACE_PREFIX = "cpg0037-oasis/axiom/workspace/"
WORKSPACE_DL_PREFIX = "cpg0037-oasis/axiom/workspace_dl/"
# List of production batches
PROD_BATCHES = ["prod_25", "prod_26", "prod_27", "prod_30"]
# Hardcoded model identifier
MODEL_IDENTIFIER = "dinov2_b_vitl14_fieldnorm_825c11"
def list_s3_objects(prefix):
"""List all objects in S3 with the given prefix."""
objects = []
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=BUCKET_NAME, Prefix=prefix):
if "Contents" in page:
objects.extend(page["Contents"])
return objects
def verify_workspace_structure():
"""Verify the structure of files in the workspace directory."""
expected_structure = {
"segmentation": ["segmentation.parquet"],
"qc": ["image_metrics.parquet"],
"metadata": [
"metadata.parquet",
"biochem.parquet",
],
}
errors = defaultdict(list)
for prod_batch in PROD_BATCHES:
for folder, expected_files in expected_structure.items():
prefix = f"{WORKSPACE_PREFIX}{folder}/{prod_batch}/"
objects = list_s3_objects(prefix)
plates = set()
for obj in objects:
key = obj["Key"]
parts = key.split("/")
if (
len(parts) != 7
): # cpg0037-oasis/axiom/workspace/folder/prod_batch/plate/file
errors[prod_batch].append(f"Unexpected key structure: {key}")
continue
plate = parts[-2]
filename = parts[-1]
plates.add(plate)
if filename not in expected_files:
errors[prod_batch].append(f"Unexpected file in {folder}: {key}")
# Check if all plates have all expected files
for plate in plates:
for expected_file in expected_files:
expected_key = f"{prefix}{plate}/{expected_file}"
if not any(obj["Key"] == expected_key for obj in objects):
errors[prod_batch].append(f"Missing file: {expected_key}")
return errors
def verify_workspace_dl_structure():
"""Verify the structure of files in the workspace_dl directory."""
errors = defaultdict(list)
for prod_batch in PROD_BATCHES:
prefix = f"{WORKSPACE_DL_PREFIX}profiles/{MODEL_IDENTIFIER}/{prod_batch}/"
objects = list_s3_objects(prefix)
plates = set()
for obj in objects:
key = obj["Key"]
parts = key.split("/")
if (
len(parts) != 8
): # cpg0037-oasis/axiom/workspace_dl/profiles/dinov2_b.../prod_batch/plate/file
errors[prod_batch].append(
f"Unexpected key structure: {key}. len(parts) = {len(parts)}"
)
continue
plate = parts[-2]
filename = parts[-1]
plates.add(plate)
if filename != f"{plate}.parquet":
errors[prod_batch].append(f"Unexpected filename: {key}")
# Check if all plates have their corresponding file
for plate in plates:
expected_key = f"{prefix}{plate}/{plate}.parquet"
if not any(obj["Key"] == expected_key for obj in objects):
errors[prod_batch].append(f"Missing file: {expected_key}")
return errors
def main():
logging.info("Starting S3 file structure verification")
workspace_errors = verify_workspace_structure()
workspace_dl_errors = verify_workspace_dl_structure()
if not workspace_errors and not workspace_dl_errors:
logging.info("Verification completed. All files are correctly structured.")
else:
logging.error("Verification completed. Errors were found:")
for prod_batch, errors in workspace_errors.items():
for error in errors:
logging.error(f"{prod_batch}: {error}")
for prod_batch, errors in workspace_dl_errors.items():
for error in errors:
logging.error(f"{prod_batch}: {error}")
if __name__ == "__main__":
main()Log: |
Beta Was this translation helpful? Give feedback.
-
|
For our notes -- we also have an internal email thread discussing this "Updating metadata.parquet files / blocklists" |
Beta Was this translation helpful? Give feedback.
-
The data from Axiom will kick us off!
We will follow the process documented here https://github.com/broadinstitute/cellpainting-gallery/wiki/Data-Transfer-Workflow
Axiom team, could you please answer the following:
images,analysis,backend,load_data_csv,profiles. optional components:pipelines,qc, etc.). Note thatmetadatais required.cpg0037-oasisThen, please orient yourself to the prep in https://broadinstitute.github.io/cellpainting-gallery/contributing_to_cpg.html#preparing-for-data-deposition
For Broadies, here's our checklist from https://github.com/broadinstitute/cellpainting-gallery/wiki/Data-Transfer-Workflow
Champion: Shantanu SIngh
Dataset name:
cpg0037-oasiscpg0031) by incrementing the identifier of the most recent dataset in the dataset list, and update the list with the new identifier.cpg0031-caicedo-cmvip).Beta Was this translation helpful? Give feedback.
All reactions