Invalid columns in compact manifest for AnVIL

[hannes-ucsc]

species for example, but there could be others. There may also be missing columns or extra columns. We didn't put a lot of effort into curating the manifest config for AnVIL. The default approach should be to include all fields as columns in a compact manifest and to derive the manifest config from the field mapping for the /index/… endpoints.

The HCA manifest is a hand-curated set of fields but we've learned that no matter what set of fields we pick, there will always be people requesting yet another field to be included, so maintaining that set became a whack-a-mole. There will always be people who request that fields be removed because they don't see a use case for those fields, which seems short-sighted/pedantic to me. The only valid reasons not to include a field should be technical limitations, e.g., size, cardinality.

The current manifest config for AnVIL also uses a naming convention that is inconsistent with the naming of fields in the /index/… endpoint responses. The Data Browser doesn't currently expose the compact manifest so we should be able to rename columns using the same dotted convention used for the /index/… endpoints which itself is based on the naming of fields in the AnVIL schema.

[edit] Quick link to important comment detailing the implementation plan.

[hannes-ucsc]

For demo, show compact manifest on anvilprod. Review column headers. Show that species column is gone and that organism_type is there. Compare manifest with /index/files response. Download a file using a URL from the manifest.

[hannes-ucsc]

Discovered during demo that the manifest contains additional columns compared to the /index/files endpoint, for example, activities.reference_assembly.

[hannes-ucsc]

Additionally, the service response contains the files.uuid field which the manifest omits. It seems to have the same value as files.document_id. It could be that the service response field is redundant.

[hannes-ucsc]

The manifest has files.file_size. The service response has files.size. Same for .file_name vs .name. And .file_url and .url

[hannes-ucsc]

files.accessible is missing from manifest.

[achave11-ucsc]

Assignee to consider next steps.

[hannes-ucsc]

Spike in anvilprod to systematically compare the set of columns in the manifest with that of the inner entity properties in the /index/files endpoint. The set comparison should yield three sets, that of columns only in the manifest, that of properties only the endpoint and that of fields with other inconsistencies, e.g. a type mismatch.

[dsotirho-ucsc]

Only in the manifest

diagnoses.document_id
diagnoses.source_datarepo_row_ids
diagnoses.diagnosis_id

Only in the /index/{entity} response

activities.accessible
biosamples.accessible
bundles.accessible
datasets.description
datasets.accessible
donors.accessible
files.version
files.uuid
files.size (duplicate of `files.file_size`)
files.name (duplicate of `files.file_name`)
files.accessible

Mismatches

manifest          | /index/{entity} repsonse
------------------------------------------
bundle_uuid       | bundles.bundle_uuid
bundle_version    | bundles.bundle_version
files.file_url    | files.url
source_id         | sources.source_id
source_spec       | sources.source_spec

[hannes-ucsc]

For demo, repeat the spike (before demo) and review differences (during demo). I would like to be present at that demo.