-
Notifications
You must be signed in to change notification settings - Fork 1
Data Assimilation
Nirav Raje edited this page Apr 7, 2022
·
4 revisions
We explored different approaches for NASA MERRA data ingestion and finalized on using subset APIs with aggregation.
We first break down the data into small chunks and then convert them into a cloud-friendly JSON format for ease of transfer within our microservice architecture. We are currently storing the processed JSON data in our Redis store which we have configured to be persistent.
Our frontend will utilize a polling mechanism to query our intermediate Redis data set.
Security Note: We have configured Kubeseal on our master node such that it uses private key of master to perform encryption of the secret values provided in the Kubernetes YAML file. Thus, our NASA credentials are not exposed in our GitHub repository.
Issue:
- Testing Overview
- Load Testing
- Overall System Load Testing
- Spike Testing
- Fault Tolerance Testing
- Conclusion & Future Improvements
- Data Assimilation
- Architecture Improvements: Message Queues, Caching and Polling
- CI/CD and Infrastructure Deployment
- Visualization
- Custos Deployment Status
- Rancher Setup
- Kubernetes Cluster Deployment using Rancher
- Setting cert-manager, keycloak, consul, vault and MySQL
- Custos Deployment
- JMeter Testing for Custos Deployment with Python SDK
- Custos - Suggested Improvements