Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metagenomics Exchange upload command and related bookkeeping code #350

Merged
merged 49 commits into from
Feb 29, 2024
Merged

Metagenomics Exchange upload command and related bookkeeping code #350

merged 49 commits into from
Feb 29, 2024

Commits on Jan 29, 2024

  1. adds incremental ebi search dumping for analysisjobs

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    0a8a761 View commit details
    Browse the repository at this point in the history

  2. integrates indexable sample data into analyses ebi search dump

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    4d51fe3 View commit details
    Browse the repository at this point in the history

  3. adds ebi search dump for studies/projects (plus some tweaks to analyses)

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    240d8f4 View commit details
    Browse the repository at this point in the history

  4. resolve migration merge conflicts

    @SandyRogers @mberacochea
    SandyRogers authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    ed97ccb View commit details
    Browse the repository at this point in the history

  5. adds support for chunking of analysis-runs ebi search dump

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    ab829f4 View commit details
    Browse the repository at this point in the history

  6. bugfixes on analysis-run dumper

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    fcb9300 View commit details
    Browse the repository at this point in the history

  7. Add metagenomics exchange support

    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    6438c3b View commit details
    Browse the repository at this point in the history

  8. Added 2 tests for ME, population script WIP

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    64f4900 View commit details
    Browse the repository at this point in the history

  9. Move metagenomic exchange commands to separate Class. Tests fail in P… 

    …OST request
    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    7484116 View commit details
    Browse the repository at this point in the history

  10. My revised version: 

    - Refactored the cli wrapper
- Refactored the settings to be in line with the rest
- Modified the unit tests
    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    fee19bd View commit details
    Browse the repository at this point in the history

  11. Modified ME dj function to add new ME records, fixed tests for ME and… 

    … api
    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    e7d9b17 View commit details
    Browse the repository at this point in the history

  12. Move test for mock API to test_mgx_api.py

    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    2e0b019 View commit details
    Browse the repository at this point in the history

  13. Add settings

    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    c92c381 View commit details
    Browse the repository at this point in the history

  14. change models and add patch and delete to ME

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    bdac39c View commit details
    Browse the repository at this point in the history

  15. Add delete and patch commands with tests to ME API

    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    65e64e0 View commit details
    Browse the repository at this point in the history

  16. add population scripts and tests for dry-run

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    3c664e6 View commit details
    Browse the repository at this point in the history

  17. add population scripts and tests for dry-run

    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    f082415 View commit details
    Browse the repository at this point in the history

  18. change api adress

    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    bb1ce81 View commit details
    Browse the repository at this point in the history

  19. change api adress

    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    0f309c4 View commit details
    Browse the repository at this point in the history

  20. WIP

    @KateSakharova @mberacochea
    KateSakharova authored and mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    83b6b3a View commit details
    Browse the repository at this point in the history

  21. Minor refactor of the MGX indexation process WIP

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    97e226e View commit details
    Browse the repository at this point in the history

  22. This is still a WIP - MGX

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    e229f5c View commit details
    Browse the repository at this point in the history

  23. Fixed the migrations for EBI Search and MGX. 

    The test tests/me/test_populate_metagenomics_exchange.py::TestMeAPI::test_removals_dry_mode it's working now.
    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    a6e9eda View commit details
    Browse the repository at this point in the history

  24. Fix some ME tests

    @KateSakharova
    KateSakharova committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    2f83224 View commit details
    Browse the repository at this point in the history

  25. WIP - migration tidy up

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    5961f7d View commit details
    Browse the repository at this point in the history

  26. Merge branch 'feature/metagenomics_exchange' of github.com:EBI-Metage… 

    …nomics/emgapi into feature/metagenomics_exchange
    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    02d3943 View commit details
    Browse the repository at this point in the history

  27. Adjust field names to new convention (last_ebi_search_indexed)

    @mberacochea
    mberacochea committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    60d2a37 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2024

  1. WIP testing

    @KateSakharova
    KateSakharova committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    ad539e7 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. Tests for populate ME

    @KateSakharova
    KateSakharova committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    986f222 View commit details
    Browse the repository at this point in the history

  2. rm dev from tests and leave only DEV API

    @KateSakharova
    KateSakharova committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    916ce45 View commit details
    Browse the repository at this point in the history

  3. fix pipeline mock

    @KateSakharova
    KateSakharova committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    948caa8 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. fixes

    @KateSakharova
    KateSakharova committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    6a55c40 View commit details
    Browse the repository at this point in the history

  2. add mock to patch test

    @KateSakharova
    KateSakharova committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    18d4641 View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. some fixes after review

    @KateSakharova
    KateSakharova committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    bfb1b54 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. Rename EBI Seach last_indexed fields

    @mberacochea
    mberacochea committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    7dbbb14 View commit details
    Browse the repository at this point in the history

  2. Add an extra migrate step to rename last_indexed -> last_ebi_search_i… 

    …ndexed

This is needed because RenameField doesn't allow you to change
the db_column name.

With this change it seems to work, and the data should be kept:

```sql
--
-- Alter field last_indexed on analysisjob
--
ALTER TABLE "ANALYSIS_JOB" RENAME COLUMN "LAST_INDEXED" TO "LAST_EBI_SEARCH_INDEXED";
--
-- Alter field last_indexed on study
--
ALTER TABLE "STUDY" RENAME COLUMN "LAST_INDEXED" TO "LAST_EBI_SEARCH_INDEXED";
--
-- Rename field last_indexed on analysisjob to last_ebi_search_indexed
--
--
-- Rename field last_indexed on study to last_ebi_search_indexed
--
```
    @mberacochea
    mberacochea committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    4a8f3b4 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2024

  1. Merge pull request #325 from EBI-Metagenomics/feature/metagenomics_ex… 

    …change

Feature/metagenomics exchange
    @mberacochea
    mberacochea authored Feb 8, 2024
    Configuration menu
    Copy the full SHA
    633b3f7 View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2024

  1. Fixes on the MGX based on feedback

    @mberacochea
    mberacochea committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    f3daefe View commit details
    Browse the repository at this point in the history

  2. Bump version

    @mberacochea
    mberacochea committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    f0a72ff View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. A few bug fixes for the metagenomics exchange command and api wrapper. 

    Add support for Assembly and Run based jobs (both).
Remove the public check, we only submit public data to the exchange.
Normalize argument names in the MGX wrapper.
Added some docstrings
    @mberacochea
    mberacochea committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    b2d60bd View commit details
    Browse the repository at this point in the history

  2. Fix typos in method args

    @mberacochea
    mberacochea committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    e3e9cfb View commit details
    Browse the repository at this point in the history

  3. Typo sequence_accesion -> sequence_accession

    @mberacochea
    mberacochea committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    afa3f18 View commit details
    Browse the repository at this point in the history

  4. Send the sequence_accession, not a Run object

    @mberacochea
    mberacochea committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    c3ba441 View commit details
    Browse the repository at this point in the history

  5. Add a bit of logging when posting a record to MGX

    @mberacochea
    mberacochea committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    4b145c0 View commit details
    Browse the repository at this point in the history

  6. Capture the API error data response, if present in the response. 

    The MGX API returns a json with the error message, which is useful
for debugging.
    @mberacochea
    mberacochea committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    6235f03 View commit details
    Browse the repository at this point in the history

  7. Mock the Metagenomics Exchange API

    @mberacochea
    mberacochea committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    fe6d2b8 View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2024

  1. Fix mgx_api check_analysis invocation typo

    @mberacochea
    mberacochea committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    783f275 View commit details
    Browse the repository at this point in the history

  2. Mark piece of code to revisit and improve

    @mberacochea
    mberacochea committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    1b031df View commit details
    Browse the repository at this point in the history

  3. Merge pull request #351 from EBI-Metagenomics/bugfix/metagenomics-exc… 

    …hange-bug-fixes

Fixes for a few bugs I encountered while testing the command
    @mberacochea
    mberacochea authored Feb 19, 2024
    Configuration menu
    Copy the full SHA
    c98171c View commit details
    Browse the repository at this point in the history