KeyError at step 1 when using fragpipe-generated sdrf template

[levitsky]

Hi! I am uploading an SDRF file produced by FragPIPE (here is an example with only one row to demonstrate the issue: 1file.sdrf.txt. I renamed it from tsv to txt to be able to share it here)

When I go to page 1: Mapping local metadata, I get this error message:

KeyError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app).
Traceback:
File "/home/adminuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
File "/mount/src/lessdrf/pages/1_1. Mapping_local_metadata.py", line 75, in <module>
    download = st.download_button("Press to download SDRF file",ParsingModule.convert_df(template_df), "intermediate_SDRF.sdrf.tsv", help="download your SDRF file")
File "/mount/src/lessdrf/ParsingModule.py", line 381, in convert_df
    df = df[new_cols]
File "/home/adminuser/venv/lib/python3.9/site-packages/pandas/core/frame.py", line 4108, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
File "/home/adminuser/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 6200, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
File "/home/adminuser/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 6252, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")

[levitsky]

To update, here is an unredacted traceback that I got from running lesSDRF locally:

Traceback (most recent call last):
  File "/home/levitsky/.pyenv/versions/3.11.10/envs/lessdrf/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
  File "/home/levitsky/work/lesSDRF/pages/1_1. Mapping_local_metadata.py", line 75, in <module>
    download = st.download_button("Press to download SDRF file",ParsingModule.convert_df(template_df), "intermediate_SDRF.sdrf.tsv", help="download your SDRF file")
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/levitsky/work/lesSDRF/ParsingModule.py", line 381, in convert_df
    df = df[new_cols]
         ~~^^^^^^^^^^
  File "/home/levitsky/.pyenv/versions/3.11.10/envs/lessdrf/lib/python3.11/site-packages/pandas/core/frame.py", line 4108, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/levitsky/.pyenv/versions/3.11.10/envs/lessdrf/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 6200, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/levitsky/.pyenv/versions/3.11.10/envs/lessdrf/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 6252, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: "['source name'] not in index"

[levitsky]

This seems easy enough to fix, but perhaps I was not expected to upload this file at the home page, rather add it as a "local metadata file" at the next step, is that right?

I tried starting from scratch with an empty human SDRF and put in a list of file names, then uploaded the fragpipe SDRF, however I am not able to get any further with the metadata from the fragpipe file. I selected all columns that contained data and matched on comment[data file]:

It looks like it works but I don't see the imported metadata in my SDRF, it remains empty.

[levitsky]

I'm sorry to pollute this thread with every step of my thought process, but I think I have now realized how the local metadata mapping is supposed to work. I should not match on data file, but on every other column that I want to have in my SDRF. This is an awkward process and half of the columns cannot be transferred this way at all, so I suppose for the use case with partial SDRF from Fragpipe it's easier to allow uploading the whole file at the home page and just work around the missing source names.

[TineClaeys]

Hi Lev
Thanks for taking the time to look into this! The partial FragPipe files do not contain all columns which makes it indeed not optimal in lesSDRF with the current upload and mapping functions.
To allow for upload on the Home page it should not only add the source name column but all the columns that are in the selected template, otherwise it's not a valid SDRF.
I suggest we make another page in the app 'Technical SDRF upload' that can allow the upload of the file, add the columns from the selected template and does a validation of the input of the technical SDRF. What do you think? :)

[levitsky]

Hi Tine,

It would be great to have this functionality! I wonder, though, if it is worth it to make it a separate page as opposed to just processing it on the home page. As the developer, the pros and cons would be more obvious to you, but as a new user, I would be happy if what I tried to do initially had just worked. Can we add any missing columns (and later require filling them with values) and validate the existing ones when an SDRF is uploaded?