Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory for kraken2 #36

Open
wm75 opened this issue Aug 7, 2023 · 4 comments
Open

Memory for kraken2 #36

wm75 opened this issue Aug 7, 2023 · 4 comments

Comments

@wm75
Copy link

wm75 commented Aug 7, 2023

The memory requirements of this tool are largely dependent on the DB size you're using with it.
For DB sizes see:
https://benlangmead.github.io/aws-indexes/k2
As you can see, there are dramatic size differences.

Can this information be used somehow to guide the memory assigned to the tool?

@bgruening
Copy link
Member

If you provide a database=memory mapping we could use that I think.

@natefoo
Copy link
Member

natefoo commented Apr 9, 2024

I am using DB size + 24 GB:

  toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/.*:
    cores: 16
    # TODO: make a helper
    mem: |
      import os
      mem = 64
      table_name = "kraken2_databases"
      lookup_column = "value"
      value_column = "path"
      value_template = "{value}/hash.k2d"
      options = job.get_param_values(app)
      lookup_value = options["kraken2_database"]
      table_value = app.tool_data_tables.get(table_name).get_entry(lookup_column, lookup_value, value_column)
      if table_value is not None:
          table_value = value_template.format(value=table_value)
          try:
              mem = os.path.getsize(table_value)/1024**3
              log.debug("Data table '%s' lookup '%s=%s: %s=%s': %s GB",
                        table_name, lookup_column, lookup_value, value_column, table_value, mem)
          except OSError:
              log.exception("Failed to get size of: %s", table_value)
      else:
          log.warning("Data table '%s' lookup '%s=%s: %s=None' returned None!, defaulting to %s",
                      table_name, lookup_column, lookup_value, value_column, mem)
      mem + 24

@nuwang
Copy link
Member

nuwang commented Apr 10, 2024

This looks like a good rule to add to the TPV db itself? Would you be able to make a PR?

@natefoo
Copy link
Member

natefoo commented Apr 10, 2024

Yes, I meant to add a helper at some point, just haven't gotten to it yet - I will do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants