[sharktank] Add perplexity CI to sharktank dashboard #466
Conversation
|
@saienduri Would like to add Perplexity CI results to this dashboard too. pytest-html doesn't auto-merge existing html files. Do we need a utility function to do this separately? |
.github/workflows/ci_eval.yaml
Outdated
| run: pytest -n 8 -v -s sharktank/tests/evaluate/perplexity_torch_test.py --longrun --llama3-8b-f16-model-path=/data/llama3.1/8b/llama8b_f16.irpa --llama3-8b-tokenizer-path=/data/llama3.1/8b/tokenizer_config.json --html=perplexity/perplexity_torch.html | ||
|
|
||
| - name: Deploy to GitHub Pages | ||
| uses: peaceiris/actions-gh-pages@v3 |
There was a problem hiding this comment.
| uses: peaceiris/actions-gh-pages@v3 | |
| uses: peaceiris/actions-gh-pages@4f9cc6602d3f66b9c108549d475ec49e8ef4d45e # v4.0.0 |
Actions should be pinned as as suggested by OpenSSF Scorecard, see https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies.
There was a problem hiding this comment.
Furthermore, is it intended to push to gh-pages with every run of the CI? This probably creates the same problems outlined in issue #395.
There was a problem hiding this comment.
This is a nightly, so I don't think it will pose any presubmit problems
Co-authored-by: Marius Brehler <[email protected]>
Co-authored-by: Marius Brehler <[email protected]>
|
I don't think we need to merge the html files themselves. Each one will live on it's own subsite with an overarching index.html that points to each of them was what I was thinking. I am planning to explore this when I get some time today |
Makes sense. I tried to push the Each folder in |
…K-Platform into update-perplexity-ci
Add perplexity CI to sharktank dashboard
Rename _vmfb to _iree
Add Perplexity scoreboard & description
Add descriptive errors for better logging