Change eval batch size #128
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #128 +/- ##
=======================================
Coverage 64.58% 64.58%
=======================================
Files 17 17
Lines 1762 1765 +3
Branches 291 292 +1
=======================================
+ Hits 1138 1140 +2
Misses 560 560
- Partials 64 65 +1 ☔ View full report in Codecov by Sentry. |
| @@ -106,6 +106,10 @@ class LanguageModelSAERunnerConfig: | |||
|
|
|||
| dead_feature_threshold: float = 1e-8 | |||
|
|
|||
| # Evals | |||
| n_eval_batches: int = 10 | |||
| n_eval_seqs: int | None = None # useful if evals cause OOM | |||
There was a problem hiding this comment.
nit: is this the batch size used for eval? If so, this would be clearer named as eval_batch_size to match the naming of store_batch_size and train_batch_size params
There was a problem hiding this comment.
ah looks like I must have commented as this was being merged 😅. Not a big issue regardless
There was a problem hiding this comment.
True. Feel free to make a PR for this.
|
Yeah, the reason I called it this was that the batch size here is the
number of sequences, but the batch size produced for the SAEs is #
sequences * context length. Having an ‘eval_batch_size=4’ in the config
seemed like it would be surprising next to a batch size of (say) 4096.
…On Wed, 8 May 2024 at 16:40, Joseph Bloom ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In sae_lens/training/config.py
<#128 (comment)>:
> @@ -106,6 +106,10 @@ class LanguageModelSAERunnerConfig:
dead_feature_threshold: float = 1e-8
+ # Evals
+ n_eval_batches: int = 10
+ n_eval_seqs: int | None = None # useful if evals cause OOM
True. Feel free to make a PR for this.
—
Reply to this email directly, view it on GitHub
<#128 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AALGU42XA7ESJNXWWGMWWQLZBJBN5AVCNFSM6AAAAABHM4UZ5KVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDANBWGAZDMMBUGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
👍 makes sense! |
|
@tomMcGrath @tomMcGrath We previously has "train_batch_size" which is a token batch size for the SAE and a "store batch size" which was a prompts batch size for generating activations. The latter was then reused during evaluations, but we've now seperated them out. I think the following names might be best:
|
|
That rename makes sense to me, I feel like that makes it more clear what's going on and reduces confusion around the difference between a "batch" of prompts vs tokens |
|
I like that rename - I just put a quick issue in to remind me to do the rename. |
* Surface # of eval batches and # of eval sequences * fix formatting * fix print statement accidentally left in
Description
Surfaces extra options for eval step: number of eval batches and number of sequences per eval batch. Adds an optional parameter to activations_store.get_batch_tokens to make this clean: now get_batch_tokens can get a number of sequences that differs from the value defined at init.
Type of change
Please delete options that are not relevant.
Checklist:
I've checked these locally but not added tests.
You have tested formatting, typing and unit tests (acceptance tests not currently in use)
make check-cito check format and linting. (you can runmake formatto format code if needed.)