Is it possible to alter MODEL_N_CTX? #934
Unanswered
davelejeune
asked this question in
Q&A
Replies: 1 comment
-
I'm struggling with the same problem. Changing values takes no effect. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
All -
It appears whatever that I set MODEL_N_CTX to is not observed when running the privateGPT.py script.
Although I've set it to 4096 it always fires up with 2048 --- assume that correlates to the 2K token limit in Chat3.5 that 4 was supposed to be able to increase to 8K?
I've also read there are limitations on what is returned as far as length. Is that limitation with GPT4All?
Finally, any recommendations on other models other than the groovy GPT4All one - perhaps even a flavor of LlamaCpp?
Below is the output I described showing that the max number of tokens is set to 2048...
Found model file at models/ggml-gpt4all-j-v1.3-groovy.bin
gptj_model_load: loading model from 'models/ggml-gpt4all-j-v1.3-groovy.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx = 2048
gptj_model_load: n_embd = 4096
gptj_model_load: n_head = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot = 64
gptj_model_load: f16 = 2
gptj_model_load: ggml ctx size = 5401.45 MB
gptj_model_load: kv self size = 896.00 MB
gptj_model_load: ................................... done
gptj_model_load: model size = 3609.38 MB / num tensors = 285
Many thanks in advance -
=-Dave
Beta Was this translation helpful? Give feedback.
All reactions