Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add inputs_embeds as alternative for input_ids #158

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Maykeye
Copy link

@Maykeye Maykeye commented Feb 3, 2024

Adding to the model.forward inputs_embeds (same name as HF models use) to give the model token embedding directly rather than token ids

The main use case is training soft prompts without adding N tokens to tokenizer that never get outputted(and then jumping through hoops to train them only without touching existing tokens)

(ETA: from more recent example: techniques like neft are much easier to do when it's possible to pass embedding directly)

@albertfgu albertfgu force-pushed the main branch 2 times, most recently from 6d45666 to 41d30ce Compare June 3, 2024 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant