Use compact prompt when generating requests via Claude 3.5

[actualwitch]

this pr adds a slim version of system prompt for ai request generation and a suite of tests to evaluate its effects on the results (using goose-quotes from examples). as running these on each commit would be rather expensive and claude api is not fully deterministic at temp=0, they are not run by default, but instructions are provided inside the test.

[pkg-pr-new]

Open in Stackblitz

pnpm add https://pkg.pr.new/fiberplane/fpx/@fiberplane/studio@277

pnpm add https://pkg.pr.new/fiberplane/fpx/@fiberplane/hono-otel@277

commit: 48089a4

[brettimus]

noice - just looked over the prompt changes, will test locally tomorra

[brettimus]

haven't fully finished the vibe check, but one thing i did notice is that it's a little flaky with query params now?

tested on a few routes in goose quotes that accepted query params and every once in a while it would add them, but most of the time not.

old prompt always adds query params fwiw.

other notes:

• It's not taking much of a cue from the request history. If i just created a resource, it doesn't use the ID of that resource. However, this problem already appears to happen in the status quo with claude 👀 so not a regression. i think it might have to do with how chunky the history of requests is

• It correctly determined multipart form data for a request! Wild

• Hono url encoded forms were hard to sniff out, but i also think that's an existing problem

Will keep testing a bit more this eve

[brettimus]

separate question: you said you tested on our data we'd collected from our internal honcathons. do you have scripts available for those tests? how did you go about that?

[brettimus]

oh also! one more thing. it does tend to add the fpx-trace-id header into requests, which is a no-no. should probably add that instruction back to the system prompt

[actualwitch]

i had a bit in prompt to emphasize reusing the data from history, wasnt sure if that would be preferable so i left it out, will put it back. the fpx-trace-id is being added because we include it in history of requests, probably will disappear if we filter the headers there
i used https://github.com/actualwitch/experiment to explore the csv with our honcathon data

[actualwitch]

i have modified the inference logic to remove trace header from history entries, so it shouldn't be resurfacing in responses anymore. i also added a bit to integrate previous requests data, so that should be resolved too