Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Big interpreter rewrite #5406

Draft
wants to merge 75 commits into
base: main
Choose a base branch
from

Conversation

StachuDotNet
Copy link
Member

@StachuDotNet StachuDotNet commented Jul 20, 2024

This is a big PR that rewrites the Runtime and surrounding bits (esp. RuntimeTypes, the Interpreter, and PT2RT) to not stack-overflow so trivially (#5222). It should also lay the groundwork for improving overall runtime performance.

While working towards that goal, I accidentally handled many other things that have been bugging me for a while.

Here's what's been done towards the stated goal:

  • RuntimeTypes no longer has a recursive Expr concept, but rather has Instructions
  • the Interpreter thus is no longer tree-based, but is now a register-based virtual machine bytecode interpreter
    • when calling upon a Package Function or a Lambda, we append a new "call frame" to the virtual machine, to be processed in another iteration of the main interpreter loop
    • so: it's no longer trivial to reach stack overflows (the actual goal)
  • PT2RT is now notably more complicated, to map PT.Exprs (and fns, etc) to RT.Instructions
  • I think that the interpreter is generally easier to understand since these changes
  • a bunch of lower-level Interpreter and PT2RT tests
  • temp: SqlCompiler and DB.query have been gutted - I couldn't figure out a way to make them work in this new world, so we'll have to circle back to that

Here's the out-of-scope work that got done:

  • RuntimeErrors (RTEs) are now much better, and always structured -- OldStringErrorTODO is gone
  • non-recursive runtime type-checking is now complete[1]
  • partial-application works (!) for "applys" to both fns and lambdas
  • we reduced a few Builtins in favor of direct impl in package functions
  • we no longer build our tree-sitter parser to WASM -- we haven't been using that in a bit. the code is still there for when we need it, just commented
  • tidied a lot of test .dark files

[1] so, this closes #5222, closes #5130, closes #5071

There's some follow-up work to be done:

  • bring back SqlCompiler and DB.query
  • better handling of RTEs, generally
    • especially dev experience (understanding what went wrong)
  • more serious performance-tuning
  • more work around async and concurrent function calls

@StachuDotNet StachuDotNet force-pushed the big-interpreter-rewrite branch 3 times, most recently from 21af1ef to 8a3496c Compare July 31, 2024 19:51
@StachuDotNet StachuDotNet force-pushed the big-interpreter-rewrite branch 3 times, most recently from 4abbe16 to 29faccd Compare August 13, 2024 13:57
@StachuDotNet
Copy link
Member Author

StachuDotNet commented Aug 14, 2024

thoughts on implemented match that I wanted to record in case my current plan doesn't end up working out: https://gist.github.com/StachuDotNet/5a8923721b255ad2a91c1c6dcc3e9b8e

Edit: it worked out (really well!) but might as well keep this around.

@StachuDotNet StachuDotNet force-pushed the big-interpreter-rewrite branch 2 times, most recently from 2c4303b to 6ae7196 Compare August 15, 2024 16:13
@StachuDotNet StachuDotNet force-pushed the big-interpreter-rewrite branch 4 times, most recently from 713821d to 4299937 Compare September 18, 2024 15:07
@StachuDotNet StachuDotNet force-pushed the big-interpreter-rewrite branch 2 times, most recently from b19fc85 to 356bb70 Compare November 5, 2024 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: List.map Causes Stack Overflow Error on Large Lists
2 participants