To-Do.txt

This file contains my personal notes/to-do list.
Don't take anything listed here for granted. The stuff I write here changes a lot!
- Pierre

To-Do:
  > ASTUnit : Represents a source file, contains declarations. (Done)
  > IdentifierTable & IdentifierInfo (Done)
  > ASTContext (Done)
  > Move stuf to /Fox/Basic (Done)
  > ASTContext : Types (Done)
  > New type system & deletion of /Common/Types (Done!)
  > Replace strings in the AST by IdentifierInfo* pointers (Done)  
  > ParsingResult rework (Done)
  > AST Improvements (Done)
  > Add ASTContext,parser & ast support for arrays types & Array literals. (Done)
  > Parser recovery improvements (Done)
  > Parsing Result Rework (Again...) (Done)
  > DeclRecorder (Done)
  > Parser support for DeclRecorder with a RAIIContext object. (Done)
  > UniquePtrVector & DereferenceIterators + Minimizing interaction with unique pointers for AST Visitors(Done)
  > Major AST "Cleanup" in preparation for Visitor pattern rework (Done)
  > Write more Unit tests (Done)
  > Visitor system rework (Done)
  > Delete old ASTVisitor & Dumper  (Done)
  > Rewrite dumper under the new system  (Done)
  > Make "MapDataIterator" work (Done)
  > ASTDumper overhaul (Done)
  > Write tests for the new visitor system (I'll do more when needed)
  > Sourceloc & SourceManager system (Done)
  > Update lexer/token to add a sourcerange/loc to every single token (Done)
  > Update the Parser so it can return more SourceLoc info. (Done)
  > Update the AST so all nodes have a sourceloc/range too wherever needed. (Done)
  > Write Decl Locs Test (Done)
  > First AST Code Review (Done)
  > Remove "Fake" Recovery nodes in the AST and find a proper, better solution to this problem. (Done)
  > Update Diagnostics & DiagnosticsEngine to support SourceRange/SourceLoc (Done)
  > Namespace/Name change (Done)
  > Delete the ContepromoteSeverityIfNeededxt (Done)
  > Update Lexer to use the DiagnosticEngine (Done)
  > Update Parser to use DiagnosticEngine + ParserDiagnostics Rework (Done)  
  > SourceManager improvements : Line table & Renaming SourceManager.hpp/.cpp to Source.hpp/.cpp (Done)
  > Small code review (error handling checking) (Done)
  > Rename DeclRecorder to DeclContext.(Done)
  > Rework the DiagnosticEngine::report function (Done)
  > Add classof to AST Nodes (Done)
  > Replace dynamic_cast with llvm's dyn_cast (Done)
  > Compile without RTTI (Done)
  > Diagnostics Overhaul (Done)
  > Finish the LinearAllocator (Done)
  > AST: Allocator: Move from unique_ptrs to raw allocated pointers (Done)
  > AST: Divide the hierarchies (Done)
  > Review code of expr/stmt/decl/astnode (Done)
  > AST code review (again) (Done)
  > Finish the PtrVariant (Done)
  > parseFuncDecl refactoring (Done)
  > Minimize ExprVector copies in the ArrayLiteral & FuncCall exprs (Done)
  > Add custom return types for expr/decl/stmt/types for the ASTVisitor (Done)
  > Add "Extra Range" to Diagnostics (Done)
  > Add Type* to all exprs (Done)
  > Add LValueType (Done)
  > Add ValueDecl (Done)
  > Remove QualType (Done)
  > Update PrimitiveType (Done)
  > Begin basic Sema functions (Done)
  > Switch to library-based architecture, splitting includes & headers (Done)
  > ResultObject (Done)
  > ASTWalker (Done)
  > Add ErrorType (Done)
  > Bulk of the Type rework (Done)
  > Finish Type Rework/Solve last issues (Done)
  > Switching to ::create or ::get idiom for Types instead of creation through ASTContext (Done)
  > Constrained Types (Done)
  > Type toString rewrite (TypePrinter) (Done)
  > Constraint Printer (toString & toDebugString) (Done)
  > Type debug print (toDebugString) (Done)
  > Removing ParensExpr (Done)
  > ConstrainedType in Semantic Analysis (Done)
  > Fix Array Literal Inference (Done)
  > Add DiagnosticVerifier (Done)
  > Sema: Replacing ConstrainedType with CellType (Done)
  > Create better unwrapping functions to effectively unwrap types (Done)
  > Adding -werr for LIT test suite (Done)
  > Replace BuiltinType by BasicType (Done)
  > Properly use Type in place of TypeBase* (Done)
  > Refactoring - fixing bad terminology (Done)
  > Sema::typecheckExprOfType (Done)
  > Sema::isDowncast (Done)
  > Sema: Basic Exprs Type-Checking + tests (Done)
  > Sema: Basic Stmt checking + tests (Done)
  > Identifier & IdentifierTable rework (Done)
  > const correctness update: no more double getters for getters of pointers (Done)
  > UnitDecl rework (Done)
  > Add features to get a Lexical view of a DeclContext (Done)
  > Add ASTContext& to UnitDecl (Done)
  > Remove Empty constructors in AST nodes (Done)
  > Switch whole AST to ::create Idiom (Done)
  > FuncDecl is not a DeclContext anymore (Done)
  > Decl::Parent to represent declaration parents (Done)
  > Remove Parser::state + Cleanup Parser.hpp (Done)
  > (Parser) Implement grammar 0.14 in the (Done)
  > DeclContext Improvements: Lexical view is the default, Lookup Map generated on demand (Done)
  > Unqualified Lookup (Done)
  > Finish Decl Checking (Done)
  > Add an Identifier range for NamedDecls (Done)
  > UnitDecl is no longer a NamedDecl (Done)
  > ASTContext stores DiagnosticEngine& and SourceManager& (Done)
  > Update StmtChecker to correctly open scopes where appropriate (Done)
  > ASTDumper overhaul (Done)
  > Add an IdentifierRange to NamedDecls (Done)
  > Diagnostic/DiagnosticEngine code cleanup/review (Done)
  > AST ParamList (Done)
  > Not all AST Nodes are trivially destructible, creating memory leaks (Fixed)
  > Subscript works on String & Uninferred types (Done)
  > ASTContext cleanups test (Done)
  > Fix "couldn't infer type of this expression" cascading errors in ExprFinalizer (Done)
  > Proper SourceRange for UnitDecls (Done)
  > FunctionType + FunctionType tests (Done)
  > Name binding logic (Done)
  > Semantic analysis for using variable within their own initial value (Done)
  > CallExpr checking (Done)
  > ReturnStmt checking (Done)
  > Disable function types within arrays (Done)
  > Write tests for interaction of a FunctionType with other operations (Done)
  > Introduce and use "isWellFormed" in Sema (Done)
  > Drop the "Moonshot" name and only use "Fox" as the project name (Done) 
  > Booleans are not numeric types anymore (Done)
  > Switch Fox to full blown strong typing (Done)
  > Make the Type static factories return a Type instead of a TypeBase* (Done)
  > Add "AnyLiteralExpr" as a common base for LiteralExprs (Done)
  > Refactor "getRange" in Decl/Stmt/Expr (Done)
  > Make the Type getters return "Type" instead of pointers (Done)
  > Replace CellTypes with a proper "TypeVariableType" (Done)
  > SourceManager rework/cleanup (Done)
  > TypeLoc update (Done)
  > Parser result object rework (Done)
  > Split DeclContext into DeclContext and LookupContext (Done)
  > FuncDecl is a DeclContext again + Simplify the Decl Parent logic (Done)
  > Split SourceLoc & SourceManager headers (Done)
  > SourceManager interface improvements (Done)
  > DiagnosticEngine interface improvements (Done)
  > LinearAllocator overhaul (Done)
  > DeclContext - LookupContext merged + interface improvement/rework (Done)
  > UTF8 Handling improvements (Done)
  > Add local lookups in DeclContext (Done)
  > Remove LocalScope
  > Rename getBegin/getEnd/getRange to getBeginLoc/getEndLoc/getSourceRange (Done)
  > Polish the DeclContext changes (Done)
  > Purge the codebase of trivial getters and setters of references (Done)
  > Move TypeWalker/TypeVisitor out of ASTWalker.hpp/ASTVisitor.hpp in their own files (Done)
  > Make TypeBase::isXXXType see through LValues (Done)
  > Lexer Rework (Done)
  > String Normalization + ASTDump of normalized strings (Done)
  > Parser Recovery Improvements (Done)
  > Document the public interface with Doxygen-style comments (Done)
  > Removing StringManipulator (Done)
  > Polishing the Lexer Rework + Adding EOF token (Done)
  > StableVectorIterator (Done)
  > Remove FunctionTypeParam (Done)
  > Fixed immutability issues (Done)
  > Added toString '$' unary operator (implicitely calls the appropriate toString func) (Done)
  > Stop taking the ';' into account in VarDecl/ReturnStmt's range.

  > VM, Bytecode, BytecodeGen (Currently being worked on, see Backend To-Do.txt)

  > Do a BIG code re-read in BCGen and Sema. They have grown quite a bit lately and they could use
    a serious code review to fix some ugly parts.
    -> Sema (done)
    -> BCGen

  > Consider removing TypeLoc, it's nearly useless, it's just a SourceRange/Type pack that's
    never used outside AST Nodes.
  > Break and Continue Statements
  > Warn if a "let" variable doesn't have an initializer

In no particular order:
  > Implement more features (see below)
  > Small "Make it Right" phase, fix things in the interpreter I'm not fully satisfied of,
    and try to do some work in the High/Medium priority to-dos below.
  > Proper VM Debug info & error handling mechanism. Possibly exception-based.

Features I want to add (in order of priority)
  > UFCS
    > Use an "UnresolvedDotExpr" that also captures the argument list for easy resolution of UFCS. It also allows "disabling" UFCS
     like this: (a.b)(c) (instead of a.b() which might be interpreted as b(a, c))
  > Function Overloading (on number & type of arguments)
  > Ternary operator
  > Allow omission of type annotations in local & global variable declarations
    > To Implement function overloads, add another "SemaType" that represents a
      disjunction of FunctionTypes* (the candidates) + the ID of the selected candidate.
    > It'd also be a good idea to give 'TypeVariableType' and 'OverloadType' a common base
      called 'SemaType'.
  > Support multiple files
  > Tuples
  > Unions & is-expression
  > Structs
  > Pattern in variable declarations
  > Allow variable declarations in `if`s
  > for-in loops
  > Functions as first-class citizens
  > Typedefs

Other things that aren't part of the main pipeline:
  High priority:
    > DiagnosticVerifier should be updated to support a diag at any line

    > Implement "lex on demand" + Improve the parser
      > Lex on demand, Remove TokenVector
      > Parser consume methods should return a SourceLoc instead of a range to eliminate
        the dozen of .getBeginLoc() calls in the parser. This change is just to reduce
        verbosity and make the code more readable.
        Ideally, I should have 2 consume methods, one that returns locs, and one that
        returns range
  
    > Split lookupmap implementation depending on if it needs ScopeInfo or not, 
      so not all DeclContext pay the price of storing the ScopeInfo if they don't need it

    > ASTVerifier: Checks that the AST is well formed: a nice debug tool!

    > Multiple allocator kids in the ASTContext: Perma (for the resolved ASTNodes)
      Unresolveds (for unresolved ASTNode/Types) and Sema (for SemaTypes)
      Sema pools are created and managed by Sema using a RAII object. Unresolveds pool
      is always there and can be reset separately from the Perma pool at any moment, but usually
      they'll be freed past semantic analysis.
        ASTContext::Arena class: type maps, the allocator and cleanups
        
  Medium priority:  
    > Parser diagnostics could be better in some places, especially when there are unexpected tokens
      in the input.

    > Find a better fix for recursive dependecy of global variable initializers, e.g. scenarios
      like 
        let a : int = b;
        let b : int = a;  
      (this doesn't work if they're in the same file, but if they're in different files, it should)
      Currently the fix is to disallow referencing other declarations inside a global var initializer,
      but this might be a bit too restrictive. I'd like to give some more freedom there.

    > Add a "RAIIIgnoreAll" object to ignore all diagnostics temporarily, and
      remove setIgnoreAll(value) (replace by ignoreAll() returning RAIIIgnoreAll)

    > DiagnosticVerifier: Add a syntax to expect a diag multiple times, e.g. 'expect-error(3)@+1'
      -> Use it in tests\VerifyMode\Sema\Expr\call.fox

    > Rewrite the mess that is ASTDumper.

    > DiagnosticEngine overhaul: report(..) should return a Diagnostic::Builder object:
      a lightweight RAII object. The actual Diagnostic data should be contained 
      inside the DiagnosticEngine. 
      TODO: Prove that this will bring a perf improvement.

  Low priority:
    This is a list of "low-priority" stuff that needs to be done.

    > Getting Exception-Free:
        > Create my own "checked" wrapper around the unchecked functions of utfcpp, and stop
          using utfcpp's checked.h header, so I can be fully exception-free.

    > Add proper unicode support for StreamDiagConsumer's caret line, and re-enable it for
      every source line (even those with non ASCII char).
        utf8proc would be great for this: https://github.com/JuliaStrings/utf8proc

    > Fix warnings generated by -Wreorder on CLang
    > Re-create the wiki on GitHub and explain the semantics of Fox.
    > Add more AST tests like ASTTest.ExprFlags, which tests that flags works as intended in the AST
    > Allow cast expr chaining (update grammar & parser)
      This isn't really a commonly used feature, but I hate seeing errors just because I type
      "foo as bool as string" in tests.

  Maybe someday: 
    > Rethink the const-correctness policy, especially when it comes to getters that return pointers in the AST.