Implement TypeScript in New Expensify

[roryabraham]

Proposal: Utilize TypeScript to reduce bugs in NewDot

📄 Link to the design doc

docs.google.com/document/d/1LNcgYeYyovQ88gNjLIgBqBReLtbLAbxeAbna9QKxeAQ/edit#

🖇️ Other links

• TypeScript configurations
• File by file migration plan

Strategy

Our strategic roadmap for the next couple years is Reunification – moving all of our existing users from OldDot to NewDot, and it’s going to look like a completely new product. During this transition period, there’s an increased risk for churn, and stability of the platform is more important than ever. Any lines of defense between our customers and bad code will pay dividends during any tumultuous times to come.

Problem

JavaScript has been very popular for front-end development for a long time, and over the years it has evolved with new developer-friendly features that make it easier to GSD. Overall, it’s pretty great. However, it is still plagued by one fundamental problem: runtime errors stemming from incorrect types. This common class of problem leads to bugs, ranging in severity from showing weird things on the screen to crashing the whole app.

While there are many bugs that can be caused by incorrect types, the most common example of a runtime type safety error is Cannot access property X of undefined. A quick GitHub search reveals that over the last couple years we have introduced hundreds of this type of bug into the NewDot codebase, each of them stemming from this same root cause – attempting to access a property on a variable whose value is undefined. When this occurs, the app crashes.

Solution

Let's commit to adopting TypeScript in New Expensify, an awesome superset of JavaScript that gives us the benefits of using a compiled language like C++! That way, type errors can be caught at compile-time instead of run-time, and a whole class of common bugs will be eliminated from our product. #BugZero here we come.

Furthermore, let’s engage the near-limitless capacity of our open-source community to do it in parallel to our existing roadmap, without pausing any other feature development in the meantime.

Tasks

• Post Proposal (full Problem/Solution statement) in #expensify-open-source
• Wait at least one full business day, and until the post has a majority (2/3) of positive reactions (👍)
• Paste Proposal in the space above with a link to the Slack thread
• Email [email protected] and paste in the Proposal
• Fill out the High-level overview of the problem, Timeline, and Terminology sections of the Design Doc
• Email [email protected] (continue the same email chain as before) with the link to your Design Doc
• Host a pre-design meeting (example) in #expensify-open-source to discuss any necessary details in public before filling out the High-level of proposed solution section.
• Fill out the High-level of proposed solution section
• Email [email protected] again with links to the doc and pre-design conversation in Slack
• Add the DesignDocReview label to get the High-level of proposed solution section reviewed
• Respond to any questions or concerns and bring up blockers in Slack to get a consensus if necessary
• Confirm that the doc has the minimum necessary number of reviews before proceeding
• Host another pre-design meeting in #expensify-open-source to ask for engineering feedback on the technical solution.
• Fill out the Detailed implementation of the solution and related sections.
• Re-add the DesignDocReview label to this issue
• Respond to any questions or concerns and bring up blockers in Slack to get consensus if necessary
• Confirm that the doc has the minimum necessary number of reviews before proceeding
• Email [email protected] one last time to let them know the Design Doc is moving into the implementation phase
• Implement the changes
• Send out a follow up email to [email protected] once everything has been implemented and do a Project Wrap-Up retrospective that provides:
  • Summary of what we accomplished with this project
  • What went well?
  • What could we have done better?
  • What did we learn?

[roryabraham]

Here's a high-level overview of what I'm imagining for the rollout plan:

flowchart LR
    A(Setup TypeScript for E/App source code)
    X(Setup TypeScript for GitHub Actions)
    Y(Setup TypeScript for Jest)

    B(Laying Foundation)
    C(Materials and Plumbing)
    D(Construction)
    E(Finishings)
    
    F(Implement TypeScript in GitHub Actions)
    G(Implement TypeScript for Jest tests)
    
    subgraph Main App
        A --> B --> C --> D --> E
    end
    
    subgraph GitHub Actions
        direction LR
        X --> F
    end
    
    subgraph Jest
        Y ----> G
        C --> G
    end
    
    H(Setup TypeScript in react-native-onyx)
    I(Implement TypeScript in react-native-onyx)
    
    subgraph react-native-onyx
        direction LR
        H --> I
    end

Loading

Most of this is pretty self-explanatory, but I'll go into some extra detail for the rollout plan for the main source code.

• Setup:
  • Add TypeScript tooling for local development and in the deploy process.
  • Migrate App.js to App.ts
• Laying Foundation:
  • Create type definitions for key entities (such as Report and Policy)
  • Migrate simple, widely used files that will use mostly inferred types (such as CONST and ONYXKEYS)
  • After this phase, disallow new JS files
• Materials and Plumbing:
  • Migrate low-level components and utilities that don’t depend much on other files. Because these files generally don’t have many dependencies, it’s less likely that we’ll scope creep, have tons of conflicts, or find that they need to be on HOLD for other migrations.
  • Files that don’t have other dependencies are like “materials”. Starting with most of the basic materials in place will make the rest of the construction go faster and smoother.
  • However, we won’t delay the main “construction phase” to get all the materials in place, just the most common and critical ones.
• Construction:
  • Migrate our most commonly used and most critical components and libraries.
• Finishings:
  • Finish migrating, until there are no JS files in our App source code

We can use deprank to determine the number of dependencies and dependents on a file-by-file basis.

[roryabraham]

Posted a second predesign here yesterday

[roryabraham]

Started working on a doc here: https://docs.google.com/document/d/1LNcgYeYyovQ88gNjLIgBqBReLtbLAbxeAbna9QKxeAQ/edit#

[neil-marcellini]

I'm going to help Rory with this by asking external agencies to collaborate on a pre-design / writing the doc, and then making sure the process is going well.

[neil-marcellini]

I didn't get to this today.

[neil-marcellini]

I posted to the Callstack and Software Mansion Slack channels asking for proposals on how they would plan and implement this project.

https://expensify.slack.com/archives/C03UK30EA1Z/p1684525440666989
https://expensify.slack.com/archives/C04878MDF34/p1684525456740769

[melvin-bot]

@neil-marcellini, @roryabraham Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[neil-marcellini]

I got a reply from Callstack and SWM wants a day or two to post something.

[neil-marcellini]

I created #typescript-new-expensify and invited CK and SWM. @hayata-suenaga is helping out now too!

I still need to read through the proposal that software mansion already wrote, and then I’ll let them know that we’re waiting for their joint proposal and ask on an ETA for that. Maybe Hayata could tackle that?

[hayata-suenaga]

@neil-marcellini thank you for adding me to this issue

I thought you asked for proposals separately from Software Mansion and CallStack. Do they know they have to produce a joint proposal?

[hayata-suenaga]

okay CallStack is willing to collaborate

2. Proposal: We will review the proposal published in March by Fabio and other proposals and comments from contributors to gather the most relevant information and update the proposal and plan in cooperation with SWM

from Tomek

[hayata-suenaga]

and Software Mansion is saying they're already in touch with CallStack

This is our proposal on how to tackle migration, and we are already in touch with Callstack to compile a common one

From Michal

[roryabraham]

I went ahead and emailed out the project wrap-up. The last things couple items are still lingering, but 99.9% of this project is done

[roryabraham]

One of the last two wrap-up issues was rejected by maintainer so we closed the issue as not planned. That leaves only #43055

[fabioh8010]

Another Post-migration weekly update on the Slack channel!

[neil-marcellini]

We're incredibly close to finishing this for good, not overdue!

[neil-marcellini]

Head's up, I'll be working 50% for the rest of this week and next 🌲

[roryabraham]

@ShridharGoel reports that the last PR to close this out will be ready for review in the next couple days

[fabioh8010]

Another Post-migration weekly update on the Slack channel!

[roryabraham]

Just merged Expensify/eslint-config-expensify#114 earlier today

[roryabraham]

Apparently some delays for personal reasons on the part of contributor for the last PR.

hopefully we can close this out by EOW 🤞🏼

[neil-marcellini]

Still waiting on that last one.

[fabioh8010]

Another Post-migration weekly update on the Slack channel!

[neil-marcellini]

Here's the latest update.

@roryabraham it seems like it's going to take quite a while before withOnyx is deprecated. Can we sent the project wrap up sooner?

[roryabraham]

I already sent the project-wrap up for this project. The only thing left in this project is #43055

[roryabraham]

I can track the official close of the project in https://github.com/Expensify/Expensify/issues/413336. We can close this issue. Thanks so much everyone for all your hard work and consistent communication. This was a huge project, and it's already hard to remember what it was like before we used TypeScript.

Huge thanks in particular to @fabioh8010 and @blazejkustra, who were the real champions of this project.