[Tracking] Add email tracking for email deliverability and open/click rates.

[danielrvidal]

High level design doc

Proposal: Add email tracking for email deliverability and open/click rates.

Problem: Recipients are not reliably clicking on IOUs from senders, and thus never coming into the product. We know a non zero percentage of users ignore the emails we send, but we don’t actually know how bad the problem is. We rely on email notifications to pull viral leads into our product. However, we’re unable to determine if our emails are making this problem bigger by not being engaging and compelling for our users, or even worse not getting delivered. This lack of insight prevents us from optimizing our email communications, resulting in decreased user engagement, decreased understanding of our value, decreased conversion rates, and ultimately hampers our ability to drive sustainable growth.

Solution: Let’s start tracking email deliverability and open/click through rates with all relevant notifications so we know how big the problem is and can a/b test email communications to increase conversion of invited users to active users.

1. Let’s start by tracking the emails we send for one invite type: IOUs for new users.
2. We already use custom subject line to trigger an immediate unread message notification to the IOU recipient (added in this PR). Let’s add a tracker in this same place that identifies the type of email.

• It might look something like $this->parameters['trackingKey'] = 'IOUnewUser'

3. We’ll send this tracker as a tag to Mailgun

• Here is an example of where we’ll put it.

4. Mailgun is already set up to measure email deliverability and open/click-through rates for all emails. All we need to do now is distinguish an IOU notification vs. any other unread message notification and the tracker accomplishes that

• Note: Mailgun only covers 50% of our emails. But so long as we track the control and test versions and compare them within that 50% pool, and then wait for statistical significance, we should learn if/which notifications are effective.

5. We’ll run an inbox placement test which sends the IOU notification to various mail providers and lets us know how often the specific notification is getting caught by spam.
6. We will get a baseline on the current notification open/click rate and validate this design before we introduce any A/B testing

• A/b testing the open rate of IOUs for new users using different subject lines will likely be our first test which is why we chose that, but the a/b test will come later.

Tasks

• Post Proposal (full Problem/Solution statement) in #expensify-open-source
• Wait at least one full business day, and until the post has a majority (2/3) of positive reactions (👍)
• Paste Proposal in the space above with a link to the Slack thread
• Email [email protected] and paste in the Proposal
• Fill out the High-level overview of the problem, Timeline, and Terminology sections of the Design Doc
• Email [email protected] (continue the same email chain as before) with the link to your Design Doc
• Host a pre-design meeting (example) in #expensify-open-source to discuss any necessary details in public before filling out the High-level of proposed solution section.
• Fill out the High-level of proposed solution section
• Email [email protected] again with links to the doc and pre-design conversation in Slack
• Add the DesignDocReview label to get the High-level of proposed solution section reviewed
• Respond to any questions or concerns and bring up blockers in Slack to get a consensus if necessary
• Confirm that the doc has the minimum necessary number of reviews before proceeding
• Host another pre-design meeting in #expensify-open-source to ask for engineering feedback on the technical solution.
• Fill out the Detailed implementation of the solution and related sections.
• Re-add the DesignDocReview label to this issue
• Respond to any questions or concerns and bring up blockers in Slack to get consensus if necessary
• Confirm that the doc has the minimum necessary number of reviews before proceeding
• Email [email protected] one last time to let them know the Design Doc is moving into the implementation phase
• Implement the changes
• Add regression tests so that QA can test your feature with every deploy (instructions)
• Send out a follow up email to [email protected] once everything has been implemented and do a Project Wrap-Up retrospective that provides:
  • Summary of what we accomplished with this project
  • What went well?
  • What could we have done better?
  • What did we learn?

[danielrvidal]

Here are the next steps as of now

• @justinpersaud - Give access to Mailgun: https://github.com/Expensify/Expensify/issues/426298
• @justinpersaud - Update IOUs to have the tag: https://github.com/Expensify/Expensify/issues/426303
• @justinpersaud - Update Mailgun to store the tags for IOUs: https://github.com/Expensify/Expensify/issues/426306

From there we'll plan the next steps once those are done.

[justinpersaud]

Workong on it this week but I'm just trying to wrap up some SOX audit issues that keep coming back from EY + finishing up blocking spam with ChatGPT for #fireroom-05-23-spam

[danielrvidal]

We should have the doc out today or tomorrow!

[justinpersaud]

I updated the design doc and filled in the Data Storage / API section

[danielrvidal]

High level design doc

The HL is ready for reviews. Please give it a read and ask questions in the doc or room. Assigning the review label now!

[melvin-bot]

👋 Hello Generalist Track Team - you have been assigned to review this High Level Design Doc. Check out this Stack Overflow for some tips on reviewing a design doc. Once you are done, simply press the Add "Reviewed Doc" comment button in the right hand side K2 panel or follow these instructions.

---

• @srikarparsi (Expensifier / Graduate) - High-level design doc review (Expensifier / Graduate) - Add email tracking for email deliverability and open/click rates. #49304
• @Christinadobrzyn (Expensifier / Graduate) - High-level design doc review (Expensifier / Graduate) - Add email tracking for email deliverability and open/click rates. #49305
• @zsgreenwald (Project Manager) - High-level design doc review (Project Manager) - Add email tracking for email deliverability and open/click rates. #49306
• @tylerkaraszewski (Project Manager) - High-level design doc review (Project Manager) - Add email tracking for email deliverability and open/click rates. #49307
• @twisterdotcom (Product Manager) - High-level design doc review (Product Manager) - Add email tracking for email deliverability and open/click rates. #49308
• @stephanieelliott (Product Manager) - High-level design doc review (Product Manager) - Add email tracking for email deliverability and open/click rates. #49309
• @davidcardoza (Generalist) - High-level design doc review (Generalist) - Add email tracking for email deliverability and open/click rates. #49310
• @garrettmknight (Generalist) - High-level design doc review (Generalist) - Add email tracking for email deliverability and open/click rates. #49311
• @MarcFillari (Accounting Technical Team) - High-level design doc review (Accounting Technical Team) - Add email tracking for email deliverability and open/click rates. #49312
• @CherylWalsh (Marketing Team) - High-level design doc review (Marketing Team) - Add email tracking for email deliverability and open/click rates. #49313

[techievivek]

I have read and reviewed this Design Doc!

[iwiznia]

I have read and reviewed this Design Doc!

[flodnv]

I have read and reviewed this Design Doc!

[melvin-bot]

👋 Hello Generalist Track Team - you have been assigned to review this Detailed Design Doc. Check out this Stack Overflow for some tips on reviewing a design doc. Once you are done, simply press the Add "Reviewed Doc" comment button in the right hand side K2 panel or follow these instructions.

---

• @zsgreenwald (Any Expensifiers, Non-Engineering Graduates or Project Managers) - Detailed design doc review (Any Expensifiers, Non-Engineering Graduates or Project Managers) - [Tracking] Add email tracking for email deliverability and open/click rates. #50405
• @CherylWalsh (Any Expensifiers, Non-Engineering Graduates or Project Managers) - Detailed design doc review (Any Expensifiers, Non-Engineering Graduates or Project Managers) - [Tracking] Add email tracking for email deliverability and open/click rates. #50406
• @garrettmknight (Non-engineering Product Managers or Generalists) - Detailed design doc review (Non-engineering Product Managers or Generalists) - [Tracking] Add email tracking for email deliverability and open/click rates. #50407
• @davidcardoza (Non-engineering Product Managers or Generalists) - Detailed design doc review (Non-engineering Product Managers or Generalists) - [Tracking] Add email tracking for email deliverability and open/click rates. #50408
• @tylerkaraszewski (Engineering Graduates) - Detailed design doc review (Engineering Graduates) - [Tracking] Add email tracking for email deliverability and open/click rates. #50409
• @aldo-expensify (Engineering Graduates) - Detailed design doc review (Engineering Graduates) - [Tracking] Add email tracking for email deliverability and open/click rates. #50410
• @chiragsalian (Engineering Graduates) - Detailed design doc review (Engineering Graduates) - [Tracking] Add email tracking for email deliverability and open/click rates. #50411
• @puneetlath (Engineering Product Managers or Generalists) - Detailed design doc review (Engineering Product Managers or Generalists) - [Tracking] Add email tracking for email deliverability and open/click rates. #50412

[justinpersaud]

Wrapping up some discussions in the detailed portion

[puneetlath]

I have read and reviewed this Design Doc!

[garrettmknight]

I have read and reviewed this Design Doc!

[cead22]

I have read and reviewed this Design Doc!

[CherylWalsh]

I have read and reviewed this Design Doc!

[tylerkaraszewski]

I have read and reviewed this Design Doc!

[chiragsalian]

I have read and reviewed this Design Doc!

[justinpersaud]

Implementation for this is in review

[JmillsExpensify]

I have read and reviewed this Design Doc!

[justinpersaud]

Working on some last bit of bugs with the IOU tracking