Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Robyn Budget Allocator max_budget scenario should scale x_hist_carryover for accurate response simulation #1014

Open
NumesSanguis opened this issue Jul 4, 2024 · 2 comments

Comments

@NumesSanguis
Copy link

Project Robyn

Describe issue

Simulated Adstock effect is wrongly calculated in max_budget scenario. The Adstock effect is not changed to reflect this budget change when simulated with a different budget. If I understand the code correctly.

The function objective.channel in eval_f, which calls fx_objective.chanel calculates Adstock with: xAdstocked <- x + mean(x_hist_carryover).
is used to calculate the response:

optmResponseUnit <- -eval_f(optmSpendUnit)[["objective.channel"]]

https://github.com/facebookexperimental/Robyn/blob/main/R/R/allocator.R#L516

Due to using the mean, the number of weeks simulated does not matter. However, if the budget gets very small, the mean(x_hist_carryover)part is much bigger thanx`, meaning the whole simulated response is dominated by historical carryover at a different budget level.

Possible solution

Get the ratio: "mean historical spend" / "weekly budget" and multiply with the Adstock effect to scale the effect.
However, I'm not sure if this is accurate. It might be that the Adstock effect is different at different spend levels though.

Environment & Robyn version

  • Robyn version: 3.11.0
@NumesSanguis
Copy link
Author

Sugested solution

Instead of using average historical Adstock effect:

xAdstocked <- x + mean(x_hist_carryover)

Use the carryover response ratio as shown in the "Immediate vs. Carryover Response Percentage". If e.g. carry over is 30% for a channel, then for that channel:

xAdstocked <- x * (carry_over_rate + 1)  # x * 1.3

The immediate response triggers in that week (assuming weekly data), e.g. 70%. The remaining 30% over the following x weeks. If we assume for the simulator that previous weeks (before week 1 of the simulation) has been on the same spend level, then the sum of all previous week is equal to the 30%, simplifying it to just x * 1.3.

This way changing the total budget in the simulator will be using the correct Adstock strength rather than historical (which can be at a different spend level)

For this, you only need to capture an array with a % value of carryover for each media channel and pass this along when calling the funcs like fx_objective.

@AdimDrewnik
Copy link

This whole budget allocation tool is so problematic. I have also noticed this problem. When selecting short time range carryover effect dominates, but when selecting longer time range carryover effect vanishes.
But the main problem with allocator is totally broken total response calculation. For example, from model I am getting that marketing spend is responsible for 1M of revenue, but allocator shows a number not even close to that, more than half of that which makes no sense.
Also, what about channels with long delays from Weibull PDF. If a channel has few weeks of delay it is punished by allocator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants