Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducing Rule-based baseline and not matching paper results #30

Open
lihkinVerma opened this issue Apr 16, 2024 · 1 comment
Open

Comments

@lihkinVerma
Copy link

Hey Authors,

I tried replicating the results for rule-based baseline.
In paper, the mentioned metrics for the same are: Score / SR = 45.8 / 19%
while replicating it, I am getting folloiwng values for metrics: Score / SR = 26.27 / 3.59%

The values of all reward variables are also not matching the paper's baseline. I obtained
r_type: 0.5826
r_attr: 0.4108
r_option: 0.0
r_price: 0.0632

Can you check for the anomaly?

@ai-nikolai
Copy link
Contributor

ai-nikolai commented May 25, 2024

Hey @lihkinVerma ,

I am also interested in replicating scores.

How do you run the above? Specifically:

  1. How do you run Webshop (which server / env do you use?) (e.g. did you use ./setup.sh -d small and then ./run_dev.sh)
  2. How do you run the baseline?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants