Add cheap way of deciding whether to re-run predictions for long-running markets

[evangriffiths]

Follow-on from #485

Current behaviour, defined in DeployableTraderAgent.verify_market, is for agent to skip the market if it has bet on it in the last 24 hours.

If agents place bets on long running markets, this will be too frequent. But it could involve some combination of:

• using a news API, and seeing if something relevant has happened since the last bet (must be v. cheap to be worth using instead of a full prediction)
• time since last bet
• time til market close

[gabrielfior]

using a news API, and seeing if something relevant has happened since the last bet (must be v. cheap to be worth using instead of a full prediction)

This is the hardest but also the best option for me. One option that comes to mind is e.g. monitoring "twitter trends" from the last week and seeing if it has high correlation with any market on Omen and order markets by that. This is kind of an alternative to the newsfeed API.

[kongzii]

We could use our market-insights API for this. It uses Tavily to get news. It's cached for 3 days and we expect that it will be used a lot, so the "price per agent" is cheap.

[evangriffiths]

I've just had a look at this. It's a nice idea, but the the output is not really suitable for the use case. e.g. for the market with title "Will there be more recorded forest fires in the US and Canada in 2024 than 2023?"

the summary you get is

'Forest fires have been increasingly severe, with 2023 marking the worst year on record globally. In Canada, the 2023 wildfire season was unprecedented, burning six times more forest than the average of the previous 20 years. Experts attribute this to climate change, though they note that years like 2023 might be rare. As 2024 unfolds, early signs suggest another intense fire season. Consider these factors and conduct thorough research before making any predictions in the market.'

What we want is different in 2 ways:

• We only care about information/news that is newer than a date specified by the caller of the api
• We don't really care about the content of the news - just the fact that there is/isn't new news.

But I forgot that Tavily also have a news API feature (see docs), where you can do e.g. search(topic="news", days=14). I'll give that a go!

Also good point on the caching. The queries will be identical for all agents, so it makes sense to use it. Even just using existing TavilyStorage should save a lot.