Weird recommendation behavior

[hoguej]

When I dislike a course, it recommends it to similar users.

class user < ActiveRecord::Base
  has_many :courses
  recommends :course
end

user1.like(Course.find(2622))
user1.like(Course.find(2431))
user1.like(Course.find(1466))
user1.dislike(Course.find(315))

user2.like(Course.find(2622))
user2.like(Course.find(2431)
user2.recommended_courses.map{ |c| c.id }
 => [315, 1466]

user2.liked_course_ids
=> [2431, 2622]

user1.liked_course_ids
 => [1466, 2431, 2622]

user1.disliked_course_ids
 => [315]

[davidcelis]

Ah, yeah. The reason for this is that all recommendation values (or the top N values if you've used config.nearest_neighbors) get stored in Redis. So if you've only been compared with a few other users, it's possible that your top recommendations would include a recommendation with a score of less than 0 (which happens when similar users dislike that item). I wonder if it would be best to exclude any item with a negative score from returned recommendations?

[hoguej]

Should at least be sorted by highest score, right? 315 should have a negative score, and 1466 is positive. You'd expect 1466 to be first?

[hoguej]

But would also make sense to remove negative scores from recommended_(). maybe expose that through a new method not_recommended_ or something.

[davidcelis]

It's definitely sorted by highest score. Are you sure that 315 has a negative score, or a lower score than 1466? I'd inspect the Redis ZSET directly. Are there other ratings at work here?

[hoguej]

recommendable:users:134255:recommended_courses

value score
315 -1
1466 1

[hoguej]

I'm not sure what you mean by ratings, exactly.

Checking redis, it seems like the data is consistent with what I'd expect from the steps I followed in the first post.

[hoguej]

So, my course class has a rated? method, unrelated to recommendable. Wonder if that's causing a problem.

[hoguej]

Rename that method and reran everything. Same result. I guess next step is to create a minimal example app and try to reproduce the problem there.

[davidcelis]

Sorry, typoed that. I meant to ask if there are any other ratings (likes/dislikes) at play here.

[davidcelis]

Anyway, please do try to reproduce it in a codebase I can clone down and take a look at. Thanks!

[hoguej]

I've created a simple sample app, but noticed something that is probably the culprit.

Notice, that we look for (5,3,4) but we get back [3,4,5]. This is rails 3. Not sure if rails 4 handles this more appropriately.

1.9.3-p484 :001 > user2 = User.find(2); user2.recommended_courses.map{ |c| c.id }
User Load (2.9ms) SELECT "users".* FROM "users" WHERE "users"."id" = ? LIMIT 1 [["id", 2]]
Course Load (0.1ms) SELECT "courses".* FROM "courses" WHERE "courses"."id" IN (5, 3, 4)
=> [3, 4, 5]
1.9.3-p484 :002 >

[hoguej]

I do something like this to deal with this same situation from solr. Not the most efficient, but database agnostic. If I could access the raw scores, I could do something like that myself.

courses.sort!{ |x,y| course_ids.index(x.id.to_s) <=> course_ids.index(y.id.to_s) }

[davidcelis]

Shoot, you're right: https://github.com/davidcelis/recommendable/blob/master/lib/recommendable/rater/recommender.rb#L21-L27

I thought that I was correctly sorting the records before returning them to the user, but that's not the case. So while that list does contain the best recommendations, they're not sorted within that list. I probably did this originally to keep that query chainable, but I'm willing to wager that it's more important for that list to be in the right order.

[hoguej]

So while that list does contain the best recommendations
Except that 4 should not show up at all since it was disliked. At most, it should show up last. Because the negative numbers are in there, and it's out of order, it's giving the wrong answer.

Also, seems weird it would even be in the return result, since there's no way to know where the positive stops and the negative starts. You could argue, with a fully populated recommendation engine, it would probably give the correct result. Might be true where you are getting 10 results back, but false if getting back 100's of recommendations.

I can take a crack at a patch, but I'm new to this lib and redis.

[davidcelis]

Good point. I think that at the very least, items with negative scores should never be recommended. I can definitely fix that much, but I'll need to think a bit more on sacrificing the chainability of querying against those recommendations for having them ordered correctly. There is a tradeoff there, but I'm leaning towards ordering them correctly.

[hoguej]

Wonder if we can dip into the ActiveRecord class to lazily store the order.

On Fri, Apr 25, 2014 at 3:26 PM, David Celis [email protected]:

Good point. I think that at the very least, items with negative scores
should never be recommended. I can definitely fix that much, but I'll need
to think a bit more on sacrificing the chainability of querying against
those recommendations for having them ordered correctly. There is a
tradeoff there, but I'm leaning towards ordering them correctly.

Reply to this email directly or view it on GitHubhttps://github.com//issues/91#issuecomment-41430409
.

[davidcelis]

@hoguej I decided to abandon leaving the recommended_things methods chainable so that recommendations are sorted correctly. This has landed as of a0e77bd. I'll leave this open if you want to confirm that it works.

[hoguej]

I'll take a look this week. Thanks for the update. When I get to it, I'll expose my test app.

[hoguej]

Sorry, got ill last week. Still on my agenda.