python[patch]: evaluators can return primitives

[baskaryan]

from langsmith import evaluate

def foo(run, example):
    return 0

def bar(run, example):
    return "long"

# removed, needs to be list of dict
# def baz(run, example):
#     return [0, 0.2, "how are ya"]

def app(inputs):
    return inputs

evaluate(app, data="Sample Dataset 3", evaluators=[foo, bar, baz])

[baskaryan]

how ^ example gets logged

[hinthornw]

The list return type seems unexpected to me - I wouldn't expect it to make separate keys for each value

[baskaryan]

The list return type seems unexpected to me - I wouldn't expect it to make separate keys for each value

are we able to support list values as feedback?

[hinthornw]

The list return type seems unexpected to me - I wouldn't expect it to make separate keys for each value

are we able to support list values as feedback?

Only dict or string it seems.

Maybe doing the _ix suffix is preferable. I'm not sure honestly. Probalby less surprising than just logging duplicates to the same key.

It just messes with the experiment averages (we'd be averaging over index 1 and over index 2 )

[baskaryan]

The list return type seems unexpected to me - I wouldn't expect it to make separate keys for each value

are we able to support list values as feedback?

Only dict or string it seems.

Maybe doing the _ix suffix is preferable. I'm not sure honestly. Probalby less surprising than just logging duplicates to the same key.

It just messes with the experiment averages (we'd be averaging over index 1 and over index 2 )

feel less strongly about list behavior either way, think main use case is supporting int/float/bool/str

[hinthornw]

Maybe let's land with the numeric and string value support but hold off on list behavior?

Maybe add support for list[evaluationresultlike] to make the "results" key not necessary

[jakerachleff]

So if I have four categories that are numbers for some reason (I'm classifying college class levels and they're 100 level, 200 level, 300 level, etc), then I should do str(value) to explicitly use categorical scores?

[baskaryan]

or return as {"value": 200} (something for us to clearly document)

[jakerachleff]

maybe good to confirm that a list of ints, for example, doesn't work?

[hinthornw]

is i needed anymore?

[agola11]

I think it's worth holding off on the categorical dict case for now and just interpreting strings as categories. We need to think about the UX for allowing users to specify configuration for the feedback users are sending with their evaluators