More flexible variable selection

[aseyboldt]

I often would want to select subsets of variables with both positive and negative conditions. Things like: "give me a traceplot of all variables that contain in their name sigma but not foo". At first it looks like arviz already supports this, you can just pass var_names=["sigma", "~foo"], filter_items="like". This doesn't do what I would have expected though, if negative selections are present it simply ignores the positive ones.

This however raises the question how the combination of negative and positive filters should be interpreted. I can think of several options of how to do this:

1. This is what I just assumed it would do when I first entered multiple filters: All filters are sorted into negative and positive ones. It then selects all variables that fulfill at least one of the positive filters but none of the negative ones. If there are no positive filters it returns all variables that are not excluded by at least of one the negative ones.
2. We stick with interpreting the filters as filters that are ored together. So ["sigma", "~foo"] returns all variables that either contain sigma or do not contain foo. I think this would be the most consistent, because it treats the negative selectors the same as the positive ones, but I don't think that is what users would usually want to achieve?
3. Extend the logic of the selector a bit, so that it is possible to specify combinations of and and or selectors. So we could do the same thing as in 2., but additionally allow lists as elements in var_names, and interpret those lists as and conditions. So var_names=[["foo", "~bar"], "sigma"] would show all variables that either contain sigma, or that contain foo but not bar. I think I like the flexibility of the last one, but I'm not sure if this is maybe getting to complicated...

(cc @ColCarroll @OriolAbril @cluhmann)

[OriolAbril]

I think both 1 and 2 are good options. 1 should probably be relatively easy to implement with: split var_names between include and exclude filters, subset using include filters, then subset its result using exclude filters. While also being conceptually simple and intuitive for users.

I personally prefer 1, as 2 is in 99% of the cases equivalent to "ignore the include filter".

Option 3 sounds a bit too complicated, and I think we should ask/expect users who want these kinds of "advanced" behaviour to use regex option instead of like.

Extra note: what I think happens currently is "ignore the include filter" when both are present. It is not option 2 as described here. We could also stick to this, but then we should probably warn when both include and exclude filters are present.

Examples

lst = ["foo", "bar", "foobar", "foo_x", "bar_x", "x"]
az.utils._subset_list(["foo", "~bar"], lst, filter_items="like")
['foo', 'foo_x', 'x']  # current behaviour
['foo', 'foo_x']  # option 1
['foo', 'foobar', 'foo_x', 'x']  # option 2

[cluhmann]

Extra note: what I think happens currently is "ignore the include filter" when both are present. It is not option 2 as described here. We could also stick to this, but then we should probably warn when both include and exclude filters are present.

This seems to be true and is definitely unexpected.

lst = ["abc", "foo", "bar", "foobar", "foo_x", "bar_x", "x"]
az.utils._subset_list(["foo", "~bar"], lst, filter_items="like")
# ['abc', 'foo', 'foo_x', 'x']

[ColCarroll]

I'll add a weird option 4 that I think we should lean into using fnmatch for the "like" case, since it saves users from having to learn a new dialect. That is, you'd write something like

lst = ["abc", "foo", "bar", "foobar", "foo_x", "bar_x", "x"]
az.utils._subset_list(["foo*", "~*bar"], lst, filter_items="like")

to get "everything starting with foo, but nothing ending with bar", so just [foo, foo_x].

[OriolAbril]

I see this as an alternative implementation of option 1. It modifies a bit the like mode for fnmatch, but it is an intersection of conditions, that is, elements in the returned list fulfill all provided conditions (whatever those are)