Add ComplexFilterBackend

README.rst

@@ -44,6 +44,7 @@ Features
 * Easy filtering across relationships.
 * Support for method filtering across relationships.
 * Automatic filter negation with a simple ``param!=value`` syntax.
+* Backend for complex operations on multiple filtered querysets. eg, ``q1 | q2``.
 
 
 Requirements
@@ -63,8 +64,8 @@ Installation
     $ pip install djangorestframework-filters
 
 
-Usage
------
+``FilterSet`` usage
+-------------------
 
 Upgrading from ``django-filter`` to ``django-rest-framework-filters`` is straightforward:
 
@@ -407,6 +408,149 @@ The recommended solutions are to either:
     ?publish_date__range=2016-01-01,2016-02-01
 
 
+Complex Operations
+------------------
+
+The ``ComplexFilterBackend`` defines a custom querystring syntax and encoding process that enables the expression of
+`complex queries`_. This syntax extends standard querystrings with the ability to define multiple sets of parameters
+and operators for how the queries should be combined.
+
+.. _`complex queries`: https://docs.djangoproject.com/en/2.0/topics/db/queries/#complex-lookups-with-q-objects
+
+----
+
+**!** Note that this feature is experimental. Bugs may be encountered, and the backend is subject to change.
+
+----
+
+To understand the backend more fully, consider a query to find all articles that contain titles starting with either
+"Who" or "What". The underlying query could be represented with the following:
+
+.. code-block:: python
+
+    q1 = Article.objects.filter(title__startswith='Who')
+    q2 = Article.objects.filter(title__startswith='What')
+    return q1 | q2
+
+Now consider the query, but modified with upper and lower date bounds:
+
+.. code-block:: python
+
+    q1 = Article.objects.filter(title__startswith='Who').filter(publish_date__lte='2005-01-01')
+    q2 = Article.objects.filter(title__startswith='What').filter(publish_date__gte='2010-01-01')
+    return q1 | q2
+
+Using just a ``FilterSet``, it is certainly feasible to represent the former query by writing a custom filter class.
+However, it is less feasible with the latter query, where multiple sets of varying data types and lookups need to be
+validated. In contrast, the ``ComplexFilterBackend`` can create this complex query through the arbitrary combination
+of a simple filter. To support the above, the querystring needs to be created with minimal changes. Unencoded example:
+
+.. code-block::
+
+    (title__startswith=Who&publish_date__lte=2005-01-01) | (title__startswith=What&publish_date__gte=2010-01-01)
+
+By default, the backend combines queries with both ``&`` (AND) and ``|`` (OR), and supports unary negation ``~``. E.g.,
+
+.. code-block::
+
+    (param1=value1) & (param2=value2) | ~(param3=value3)
+
+The backend supports both standard and complex queries. To perform complex queries, the query must be encoded and set
+as the value of the ``complex_filter_param`` (defaults to ``filters``). To perform standard queries, use the backend
+in the same manner as the ``DjangoFilterBackend``.
+
+
+Configuring ``ComplexFilterBackend``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Similar to other backends, ``ComplexFilterBackend`` must be added to a view's ``filter_backends`` atribute. Either add
+it to the ``DEFAULT_FILTER_BACKENDS`` setting, or set it as a backend on the view class.
+
+.. code-block:: python
+
+    REST_FRAMEWORK = {
+        'DEFAULT_FILTER_BACKENDS': (
+            'rest_framework_filters.backends.ComplexFilterBackend',
+        ),
+    }
+
+    # or
+
+    class MyViewSet(generics.ListAPIView):
+        filter_backends = (rest_framework_filters.backends.ComplexFilterBackend, )
+        ...
+
+You may customize how queries are combined by subclassing ``ComplexFilterBackend`` and overriding the ``operators``
+attribute. ``operators`` is a map of operator symbols to functions that combine two querysets. For example, the map
+can be overridden to use the ``QuerySet.intersection()`` and ``QuerySet.union()`` instead of ``&`` and ``|``.
+
+.. code-block:: python
+
+    class CustomizedBackend(ComplexFilterBackend):
+        operators = {
+            '&': QuerySet.intersection,
+            '|': QuerySet.union,
+            '-': QuerySet.difference,
+        }
+
+Unary ``negation`` relies on ORM internals and may be buggy in certain circumstances. If there are issues with this
+feature, it can be disabled by setting the ``negation`` attribute to ``False`` on the backend class. If you do
+experience bugs, please open an issue on the `bug tracker`_.
+
+.. _`bug tracker`: https://github.com/philipn/django-rest-framework-filters/issues/
+
+
+Complex querystring encoding
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Below is the procedure for encoding a complex query:
+
+* Convert the query paramaters into individual querystrings.
+* URL-encode the individual querystrings.
+* Wrap the encoded strings in parentheses, and join with operators.
+* URL-encode the entire querystring.
+* Set as the value to the complex filter param (default: ``filters``).
+
+Using the first example, these steps can be visualized as so:
+
+* ``title__startswith=Who``, ``title__startswith=What``
+* ``title__startswith%3DWho``, ``title__startswith%3DWhat``
+* ``(title__startswith%3DWho) | (title__startswith%3DWhat)``
+* ``%28title__startswith%253DWho%29%20%7C%20%28title__startswith%253DWhat%29``
+* ``filters=%28title__startswith%253DWho%29%20%7C%20%28title__startswith%253DWhat%29``
+
+
+Error handling
+~~~~~~~~~~~~~~
+
+``ComplexFilterBackend`` will raise any decoding errors under the complex filtering parameter name. For example,
+
+.. code-block:: json
+
+    {
+        "filters": [
+            "Invalid querystring operator. Matched: 'foo'."
+        ]
+    }
+
+When filtering the querysets, filterset validation errors will be collected and raised under the complex filtering
+parameter name, then under the filterset's decoded querystring. For a complex query like ``(a=1&b=2) | (c=3&d=4)``,
+errors would be raised like so:
+
+.. code-block:: json
+
+    {
+        "filters": {
+            "a=1&b=2": {
+                "a": ["..."]
+            },
+            "c=3&d=4": {
+                "c": ["..."]
+            }
+        }
+    {
+
+
 Migrating to 1.0
 ----------------
 

rest_framework_filters/backends.py

@@ -1,7 +1,10 @@
 from contextlib import contextmanager
 
+from django.http import QueryDict
 from django_filters.rest_framework import backends
+from rest_framework.exceptions import ValidationError
 
+from .complex_ops import combine_complex_queryset, decode_complex_ops
 from .filterset import FilterSet
 
 
@@ -36,3 +39,48 @@ def to_html(self, request, queryset, view):
         # us to avoid maintenance issues with code duplication.
         with self.patch_for_rendering(request):
             return super(DjangoFilterBackend, self).to_html(request, queryset, view)
+
+
+class ComplexFilterBackend(DjangoFilterBackend):
+    complex_filter_param = 'filters'
+    operators = None
+    negation = True
+
+    def filter_queryset(self, request, queryset, view):
+        if self.complex_filter_param not in request.query_params:
+            return super(ComplexFilterBackend, self).filter_queryset(request, queryset, view)
+
+        # Decode the set of complex operations
+        encoded_querystring = request.query_params[self.complex_filter_param]
+        try:
+            complex_ops = decode_complex_ops(encoded_querystring, self.operators, self.negation)
+        except ValidationError as exc:
+            raise ValidationError({self.complex_filter_param: exc.detail})
+
+        # Collect the individual filtered querysets
+        querystrings = [op.querystring for op in complex_ops]
+        try:
+            querysets = self.get_filtered_querysets(querystrings, request, queryset, view)
+        except ValidationError as exc:
+            raise ValidationError({self.complex_filter_param: exc.detail})
+
+        return combine_complex_queryset(querysets, complex_ops)
+
+    def get_filtered_querysets(self, querystrings, request, queryset, view):
+        parent = super(ComplexFilterBackend, self)
+        original_GET = request._request.GET
+
+        querysets, errors = [], {}
+        for qs in querystrings:
+            request._request.GET = QueryDict(qs)
+            try:
+                result = parent.filter_queryset(request, queryset, view)
+                querysets.append(result)
+            except ValidationError as exc:
+                errors[qs] = exc.detail
+            finally:
+                request._request.GET = original_GET
+
+        if errors:
+            raise ValidationError(errors)
+        return querysets

rest_framework_filters/complex_ops.py

@@ -0,0 +1,94 @@
+import re
+from collections import namedtuple
+from urllib.parse import unquote
+
+from django.db.models import QuerySet
+from django.utils.translation import ugettext as _
+
+from rest_framework.serializers import ValidationError
+
+from rest_framework_filters.utils import lookahead
+
+
+# originally based on: https://regex101.com/r/5rPycz/1
+# current iteration: https://regex101.com/r/5rPycz/3
+# special thanks to @JohnDoe2 on the #regex IRC channel!
+# matches groups of "<negate>(<encoded querystring>)<set op>"
+COMPLEX_OP_RE = re.compile(r'()\(([^)]+)\)([^(]*?(?=\())?')
+COMPLEX_OP_NEG_RE = re.compile(r'(~?)\(([^)]+)\)([^(]*?(?=~\(|\())?')
+COMPLEX_OPERATORS = {
+    '&': QuerySet.__and__,
+    '|': QuerySet.__or__,
+}
+
+ComplexOp = namedtuple('ComplexOp', ['querystring', 'negate', 'op'])
+
+
+def decode_complex_ops(encoded_querystring, operators=None, negation=True):
+    """
+    Returns a list of (querystring, negate, op) tuples that represent complex operations.
+
+    This function will raise a `ValidationError`s if:
+    - the individual querystrings are not wrapped in parentheses
+    - the set operators do not match the provided `operators`
+    - there is trailing content after the ending querysting
+
+    Ex::
+
+        # unencoded query: (a=1) & (b=2) | ~(c=3)
+        >>> s = '%28a%253D1%29%20%26%20%28b%253D2%29%20%7C%20%7E%28c%253D3%29'
+        >>> decode_querystring_ops(s)
+        [
+            ('a=1', False, QuerySet.__and__),
+            ('b=2', False, QuerySet.__or__),
+            ('c=3', True, None),
+        ]
+    """
+    complex_op_re = COMPLEX_OP_NEG_RE if negation else COMPLEX_OP_RE
+    if operators is None:
+        operators = COMPLEX_OPERATORS
+
+    # decode into: (a%3D1) & (b%3D2) | ~(c%3D3)
+    decoded_querystring = unquote(encoded_querystring)
+    matches = [m for m in complex_op_re.finditer(decoded_querystring)]
+
+    if not matches:
+        msg = _("Unable to parse querystring. Decoded: '%(decoded)s'.")
+        raise ValidationError(msg % {'decoded': decoded_querystring})
+
+    results, errors = [], []
+    for match, has_next in lookahead(matches):
+        negate, querystring, op = match.groups()
+
+        negate = negate == '~'
+        querystring = unquote(querystring)
+        op_func = operators.get(op.strip()) if op else None
+        if op_func is None and has_next:
+            msg = _("Invalid querystring operator. Matched: '%(op)s'.")
+            errors.append(msg % {'op': op})
+
+        results.append(ComplexOp(querystring, negate, op_func))
+
+    trailing_chars = decoded_querystring[matches[-1].end():]
+    if trailing_chars:
+        msg = _("Ending querystring must not have trailing characters. Matched: '%(chars)s'.")
+        errors.append(msg % {'chars': trailing_chars})
+
+    if errors:
+        raise ValidationError(errors)
+
+    return results
+
+
+def combine_complex_queryset(querysets, complex_ops, negation=True):
+    # Negate querysets
+    for queryset, op in zip(querysets, complex_ops):
+        if negation and op.negate:
+            queryset.query.where.negate()
+
+    # Combine querysets
+    combined = querysets[0]
+    for queryset, op in zip(querysets[1:], complex_ops[:-1]):
+        combined = op.op(combined, queryset)
+
+    return combined

rest_framework_filters/utils.py

@@ -62,3 +62,13 @@ def lookups_for_transform(transform):
             lookups.append(expr)
 
     return lookups
+
+
+def lookahead(iterable):
+    it = iter(iterable)
+    current = next(it)
+
+    for value in it:
+        yield current, True
+        current = value
+    yield current, False