Merge branch 'regex-query-shortcuts'

docs/guide/querying.rst

@@ -68,6 +68,16 @@ Available operators are as follows:
 * ``size`` -- the size of the array is 
 * ``exists`` -- value for field exists
 
+The following operators are available as shortcuts to querying with regular
+expressions:
+
+* ``contains`` -- string field contains value
+* ``icontains`` -- string field contains value (case insensitive)
+* ``startswith`` -- string field starts with value
+* ``istartswith`` -- string field starts with value (case insensitive)
+* ``endswith`` -- string field ends with value
+* ``iendswith`` -- string field ends with value (case insensitive)
+
 Limiting and skipping results
 =============================
 Just as with traditional ORMs, you may limit the number of results returned, or
@@ -232,6 +242,109 @@ calling it with keyword arguments::
    natively supported by MongoDB -- they are compiled to Javascript and sent
    to the server for execution.
 
+Server-side javascript execution
+================================
+Javascript functions may be written and sent to the server for execution. The
+result of this is the return value of the Javascript function. This
+functionality is accessed through the
+:meth:`~mongoengine.queryset.QuerySet.exec_js` method on
+:meth:`~mongoengine.queryset.QuerySet` objects. Pass in a string containing a
+Javascript function as the first argument.
+
+The remaining positional arguments are names of fields that will be passed into
+you Javascript function as its arguments. This allows functions to be written
+that may be executed on any field in a collection (e.g. the
+:meth:`~mongoengine.queryset.QuerySet.sum` method, which accepts the name of
+the field to sum over as its argument). Note that field names passed in in this
+manner are automatically translated to the names used on the database (set
+using the :attr:`name` keyword argument to a field constructor).
+
+Keyword arguments to :meth:`~mongoengine.queryset.QuerySet.exec_js` are
+combined into an object called :attr:`options`, which is available in the
+Javascript function. This may be used for defining specific parameters for your
+function.
+
+Some variables are made available in the scope of the Javascript function:
+
+* ``collection`` -- the name of the collection that corresponds to the
+  :class:`~mongoengine.Document` class that is being used; this should be
+  used to get the :class:`Collection` object from :attr:`db` in Javascript
+  code
+* ``query`` -- the query that has been generated by the
+  :class:`~mongoengine.queryset.QuerySet` object; this may be passed into
+  the :meth:`find` method on a :class:`Collection` object in the Javascript
+  function
+* ``options`` -- an object containing the keyword arguments passed into
+  :meth:`~mongoengine.queryset.QuerySet.exec_js`
+
+The following example demonstrates the intended usage of
+:meth:`~mongoengine.queryset.QuerySet.exec_js` by defining a function that sums
+over a field on a document (this functionality is already available throught
+:meth:`~mongoengine.queryset.QuerySet.sum` but is shown here for sake of
+example)::
+
+    def sum_field(document, field_name, include_negatives=True):
+        code = """
+        function(sumField) {
+            var total = 0.0;
+            db[collection].find(query).forEach(function(doc) {
+                var val = doc[sumField];
+                if (val >= 0.0 || options.includeNegatives) {
+                    total += val;
+                }
+            });
+            return total;
+        }
+        """
+        options = {'includeNegatives': include_negatives}
+        return document.objects.exec_js(code, field_name, **options)
+
+As fields in MongoEngine may use different names in the database (set using the
+:attr:`name` keyword argument to a :class:`Field` constructor), a mechanism
+exists for replacing MongoEngine field names with the database field names in
+Javascript code. When accessing a field on a collection object, use
+square-bracket notation, and prefix the MongoEngine field name with a tilde.
+The field name that follows the tilde will be translated to the name used in
+the database. Note that when referring to fields on embedded documents,
+the name of the :class:`~mongoengine.EmbeddedDocumentField`, followed by a dot,
+should be used before the name of the field on the embedded document. The
+following example shows how the substitutions are made::
+
+    class Comment(EmbeddedDocument):
+        content = StringField(name='body')
+
+    class BlogPost(Document):
+        title = StringField(name='doctitle')
+        comments = ListField(EmbeddedDocumentField(Comment), name='cs')
+
+    # Returns a list of dictionaries. Each dictionary contains a value named
+    # "document", which corresponds to the "title" field on a BlogPost, and
+    # "comment", which corresponds to an individual comment. The substitutions
+    # made are shown in the comments.
+    BlogPost.objects.exec_js("""
+    function() {
+        var comments = [];
+        db[collection].find(query).forEach(function(doc) {
+            // doc[~comments] -> doc["cs"]
+            var docComments = doc[~comments];
+
+            for (var i = 0; i < docComments.length; i++) {
+                // doc[~comments][i] -> doc["cs"][i]
+                var comment = doc[~comments][i];
+
+                comments.push({
+                    // doc[~title] -> doc["doctitle"]
+                    'document': doc[~title],
+
+                    // comment[~comments.content] -> comment["body"]
+                    'comment': comment[~comments.content]
+                });
+            }
+        });
+        return comments;
+    }
+    """)
+
 .. _guide-atomic-updates:
 
 Atomic updates

mongoengine/fields.py

@@ -39,8 +39,25 @@ class StringField(BaseField):
     def lookup_member(self, member_name):
         return None
 
+    def prepare_query_value(self, op, value):
+        if not isinstance(op, basestring):
+            return value
 
-class URLField(BaseField):
+        if op.lstrip('i') in ('startswith', 'endswith', 'contains'):
+            flags = 0
+            if op.startswith('i'):
+                flags = re.IGNORECASE
+                op = op.lstrip('i')
+
+            regex = r'%s'
+            if op == 'startswith':
+                regex = r'^%s'
+            elif op == 'endswith':
+                regex = r'%s$'
+            value = re.compile(regex % value, flags)
+        return value
+
+class URLField(StringField):
     """A field that validates input as a URL.
     """
 

mongoengine/queryset.py

@@ -1,6 +1,7 @@
 from connection import _get_db
 
 import pymongo
+import re
 import copy
 
 
@@ -27,6 +28,8 @@ class OperationError(Exception):
     pass
 
 
+RE_TYPE = type(re.compile(''))
+
 class Q(object):
     
     OR = '||'
@@ -46,6 +49,8 @@ class Q(object):
                 'return this.%(field)s.indexOf(a) != -1 })'),
         'size': 'this.%(field)s.length == %(value)s',
         'exists': 'this.%(field)s != null',
+        'regex_eq': '%(value)s.test(this.%(field)s)',
+        'regex_ne': '!%(value)s.test(this.%(field)s)',
     }
     
     def __init__(self, **query):
@@ -90,24 +95,41 @@ class Q(object):
                 for j, (op, value) in enumerate(value.items()):
                     # Create a custom variable name for this operator
                     op_value_name = '%so%s' % (value_name, j)
+                    # Construct the JS that uses this op
+                    value, operation_js = self._build_op_js(op, key, value,
+                                                            op_value_name)
                     # Update the js scope with the value for this op
                     js_scope[op_value_name] = value
-                    # Construct the JS that uses this op
-                    operation_js = Q.OPERATORS[op.strip('$')] % {
-                        'field': key, 
-                        'value': op_value_name
-                    }
                     js.append(operation_js)
             else:
-                js_scope[value_name] = value
                 # Construct the JS for this field
-                field_js = Q.OPERATORS[op.strip('$')] % {
+                value, field_js = self._build_op_js(op, key, value, value_name)
-                    'field': key, 
+                js_scope[value_name] = value
-                    'value': value_name
-                }
                 js.append(field_js)
         return ' && '.join(js)
 
+    def _build_op_js(self, op, key, value, value_name):
+        """Substitute the values in to the correct chunk of Javascript.
+        """
+        if isinstance(value, RE_TYPE):
+            # Regexes are handled specially
+            if op.strip('$') == 'ne':
+                op_js = Q.OPERATORS['regex_ne']
+            else:
+                op_js = Q.OPERATORS['regex_eq']
+        else:
+            op_js = Q.OPERATORS[op.strip('$')]
+
+        # Comparing two ObjectIds in Javascript doesn't work..
+        if isinstance(value, pymongo.objectid.ObjectId):
+            value = str(value)
+
+        # Perform the substitution
+        operation_js = op_js % {
+            'field': key, 
+            'value': value_name
+        }
+        return value, operation_js
 
 class QuerySet(object):
     """A set of results returned from a query. Wraps a MongoDB cursor, 
@@ -274,13 +296,15 @@ class QuerySet(object):
         """
         operators = ['ne', 'gt', 'gte', 'lt', 'lte', 'in', 'nin', 'mod',
                      'all', 'size', 'exists']
+        match_operators = ['contains', 'icontains', 'startswith', 
+                           'istartswith', 'endswith', 'iendswith']
 
         mongo_query = {}
         for key, value in query.items():
             parts = key.split('__')
             # Check for an operator and transform to mongo-style if there is
             op = None
-            if parts[-1] in operators:
+            if parts[-1] in operators + match_operators:
                 op = parts.pop()
 
             if _doc_cls:
@@ -290,13 +314,15 @@ class QuerySet(object):
 
                 # Convert value to proper value
                 field = fields[-1]
-                if op in (None, 'ne', 'gt', 'gte', 'lt', 'lte'):
+                singular_ops = [None, 'ne', 'gt', 'gte', 'lt', 'lte']
+                singular_ops += match_operators
+                if op in singular_ops:
                     value = field.prepare_query_value(op, value)
                 elif op in ('in', 'nin', 'all'):
                     # 'in', 'nin' and 'all' require a list of values
                     value = [field.prepare_query_value(op, v) for v in value]
 
-            if op:
+            if op and op not in match_operators:
                 value = {'$' + op: value}
 
             key = '.'.join(parts)
@@ -372,7 +398,7 @@ class QuerySet(object):
     def in_bulk(self, object_ids):
         """Retrieve a set of documents by their ids.
         
-        :param object_ids: a list or tuple of ``ObjectId``s
+        :param object_ids: a list or tuple of ``ObjectId``\ s
         :rtype: dict of ObjectIds as keys and collection-specific
                 Document subclasses as values.
         """
@@ -454,7 +480,7 @@ class QuerySet(object):
         
             post = BlogPost.objects(...).only("title")
         
-        :param *fields: fields to include
+        :param fields: fields to include
         """
         self._loaded_fields = []
         for field in fields:
@@ -604,6 +630,21 @@ class QuerySet(object):
     def __iter__(self):
         return self
 
+    def _sub_js_fields(self, code):
+        """When fields are specified with [~fieldname] syntax, where 
+        *fieldname* is the Python name of a field, *fieldname* will be 
+        substituted for the MongoDB name of the field (specified using the
+        :attr:`name` keyword argument in a field's constructor).
+        """
+        def field_sub(match):
+            # Extract just the field name, and look up the field objects
+            field_name = match.group(1).split('.')
+            fields = QuerySet._lookup_field(self._document, field_name)
+            # Substitute the correct name for the field into the javascript
+            return '["%s"]' % fields[-1].name
+
+        return re.sub('\[\s*~([A-z_][A-z_0-9.]+?)\s*\]', field_sub, code)
+
     def exec_js(self, code, *fields, **options):
         """Execute a Javascript function on the server. A list of fields may be
         provided, which will be translated to their correct names and supplied
@@ -613,12 +654,21 @@ class QuerySet(object):
         current query; and ``options``, which is an object containing any
         options specified as keyword arguments.
 
+        As fields in MongoEngine may use different names in the database (set
+        using the :attr:`name` keyword argument to a :class:`Field` 
+        constructor), a mechanism exists for replacing MongoEngine field names
+        with the database field names in Javascript code. When accessing a 
+        field, use square-bracket notation, and prefix the MongoEngine field
+        name with a tilde (~).
+
         :param code: a string of Javascript code to execute
         :param fields: fields that you will be using in your function, which
             will be passed in to your function as arguments
         :param options: options that you want available to the function 
             (accessed in Javascript through the ``options`` object)
         """
+        code = self._sub_js_fields(code)
+
         fields = [QuerySet._translate_field_name(self._document, f)
                   for f in fields]
         collection = self._document._meta['collection']

tests/queryset.py

@@ -186,6 +186,59 @@ class QuerySetTest(unittest.TestCase):
         person = self.Person.objects.get(age=50)
         self.assertEqual(person.name, "User C")
 
+    def test_regex_query_shortcuts(self):
+        """Ensure that contains, startswith, endswith, etc work.
+        """
+        person = self.Person(name='Guido van Rossum')
+        person.save()
+
+        # Test contains
+        obj = self.Person.objects(name__contains='van').first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(name__contains='Van').first()
+        self.assertEqual(obj, None)
+        obj = self.Person.objects(Q(name__contains='van')).first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(Q(name__contains='Van')).first()
+        self.assertEqual(obj, None)
+
+        # Test icontains
+        obj = self.Person.objects(name__icontains='Van').first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(Q(name__icontains='Van')).first()
+        self.assertEqual(obj, person)
+
+        # Test startswith
+        obj = self.Person.objects(name__startswith='Guido').first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(name__startswith='guido').first()
+        self.assertEqual(obj, None)
+        obj = self.Person.objects(Q(name__startswith='Guido')).first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(Q(name__startswith='guido')).first()
+        self.assertEqual(obj, None)
+
+        # Test istartswith
+        obj = self.Person.objects(name__istartswith='guido').first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(Q(name__istartswith='guido')).first()
+        self.assertEqual(obj, person)
+
+        # Test endswith
+        obj = self.Person.objects(name__endswith='Rossum').first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(name__endswith='rossuM').first()
+        self.assertEqual(obj, None)
+        obj = self.Person.objects(Q(name__endswith='Rossum')).first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(Q(name__endswith='rossuM')).first()
+        self.assertEqual(obj, None)
+
+        # Test iendswith
+        obj = self.Person.objects(name__iendswith='rossuM').first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(Q(name__iendswith='rossuM')).first()
+        self.assertEqual(obj, person)
 
     def test_filter_chaining(self):
         """Ensure filters can be chained together.
@@ -356,6 +409,11 @@ class QuerySetTest(unittest.TestCase):
         post6 = BlogPost(published=False)
         post6.save()
 
+        # Check ObjectId lookup works
+        obj = BlogPost.objects(id=post1.id).first()
+        self.assertEqual(obj, post1)
+
+        # Check Q object combination
         date = datetime(2010, 1, 10)
         q = BlogPost.objects(Q(publish_date__lte=date) | Q(published=True))
         posts = [post.id for post in q]
@@ -376,6 +434,26 @@ class QuerySetTest(unittest.TestCase):
         self.assertEqual(len(self.Person.objects(Q(age__in=[20]))), 2)
         self.assertEqual(len(self.Person.objects(Q(age__in=[20, 30]))), 3)
 
+    def test_q_regex(self):
+        """Ensure that Q objects can be queried using regexes.
+        """
+        person = self.Person(name='Guido van Rossum')
+        person.save()
+
+        import re
+        obj = self.Person.objects(Q(name=re.compile('^Gui'))).first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(Q(name=re.compile('^gui'))).first()
+        self.assertEqual(obj, None)
+
+        obj = self.Person.objects(Q(name=re.compile('^gui', re.I))).first()
+        self.assertEqual(obj, person)
+
+        obj = self.Person.objects(Q(name__ne=re.compile('^bob'))).first()
+        self.assertEqual(obj, person)
+        obj = self.Person.objects(Q(name__ne=re.compile('^Gui'))).first()
+        self.assertEqual(obj, None)
+
     def test_exec_js_query(self):
         """Ensure that queries are properly formed for use in exec_js.
         """
@@ -420,6 +498,58 @@ class QuerySetTest(unittest.TestCase):
 
         BlogPost.drop_collection()
 
+    def test_exec_js_field_sub(self):
+        """Ensure that field substitutions occur properly in exec_js functions.
+        """
+        class Comment(EmbeddedDocument):
+            content = StringField(name='body')
+
+        class BlogPost(Document):
+            name = StringField(name='doc-name')
+            comments = ListField(EmbeddedDocumentField(Comment), name='cmnts')
+
+        BlogPost.drop_collection()
+
+        comments1 = [Comment(content='cool'), Comment(content='yay')]
+        post1 = BlogPost(name='post1', comments=comments1)
+        post1.save()
+
+        comments2 = [Comment(content='nice stuff')]
+        post2 = BlogPost(name='post2', comments=comments2)
+        post2.save()
+
+        code = """
+        function getComments() {
+            var comments = [];
+            db[collection].find(query).forEach(function(doc) {
+                var docComments = doc[~comments];
+                for (var i = 0; i < docComments.length; i++) {
+                    comments.push({
+                        'document': doc[~name],
+                        'comment': doc[~comments][i][~comments.content]
+                    });
+                }
+            });
+            return comments;
+        }
+        """
+        
+        sub_code = BlogPost.objects._sub_js_fields(code)
+        code_chunks = ['doc["cmnts"];', 'doc["doc-name"],', 
+                       'doc["cmnts"][i]["body"]']
+        for chunk in code_chunks:
+            self.assertTrue(chunk in sub_code)
+
+        results = BlogPost.objects.exec_js(code)
+        expected_results = [
+            {u'comment': u'cool', u'document': u'post1'}, 
+            {u'comment': u'yay', u'document': u'post1'}, 
+            {u'comment': u'nice stuff', u'document': u'post2'},
+        ]
+        self.assertEqual(results, expected_results)
+
+        BlogPost.drop_collection()
+
     def test_delete(self):
         """Ensure that documents are properly deleted from the database.
         """