pub/mongoengine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'regex-query-shortcuts'

Browse Source
This commit is contained in:
Harry Marr 2010-02-28 17:38:03 +00:00
commit 5e2c5fa97b
4 changed files with 326 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

113
docs/guide/querying.rst
View File

@ -68,6 +68,16 @@ Available operators are as follows:
* ``size`` -- the size of the array is
* ``exists`` -- value for field exists
The following operators are available as shortcuts to querying with regular
expressions:
* ``contains`` -- string field contains value
* ``icontains`` -- string field contains value (case insensitive)
* ``startswith`` -- string field starts with value
* ``istartswith`` -- string field starts with value (case insensitive)
* ``endswith`` -- string field ends with value
* ``iendswith`` -- string field ends with value (case insensitive)
Limiting and skipping results
=============================
Just as with traditional ORMs, you may limit the number of results returned, or
@ -232,6 +242,109 @@ calling it with keyword arguments::
natively supported by MongoDB -- they are compiled to Javascript and sent
to the server for execution.
Server-side javascript execution
================================
Javascript functions may be written and sent to the server for execution. The
result of this is the return value of the Javascript function. This
functionality is accessed through the
:meth:`~mongoengine.queryset.QuerySet.exec_js` method on
:meth:`~mongoengine.queryset.QuerySet` objects. Pass in a string containing a
Javascript function as the first argument.
The remaining positional arguments are names of fields that will be passed into
you Javascript function as its arguments. This allows functions to be written
that may be executed on any field in a collection (e.g. the
:meth:`~mongoengine.queryset.QuerySet.sum` method, which accepts the name of
the field to sum over as its argument). Note that field names passed in in this
manner are automatically translated to the names used on the database (set
using the :attr:`name` keyword argument to a field constructor).
Keyword arguments to :meth:`~mongoengine.queryset.QuerySet.exec_js` are
combined into an object called :attr:`options`, which is available in the
Javascript function. This may be used for defining specific parameters for your
function.
Some variables are made available in the scope of the Javascript function:
* ``collection`` -- the name of the collection that corresponds to the
:class:`~mongoengine.Document` class that is being used; this should be
used to get the :class:`Collection` object from :attr:`db` in Javascript
code
* ``query`` -- the query that has been generated by the
:class:`~mongoengine.queryset.QuerySet` object; this may be passed into
the :meth:`find` method on a :class:`Collection` object in the Javascript
function
* ``options`` -- an object containing the keyword arguments passed into
:meth:`~mongoengine.queryset.QuerySet.exec_js`
The following example demonstrates the intended usage of
:meth:`~mongoengine.queryset.QuerySet.exec_js` by defining a function that sums
over a field on a document (this functionality is already available throught
:meth:`~mongoengine.queryset.QuerySet.sum` but is shown here for sake of
example)::
def sum_field(document, field_name, include_negatives=True):
code = """
function(sumField) {
var total = 0.0;
db[collection].find(query).forEach(function(doc) {
var val = doc[sumField];
if (val >= 0.0 || options.includeNegatives) {
total += val;
}
});
return total;
}
"""
options = {'includeNegatives': include_negatives}
return document.objects.exec_js(code, field_name, **options)
As fields in MongoEngine may use different names in the database (set using the
:attr:`name` keyword argument to a :class:`Field` constructor), a mechanism
exists for replacing MongoEngine field names with the database field names in
Javascript code. When accessing a field on a collection object, use
square-bracket notation, and prefix the MongoEngine field name with a tilde.
The field name that follows the tilde will be translated to the name used in
the database. Note that when referring to fields on embedded documents,
the name of the :class:`~mongoengine.EmbeddedDocumentField`, followed by a dot,
should be used before the name of the field on the embedded document. The
following example shows how the substitutions are made::
class Comment(EmbeddedDocument):
content = StringField(name='body')
class BlogPost(Document):
title = StringField(name='doctitle')
comments = ListField(EmbeddedDocumentField(Comment), name='cs')
# Returns a list of dictionaries. Each dictionary contains a value named
# "document", which corresponds to the "title" field on a BlogPost, and
# "comment", which corresponds to an individual comment. The substitutions
# made are shown in the comments.
BlogPost.objects.exec_js("""
function() {
var comments = [];
db[collection].find(query).forEach(function(doc) {
// doc[~comments] -> doc["cs"]
var docComments = doc[~comments];
for (var i = 0; i < docComments.length; i++) {
// doc[~comments][i] -> doc["cs"][i]
var comment = doc[~comments][i];
comments.push({
// doc[~title] -> doc["doctitle"]
'document': doc[~title],
// comment[~comments.content] -> comment["body"]
'comment': comment[~comments.content]
});
}
});
return comments;
}
""")
.. _guide-atomic-updates:
Atomic updates

19
mongoengine/fields.py
View File

@ -39,8 +39,25 @@ class StringField(BaseField):
def lookup_member(self, member_name):
return None
def prepare_query_value(self, op, value):
if not isinstance(op, basestring):
return value
class URLField(BaseField):
if op.lstrip('i') in ('startswith', 'endswith', 'contains'):
flags = 0
if op.startswith('i'):
flags = re.IGNORECASE
op = op.lstrip('i')
regex = r'%s'
if op == 'startswith':
regex = r'^%s'
elif op == 'endswith':
regex = r'%s$'
value = re.compile(regex % value, flags)
return value
class URLField(StringField):
"""A field that validates input as a URL.
"""

80
mongoengine/queryset.py
View File

@ -1,6 +1,7 @@
from connection import _get_db
import pymongo
import re
import copy
@ -27,6 +28,8 @@ class OperationError(Exception):
pass
RE_TYPE = type(re.compile(''))
class Q(object):
OR = '||'
@ -46,6 +49,8 @@ class Q(object):
'return this.%(field)s.indexOf(a) != -1 })'),
'size': 'this.%(field)s.length == %(value)s',
'exists': 'this.%(field)s != null',
'regex_eq': '%(value)s.test(this.%(field)s)',
'regex_ne': '!%(value)s.test(this.%(field)s)',
}
def __init__(self, **query):
@ -90,24 +95,41 @@ class Q(object):
for j, (op, value) in enumerate(value.items()):
# Create a custom variable name for this operator
op_value_name = '%so%s' % (value_name, j)
# Construct the JS that uses this op
value, operation_js = self._build_op_js(op, key, value,
op_value_name)
# Update the js scope with the value for this op
js_scope[op_value_name] = value
# Construct the JS that uses this op
operation_js = Q.OPERATORS[op.strip('$')] % {
'field': key,
'value': op_value_name
}
js.append(operation_js)
else:
js_scope[value_name] = value
# Construct the JS for this field
field_js = Q.OPERATORS[op.strip('$')] % {
'field': key,
'value': value_name
}
value, field_js = self._build_op_js(op, key, value, value_name)
js_scope[value_name] = value
js.append(field_js)
return ' && '.join(js)
def _build_op_js(self, op, key, value, value_name):
"""Substitute the values in to the correct chunk of Javascript.
"""
if isinstance(value, RE_TYPE):
# Regexes are handled specially
if op.strip('$') == 'ne':
op_js = Q.OPERATORS['regex_ne']
else:
op_js = Q.OPERATORS['regex_eq']
else:
op_js = Q.OPERATORS[op.strip('$')]
# Comparing two ObjectIds in Javascript doesn't work..
if isinstance(value, pymongo.objectid.ObjectId):
value = str(value)
# Perform the substitution
operation_js = op_js % {
'field': key,
'value': value_name
}
return value, operation_js
class QuerySet(object):
"""A set of results returned from a query. Wraps a MongoDB cursor,
@ -274,13 +296,15 @@ class QuerySet(object):
"""
operators = ['ne', 'gt', 'gte', 'lt', 'lte', 'in', 'nin', 'mod',
'all', 'size', 'exists']
match_operators = ['contains', 'icontains', 'startswith',
'istartswith', 'endswith', 'iendswith']
mongo_query = {}
for key, value in query.items():
parts = key.split('__')
# Check for an operator and transform to mongo-style if there is
op = None
if parts[-1] in operators:
if parts[-1] in operators + match_operators:
op = parts.pop()
if _doc_cls:
@ -290,13 +314,15 @@ class QuerySet(object):
# Convert value to proper value
field = fields[-1]
if op in (None, 'ne', 'gt', 'gte', 'lt', 'lte'):
singular_ops = [None, 'ne', 'gt', 'gte', 'lt', 'lte']
singular_ops += match_operators
if op in singular_ops:
value = field.prepare_query_value(op, value)
elif op in ('in', 'nin', 'all'):
# 'in', 'nin' and 'all' require a list of values
value = [field.prepare_query_value(op, v) for v in value]
if op:
if op and op not in match_operators:
value = {'$' + op: value}
key = '.'.join(parts)
@ -372,7 +398,7 @@ class QuerySet(object):
def in_bulk(self, object_ids):
"""Retrieve a set of documents by their ids.
:param object_ids: a list or tuple of ``ObjectId``s
:param object_ids: a list or tuple of ``ObjectId``\ s
:rtype: dict of ObjectIds as keys and collection-specific
Document subclasses as values.
"""
@ -454,7 +480,7 @@ class QuerySet(object):
post = BlogPost.objects(...).only("title")
:param *fields: fields to include
:param fields: fields to include
"""
self._loaded_fields = []
for field in fields:
@ -604,6 +630,21 @@ class QuerySet(object):
def __iter__(self):
return self
def _sub_js_fields(self, code):
"""When fields are specified with [~fieldname] syntax, where
*fieldname* is the Python name of a field, *fieldname* will be
substituted for the MongoDB name of the field (specified using the
:attr:`name` keyword argument in a field's constructor).
"""
def field_sub(match):
# Extract just the field name, and look up the field objects
field_name = match.group(1).split('.')
fields = QuerySet._lookup_field(self._document, field_name)
# Substitute the correct name for the field into the javascript
return '["%s"]' % fields[-1].name
return re.sub('\[\s*~([A-z_][A-z_0-9.]+?)\s*\]', field_sub, code)
def exec_js(self, code, *fields, **options):
"""Execute a Javascript function on the server. A list of fields may be
provided, which will be translated to their correct names and supplied
@ -613,12 +654,21 @@ class QuerySet(object):
current query; and ``options``, which is an object containing any
options specified as keyword arguments.
As fields in MongoEngine may use different names in the database (set
using the :attr:`name` keyword argument to a :class:`Field`
constructor), a mechanism exists for replacing MongoEngine field names
with the database field names in Javascript code. When accessing a
field, use square-bracket notation, and prefix the MongoEngine field
name with a tilde (~).
:param code: a string of Javascript code to execute
:param fields: fields that you will be using in your function, which
will be passed in to your function as arguments
:param options: options that you want available to the function
(accessed in Javascript through the ``options`` object)
"""
code = self._sub_js_fields(code)
fields = [QuerySet._translate_field_name(self._document, f)
for f in fields]
collection = self._document._meta['collection']

130
tests/queryset.py
View File

@ -186,6 +186,59 @@ class QuerySetTest(unittest.TestCase):
person = self.Person.objects.get(age=50)
self.assertEqual(person.name, "User C")
def test_regex_query_shortcuts(self):
"""Ensure that contains, startswith, endswith, etc work.
"""
person = self.Person(name='Guido van Rossum')
person.save()
# Test contains
obj = self.Person.objects(name__contains='van').first()
self.assertEqual(obj, person)
obj = self.Person.objects(name__contains='Van').first()
self.assertEqual(obj, None)
obj = self.Person.objects(Q(name__contains='van')).first()
self.assertEqual(obj, person)
obj = self.Person.objects(Q(name__contains='Van')).first()
self.assertEqual(obj, None)
# Test icontains
obj = self.Person.objects(name__icontains='Van').first()
self.assertEqual(obj, person)
obj = self.Person.objects(Q(name__icontains='Van')).first()
self.assertEqual(obj, person)
# Test startswith
obj = self.Person.objects(name__startswith='Guido').first()
self.assertEqual(obj, person)
obj = self.Person.objects(name__startswith='guido').first()
self.assertEqual(obj, None)
obj = self.Person.objects(Q(name__startswith='Guido')).first()
self.assertEqual(obj, person)
obj = self.Person.objects(Q(name__startswith='guido')).first()
self.assertEqual(obj, None)
# Test istartswith
obj = self.Person.objects(name__istartswith='guido').first()
self.assertEqual(obj, person)
obj = self.Person.objects(Q(name__istartswith='guido')).first()
self.assertEqual(obj, person)
# Test endswith
obj = self.Person.objects(name__endswith='Rossum').first()
self.assertEqual(obj, person)
obj = self.Person.objects(name__endswith='rossuM').first()
self.assertEqual(obj, None)
obj = self.Person.objects(Q(name__endswith='Rossum')).first()
self.assertEqual(obj, person)
obj = self.Person.objects(Q(name__endswith='rossuM')).first()
self.assertEqual(obj, None)
# Test iendswith
obj = self.Person.objects(name__iendswith='rossuM').first()
self.assertEqual(obj, person)
obj = self.Person.objects(Q(name__iendswith='rossuM')).first()
self.assertEqual(obj, person)
def test_filter_chaining(self):
"""Ensure filters can be chained together.
@ -356,6 +409,11 @@ class QuerySetTest(unittest.TestCase):
post6 = BlogPost(published=False)
post6.save()
# Check ObjectId lookup works
obj = BlogPost.objects(id=post1.id).first()
self.assertEqual(obj, post1)
# Check Q object combination
date = datetime(2010, 1, 10)
q = BlogPost.objects(Q(publish_date__lte=date) | Q(published=True))
posts = [post.id for post in q]
@ -376,6 +434,26 @@ class QuerySetTest(unittest.TestCase):
self.assertEqual(len(self.Person.objects(Q(age__in=[20]))), 2)
self.assertEqual(len(self.Person.objects(Q(age__in=[20, 30]))), 3)
def test_q_regex(self):
"""Ensure that Q objects can be queried using regexes.
"""
person = self.Person(name='Guido van Rossum')
person.save()
import re
obj = self.Person.objects(Q(name=re.compile('^Gui'))).first()
self.assertEqual(obj, person)
obj = self.Person.objects(Q(name=re.compile('^gui'))).first()
self.assertEqual(obj, None)
obj = self.Person.objects(Q(name=re.compile('^gui', re.I))).first()
self.assertEqual(obj, person)
obj = self.Person.objects(Q(name__ne=re.compile('^bob'))).first()
self.assertEqual(obj, person)
obj = self.Person.objects(Q(name__ne=re.compile('^Gui'))).first()
self.assertEqual(obj, None)
def test_exec_js_query(self):
"""Ensure that queries are properly formed for use in exec_js.
"""
@ -420,6 +498,58 @@ class QuerySetTest(unittest.TestCase):
BlogPost.drop_collection()
def test_exec_js_field_sub(self):
"""Ensure that field substitutions occur properly in exec_js functions.
"""
class Comment(EmbeddedDocument):
content = StringField(name='body')
class BlogPost(Document):
name = StringField(name='doc-name')
comments = ListField(EmbeddedDocumentField(Comment), name='cmnts')
BlogPost.drop_collection()
comments1 = [Comment(content='cool'), Comment(content='yay')]
post1 = BlogPost(name='post1', comments=comments1)
post1.save()
comments2 = [Comment(content='nice stuff')]
post2 = BlogPost(name='post2', comments=comments2)
post2.save()
code = """
function getComments() {
var comments = [];
db[collection].find(query).forEach(function(doc) {
var docComments = doc[~comments];
for (var i = 0; i < docComments.length; i++) {
comments.push({
'document': doc[~name],
'comment': doc[~comments][i][~comments.content]
});
}
});
return comments;
}
"""
sub_code = BlogPost.objects._sub_js_fields(code)
code_chunks = ['doc["cmnts"];', 'doc["doc-name"],',
'doc["cmnts"][i]["body"]']
for chunk in code_chunks:
self.assertTrue(chunk in sub_code)
results = BlogPost.objects.exec_js(code)
expected_results = [
{u'comment': u'cool', u'document': u'post1'},
{u'comment': u'yay', u'document': u'post1'},
{u'comment': u'nice stuff', u'document': u'post2'},
]
self.assertEqual(results, expected_results)
BlogPost.drop_collection()
def test_delete(self):
"""Ensure that documents are properly deleted from the database.
"""