Bump to v0.2.1

_types index prepended to user defined indexes

Conflicts:
	mongoengine/queryset.py

docs/apireference.rst

@@ -27,6 +27,8 @@ Querying
 .. autoclass:: mongoengine.queryset.QuerySet
    :members:
 
+   .. automethod:: mongoengine.queryset.QuerySet.__call__
+   
 .. autofunction:: mongoengine.queryset.queryset_manager
 
 Fields

docs/changelog.rst

@@ -2,7 +2,22 @@
 Changelog
 =========
 
-Changes is v0.1.3
+Changes in v0.2.1
+=================
+- Added a MongoEngine backend for Django sessions
+- Added force_insert to Document.save()
+- Improved querying syntax for ListField and EmbeddedDocumentField
+- Added support for user-defined primary keys (_ids in MongoDB)
+
+Changes in v0.2
+===============
+- Added Q class for building advanced queries
+- Added QuerySet methods for atomic updates to documents
+- Fields may now specify ``unique=True`` to enforce uniqueness across a collection
+- Added option for default document ordering
+- Fixed bug in index definitions
+
+Changes in v0.1.3
 =================
 - Added Django authentication backend
 - Added Document.meta support for indexes, which are ensured just before 

docs/django.rst

@@ -0,0 +1,42 @@
+=============================
+Using MongoEngine with Django
+=============================
+
+Connecting
+==========
+In your **settings.py** file, ignore the standard database settings (unless you
+also plan to use the ORM in your project), and instead call 
+:func:`~mongoengine.connect` somewhere in the settings module.
+
+Authentication
+==============
+MongoEngine includes a Django authentication backend, which uses MongoDB. The
+:class:`~mongoengine.django.auth.User` model is a MongoEngine 
+:class:`~mongoengine.Document`, but implements most of the methods and 
+attributes that the standard Django :class:`User` model does - so the two are
+moderately compatible. Using this backend will allow you to store users in 
+MongoDB but still use many of the Django authentication infrastucture (such as
+the :func:`login_required` decorator and the :func:`authenticate` function). To
+enable the MongoEngine auth backend, add the following to you **settings.py**
+file::
+    
+    AUTHENTICATION_BACKENDS = (
+        'mongoengine.django.auth.MongoEngineBackend',
+    )
+
+The :mod:`~mongoengine.django.auth` module also contains a 
+:func:`~mongoengine.django.auth.get_user` helper function, that takes a user's
+:attr:`id` and returns a :class:`~mongoengine.django.auth.User` object.
+
+Sessions
+========
+Django allows the use of different backend stores for its sessions. MongoEngine
+provides a MongoDB-based session backend for Django, which allows you to use
+sessions in you Django application with just MongoDB. To enable the MongoEngine
+session backend, ensure that your settings module has
+``'django.contrib.sessions.middleware.SessionMiddleware'`` in the
+``MIDDLEWARE_CLASSES`` field  and ``'django.contrib.sessions'`` in your
+``INSTALLED_APPS``. From there, all you need to do is add the following line
+into you settings module::
+
+    SESSION_ENGINE = 'mongoengine.django.sessions'

docs/userguide.rst

@@ -138,6 +138,21 @@ field::
 The :class:`User` object is automatically turned into a reference behind the
 scenes, and dereferenced when the :class:`Page` object is retrieved.
 
+Uniqueness constraints
+^^^^^^^^^^^^^^^^^^^^^^
+MongoEngine allows you to specify that a field should be unique across a
+collection by providing ``unique=True`` to a :class:`~mongoengine.Field`\ 's
+constructor. If you try to save a document that has the same value for a unique
+field as a document that is already in the database, a 
+:class:`~mongoengine.OperationError` will be raised. You may also specify
+multi-field uniqueness constraints by using :attr:`unique_with`, which may be
+either a single field name, or a list or tuple of field names::
+
+    class User(Document):
+        username = StringField(unique=True)
+        first_name = StringField()
+        last_name = StringField(unique_with='last_name')
+
 Document collections
 --------------------
 Document classes that inherit **directly** from :class:`~mongoengine.Document`
@@ -168,6 +183,62 @@ The following example shows a :class:`Log` document that will be limited to
         ip_address = StringField()
         meta = {'max_documents': 1000, 'max_size': 2000000}
 
+Indexes
+-------
+You can specify indexes on collections to make querying faster. This is done
+by creating a list of index specifications called :attr:`indexes` in the
+:attr:`~mongoengine.Document.meta` dictionary, where an index specification may
+either be a single field name, or a tuple containing multiple field names. A
+direction may be specified on fields by prefixing the field name with a **+**
+or a **-** sign. Note that direction only matters on multi-field indexes. ::
+
+    class Page(Document):
+        title = StringField()
+        rating = StringField()
+        meta = {
+            'indexes': ['title', ('title', '-rating')]
+        }
+        
+Ordering
+--------
+A default ordering can be specified for your
+:class:`~mongoengine.queryset.QuerySet` using the :attr:`ordering` attribute of
+:attr:`~mongoengine.Document.meta`.  Ordering will be applied when the
+:class:`~mongoengine.queryset.QuerySet` is created, and can be overridden by
+subsequent calls to :meth:`~mongoengine.queryset.QuerySet.order_by`. ::
+
+    from datetime import datetime
+
+    class BlogPost(Document):
+        title = StringField()
+        published_date = DateTimeField()
+
+        meta = {
+            'ordering': ['-published_date']
+        }
+
+    blog_post_1 = BlogPost(title="Blog Post #1")
+    blog_post_1.published_date = datetime(2010, 1, 5, 0, 0 ,0))
+
+    blog_post_2 = BlogPost(title="Blog Post #2") 
+    blog_post_2.published_date = datetime(2010, 1, 6, 0, 0 ,0))
+
+    blog_post_3 = BlogPost(title="Blog Post #3")
+    blog_post_3.published_date = datetime(2010, 1, 7, 0, 0 ,0))
+
+    blog_post_1.save()
+    blog_post_2.save()
+    blog_post_3.save()
+
+    # get the "first" BlogPost using default ordering
+    # from BlogPost.meta.ordering
+    latest_post = BlogPost.objects.first() 
+    self.assertEqual(latest_post.title, "Blog Post #3")
+
+    # override default ordering, order BlogPosts by "published_date"
+    first_post = BlogPost.objects.order_by("+published_date").first()
+    self.assertEqual(first_post.title, "Blog Post #1")
+
 Document inheritance
 --------------------
 To create a specialised type of a :class:`~mongoengine.Document` you have
@@ -247,8 +318,25 @@ saved::
     >>> page.id
     ObjectId('123456789abcdef000000000')
 
-Alternatively, you may explicitly set the :attr:`id` before you save the
-document, but the id must be a valid PyMongo :class:`ObjectId`.
+Alternatively, you may define one of your own fields to be the document's
+"primary key" by providing ``primary_key=True`` as a keyword argument to a
+field's constructor. Under the hood, MongoEngine will use this field as the
+:attr:`id`; in fact :attr:`id` is actually aliased to your primary key field so
+you may still use :attr:`id` to access the primary key if you want::
+
+    >>> class User(Document):
+    ...     email = StringField(primary_key=True)
+    ...     name = StringField()
+    ...
+    >>> bob = User(email='bob@example.com', name='Bob')
+    >>> bob.save()
+    >>> bob.id == bob.email == 'bob@example.com'
+    True
+
+.. note::
+   If you define your own primary key field, the field implicitly becomes
+   required, so a :class:`ValidationError` will be thrown if you don't provide
+   it.
 
 Querying the database
 =====================
@@ -383,3 +471,64 @@ would be generating "tag-clouds"::
     from operator import itemgetter
     top_tags = sorted(tag_freqs.items(), key=itemgetter(1), reverse=True)[:10]
 
+Advanced queries
+----------------
+Sometimes calling a :class:`~mongoengine.queryset.QuerySet` object with keyword
+arguments can't fully express the query you want to use -- for example if you
+need to combine a number of constraints using *and* and *or*. This is made 
+possible in MongoEngine through the :class:`~mongoengine.queryset.Q` class.
+A :class:`~mongoengine.queryset.Q` object represents part of a query, and
+can be initialised using the same keyword-argument syntax you use to query
+documents. To build a complex query, you may combine 
+:class:`~mongoengine.queryset.Q` objects using the ``&`` (and) and ``|`` (or)
+operators. To use :class:`~mongoengine.queryset.Q` objects, pass them in
+as positional arguments to :attr:`Document.objects` when you filter it by
+calling it with keyword arguments::
+
+    # Get published posts
+    Post.objects(Q(published=True) | Q(publish_date__lte=datetime.now()))
+
+    # Get top posts
+    Post.objects((Q(featured=True) & Q(hits__gte=1000)) | Q(hits__gte=5000))
+
+.. warning::
+   Only use these advanced queries if absolutely necessary as they will execute
+   significantly slower than regular queries. This is because they are not
+   natively supported by MongoDB -- they are compiled to Javascript and sent
+   to the server for execution.
+
+Atomic updates
+--------------
+Documents may be updated atomically by using the
+:meth:`~mongoengine.queryset.QuerySet.update_one` and
+:meth:`~mongoengine.queryset.QuerySet.update` methods on a 
+:meth:`~mongoengine.queryset.QuerySet`. There are several different "modifiers"
+that you may use with these methods:
+
+* ``set`` -- set a particular value
+* ``unset`` -- delete a particular value (since MongoDB v1.3+)
+* ``inc`` -- increment a value by a given amount
+* ``dec`` -- decrement a value by a given amount
+* ``push`` -- append a value to a list
+* ``push_all`` -- append several values to a list
+* ``pull`` -- remove a value from a list
+* ``pull_all`` -- remove several values from a list
+
+The syntax for atomic updates is similar to the querying syntax, but the 
+modifier comes before the field, not after it::
+    
+    >>> post = BlogPost(title='Test', page_views=0, tags=['database'])
+    >>> post.save()
+    >>> BlogPost.objects(id=post.id).update_one(inc__page_views=1)
+    >>> post.reload()  # the document has been changed, so we need to reload it
+    >>> post.page_views
+    1
+    >>> BlogPost.objects(id=post.id).update_one(set__title='Example Post')
+    >>> post.reload()
+    >>> post.title
+    'Example Post'
+    >>> BlogPost.objects(id=post.id).update_one(push__tags='nosql')
+    >>> post.reload()
+    >>> post.tags
+    ['database', 'nosql']
+

mongoengine/__init__.py

@@ -12,7 +12,7 @@ __all__ = (document.__all__ + fields.__all__ + connection.__all__ +
 
 __author__ = 'Harry Marr'
 
-VERSION = (0, 1, 3)
+VERSION = (0, 2, 1)
 
 def get_version():
     version = '%s.%s' % (VERSION[0], VERSION[1])

mongoengine/base.py

@@ -1,4 +1,4 @@
-from queryset import QuerySetManager
+from queryset import QuerySet, QuerySetManager
 
 import pymongo
 
@@ -12,10 +12,14 @@ class BaseField(object):
     may be added to subclasses of `Document` to define a document's schema.
     """
     
-    def __init__(self, name=None, required=False, default=None):
-        self.name = name
-        self.required = required
+    def __init__(self, name=None, required=False, default=None, unique=False,
+                 unique_with=None, primary_key=False):
+        self.name = name if not primary_key else '_id'
+        self.required = required or primary_key
         self.default = default
+        self.unique = bool(unique or unique_with)
+        self.unique_with = unique_with
+        self.primary_key = primary_key
 
     def __get__(self, instance, owner):
         """Descriptor for retrieving a value from a field in a document. Do 
@@ -69,7 +73,7 @@ class ObjectIdField(BaseField):
 
     def to_mongo(self, value):
         if not isinstance(value, pymongo.objectid.ObjectId):
-            return pymongo.objectid.ObjectId(value)
+            return pymongo.objectid.ObjectId(str(value))
         return value
 
     def prepare_query_value(self, value):
@@ -136,6 +140,9 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
         collection = name.lower()
         
         simple_class = True
+        id_field = None
+        base_indexes = []
+
         # Subclassed documents inherit collection from superclass
         for base in bases:
             if hasattr(base, '_meta') and 'collection' in base._meta:
@@ -149,16 +156,23 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
                     simple_class = False
                 collection = base._meta['collection']
 
+                id_field = id_field or base._meta.get('id_field')
+                base_indexes += base._meta.get('indexes', [])
+
         meta = {
             'collection': collection,
             'allow_inheritance': True,
             'max_documents': None,
             'max_size': None,
-            'indexes': [] # indexes to be ensured at runtime
+            'ordering': [], # default ordering applied at runtime
+            'indexes': [], # indexes to be ensured at runtime
+            'id_field': id_field,
         }
 
         # Apply document-defined meta options
         meta.update(attrs.get('meta', {}))
+
+        meta['indexes'] += base_indexes
         
         # Only simple classes - direct subclasses of Document - may set
         # allow_inheritance to False
@@ -167,13 +181,55 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
                              '"allow_inheritance" to False')
         attrs['_meta'] = meta
 
-        attrs['id'] = ObjectIdField(name='_id')
-
         # Set up collection manager, needs the class to have fields so use
         # DocumentMetaclass before instantiating CollectionManager object
         new_class = super_new(cls, name, bases, attrs)
         new_class.objects = QuerySetManager()
 
+        unique_indexes = []
+        for field_name, field in new_class._fields.items():
+            # Generate a list of indexes needed by uniqueness constraints
+            if field.unique:
+                field.required = True
+                unique_fields = [field_name]
+
+                # Add any unique_with fields to the back of the index spec
+                if field.unique_with:
+                    if isinstance(field.unique_with, basestring):
+                        field.unique_with = [field.unique_with]
+
+                    # Convert unique_with field names to real field names
+                    unique_with = []
+                    for other_name in field.unique_with:
+                        parts = other_name.split('.')
+                        # Lookup real name
+                        parts = QuerySet._lookup_field(new_class, parts)
+                        name_parts = [part.name for part in parts]
+                        unique_with.append('.'.join(name_parts))
+                        # Unique field should be required
+                        parts[-1].required = True
+                    unique_fields += unique_with
+
+                # Add the new index to the list
+                index = [(f, pymongo.ASCENDING) for f in unique_fields]
+                unique_indexes.append(index)
+
+            # Check for custom primary key
+            if field.primary_key:
+                if not new_class._meta['id_field']:
+                    new_class._meta['id_field'] = field_name
+                    # Make 'Document.id' an alias to the real primary key field
+                    new_class.id = field
+                    #new_class._fields['id'] = field
+                else:
+                    raise ValueError('Cannot override primary key field')
+
+        new_class._meta['unique_indexes'] = unique_indexes
+
+        if not new_class._meta['id_field']:
+            new_class._meta['id_field'] = 'id'
+            new_class.id = new_class._fields['id'] = ObjectIdField(name='_id')
+
         return new_class
 
 
@@ -236,6 +292,18 @@ class BaseDocument(object):
     def __len__(self):
         return len(self._data)
 
+    def __repr__(self):
+        try:
+            u = unicode(self)
+        except (UnicodeEncodeError, UnicodeDecodeError):
+            u = '[Bad Unicode data]'
+        return u'<%s: %s>' % (self.__class__.__name__, u)
+
+    def __str__(self):
+        if hasattr(self, '__unicode__'):
+            return unicode(self).encode('utf-8')
+        return '%s object' % self.__class__.__name__
+
     def to_mongo(self):
         """Return data dictionary ready for use with MongoDB.
         """

mongoengine/django/sessions.py

@@ -0,0 +1,63 @@
+from django.contrib.sessions.backends.base import SessionBase, CreateError
+from django.core.exceptions import SuspiciousOperation
+from django.utils.encoding import force_unicode
+
+from mongoengine.document import Document
+from mongoengine import fields
+from mongoengine.queryset import OperationError
+
+from datetime import datetime
+
+
+class MongoSession(Document):
+    session_key = fields.StringField(primary_key=True, max_length=40)
+    session_data = fields.StringField()
+    expire_date = fields.DateTimeField()
+    
+    meta = {'collection': 'django_session', 'allow_inheritance': False}
+
+
+class SessionStore(SessionBase):
+    """A MongoEngine-based session store for Django.
+    """
+
+    def load(self):
+        try:
+            s = MongoSession.objects(session_key=self.session_key,
+                                     expire_date__gt=datetime.now())[0]
+            return self.decode(force_unicode(s.session_data))
+        except (IndexError, SuspiciousOperation):
+            self.create()
+            return {}
+
+    def exists(self, session_key):
+        return bool(MongoSession.objects(session_key=session_key).first())
+
+    def create(self):
+        while True:
+            self.session_key = self._get_new_session_key()
+            try:
+                self.save(must_create=True)
+            except CreateError:
+                continue
+            self.modified = True
+            self._session_cache = {}
+            return
+
+    def save(self, must_create=False):
+        s = MongoSession(session_key=self.session_key)
+        s.session_data = self.encode(self._get_session(no_load=must_create))
+        s.expire_date = self.get_expiry_date()
+        try:
+            s.save(force_insert=must_create, safe=True)
+        except OperationError:
+            if must_create:
+                raise CreateError
+            raise
+
+    def delete(self, session_key=None):
+        if session_key is None:
+            if self.session_key is None:
+                return
+            session_key = self.session_key
+        MongoSession.objects(session_key=session_key).delete()

mongoengine/document.py

@@ -1,9 +1,12 @@
 from base import (DocumentMetaclass, TopLevelDocumentMetaclass, BaseDocument,
                   ValidationError)
+from queryset import OperationError
 from connection import _get_db
 
+import pymongo
 
-__all__ = ['Document', 'EmbeddedDocument']
+
+__all__ = ['Document', 'EmbeddedDocument', 'ValidationError', 'OperationError']
 
 
 class EmbeddedDocument(BaseDocument):
@@ -44,32 +47,65 @@ class Document(BaseDocument):
     maximum size of the collection in bytes. If :attr:`max_size` is not 
     specified and :attr:`max_documents` is, :attr:`max_size` defaults to 
     10000000 bytes (10MB).
+
+    Indexes may be created by specifying :attr:`indexes` in the :attr:`meta`
+    dictionary. The value should be a list of field names or tuples of field 
+    names. Index direction may be specified by prefixing the field names with
+    a **+** or **-** sign.
     """
 
     __metaclass__ = TopLevelDocumentMetaclass
 
-    def save(self):
+    def save(self, safe=True, force_insert=False):
         """Save the :class:`~mongoengine.Document` to the database. If the
         document already exists, it will be updated, otherwise it will be
         created.
+
+        If ``safe=True`` and the operation is unsuccessful, an 
+        :class:`~mongoengine.OperationError` will be raised.
+
+        :param safe: check if the operation succeeded before returning
+        :param force_insert: only try to create a new document, don't allow 
+            updates of existing documents
         """
         self.validate()
-        object_id = self.__class__.objects._collection.save(self.to_mongo())
-        self.id = self._fields['id'].to_python(object_id)
+        doc = self.to_mongo()
+        try:
+            collection = self.__class__.objects._collection
+            if force_insert:
+                object_id = collection.insert(doc, safe=safe)
+            else:
+                object_id = collection.save(doc, safe=safe)
+        except pymongo.errors.OperationFailure, err:
+            message = 'Could not save document (%s)'
+            if 'duplicate key' in str(err):
+                message = 'Tried to save duplicate unique keys (%s)'
+            raise OperationError(message % str(err))
+        id_field = self._meta['id_field']
+        self[id_field] = self._fields[id_field].to_python(object_id)
 
-    def delete(self):
+    def delete(self, safe=False):
         """Delete the :class:`~mongoengine.Document` from the database. This
         will only take effect if the document has been previously saved.
+
+        :param safe: check if the operation succeeded before returning
         """
-        object_id = self._fields['id'].to_mongo(self.id)
-        self.__class__.objects(id=object_id).delete()
+        id_field = self._meta['id_field']
+        object_id = self._fields[id_field].to_mongo(self[id_field])
+        try:
+            self.__class__.objects(**{id_field: object_id}).delete(safe=safe)
+        except pymongo.errors.OperationFailure, err:
+            raise OperationError('Could not delete document (%s)' % str(err))
 
     def reload(self):
         """Reloads all attributes from the database.
+
+        .. versionadded:: 0.1.2
         """
-        obj = self.__class__.objects(id=self.id).first()
+        id_field = self._meta['id_field']
+        obj = self.__class__.objects(**{id_field: self[id_field]}).first()
         for field in self._fields:
-            setattr(self, field, getattr(obj, field))
+            setattr(self, field, obj[field])
 
     def validate(self):
         """Ensure that all fields' values are valid and that required fields

mongoengine/fields.py

@@ -82,6 +82,8 @@ class FloatField(BaseField):
 
 class BooleanField(BaseField):
     """A boolean field type.
+
+    .. versionadded:: 0.1.2
     """
     
     def to_python(self, value):
@@ -131,6 +133,9 @@ class EmbeddedDocumentField(BaseField):
     def lookup_member(self, member_name):
         return self.document._fields.get(member_name)
 
+    def prepare_query_value(self, value):
+        return self.to_mongo(value)
+
 
 class ListField(BaseField):
     """A list field that wraps a standard field, allowing multiple instances
@@ -163,6 +168,9 @@ class ListField(BaseField):
             raise ValidationError('All items in a list field must be of the '
                                   'specified type')
 
+    def prepare_query_value(self, value):
+        return self.field.to_mongo(value)
+
     def lookup_member(self, member_name):
         return self.field.lookup_member(member_name)
 

mongoengine/queryset.py

@@ -1,15 +1,106 @@
 from connection import _get_db
 
 import pymongo
+import copy
 
 
-__all__ = ['queryset_manager', 'InvalidQueryError', 'InvalidCollectionError']
+__all__ = ['queryset_manager', 'Q', 'InvalidQueryError', 
+           'InvalidCollectionError']
+
+# The maximum number of items to display in a QuerySet.__repr__
+REPR_OUTPUT_SIZE = 20
 
 
 class InvalidQueryError(Exception):
     pass
 
 
+class OperationError(Exception):
+    pass
+
+
+class Q(object):
+    
+    OR = '||'
+    AND = '&&'
+    OPERATORS = {
+        'eq': 'this.%(field)s == %(value)s',
+        'neq': 'this.%(field)s != %(value)s',
+        'gt': 'this.%(field)s > %(value)s',
+        'gte': 'this.%(field)s >= %(value)s',
+        'lt': 'this.%(field)s < %(value)s',
+        'lte': 'this.%(field)s <= %(value)s',
+        'lte': 'this.%(field)s <= %(value)s',
+        'in': 'this.%(field)s.indexOf(%(value)s) != -1',
+        'nin': 'this.%(field)s.indexOf(%(value)s) == -1',
+        'mod': '%(field)s %% %(value)s',
+        'all': ('%(value)s.every(function(a){'
+                'return this.%(field)s.indexOf(a) != -1 })'),
+        'size': 'this.%(field)s.length == %(value)s',
+        'exists': 'this.%(field)s != null',
+    }
+    
+    def __init__(self, **query):
+        self.query = [query]
+
+    def _combine(self, other, op):
+        obj = Q()
+        obj.query = ['('] + copy.deepcopy(self.query) + [op]
+        obj.query += copy.deepcopy(other.query) + [')']
+        return obj
+
+    def __or__(self, other):
+        return self._combine(other, self.OR)
+
+    def __and__(self, other):
+        return self._combine(other, self.AND)
+
+    def as_js(self, document):
+        js = []
+        js_scope = {}
+        for i, item in enumerate(self.query):
+            if isinstance(item, dict):
+                item_query = QuerySet._transform_query(document, **item)
+                # item_query will values will either be a value or a dict
+                js.append(self._item_query_as_js(item_query, js_scope, i))
+            else:
+                js.append(item)
+        return pymongo.code.Code(' '.join(js), js_scope)
+
+    def _item_query_as_js(self, item_query, js_scope, item_num):
+        # item_query will be in one of the following forms
+        #    {'age': 25, 'name': 'Test'}
+        #    {'age': {'$lt': 25}, 'name': {'$in': ['Test', 'Example']}
+        #    {'age': {'$lt': 25, '$gt': 18}}
+        js = []
+        for i, (key, value) in enumerate(item_query.items()):
+            op = 'eq'
+            # Construct a variable name for the value in the JS
+            value_name = 'i%sf%s' % (item_num, i)
+            if isinstance(value, dict):
+                # Multiple operators for this field
+                for j, (op, value) in enumerate(value.items()):
+                    # Create a custom variable name for this operator
+                    op_value_name = '%so%s' % (value_name, j)
+                    # Update the js scope with the value for this op
+                    js_scope[op_value_name] = value
+                    # Construct the JS that uses this op
+                    operation_js = Q.OPERATORS[op.strip('$')] % {
+                        'field': key, 
+                        'value': op_value_name
+                    }
+                    js.append(operation_js)
+            else:
+                js_scope[value_name] = value
+                # Construct the JS for this field
+                field_js = Q.OPERATORS[op.strip('$')] % {
+                    'field': key, 
+                    'value': value_name
+                }
+                js.append(field_js)
+        return ' && '.join(js)
+
+
 class QuerySet(object):
     """A set of results returned from a query. Wraps a MongoDB cursor, 
     providing :class:`~mongoengine.Document` objects as the results.
@@ -17,8 +108,11 @@ class QuerySet(object):
     
     def __init__(self, document, collection):
         self._document = document
-        self._collection = collection
+        self._collection_obj = collection
+        self._accessed_collection = False
         self._query = {}
+        self._where_clauses = []
+
         # If inheritance is allowed, only return instances and instances of
         # subclasses of the class being used
         if document._meta.get('allow_inheritance'):
@@ -27,43 +121,79 @@ class QuerySet(object):
         
     def ensure_index(self, key_or_list):
         """Ensure that the given indexes are in place.
+
+        :param key_or_list: a single index key or a list of index keys (to
+            construct a multi-field index); keys may be prefixed with a **+**
+            or a **-** to determine the index ordering
         """
         if isinstance(key_or_list, basestring):
-            # single-field indexes needn't specify a direction
-            if key_or_list.startswith("-") or key_or_list.startswith("+"):
-                key_or_list = key_or_list[1:]
-            self._collection.ensure_index(key_or_list)
-        elif isinstance(key_or_list, (list, tuple)):
-            index_list = []
-            for key in key_or_list:
-                if key.startswith("-"):
-                    index_list.append((key[1:], pymongo.DESCENDING))
-                else:
-                    if key.startswith("+"):
-                        key = key[1:]
-                    index_list.append((key, pymongo.ASCENDING))
-            self._collection.ensure_index(index_list)
+            key_or_list = [key_or_list]
+
+        index_list = []
+        # If _types is being used, prepend it to every specified index
+        if self._document._meta.get('allow_inheritance'):
+            index_list.append(('_types', 1))
+
+        for key in key_or_list:
+            # Get direction from + or -
+            direction = pymongo.ASCENDING
+            if key.startswith("-"):
+                direction = pymongo.DESCENDING
+            if key.startswith(("+", "-")):
+                    key = key[1:]
+            # Use real field name
+            key = QuerySet._translate_field_name(self._document, key)
+            index_list.append((key, direction))
+        self._collection.ensure_index(index_list)
         return self
 
-    def __call__(self, **query):
+    def __call__(self, *q_objs, **query):
         """Filter the selected documents by calling the 
         :class:`~mongoengine.QuerySet` with a query.
+
+        :param q_objs: :class:`~mongoengine.Q` objects to be used in the query
+        :param query: Django-style query keyword arguments
         """
-        
-        # ensure document-defined indexes are created
-        if self._document._meta['indexes']:
-            for key_or_list in self._document._meta['indexes']:
-                # print "key", key_or_list
-                self.ensure_index(key_or_list)
-        
+        for q in q_objs:
+            self._where_clauses.append(q.as_js(self._document))
         query = QuerySet._transform_query(_doc_cls=self._document, **query)
         self._query.update(query)
         return self
 
+    @property
+    def _collection(self):
+        """Property that returns the collection object. This allows us to
+        perform operations only if the collection is accessed.
+        """
+        if not self._accessed_collection:
+            self._accessed_collection = True
+
+            # Ensure document-defined indexes are created
+            if self._document._meta['indexes']:
+                for key_or_list in self._document._meta['indexes']:
+                    self.ensure_index(key_or_list)
+
+            # Ensure indexes created by uniqueness constraints
+            for index in self._document._meta['unique_indexes']:
+                self._collection.ensure_index(index, unique=True)
+
+            # If _types is being used (for polymorphism), it needs an index
+            if '_types' in self._query:
+                self._collection.ensure_index('_types')
+        return self._collection_obj
+
     @property
     def _cursor(self):
         if not self._cursor_obj:
             self._cursor_obj = self._collection.find(self._query)
+            # Apply where clauses to cursor
+            for js in self._where_clauses:
+                self._cursor_obj.where(js)
+            
+            # apply default ordering
+            if self._document._meta['ordering']:
+                self.order_by(*self._document._meta['ordering'])
+            
         return self._cursor_obj
 
     @classmethod
@@ -89,10 +219,12 @@ class QuerySet(object):
         return fields
 
     @classmethod
-    def _translate_field_name(cls, doc_cls, parts):
+    def _translate_field_name(cls, doc_cls, field, sep='.'):
         """Translate a field attribute name to a database field name.
         """
-        return [field.name for field in QuerySet._lookup_field(doc_cls, parts)]
+        parts = field.split(sep)
+        parts = [f.name for f in QuerySet._lookup_field(doc_cls, parts)]
+        return '.'.join(parts)
 
     @classmethod
     def _transform_query(cls, _doc_cls=None, **query):
@@ -144,11 +276,13 @@ class QuerySet(object):
 
     def with_id(self, object_id):
         """Retrieve the object matching the id provided.
-        """
-        if not isinstance(object_id, pymongo.objectid.ObjectId):
-            object_id = pymongo.objectid.ObjectId(str(object_id))
 
-        result = self._collection.find_one(object_id)
+        :param object_id: the value for the id of the document to look up
+        """
+        id_field = self._document._meta['id_field']
+        object_id = self._document._fields[id_field].to_mongo(object_id)
+
+        result = self._collection.find_one({'_id': object_id})
         if result is not None:
             result = self._document._from_son(result)
         return result
@@ -169,6 +303,8 @@ class QuerySet(object):
     def limit(self, n):
         """Limit the number of returned documents to `n`. This may also be
         achieved using array-slicing syntax (e.g. ``User.objects[:5]``).
+
+        :param n: the maximum number of objects to return
         """
         self._cursor.limit(n)
         # Return self to allow chaining
@@ -177,6 +313,8 @@ class QuerySet(object):
     def skip(self, n):
         """Skip `n` documents before returning the results. This may also be
         achieved using array-slicing syntax (e.g. ``User.objects[5:]``).
+
+        :param n: the number of objects to skip before returning results
         """
         self._cursor.skip(n)
         return self
@@ -197,6 +335,9 @@ class QuerySet(object):
         """Order the :class:`~mongoengine.queryset.QuerySet` by the keys. The
         order may be specified by prepending each of the keys by a + or a -.
         Ascending order is assumed.
+
+        :param keys: fields to order the query results by; keys may be
+            prefixed with **+** or **-** to determine the ordering direction
         """
         key_list = []
         for key in keys:
@@ -213,17 +354,112 @@ class QuerySet(object):
     def explain(self, format=False):
         """Return an explain plan record for the 
         :class:`~mongoengine.queryset.QuerySet`\ 's cursor.
+
+        :param format: format the plan before returning it
         """
+
         plan = self._cursor.explain()
         if format:
             import pprint
             plan = pprint.pformat(plan)
         return plan
         
-    def delete(self):
+    def delete(self, safe=False):
         """Delete the documents matched by the query.
+
+        :param safe: check if the operation succeeded before returning
         """
-        self._collection.remove(self._query)
+        self._collection.remove(self._query, safe=safe)
+
+    @classmethod
+    def _transform_update(cls, _doc_cls=None, **update):
+        """Transform an update spec from Django-style format to Mongo format.
+        """
+        operators = ['set', 'unset', 'inc', 'dec', 'push', 'push_all', 'pull', 
+                     'pull_all']
+
+        mongo_update = {}
+        for key, value in update.items():
+            parts = key.split('__')
+            # Check for an operator and transform to mongo-style if there is
+            op = None
+            if parts[0] in operators:
+                op = parts.pop(0)
+                # Convert Pythonic names to Mongo equivalents
+                if op in ('push_all', 'pull_all'):
+                    op = op.replace('_all', 'All')
+                elif op == 'dec':
+                    # Support decrement by flipping a positive value's sign
+                    # and using 'inc'
+                    op = 'inc'
+                    if value > 0:
+                        value = -value
+
+            if _doc_cls:
+                # Switch field names to proper names [set in Field(name='foo')]
+                fields = QuerySet._lookup_field(_doc_cls, parts)
+                parts = [field.name for field in fields]
+
+                # Convert value to proper value
+                field = fields[-1]
+                if op in (None, 'set', 'unset', 'push', 'pull'):
+                    value = field.prepare_query_value(value)
+                elif op in ('pushAll', 'pullAll'):
+                    value = [field.prepare_query_value(v) for v in value]
+
+            key = '.'.join(parts)
+
+            if op:
+                value = {key: value}
+                key = '$' + op
+
+            if op is None or key not in mongo_update:
+                mongo_update[key] = value
+            elif key in mongo_update and isinstance(mongo_update[key], dict):
+                mongo_update[key].update(value)
+
+        return mongo_update
+
+    def update(self, safe_update=True, **update):
+        """Perform an atomic update on the fields matched by the query.
+
+        :param safe: check if the operation succeeded before returning
+        :param update: Django-style update keyword arguments
+
+        .. versionadded:: 0.2
+        """
+        if pymongo.version < '1.1.1':
+            raise OperationError('update() method requires PyMongo 1.1.1+')
+
+        update = QuerySet._transform_update(self._document, **update)
+        try:
+            self._collection.update(self._query, update, safe=safe_update, 
+                                    multi=True)
+        except pymongo.errors.OperationFailure, err:
+            if str(err) == 'multi not coded yet':
+                raise OperationError('update() method requires MongoDB 1.1.3+')
+            raise OperationError('Update failed (%s)' % str(err))
+
+    def update_one(self, safe_update=True, **update):
+        """Perform an atomic update on first field matched by the query.
+
+        :param safe: check if the operation succeeded before returning
+        :param update: Django-style update keyword arguments
+
+        .. versionadded:: 0.2
+        """
+        update = QuerySet._transform_update(self._document, **update)
+        try:
+            # Explicitly provide 'multi=False' to newer versions of PyMongo
+            # as the default may change to 'True'
+            if pymongo.version >= '1.1.1':
+                self._collection.update(self._query, update, safe=safe_update, 
+                                        multi=False)
+            else:
+                # Older versions of PyMongo don't support 'multi'
+                self._collection.update(self._query, update, safe=safe_update)
+        except pymongo.errors.OperationFailure, e:
+            raise OperationError('Update failed [%s]' % str(e))
 
     def __iter__(self):
         return self
@@ -236,6 +472,12 @@ class QuerySet(object):
         collection in use; ``query``, which is an object representing the 
         current query; and ``options``, which is an object containing any
         options specified as keyword arguments.
+
+        :param code: a string of Javascript code to execute
+        :param fields: fields that you will be using in your function, which
+            will be passed in to your function as arguments
+        :param options: options that you want available to the function 
+            (accessed in Javascript through the ``options`` object)
         """
         fields = [QuerySet._translate_field_name(self._document, f)
                   for f in fields]
@@ -252,6 +494,9 @@ class QuerySet(object):
 
     def sum(self, field):
         """Sum over the values of the specified field.
+
+        :param field: the field to sum over; use dot-notation to refer to
+            embedded document fields
         """
         sum_func = """
             function(sumField) {
@@ -266,6 +511,9 @@ class QuerySet(object):
 
     def average(self, field):
         """Average over the values of the specified field.
+
+        :param field: the field to average over; use dot-notation to refer to
+            embedded document fields
         """
         average_func = """
             function(averageField) {
@@ -286,6 +534,9 @@ class QuerySet(object):
         """Returns a dictionary of all items present in a list field across
         the whole queried set of documents, and their corresponding frequency.
         This is useful for generating tag clouds, or searching documents. 
+
+        :param list_field: the list field to use
+        :param normalize: normalize the results so they add to 1.0
         """
         freq_func = """
             function(listField) {
@@ -311,6 +562,11 @@ class QuerySet(object):
         """
         return self.exec_js(freq_func, list_field, normalize=normalize)
 
+    def __repr__(self):
+        data = list(self[:REPR_OUTPUT_SIZE + 1])
+        if len(data) > REPR_OUTPUT_SIZE:
+            data[-1] = "...(remaining elements truncated)..."
+        return repr(data)
 
 class InvalidCollectionError(Exception):
     pass

tests/document.py

@@ -225,7 +225,7 @@ class DocumentTest(unittest.TestCase):
         """Ensure that indexes are used when meta[indexes] is specified.
         """
         class BlogPost(Document):
-            date = DateTimeField(default=datetime.datetime.now)
+            date = DateTimeField(name='addDate', default=datetime.datetime.now)
             category = StringField()
             meta = {
                 'indexes': [
@@ -237,16 +237,102 @@ class DocumentTest(unittest.TestCase):
         BlogPost.drop_collection()
 
         info = BlogPost.objects._collection.index_information()
-        self.assertEqual(len(info), 0)
+        self.assertEqual(len(info), 4) # _id, types, '-date', ('cat', 'date')
 
-        BlogPost.objects()
+        # Indexes are lazy so use list() to perform query
+        list(BlogPost.objects)
         info = BlogPost.objects._collection.index_information()
-        self.assertTrue([('category', 1), ('date', -1)] in info.values())
-        # Even though descending order was specified, single-key indexes use 1
-        self.assertTrue([('date', 1)] in info.values())
+        self.assertTrue([('_types', 1), ('category', 1), ('addDate', -1)] 
+                        in info.values())
+        self.assertTrue([('_types', 1), ('addDate', -1)] in info.values())
+        
+        class ExtendedBlogPost(BlogPost):
+            title = StringField()
+            meta = {'indexes': ['title']}
 
         BlogPost.drop_collection()
 
+        list(ExtendedBlogPost.objects)
+        info = ExtendedBlogPost.objects._collection.index_information()
+        self.assertTrue([('_types', 1), ('category', 1), ('addDate', -1)] 
+                        in info.values())
+        self.assertTrue([('_types', 1), ('addDate', -1)] in info.values())
+        self.assertTrue([('_types', 1), ('title', 1)] in info.values())
+
+        BlogPost.drop_collection()
+
+    def test_unique(self):
+        """Ensure that uniqueness constraints are applied to fields.
+        """
+        class BlogPost(Document):
+            title = StringField()
+            slug = StringField(unique=True)
+
+        BlogPost.drop_collection()
+
+        post1 = BlogPost(title='test1', slug='test')
+        post1.save()
+
+        # Two posts with the same slug is not allowed
+        post2 = BlogPost(title='test2', slug='test')
+        self.assertRaises(OperationError, post2.save)
+
+        class Date(EmbeddedDocument):
+            year = IntField(name='yr')
+
+        class BlogPost(Document):
+            title = StringField()
+            date = EmbeddedDocumentField(Date)
+            slug = StringField(unique_with='date.year')
+
+        BlogPost.drop_collection()
+
+        post1 = BlogPost(title='test1', date=Date(year=2009), slug='test')
+        post1.save()
+
+        # day is different so won't raise exception
+        post2 = BlogPost(title='test2', date=Date(year=2010), slug='test')
+        post2.save()
+
+        # Now there will be two docs with the same slug and the same day: fail
+        post3 = BlogPost(title='test3', date=Date(year=2010), slug='test')
+        self.assertRaises(OperationError, post3.save)
+
+        BlogPost.drop_collection()
+
+    def test_custom_id_field(self):
+        """Ensure that documents may be created with custom primary keys.
+        """
+        class User(Document):
+            username = StringField(primary_key=True)
+            name = StringField()
+
+        User.drop_collection()
+
+        self.assertEqual(User._fields['username'].name, '_id')
+        self.assertEqual(User._meta['id_field'], 'username')
+
+        def create_invalid_user():
+            User(name='test').save() # no primary key field
+        self.assertRaises(ValidationError, create_invalid_user)
+
+        def define_invalid_user():
+            class EmailUser(User):
+                email = StringField(primary_key=True)
+        self.assertRaises(ValueError, define_invalid_user)
+        
+        user = User(username='test', name='test user')
+        user.save()
+
+        user_obj = User.objects.first()
+        self.assertEqual(user_obj.id, 'test')
+
+        user_son = User.objects._collection.find_one()
+        self.assertEqual(user_son['_id'], 'test')
+        self.assertTrue('username' not in user_son['_id'])
+        
+        User.drop_collection()
+
     def test_creation(self):
         """Ensure that document may be created using keyword arguments.
         """

tests/fields.py

@@ -231,6 +231,9 @@ class FieldTest(unittest.TestCase):
             content = StringField()
             author = ReferenceField(User)
 
+        User.drop_collection()
+        BlogPost.drop_collection()
+
         self.assertRaises(ValidationError, ReferenceField, EmbeddedDocument)
 
         user = User(name='Test User')

tests/queryset.py

@@ -1,5 +1,6 @@
 import unittest
 import pymongo
+from datetime import datetime
 
 from mongoengine.queryset import QuerySet
 from mongoengine import *
@@ -16,7 +17,7 @@ class QuerySetTest(unittest.TestCase):
         self.Person = Person
 
     def test_initialisation(self):
-        """Ensure that CollectionManager is correctly initialised.
+        """Ensure that a QuerySet is correctly initialised by QuerySetManager.
         """
         self.assertTrue(isinstance(self.Person.objects, QuerySet))
         self.assertEqual(self.Person.objects._collection.name(), 
@@ -48,6 +49,9 @@ class QuerySetTest(unittest.TestCase):
         person2 = self.Person(name="User B", age=30)
         person2.save()
 
+        q1 = Q(name='test')
+        q2 = Q(age__gte=18)
+
         # Find all people in the collection
         people = self.Person.objects
         self.assertEqual(len(people), 2)
@@ -131,6 +135,41 @@ class QuerySetTest(unittest.TestCase):
         person = self.Person.objects.with_id(person1.id)
         self.assertEqual(person.name, "User A")
 
+    def test_ordering(self):
+        """Ensure default ordering is applied and can be overridden.
+        """
+        class BlogPost(Document):
+            title = StringField()
+            published_date = DateTimeField()
+
+            meta = {
+                'ordering': ['-published_date']
+            }
+
+        BlogPost.drop_collection()
+
+        blog_post_1 = BlogPost(title="Blog Post #1", 
+                               published_date=datetime(2010, 1, 5, 0, 0 ,0))
+        blog_post_2 = BlogPost(title="Blog Post #2", 
+                               published_date=datetime(2010, 1, 6, 0, 0 ,0))
+        blog_post_3 = BlogPost(title="Blog Post #3", 
+                               published_date=datetime(2010, 1, 7, 0, 0 ,0))
+
+        blog_post_1.save()
+        blog_post_2.save()
+        blog_post_3.save()
+        
+        # get the "first" BlogPost using default ordering
+        # from BlogPost.meta.ordering
+        latest_post = BlogPost.objects.first() 
+        self.assertEqual(latest_post.title, "Blog Post #3")
+        
+        # override default ordering, order BlogPosts by "published_date"
+        first_post = BlogPost.objects.order_by("+published_date").first()
+        self.assertEqual(first_post.title, "Blog Post #1")
+
+        BlogPost.drop_collection()
+
     def test_find_embedded(self):
         """Ensure that an embedded document is properly returned from a query.
         """
@@ -141,6 +180,8 @@ class QuerySetTest(unittest.TestCase):
             content = StringField()
             author = EmbeddedDocumentField(User)
 
+        BlogPost.drop_collection()
+
         post = BlogPost(content='Had a good coffee today...')
         post.author = User(name='Test User')
         post.save()
@@ -151,6 +192,42 @@ class QuerySetTest(unittest.TestCase):
         
         BlogPost.drop_collection()
 
+    def test_q(self):
+        class BlogPost(Document):
+            publish_date = DateTimeField()
+            published = BooleanField()
+
+        BlogPost.drop_collection()
+
+        post1 = BlogPost(publish_date=datetime(2010, 1, 8), published=False)
+        post1.save()
+
+        post2 = BlogPost(publish_date=datetime(2010, 1, 15), published=True)
+        post2.save()
+
+        post3 = BlogPost(published=True)
+        post3.save()
+
+        post4 = BlogPost(publish_date=datetime(2010, 1, 8))
+        post4.save()
+
+        post5 = BlogPost(publish_date=datetime(2010, 1, 15))
+        post5.save()
+
+        post6 = BlogPost(published=False)
+        post6.save()
+
+        date = datetime(2010, 1, 10)
+        q = BlogPost.objects(Q(publish_date__lte=date) | Q(published=True))
+        posts = [post.id for post in q]
+
+        published_posts = (post1, post2, post3, post4)
+        self.assertTrue(all(obj.id in posts for obj in published_posts))
+
+        self.assertFalse(any(obj.id in posts for obj in [post5, post6]))
+
+        BlogPost.drop_collection()
+
     def test_delete(self):
         """Ensure that documents are properly deleted from the database.
         """
@@ -166,6 +243,41 @@ class QuerySetTest(unittest.TestCase):
         self.Person.objects.delete()
         self.assertEqual(len(self.Person.objects), 0)
 
+    def test_update(self):
+        """Ensure that atomic updates work properly.
+        """
+        class BlogPost(Document):
+            title = StringField()
+            hits = IntField()
+            tags = ListField(StringField())
+
+        BlogPost.drop_collection()
+
+        post = BlogPost(name="Test Post", hits=5, tags=['test'])
+        post.save()
+
+        BlogPost.objects.update(set__hits=10)
+        post.reload()
+        self.assertEqual(post.hits, 10)
+
+        BlogPost.objects.update_one(inc__hits=1)
+        post.reload()
+        self.assertEqual(post.hits, 11)
+
+        BlogPost.objects.update_one(dec__hits=1)
+        post.reload()
+        self.assertEqual(post.hits, 10)
+
+        BlogPost.objects.update(push__tags='mongo')
+        post.reload()
+        self.assertTrue('mongo' in post.tags)
+
+        BlogPost.objects.update_one(push_all__tags=['db', 'nosql'])
+        post.reload()
+        self.assertTrue('db' in post.tags and 'nosql' in post.tags)
+
+        BlogPost.drop_collection()
+
     def test_order_by(self):
         """Ensure that QuerySets may be ordered.
         """
@@ -326,9 +438,69 @@ class QuerySetTest(unittest.TestCase):
 
         BlogPost.drop_collection()
 
+
+    def test_types_index(self):
+        """Ensure that and index is used when '_types' is being used in a
+        query.
+        """
+        class BlogPost(Document):
+            date = DateTimeField()
+            meta = {'indexes': ['-date']}
+
+        # Indexes are lazy so use list() to perform query
+        list(BlogPost.objects)
+        info = BlogPost.objects._collection.index_information()
+        self.assertTrue([('_types', 1)] in info.values())
+        self.assertTrue([('_types', 1), ('date', -1)] in info.values())
+
+        BlogPost.drop_collection()
+
+        class BlogPost(Document):
+            title = StringField()
+            meta = {'allow_inheritance': False}
+
+        # _types is not used on objects where allow_inheritance is False
+        list(BlogPost.objects)
+        info = BlogPost.objects._collection.index_information()
+        self.assertFalse([('_types', 1)] in info.values())
+
+        BlogPost.drop_collection()
+
     def tearDown(self):
         self.Person.drop_collection()
 
 
+class QTest(unittest.TestCase):
+    
+    def test_or_and(self):
+        q1 = Q(name='test')
+        q2 = Q(age__gte=18)
+
+        query = ['(', {'name': 'test'}, '||', {'age__gte': 18}, ')']
+        self.assertEqual((q1 | q2).query, query)
+
+        query = ['(', {'name': 'test'}, '&&', {'age__gte': 18}, ')']
+        self.assertEqual((q1 & q2).query, query)
+
+        query = ['(', '(', {'name': 'test'}, '&&', {'age__gte': 18}, ')', '||',
+                 {'name': 'example'}, ')']
+        self.assertEqual((q1 & q2 | Q(name='example')).query, query)
+
+    def test_item_query_as_js(self):
+        """Ensure that the _item_query_as_js utilitiy method works properly.
+        """
+        q = Q()
+        examples = [
+            ({'name': 'test'}, 'this.name == i0f0', {'i0f0': 'test'}),
+            ({'age': {'$gt': 18}}, 'this.age > i0f0o0', {'i0f0o0': 18}),
+            ({'name': 'test', 'age': {'$gt': 18, '$lte': 65}}, 
+             'this.age <= i0f0o0 && this.age > i0f0o1 && this.name == i0f1', 
+             {'i0f0o0': 65, 'i0f0o1': 18, 'i0f1': 'test'}),
+        ]
+        for item, js, scope in examples:
+            test_scope = {}
+            self.assertEqual(q._item_query_as_js(item, test_scope, 0), js)
+            self.assertEqual(scope, test_scope)
+
 if __name__ == '__main__':
     unittest.main()