Bump to v0.2

_types index prepended to user defined indexes

Conflicts:
	mongoengine/queryset.py

docs/changelog.rst

@@ -2,6 +2,14 @@
 Changelog
 =========
 
+Changes is v0.1.3
+=================
+- Added Django authentication backend
+- Added Document.meta support for indexes, which are ensured just before 
+  querying takes place
+- A few minor bugfixes
+
+
 Changes in v0.1.2
 =================
 - Query values may be processed before before being used in queries

docs/django.rst

@@ -0,0 +1,29 @@
+=============================
+Using MongoEngine with Django
+=============================
+
+Connecting
+==========
+In your **settings.py** file, ignore the standard database settings (unless you
+also plan to use the ORM in your project), and instead call 
+:func:`~mongoengine.connect` somewhere in the settings module.
+
+Authentication
+==============
+MongoEngine includes a Django authentication backend, which uses MongoDB. The
+:class:`~mongoengine.django.auth.User` model is a MongoEngine 
+:class:`~mongoengine.Document`, but implements most of the methods and 
+attributes that the standard Django :class:`User` model does - so the two are
+moderately compatible. Using this backend will allow you to store users in 
+MongoDB but still use many of the Django authentication infrastucture (such as
+the :func:`login_required` decorator and the :func:`authenticate` function). To
+enable the MongoEngine auth backend, add the following to you **settings.py**
+file::
+    
+    AUTHENTICATION_BACKENDS = (
+        'mongoengine.django.auth.MongoEngineBackend',
+    )
+
+The :mod:`~mongoengine.django.auth` module also contains a 
+:func:`~mongoengine.django.auth.get_user` helper function, that takes a user's
+:attr:`id` and returns a :class:`~mongoengine.django.auth.User` object.

docs/index.rst

@@ -16,6 +16,7 @@ The source is available on `GitHub <http://github.com/hmarr/mongoengine>`_.
    tutorial
    userguide
    apireference
+   django
    changelog
 
 Indices and tables

docs/userguide.rst

@@ -138,6 +138,21 @@ field::
 The :class:`User` object is automatically turned into a reference behind the
 scenes, and dereferenced when the :class:`Page` object is retrieved.
 
+Uniqueness constraints
+^^^^^^^^^^^^^^^^^^^^^^
+MongoEngine allows you to specify that a field should be unique across a
+collection by providing ``unique=True`` to a :class:`~mongoengine.Field`\ 's
+constructor. If you try to save a document that has the same value for a unique
+field as a document that is already in the database, a 
+:class:`~mongoengine.OperationError` will be raised. You may also specify
+multi-field uniqueness constraints by using :attr:`unique_with`, which may be
+either a single field name, or a list or tuple of field names::
+
+    class User(Document):
+        username = StringField(unique=True)
+        first_name = StringField()
+        last_name = StringField(unique_with='last_name')
+
 Document collections
 --------------------
 Document classes that inherit **directly** from :class:`~mongoengine.Document`
@@ -168,6 +183,62 @@ The following example shows a :class:`Log` document that will be limited to
         ip_address = StringField()
         meta = {'max_documents': 1000, 'max_size': 2000000}
 
+Indexes
+-------
+You can specify indexes on collections to make querying faster. This is done
+by creating a list of index specifications called :attr:`indexes` in the
+:attr:`~mongoengine.Document.meta` dictionary, where an index specification may
+either be a single field name, or a tuple containing multiple field names. A
+direction may be specified on fields by prefixing the field name with a **+**
+or a **-** sign. Note that direction only matters on multi-field indexes. ::
+
+    class Page(Document):
+        title = StringField()
+        rating = StringField()
+        meta = {
+            'indexes': ['title', ('title', '-rating')]
+        }
+        
+Ordering
+--------
+A default ordering can be specified for your
+:class:`~mongoengine.queryset.QuerySet` using the :attr:`ordering` attribute of
+:attr:`~mongoengine.Document.meta`.  Ordering will be applied when the
+:class:`~mongoengine.queryset.QuerySet` is created, and can be overridden by
+subsequent calls to :meth:`~mongoengine.queryset.QuerySet.order_by`. ::
+
+    from datetime import datetime
+
+    class BlogPost(Document):
+        title = StringField()
+        published_date = DateTimeField()
+
+        meta = {
+            'ordering': ['-published_date']
+        }
+
+    blog_post_1 = BlogPost(title="Blog Post #1")
+    blog_post_1.published_date = datetime(2010, 1, 5, 0, 0 ,0))
+
+    blog_post_2 = BlogPost(title="Blog Post #2") 
+    blog_post_2.published_date = datetime(2010, 1, 6, 0, 0 ,0))
+
+    blog_post_3 = BlogPost(title="Blog Post #3")
+    blog_post_3.published_date = datetime(2010, 1, 7, 0, 0 ,0))
+
+    blog_post_1.save()
+    blog_post_2.save()
+    blog_post_3.save()
+
+    # get the "first" BlogPost using default ordering
+    # from BlogPost.meta.ordering
+    latest_post = BlogPost.objects.first() 
+    self.assertEqual(latest_post.title, "Blog Post #3")
+
+    # override default ordering, order BlogPosts by "published_date"
+    first_post = BlogPost.objects.order_by("+published_date").first()
+    self.assertEqual(first_post.title, "Blog Post #1")
+
 Document inheritance
 --------------------
 To create a specialised type of a :class:`~mongoengine.Document` you have
@@ -383,3 +454,64 @@ would be generating "tag-clouds"::
     from operator import itemgetter
     top_tags = sorted(tag_freqs.items(), key=itemgetter(1), reverse=True)[:10]
 
+Advanced queries
+----------------
+Sometimes calling a :class:`~mongoengine.queryset.QuerySet` object with keyword
+arguments can't fully express the query you want to use -- for example if you
+need to combine a number of constraints using *and* and *or*. This is made 
+possible in MongoEngine through the :class:`~mongoengine.queryset.Q` class.
+A :class:`~mongoengine.queryset.Q` object represents part of a query, and
+can be initialised using the same keyword-argument syntax you use to query
+documents. To build a complex query, you may combine 
+:class:`~mongoengine.queryset.Q` objects using the ``&`` (and) and ``|`` (or)
+operators. To use :class:`~mongoengine.queryset.Q` objects, pass them in
+as positional arguments to :attr:`Document.objects` when you filter it by
+calling it with keyword arguments::
+
+    # Get published posts
+    Post.objects(Q(published=True) | Q(publish_date__lte=datetime.now()))
+
+    # Get top posts
+    Post.objects((Q(featured=True) & Q(hits__gte=1000)) | Q(hits__gte=5000))
+
+.. warning::
+   Only use these advanced queries if absolutely necessary as they will execute
+   significantly slower than regular queries. This is because they are not
+   natively supported by MongoDB -- they are compiled to Javascript and sent
+   to the server for execution.
+
+Atomic updates
+--------------
+Documents may be updated atomically by using the
+:meth:`~mongoengine.queryset.QuerySet.update_one` and
+:meth:`~mongoengine.queryset.QuerySet.update` methods on a 
+:meth:`~mongoengine.queryset.QuerySet`. There are several different "modifiers"
+that you may use with these methods:
+
+* ``set`` -- set a particular value
+* ``unset`` -- delete a particular value (since MongoDB v1.3+)
+* ``inc`` -- increment a value by a given amount
+* ``dec`` -- decrement a value by a given amount
+* ``push`` -- append a value to a list
+* ``push_all`` -- append several values to a list
+* ``pull`` -- remove a value from a list
+* ``pull_all`` -- remove several values from a list
+
+The syntax for atomic updates is similar to the querying syntax, but the 
+modifier comes before the field, not after it::
+    
+    >>> post = BlogPost(title='Test', page_views=0, tags=['database'])
+    >>> post.save()
+    >>> BlogPost.objects(id=post.id).update_one(inc__page_views=1)
+    >>> post.reload()  # the document has been changed, so we need to reload it
+    >>> post.page_views
+    1
+    >>> BlogPost.objects(id=post.id).update_one(set__title='Example Post')
+    >>> post.reload()
+    >>> post.title
+    'Example Post'
+    >>> BlogPost.objects(id=post.id).update_one(push__tags='nosql')
+    >>> post.reload()
+    >>> post.tags
+    ['database', 'nosql']
+

mongoengine/__init__.py

@@ -12,7 +12,7 @@ __all__ = (document.__all__ + fields.__all__ + connection.__all__ +
 
 __author__ = 'Harry Marr'
 
-VERSION = (0, 1, 2)
+VERSION = (0, 2, 0)
 
 def get_version():
     version = '%s.%s' % (VERSION[0], VERSION[1])

mongoengine/base.py

@@ -1,4 +1,4 @@
-from queryset import QuerySetManager
+from queryset import QuerySet, QuerySetManager
 
 import pymongo
 
@@ -12,10 +12,13 @@ class BaseField(object):
     may be added to subclasses of `Document` to define a document's schema.
     """
     
-    def __init__(self, name=None, required=False, default=None):
+    def __init__(self, name=None, required=False, default=None, unique=False,
+                 unique_with=None):
         self.name = name
         self.required = required
         self.default = default
+        self.unique = bool(unique or unique_with)
+        self.unique_with = unique_with
 
     def __get__(self, instance, owner):
         """Descriptor for retrieving a value from a field in a document. Do 
@@ -136,6 +139,7 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
         collection = name.lower()
         
         simple_class = True
+
         # Subclassed documents inherit collection from superclass
         for base in bases:
             if hasattr(base, '_meta') and 'collection' in base._meta:
@@ -154,8 +158,13 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
             'allow_inheritance': True,
             'max_documents': None,
             'max_size': None,
+            'ordering': [], # default ordering applied at runtime
+            'indexes': [] # indexes to be ensured at runtime
         }
+
+        # Apply document-defined meta options
         meta.update(attrs.get('meta', {}))
+        
         # Only simple classes - direct subclasses of Document - may set
         # allow_inheritance to False
         if not simple_class and not meta['allow_inheritance']:
@@ -170,6 +179,35 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
         new_class = super_new(cls, name, bases, attrs)
         new_class.objects = QuerySetManager()
 
+        # Generate a list of indexes needed by uniqueness constraints
+        unique_indexes = []
+        for field_name, field in new_class._fields.items():
+            if field.unique:
+                field.required = True
+                unique_fields = [field_name]
+
+                # Add any unique_with fields to the back of the index spec
+                if field.unique_with:
+                    if isinstance(field.unique_with, basestring):
+                        field.unique_with = [field.unique_with]
+
+                    # Convert unique_with field names to real field names
+                    unique_with = []
+                    for other_name in field.unique_with:
+                        parts = other_name.split('.')
+                        # Lookup real name
+                        parts = QuerySet._lookup_field(new_class, parts)
+                        name_parts = [part.name for part in parts]
+                        unique_with.append('.'.join(name_parts))
+                        # Unique field should be required
+                        parts[-1].required = True
+                    unique_fields += unique_with
+
+                # Add the new index to the list
+                index = [(field, pymongo.ASCENDING) for field in unique_fields]
+                unique_indexes.append(index)
+        new_class._meta['unique_indexes'] = unique_indexes
+
         return new_class
 
 
@@ -202,9 +240,7 @@ class BaseDocument(object):
         return all_subclasses
 
     def __iter__(self):
-        # Use _data rather than _fields as iterator only looks at names so
-        # values don't need to be converted to Python types
-        return iter(self._data)
+        return iter(self._fields)
 
     def __getitem__(self, name):
         """Dictionary-style field access, return a field's value if present.
@@ -234,6 +270,18 @@ class BaseDocument(object):
     def __len__(self):
         return len(self._data)
 
+    def __repr__(self):
+        try:
+            u = unicode(self)
+        except (UnicodeEncodeError, UnicodeDecodeError):
+            u = '[Bad Unicode data]'
+        return u'<%s: %s>' % (self.__class__.__name__, u)
+
+    def __str__(self):
+        if hasattr(self, '__unicode__'):
+            return unicode(self).encode('utf-8')
+        return '%s object' % self.__class__.__name__
+
     def to_mongo(self):
         """Return data dictionary ready for use with MongoDB.
         """

mongoengine/django/auth.py

@@ -0,0 +1,99 @@
+from mongoengine import *
+
+from django.utils.hashcompat import md5_constructor, sha_constructor
+from django.utils.encoding import smart_str
+from django.contrib.auth.models import AnonymousUser
+
+import datetime
+
+REDIRECT_FIELD_NAME = 'next'
+
+def get_hexdigest(algorithm, salt, raw_password):
+    raw_password, salt = smart_str(raw_password), smart_str(salt)
+    if algorithm == 'md5':
+        return md5_constructor(salt + raw_password).hexdigest()
+    elif algorithm == 'sha1':
+        return sha_constructor(salt + raw_password).hexdigest()
+    raise ValueError('Got unknown password algorithm type in password')
+
+
+class User(Document):
+    """A User document that aims to mirror most of the API specified by Django
+    at http://docs.djangoproject.com/en/dev/topics/auth/#users
+    """
+    username = StringField(max_length=30, required=True)
+    first_name = StringField(max_length=30)
+    last_name = StringField(max_length=30)
+    email = StringField()
+    password = StringField(max_length=128)
+    is_staff = BooleanField(default=False)
+    is_active = BooleanField(default=True)
+    is_superuser = BooleanField(default=False)
+    last_login = DateTimeField(default=datetime.datetime.now)
+
+    def get_full_name(self):
+        """Returns the users first and last names, separated by a space.
+        """
+        full_name = u'%s %s' % (self.first_name or '', self.last_name or '')
+        return full_name.strip()
+
+    def is_anonymous(self):
+        return False
+
+    def is_authenticated(self):
+        return True
+
+    def set_password(self, raw_password):
+        """Sets the user's password - always use this rather than directly
+        assigning to :attr:`~mongoengine.django.auth.User.password` as the
+        password is hashed before storage.
+        """
+        from random import random
+        algo = 'sha1'
+        salt = get_hexdigest(algo, str(random()), str(random()))[:5]
+        hash = get_hexdigest(algo, salt, raw_password)
+        self.password = '%s$%s$%s' % (algo, salt, hash)
+
+    def check_password(self, raw_password):
+        """Checks the user's password against a provided password - always use
+        this rather than directly comparing to
+        :attr:`~mongoengine.django.auth.User.password` as the password is
+        hashed before storage.
+        """
+        algo, salt, hash = self.password.split('$')
+        return hash == get_hexdigest(algo, salt, raw_password)
+
+    @classmethod
+    def create_user(cls, username, password, email=None):
+        """Create (and save) a new user with the given username, password and
+        email address.
+        """
+        user = User(username=username, email=email)
+        user.set_password(password)
+        user.save()
+        return user
+
+
+class MongoEngineBackend(object):
+    """Authenticate using MongoEngine and mongoengine.django.auth.User.
+    """
+
+    def authenticate(self, username=None, password=None):
+        user = User.objects(username=username).first()
+        if user:
+            if password and user.check_password(password):
+                return user
+        return None
+
+    def get_user(self, user_id):
+        return User.objects.with_id(user_id)
+
+
+def get_user(userid):
+    """Returns a User object from an id (User.id). Django's equivalent takes
+    request, but taking an id instead leaves it up to the developer to store
+    the id in any way they want (session, signed cookie, etc.)
+    """
+    if not userid:
+        return AnonymousUser()
+    return MongoEngineBackend().get_user(userid) or AnonymousUser()

mongoengine/document.py

@@ -1,9 +1,12 @@
 from base import (DocumentMetaclass, TopLevelDocumentMetaclass, BaseDocument,
                   ValidationError)
+from queryset import OperationError
 from connection import _get_db
 
+import pymongo
 
-__all__ = ['Document', 'EmbeddedDocument']
+
+__all__ = ['Document', 'EmbeddedDocument', 'ValidationError', 'OperationError']
 
 
 class EmbeddedDocument(BaseDocument):
@@ -44,17 +47,27 @@ class Document(BaseDocument):
     maximum size of the collection in bytes. If :attr:`max_size` is not 
     specified and :attr:`max_documents` is, :attr:`max_size` defaults to 
     10000000 bytes (10MB).
+
+    Indexes may be created by specifying :attr:`indexes` in the :attr:`meta`
+    dictionary. The value should be a list of field names or tuples of field 
+    names. Index direction may be specified by prefixing the field names with
+    a **+** or **-** sign.
     """
 
     __metaclass__ = TopLevelDocumentMetaclass
 
-    def save(self):
+    def save(self, safe=True):
         """Save the :class:`~mongoengine.Document` to the database. If the
         document already exists, it will be updated, otherwise it will be
         created.
         """
         self.validate()
-        object_id = self.__class__.objects._collection.save(self.to_mongo())
+        doc = self.to_mongo()
+        try:
+            object_id = self.__class__.objects._collection.save(doc, safe=safe)
+        except pymongo.errors.OperationFailure, err:
+            raise OperationError('Tried to save duplicate unique keys (%s)'
+                                  % str(err))
         self.id = self._fields['id'].to_python(object_id)
 
     def delete(self):
@@ -67,11 +80,9 @@ class Document(BaseDocument):
     def reload(self):
         """Reloads all attributes from the database.
         """
-        #object_id = self._fields['id'].to_mongo(self.id)
-        #obj = self.__class__.objects(id=object_id).first()
         obj = self.__class__.objects(id=self.id).first()
         for field in self._fields:
-            setattr(self, field, getattr(obj, field))
+            setattr(self, field, obj[field])
 
     def validate(self):
         """Ensure that all fields' values are valid and that required fields

mongoengine/queryset.py

@@ -1,15 +1,106 @@
 from connection import _get_db
 
 import pymongo
+import copy
 
 
-__all__ = ['queryset_manager', 'InvalidQueryError', 'InvalidCollectionError']
+__all__ = ['queryset_manager', 'Q', 'InvalidQueryError', 
+           'InvalidCollectionError']
+
+# The maximum number of items to display in a QuerySet.__repr__
+REPR_OUTPUT_SIZE = 20
 
 
 class InvalidQueryError(Exception):
     pass
 
 
+class OperationError(Exception):
+    pass
+
+
+class Q(object):
+    
+    OR = '||'
+    AND = '&&'
+    OPERATORS = {
+        'eq': 'this.%(field)s == %(value)s',
+        'neq': 'this.%(field)s != %(value)s',
+        'gt': 'this.%(field)s > %(value)s',
+        'gte': 'this.%(field)s >= %(value)s',
+        'lt': 'this.%(field)s < %(value)s',
+        'lte': 'this.%(field)s <= %(value)s',
+        'lte': 'this.%(field)s <= %(value)s',
+        'in': 'this.%(field)s.indexOf(%(value)s) != -1',
+        'nin': 'this.%(field)s.indexOf(%(value)s) == -1',
+        'mod': '%(field)s %% %(value)s',
+        'all': ('%(value)s.every(function(a){'
+                'return this.%(field)s.indexOf(a) != -1 })'),
+        'size': 'this.%(field)s.length == %(value)s',
+        'exists': 'this.%(field)s != null',
+    }
+    
+    def __init__(self, **query):
+        self.query = [query]
+
+    def _combine(self, other, op):
+        obj = Q()
+        obj.query = ['('] + copy.deepcopy(self.query) + [op]
+        obj.query += copy.deepcopy(other.query) + [')']
+        return obj
+
+    def __or__(self, other):
+        return self._combine(other, self.OR)
+
+    def __and__(self, other):
+        return self._combine(other, self.AND)
+
+    def as_js(self, document):
+        js = []
+        js_scope = {}
+        for i, item in enumerate(self.query):
+            if isinstance(item, dict):
+                item_query = QuerySet._transform_query(document, **item)
+                # item_query will values will either be a value or a dict
+                js.append(self._item_query_as_js(item_query, js_scope, i))
+            else:
+                js.append(item)
+        return pymongo.code.Code(' '.join(js), js_scope)
+
+    def _item_query_as_js(self, item_query, js_scope, item_num):
+        # item_query will be in one of the following forms
+        #    {'age': 25, 'name': 'Test'}
+        #    {'age': {'$lt': 25}, 'name': {'$in': ['Test', 'Example']}
+        #    {'age': {'$lt': 25, '$gt': 18}}
+        js = []
+        for i, (key, value) in enumerate(item_query.items()):
+            op = 'eq'
+            # Construct a variable name for the value in the JS
+            value_name = 'i%sf%s' % (item_num, i)
+            if isinstance(value, dict):
+                # Multiple operators for this field
+                for j, (op, value) in enumerate(value.items()):
+                    # Create a custom variable name for this operator
+                    op_value_name = '%so%s' % (value_name, j)
+                    # Update the js scope with the value for this op
+                    js_scope[op_value_name] = value
+                    # Construct the JS that uses this op
+                    operation_js = Q.OPERATORS[op.strip('$')] % {
+                        'field': key, 
+                        'value': op_value_name
+                    }
+                    js.append(operation_js)
+            else:
+                js_scope[value_name] = value
+                # Construct the JS for this field
+                field_js = Q.OPERATORS[op.strip('$')] % {
+                    'field': key, 
+                    'value': value_name
+                }
+                js.append(field_js)
+        return ' && '.join(js)
+
+
 class QuerySet(object):
     """A set of results returned from a query. Wraps a MongoDB cursor, 
     providing :class:`~mongoengine.Document` objects as the results.
@@ -17,39 +108,85 @@ class QuerySet(object):
     
     def __init__(self, document, collection):
         self._document = document
-        self._collection = collection
+        self._collection_obj = collection
+        self._accessed_collection = False
         self._query = {}
+        self._where_clauses = []
+
         # If inheritance is allowed, only return instances and instances of
         # subclasses of the class being used
         if document._meta.get('allow_inheritance'):
             self._query = {'_types': self._document._class_name}
         self._cursor_obj = None
         
-    def ensure_index(self, key_or_list, direction=None):
+    def ensure_index(self, key_or_list):
         """Ensure that the given indexes are in place.
         """
         if isinstance(key_or_list, basestring):
-            # single-field indexes needn't specify a direction
-            if key_or_list.startswith("-"):
-                key_or_list = key_or_list[1:]
-            self._collection.ensure_index(key_or_list)
-        elif isinstance(key_or_list, (list, tuple)):
-            print key_or_list
-            self._collection.ensure_index(key_or_list)
+            key_or_list = [key_or_list]
+
+        index_list = []
+        # If _types is being used, prepend it to every specified index
+        if self._document._meta.get('allow_inheritance'):
+            index_list.append(('_types', 1))
+
+        for key in key_or_list:
+            # Get direction from + or -
+            direction = pymongo.ASCENDING
+            if key.startswith("-"):
+                direction = pymongo.DESCENDING
+            if key.startswith(("+", "-")):
+                    key = key[1:]
+            # Use real field name
+            key = QuerySet._translate_field_name(self._document, key)
+            index_list.append((key, direction))
+        self._collection.ensure_index(index_list)
         return self
 
-    def __call__(self, **query):
+    def __call__(self, *q_objs, **query):
         """Filter the selected documents by calling the 
         :class:`~mongoengine.QuerySet` with a query.
         """
+        for q in q_objs:
+            self._where_clauses.append(q.as_js(self._document))
         query = QuerySet._transform_query(_doc_cls=self._document, **query)
         self._query.update(query)
         return self
 
+    @property
+    def _collection(self):
+        """Property that returns the collection object. This allows us to
+        perform operations only if the collection is accessed.
+        """
+        if not self._accessed_collection:
+            self._accessed_collection = True
+
+            # Ensure document-defined indexes are created
+            if self._document._meta['indexes']:
+                for key_or_list in self._document._meta['indexes']:
+                    self.ensure_index(key_or_list)
+
+            # Ensure indexes created by uniqueness constraints
+            for index in self._document._meta['unique_indexes']:
+                self._collection.ensure_index(index, unique=True)
+
+            # If _types is being used (for polymorphism), it needs an index
+            if '_types' in self._query:
+                self._collection.ensure_index('_types')
+        return self._collection_obj
+
     @property
     def _cursor(self):
         if not self._cursor_obj:
             self._cursor_obj = self._collection.find(self._query)
+            # Apply where clauses to cursor
+            for js in self._where_clauses:
+                self._cursor_obj.where(js)
+            
+            # apply default ordering
+            if self._document._meta['ordering']:
+                self.order_by(*self._document._meta['ordering'])
+            
         return self._cursor_obj
 
     @classmethod
@@ -75,10 +212,12 @@ class QuerySet(object):
         return fields
 
     @classmethod
-    def _translate_field_name(cls, doc_cls, parts):
+    def _translate_field_name(cls, doc_cls, field, sep='.'):
         """Translate a field attribute name to a database field name.
         """
-        return [field.name for field in QuerySet._lookup_field(doc_cls, parts)]
+        parts = field.split(sep)
+        parts = [f.name for f in QuerySet._lookup_field(doc_cls, parts)]
+        return '.'.join(parts)
 
     @classmethod
     def _transform_query(cls, _doc_cls=None, **query):
@@ -132,7 +271,7 @@ class QuerySet(object):
         """Retrieve the object matching the id provided.
         """
         if not isinstance(object_id, pymongo.objectid.ObjectId):
-            object_id = pymongo.objectid.ObjectId(object_id)
+            object_id = pymongo.objectid.ObjectId(str(object_id))
 
         result = self._collection.find_one(object_id)
         if result is not None:
@@ -200,6 +339,7 @@ class QuerySet(object):
         """Return an explain plan record for the 
         :class:`~mongoengine.queryset.QuerySet`\ 's cursor.
         """
+
         plan = self._cursor.explain()
         if format:
             import pprint
@@ -211,6 +351,86 @@ class QuerySet(object):
         """
         self._collection.remove(self._query)
 
+    @classmethod
+    def _transform_update(cls, _doc_cls=None, **update):
+        """Transform an update spec from Django-style format to Mongo format.
+        """
+        operators = ['set', 'unset', 'inc', 'dec', 'push', 'push_all', 'pull', 
+                     'pull_all']
+
+        mongo_update = {}
+        for key, value in update.items():
+            parts = key.split('__')
+            # Check for an operator and transform to mongo-style if there is
+            op = None
+            if parts[0] in operators:
+                op = parts.pop(0)
+                # Convert Pythonic names to Mongo equivalents
+                if op in ('push_all', 'pull_all'):
+                    op = op.replace('_all', 'All')
+                elif op == 'dec':
+                    # Support decrement by flipping a positive value's sign
+                    # and using 'inc'
+                    op = 'inc'
+                    if value > 0:
+                        value = -value
+
+            if _doc_cls:
+                # Switch field names to proper names [set in Field(name='foo')]
+                fields = QuerySet._lookup_field(_doc_cls, parts)
+                parts = [field.name for field in fields]
+
+                # Convert value to proper value
+                field = fields[-1]
+                if op in (None, 'set', 'unset', 'push', 'pull'):
+                    value = field.prepare_query_value(value)
+                elif op in ('pushAll', 'pullAll'):
+                    value = [field.prepare_query_value(v) for v in value]
+
+            key = '.'.join(parts)
+
+            if op:
+                value = {key: value}
+                key = '$' + op
+
+            if op is None or key not in mongo_update:
+                mongo_update[key] = value
+            elif key in mongo_update and isinstance(mongo_update[key], dict):
+                mongo_update[key].update(value)
+
+        return mongo_update
+
+    def update(self, safe_update=True, **update):
+        """Perform an atomic update on the fields matched by the query.
+        """
+        if pymongo.version < '1.1.1':
+            raise OperationError('update() method requires PyMongo 1.1.1+')
+
+        update = QuerySet._transform_update(self._document, **update)
+        try:
+            self._collection.update(self._query, update, safe=safe_update, 
+                                    multi=True)
+        except pymongo.errors.OperationFailure, err:
+            if str(err) == 'multi not coded yet':
+                raise OperationError('update() method requires MongoDB 1.1.3+')
+            raise OperationError('Update failed (%s)' % str(err))
+
+    def update_one(self, safe_update=True, **update):
+        """Perform an atomic update on first field matched by the query.
+        """
+        update = QuerySet._transform_update(self._document, **update)
+        try:
+            # Explicitly provide 'multi=False' to newer versions of PyMongo
+            # as the default may change to 'True'
+            if pymongo.version >= '1.1.1':
+                self._collection.update(self._query, update, safe=safe_update, 
+                                        multi=False)
+            else:
+                # Older versions of PyMongo don't support 'multi'
+                self._collection.update(self._query, update, safe=safe_update)
+        except pymongo.errors.OperationFailure, e:
+            raise OperationError('Update failed [%s]' % str(e))
+
     def __iter__(self):
         return self
 
@@ -297,6 +517,11 @@ class QuerySet(object):
         """
         return self.exec_js(freq_func, list_field, normalize=normalize)
 
+    def __repr__(self):
+        data = list(self[:REPR_OUTPUT_SIZE + 1])
+        if len(data) > REPR_OUTPUT_SIZE:
+            data[-1] = "...(remaining elements truncated)..."
+        return repr(data)
 
 class InvalidCollectionError(Exception):
     pass

tests/document.py

@@ -221,6 +221,72 @@ class DocumentTest(unittest.TestCase):
 
         Log.drop_collection()
 
+    def test_indexes(self):
+        """Ensure that indexes are used when meta[indexes] is specified.
+        """
+        class BlogPost(Document):
+            date = DateTimeField(name='addDate', default=datetime.datetime.now)
+            category = StringField()
+            meta = {
+                'indexes': [
+                    '-date', 
+                    ('category', '-date')
+                ],
+            }
+
+        BlogPost.drop_collection()
+
+        info = BlogPost.objects._collection.index_information()
+        self.assertEqual(len(info), 4) # _id, types, '-date', ('cat', 'date')
+
+        # Indexes are lazy so use list() to perform query
+        list(BlogPost.objects)
+        info = BlogPost.objects._collection.index_information()
+        self.assertTrue([('_types', 1), ('category', 1), ('addDate', -1)] 
+                        in info.values())
+        self.assertTrue([('_types', 1), ('addDate', -1)] in info.values())
+
+        BlogPost.drop_collection()
+
+    def test_unique(self):
+        """Ensure that uniqueness constraints are applied to fields.
+        """
+        class BlogPost(Document):
+            title = StringField()
+            slug = StringField(unique=True)
+
+        BlogPost.drop_collection()
+
+        post1 = BlogPost(title='test1', slug='test')
+        post1.save()
+
+        # Two posts with the same slug is not allowed
+        post2 = BlogPost(title='test2', slug='test')
+        self.assertRaises(OperationError, post2.save)
+
+        class Date(EmbeddedDocument):
+            year = IntField(name='yr')
+
+        class BlogPost(Document):
+            title = StringField()
+            date = EmbeddedDocumentField(Date)
+            slug = StringField(unique_with='date.year')
+
+        BlogPost.drop_collection()
+
+        post1 = BlogPost(title='test1', date=Date(year=2009), slug='test')
+        post1.save()
+
+        # day is different so won't raise exception
+        post2 = BlogPost(title='test2', date=Date(year=2010), slug='test')
+        post2.save()
+
+        # Now there will be two docs with the same slug and the same day: fail
+        post3 = BlogPost(title='test3', date=Date(year=2010), slug='test')
+        self.assertRaises(OperationError, post3.save)
+
+        BlogPost.drop_collection()
+
     def test_creation(self):
         """Ensure that document may be created using keyword arguments.
         """

tests/fields.py

@@ -231,6 +231,9 @@ class FieldTest(unittest.TestCase):
             content = StringField()
             author = ReferenceField(User)
 
+        User.drop_collection()
+        BlogPost.drop_collection()
+
         self.assertRaises(ValidationError, ReferenceField, EmbeddedDocument)
 
         user = User(name='Test User')

tests/queryset.py

@@ -1,5 +1,6 @@
 import unittest
 import pymongo
+from datetime import datetime
 
 from mongoengine.queryset import QuerySet
 from mongoengine import *
@@ -16,7 +17,7 @@ class QuerySetTest(unittest.TestCase):
         self.Person = Person
 
     def test_initialisation(self):
-        """Ensure that CollectionManager is correctly initialised.
+        """Ensure that a QuerySet is correctly initialised by QuerySetManager.
         """
         self.assertTrue(isinstance(self.Person.objects, QuerySet))
         self.assertEqual(self.Person.objects._collection.name(), 
@@ -48,6 +49,9 @@ class QuerySetTest(unittest.TestCase):
         person2 = self.Person(name="User B", age=30)
         person2.save()
 
+        q1 = Q(name='test')
+        q2 = Q(age__gte=18)
+
         # Find all people in the collection
         people = self.Person.objects
         self.assertEqual(len(people), 2)
@@ -131,6 +135,41 @@ class QuerySetTest(unittest.TestCase):
         person = self.Person.objects.with_id(person1.id)
         self.assertEqual(person.name, "User A")
 
+    def test_ordering(self):
+        """Ensure default ordering is applied and can be overridden.
+        """
+        class BlogPost(Document):
+            title = StringField()
+            published_date = DateTimeField()
+
+            meta = {
+                'ordering': ['-published_date']
+            }
+
+        BlogPost.drop_collection()
+
+        blog_post_1 = BlogPost(title="Blog Post #1", 
+                               published_date=datetime(2010, 1, 5, 0, 0 ,0))
+        blog_post_2 = BlogPost(title="Blog Post #2", 
+                               published_date=datetime(2010, 1, 6, 0, 0 ,0))
+        blog_post_3 = BlogPost(title="Blog Post #3", 
+                               published_date=datetime(2010, 1, 7, 0, 0 ,0))
+
+        blog_post_1.save()
+        blog_post_2.save()
+        blog_post_3.save()
+        
+        # get the "first" BlogPost using default ordering
+        # from BlogPost.meta.ordering
+        latest_post = BlogPost.objects.first() 
+        self.assertEqual(latest_post.title, "Blog Post #3")
+        
+        # override default ordering, order BlogPosts by "published_date"
+        first_post = BlogPost.objects.order_by("+published_date").first()
+        self.assertEqual(first_post.title, "Blog Post #1")
+
+        BlogPost.drop_collection()
+
     def test_find_embedded(self):
         """Ensure that an embedded document is properly returned from a query.
         """
@@ -141,6 +180,8 @@ class QuerySetTest(unittest.TestCase):
             content = StringField()
             author = EmbeddedDocumentField(User)
 
+        BlogPost.drop_collection()
+
         post = BlogPost(content='Had a good coffee today...')
         post.author = User(name='Test User')
         post.save()
@@ -151,6 +192,42 @@ class QuerySetTest(unittest.TestCase):
         
         BlogPost.drop_collection()
 
+    def test_q(self):
+        class BlogPost(Document):
+            publish_date = DateTimeField()
+            published = BooleanField()
+
+        BlogPost.drop_collection()
+
+        post1 = BlogPost(publish_date=datetime(2010, 1, 8), published=False)
+        post1.save()
+
+        post2 = BlogPost(publish_date=datetime(2010, 1, 15), published=True)
+        post2.save()
+
+        post3 = BlogPost(published=True)
+        post3.save()
+
+        post4 = BlogPost(publish_date=datetime(2010, 1, 8))
+        post4.save()
+
+        post5 = BlogPost(publish_date=datetime(2010, 1, 15))
+        post5.save()
+
+        post6 = BlogPost(published=False)
+        post6.save()
+
+        date = datetime(2010, 1, 10)
+        q = BlogPost.objects(Q(publish_date__lte=date) | Q(published=True))
+        posts = [post.id for post in q]
+
+        published_posts = (post1, post2, post3, post4)
+        self.assertTrue(all(obj.id in posts for obj in published_posts))
+
+        self.assertFalse(any(obj.id in posts for obj in [post5, post6]))
+
+        BlogPost.drop_collection()
+
     def test_delete(self):
         """Ensure that documents are properly deleted from the database.
         """
@@ -166,6 +243,41 @@ class QuerySetTest(unittest.TestCase):
         self.Person.objects.delete()
         self.assertEqual(len(self.Person.objects), 0)
 
+    def test_update(self):
+        """Ensure that atomic updates work properly.
+        """
+        class BlogPost(Document):
+            title = StringField()
+            hits = IntField()
+            tags = ListField(StringField())
+
+        BlogPost.drop_collection()
+
+        post = BlogPost(name="Test Post", hits=5, tags=['test'])
+        post.save()
+
+        BlogPost.objects.update(set__hits=10)
+        post.reload()
+        self.assertEqual(post.hits, 10)
+
+        BlogPost.objects.update_one(inc__hits=1)
+        post.reload()
+        self.assertEqual(post.hits, 11)
+
+        BlogPost.objects.update_one(dec__hits=1)
+        post.reload()
+        self.assertEqual(post.hits, 10)
+
+        BlogPost.objects.update(push__tags='mongo')
+        post.reload()
+        self.assertTrue('mongo' in post.tags)
+
+        BlogPost.objects.update_one(push_all__tags=['db', 'nosql'])
+        post.reload()
+        self.assertTrue('db' in post.tags and 'nosql' in post.tags)
+
+        BlogPost.drop_collection()
+
     def test_order_by(self):
         """Ensure that QuerySets may be ordered.
         """
@@ -326,9 +438,69 @@ class QuerySetTest(unittest.TestCase):
 
         BlogPost.drop_collection()
 
+
+    def test_types_index(self):
+        """Ensure that and index is used when '_types' is being used in a
+        query.
+        """
+        class BlogPost(Document):
+            date = DateTimeField()
+            meta = {'indexes': ['-date']}
+
+        # Indexes are lazy so use list() to perform query
+        list(BlogPost.objects)
+        info = BlogPost.objects._collection.index_information()
+        self.assertTrue([('_types', 1)] in info.values())
+        self.assertTrue([('_types', 1), ('date', -1)] in info.values())
+
+        BlogPost.drop_collection()
+
+        class BlogPost(Document):
+            title = StringField()
+            meta = {'allow_inheritance': False}
+
+        # _types is not used on objects where allow_inheritance is False
+        list(BlogPost.objects)
+        info = BlogPost.objects._collection.index_information()
+        self.assertFalse([('_types', 1)] in info.values())
+
+        BlogPost.drop_collection()
+
     def tearDown(self):
         self.Person.drop_collection()
 
 
+class QTest(unittest.TestCase):
+    
+    def test_or_and(self):
+        q1 = Q(name='test')
+        q2 = Q(age__gte=18)
+
+        query = ['(', {'name': 'test'}, '||', {'age__gte': 18}, ')']
+        self.assertEqual((q1 | q2).query, query)
+
+        query = ['(', {'name': 'test'}, '&&', {'age__gte': 18}, ')']
+        self.assertEqual((q1 & q2).query, query)
+
+        query = ['(', '(', {'name': 'test'}, '&&', {'age__gte': 18}, ')', '||',
+                 {'name': 'example'}, ')']
+        self.assertEqual((q1 & q2 | Q(name='example')).query, query)
+
+    def test_item_query_as_js(self):
+        """Ensure that the _item_query_as_js utilitiy method works properly.
+        """
+        q = Q()
+        examples = [
+            ({'name': 'test'}, 'this.name == i0f0', {'i0f0': 'test'}),
+            ({'age': {'$gt': 18}}, 'this.age > i0f0o0', {'i0f0o0': 18}),
+            ({'name': 'test', 'age': {'$gt': 18, '$lte': 65}}, 
+             'this.age <= i0f0o0 && this.age > i0f0o1 && this.name == i0f1', 
+             {'i0f0o0': 65, 'i0f0o1': 18, 'i0f1': 'test'}),
+        ]
+        for item, js, scope in examples:
+            test_scope = {}
+            self.assertEqual(q._item_query_as_js(item, test_scope, 0), js)
+            self.assertEqual(scope, test_scope)
+
 if __name__ == '__main__':
     unittest.main()