Fixing py3 compat

MongoEngine/mongoengine#136

.travis.yml

@@ -1,5 +1,6 @@
 # http://travis-ci.org/#!/MongoEngine/mongoengine
 language: python
+services: mongodb
 python:
     - 2.5
     - 2.6

AUTHORS

@@ -123,3 +123,4 @@ that much better:
  * psychogenic
  * Stefan Wójcik
  * dimonb
+ * Garry Polley

CONTRIBUTING.rst

@@ -0,0 +1,61 @@
+Contributing to MongoEngine
+===========================
+
+MongoEngine has a large `community
+<https://raw.github.com/MongoEngine/mongoengine/master/AUTHORS>`_ and
+contributions are always encouraged. Contributions can be as simple as
+minor tweaks to the documentation. Please read these guidelines before
+sending a pull request.
+
+Bugfixes and New Features
+-------------------------
+
+Before starting to write code, look for existing `tickets
+<https://github.com/MongoEngine/mongoengine/issues?state=open>`_ or `create one
+<https://github.com/MongoEngine/mongoengine/issues>`_ for your specific
+issue or feature request. That way you avoid working on something
+that might not be of interest or that has already been addressed.  If in doubt
+post to the `user group <http://groups.google.com/group/mongoengine-users>`
+
+Supported Interpreters
+----------------------
+
+PyMongo supports CPython 2.5 and newer. Language
+features not supported by all interpreters can not be used.
+Please also ensure that your code is properly converted by
+`2to3 <http://docs.python.org/library/2to3.html>`_ for Python 3 support.
+
+Style Guide
+-----------
+
+MongoEngine aims to follow `PEP8 <http://www.python.org/dev/peps/pep-0008/>`_
+including 4 space indents and 79 character line limits.
+
+Testing
+-------
+
+All tests are run on `Travis <http://travis-ci.org/MongoEngine/mongoengine>`_
+and any pull requests are automatically tested by Travis. Any pull requests
+without tests will take longer to be integrated and might be refused.
+
+General Guidelines
+------------------
+
+- Avoid backward breaking changes if at all possible.
+- Write inline documentation for new classes and methods.
+- Write tests and make sure they pass (make sure you have a mongod
+  running on the default port, then execute ``python setup.py test``
+  from the cmd line to run the test suite).
+- Add yourself to AUTHORS.rst :)
+
+Documentation
+-------------
+
+To contribute to the `API documentation
+<http://docs.mongoengine.org/en/latest/apireference.html>`_
+just make your changes to the inline documentation of the appropriate
+`source code <https://github.com/MongoEngine/mongoengine>`_ or `rst file
+<https://github.com/MongoEngine/mongoengine/tree/master/docs>`_ in a
+branch and submit a `pull request <https://help.github.com/articles/using-pull-requests>`_.
+You might also use the github `Edit <https://github.com/blog/844-forking-with-the-edit-button>`_
+button.

README.rst

@@ -92,6 +92,4 @@ Community
 
 Contributing
 ============
-The source is available on `GitHub <http://github.com/MongoEngine/mongoengine>`_ - to
-contribute to the project, fork it on GitHub and send a pull request, all
-contributions and suggestions are welcome!
+We welcome contributions! see  the`Contribution guidelines <https://github.com/MongoEngine/mongoengine/blob/master/CONTRIBUTING.rst>`_

docs/changelog.rst

@@ -2,6 +2,35 @@
 Changelog
 =========
 
+Changes in 0.7.X
+================
+- Unicode fix for repr (MongoEngine/mongoengine#133)
+- Allow updates with match operators (MongoEngine/mongoengine#144)
+- Updated URLField - now can have a override the regex (MongoEngine/mongoengine#136)
+- Allow Django AuthenticationBackends to work with Django user (hmarr/mongoengine#573)
+- Fixed reload issue with ReferenceField where dbref=False (MongoEngine/mongoengine#138)
+
+Changes in 0.7.5
+================
+- ReferenceFields with dbref=False use ObjectId instead of strings (MongoEngine/mongoengine#134)
+  See ticket for upgrade notes (https://github.com/MongoEngine/mongoengine/issues/134)
+
+Changes in 0.7.4
+================
+- Fixed index inheritance issues - firmed up testcases (MongoEngine/mongoengine#123) (MongoEngine/mongoengine#125)
+
+Changes in 0.7.3
+================
+- Reverted EmbeddedDocuments meta handling - now can turn off inheritance (MongoEngine/mongoengine#119)
+
+Changes in 0.7.2
+================
+- Update index spec generation so its not destructive (MongoEngine/mongoengine#113)
+
+Changes in 0.7.1
+=================
+- Fixed index spec inheritance (MongoEngine/mongoengine#111)
+
 Changes in 0.7.0
 =================
 - Updated queryset.delete so you can use with skip / limit (MongoEngine/mongoengine#107)

docs/guide/defining-documents.rst

@@ -344,6 +344,10 @@ Its value can take any of the following constants:
    their :file:`models.py` in the :const:`INSTALLED_APPS` tuple.
 
 
+.. warning::
+   Signals are not triggered when doing cascading updates / deletes - if this
+   is required you must manually handle the update / delete.
+
 Generic reference fields
 ''''''''''''''''''''''''
 A second kind of reference field also exists,
@@ -465,7 +469,7 @@ If a dictionary is passed then the following options are available:
     Whether the index should be sparse.
 
 :attr:`unique` (Default: False)
-    Whether the index should be sparse.
+    Whether the index should be unique.
 
 .. note ::
 

docs/guide/signals.rst

@@ -50,4 +50,11 @@ Example usage::
     signals.post_save.connect(Author.post_save, sender=Author)
 
 
+ReferenceFields and signals
+---------------------------
+
+Currently `reverse_delete_rules` do not trigger signals on the other part of
+the relationship.  If this is required you must manually handled the
+reverse deletion.
+
 .. _blinker: http://pypi.python.org/pypi/blinker

docs/index.rst

@@ -34,10 +34,10 @@ To get help with using MongoEngine, use the `MongoEngine Users mailing list
 Contributing
 ------------
 
-The source is available on `GitHub <http://github.com/hmarr/mongoengine>`_ and
+The source is available on `GitHub <http://github.com/MongoEngine/mongoengine>`_ and
 contributions are always encouraged. Contributions can be as simple as
 minor tweaks to this documentation. To contribute, fork the project on
-`GitHub <http://github.com/hmarr/mongoengine>`_ and send a
+`GitHub <http://github.com/MongoEngine/mongoengine>`_ and send a
 pull request.
 
 Also, you can join the developers' `mailing list

docs/upgrade.rst

@@ -61,6 +61,13 @@ stored in rather than as string representations.  Your code may need to be
 updated to handle native types rather than strings keys for the results of
 item frequency queries.
 
+BinaryFields
+------------
+
+Binary fields have been updated so that they are native binary types.  If you
+previously were doing `str` comparisons with binary field values you will have
+to update and wrap the value in a `str`.
+
 0.5 to 0.6
 ==========
 

mongoengine/__init__.py

@@ -12,7 +12,7 @@ from signals import *
 __all__ = (document.__all__ + fields.__all__ + connection.__all__ +
            queryset.__all__ + signals.__all__)
 
-VERSION = (0, 7, 0)
+VERSION = (0, 7, 5)
 
 
 def get_version():

mongoengine/base.py

@@ -53,7 +53,7 @@ class ValidationError(AssertionError):
         self.message = message
 
     def __str__(self):
-        return self.message
+        return txt_type(self.message)
 
     def __repr__(self):
         return '%s(%s,)' % (self.__class__.__name__, self.message)
@@ -235,7 +235,8 @@ class BaseField(object):
         pass
 
     def _validate(self, value):
-        from mongoengine import Document, EmbeddedDocument
+        Document = _import_class('Document')
+        EmbeddedDocument = _import_class('EmbeddedDocument')
         # check choices
         if self.choices:
             is_cls = isinstance(value, (Document, EmbeddedDocument))
@@ -283,7 +284,9 @@ class ComplexBaseField(BaseField):
         if instance is None:
             # Document class being used rather than a document object
             return self
-        from fields import GenericReferenceField, ReferenceField
+
+        ReferenceField = _import_class('ReferenceField')
+        GenericReferenceField = _import_class('GenericReferenceField')
         dereference = self.field is None or isinstance(self.field,
             (GenericReferenceField, ReferenceField))
         if not self._dereference and instance._initialised and dereference:
@@ -310,6 +313,7 @@ class ComplexBaseField(BaseField):
             )
             value._dereferenced = True
             instance._data[self.name] = value
+
         return value
 
     def __set__(self, instance, value):
@@ -321,7 +325,7 @@ class ComplexBaseField(BaseField):
     def to_python(self, value):
         """Convert a MongoDB-compatible type to a Python type.
         """
-        from mongoengine import Document
+        Document = _import_class('Document')
 
         if isinstance(value, basestring):
             return value
@@ -363,7 +367,7 @@ class ComplexBaseField(BaseField):
     def to_mongo(self, value):
         """Convert a Python type to a MongoDB-compatible type.
         """
-        from mongoengine import Document
+        Document = _import_class("Document")
 
         if isinstance(value, basestring):
             return value
@@ -399,7 +403,7 @@ class ComplexBaseField(BaseField):
                         meta.get('allow_inheritance', ALLOW_INHERITANCE)
                         == False)
                     if allow_inheritance and not self.field:
-                        from fields import GenericReferenceField
+                        GenericReferenceField = _import_class("GenericReferenceField")
                         value_dict[k] = GenericReferenceField().to_mongo(v)
                     else:
                         collection = v._get_collection_name()
@@ -460,7 +464,7 @@ class ComplexBaseField(BaseField):
     @property
     def _dereference(self,):
         if not self.__dereference:
-            from dereference import DeReference
+            DeReference = _import_class("DeReference")
             self.__dereference = DeReference()  # Cached
         return self.__dereference
 
@@ -508,6 +512,10 @@ class DocumentMetaclass(type):
 
         attrs['_is_document'] = attrs.get('_is_document', False)
 
+        # EmbeddedDocuments could have meta data for inheritance
+        if 'meta' in attrs:
+            attrs['_meta'] = attrs.pop('meta')
+
         # Handle document Fields
 
         # Merge all fields from subclasses
@@ -571,6 +579,24 @@ class DocumentMetaclass(type):
                 superclasses[base._class_name] = base
                 superclasses.update(base._superclasses)
 
+            if hasattr(base, '_meta'):
+                # Warn if allow_inheritance isn't set and prevent
+                # inheritance of classes where inheritance is set to False
+                allow_inheritance = base._meta.get('allow_inheritance',
+                                                    ALLOW_INHERITANCE)
+                if (not getattr(base, '_is_base_cls', True)
+                    and allow_inheritance is None):
+                    warnings.warn(
+                        "%s uses inheritance, the default for "
+                        "allow_inheritance is changing to off by default. "
+                        "Please add it to the document meta." % name,
+                        FutureWarning
+                    )
+                elif (allow_inheritance == False and
+                      not base._meta.get('abstract')):
+                    raise ValueError('Document %s may not be subclassed' %
+                                      base.__name__)
+
         attrs['_class_name'] = '.'.join(reversed(class_name))
         attrs['_superclasses'] = superclasses
 
@@ -609,17 +635,6 @@ class DocumentMetaclass(type):
                        "field name" % field.name)
                 raise InvalidDocumentError(msg)
 
-        # Merge in exceptions with parent hierarchy
-        exceptions_to_merge = (DoesNotExist, MultipleObjectsReturned)
-        module = attrs.get('__module__')
-        for exc in exceptions_to_merge:
-            name = exc.__name__
-            parents = tuple(getattr(base, name) for base in flattened_bases
-                         if hasattr(base, name)) or (exc,)
-            # Create new exception and set to new_class
-            exception = type(name, parents, {'__module__': module})
-            setattr(new_class, name, exception)
-
         # Add class to the _document_registry
         _document_registry[new_class._class_name] = new_class
 
@@ -745,21 +760,6 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
             if hasattr(base, 'meta'):
                 meta.merge(base.meta)
             elif hasattr(base, '_meta'):
-                # Warn if allow_inheritance isn't set and prevent
-                # inheritance of classes where inheritance is set to False
-                allow_inheritance = base._meta.get('allow_inheritance',
-                                                    ALLOW_INHERITANCE)
-                if not base._is_base_cls and allow_inheritance is None:
-                    warnings.warn(
-                        "%s uses inheritance, the default for "
-                        "allow_inheritance is changing to off by default. "
-                        "Please add it to the document meta." % name,
-                        FutureWarning
-                    )
-                elif (allow_inheritance == False and
-                      not base._meta.get('abstract')):
-                    raise ValueError('Document %s may not be subclassed' %
-                                      base.__name__)
                 meta.merge(base._meta)
 
             # Set collection in the meta if its callable
@@ -825,6 +825,17 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
             new_class._fields['id'] = ObjectIdField(db_field='_id')
             new_class.id = new_class._fields['id']
 
+        # Merge in exceptions with parent hierarchy
+        exceptions_to_merge = (DoesNotExist, MultipleObjectsReturned)
+        module = attrs.get('__module__')
+        for exc in exceptions_to_merge:
+            name = exc.__name__
+            parents = tuple(getattr(base, name) for base in flattened_bases
+                         if hasattr(base, name)) or (exc,)
+            # Create new exception and set to new_class
+            exception = type(name, parents, {'__module__': module})
+            setattr(new_class, name, exception)
+
         return new_class
 
     @classmethod
@@ -936,7 +947,7 @@ class BaseDocument(object):
 
             field = None
             if not hasattr(self, name) and not name.startswith('_'):
-                from fields import DynamicField
+                DynamicField = _import_class("DynamicField")
                 field = DynamicField(db_field=name)
                 field.name = name
                 self._dynamic_fields[name] = field
@@ -1114,7 +1125,8 @@ class BaseDocument(object):
     def _get_changed_fields(self, key='', inspected=None):
         """Returns a list of all fields that have explicitly been changed.
         """
-        from mongoengine import EmbeddedDocument, DynamicEmbeddedDocument
+        EmbeddedDocument = _import_class("EmbeddedDocument")
+        DynamicEmbeddedDocument = _import_class("DynamicEmbeddedDocument")
         _changed_fields = []
         _changed_fields += getattr(self, '_changed_fields', [])
 
@@ -1245,7 +1257,9 @@ class BaseDocument(object):
         geo_indices = []
         inspected.append(cls)
 
-        from fields import EmbeddedDocumentField, GeoPointField
+        EmbeddedDocumentField = _import_class("EmbeddedDocumentField")
+        GeoPointField = _import_class("GeoPointField")
+
         for field in cls._fields.values():
             if not isinstance(field, (EmbeddedDocumentField, GeoPointField)):
                 continue
@@ -1319,10 +1333,11 @@ class BaseDocument(object):
 
     def __repr__(self):
         try:
-            u = txt_type(self)
+            u = self.__str__()
         except (UnicodeEncodeError, UnicodeDecodeError):
             u = '[Bad Unicode data]'
-        return '<%s: %s>' % (self.__class__.__name__, u)
+        repr_type = type(u)
+        return repr_type('<%s: %s>' % (self.__class__.__name__, u))
 
     def __str__(self):
         if hasattr(self, '__unicode__'):
@@ -1330,7 +1345,7 @@ class BaseDocument(object):
                 return self.__unicode__()
             else:
                 return unicode(self).encode('utf-8')
-        return '%s object' % self.__class__.__name__
+        return txt_type('%s object' % self.__class__.__name__)
 
     def __eq__(self, other):
         if isinstance(other, self.__class__) and hasattr(other, 'id'):
@@ -1479,14 +1494,30 @@ def _import_class(cls_name):
     """Cached mechanism for imports"""
     if cls_name in _class_registry:
         return _class_registry.get(cls_name)
-    if cls_name == 'Document':
-        from mongoengine.document import Document as cls
-    elif cls_name == 'EmbeddedDocument':
-        from mongoengine.document import EmbeddedDocument as cls
-    elif cls_name == 'DictField':
-        from mongoengine.fields import DictField as cls
-    elif cls_name == 'OperationError':
-        from queryset import OperationError as cls
 
-    _class_registry[cls_name] = cls
-    return cls
+    doc_classes = ['Document', 'DynamicEmbeddedDocument', 'EmbeddedDocument']
+    field_classes = ['DictField', 'DynamicField', 'EmbeddedDocumentField',
+                     'GenericReferenceField', 'GeoPointField',
+                     'ReferenceField']
+    queryset_classes = ['OperationError']
+    deref_classes = ['DeReference']
+
+    if cls_name in doc_classes:
+        from mongoengine import document as module
+        import_classes = doc_classes
+    elif cls_name in field_classes:
+        from mongoengine import fields as module
+        import_classes = field_classes
+    elif cls_name in queryset_classes:
+        from mongoengine import queryset as module
+        import_classes = queryset_classes
+    elif cls_name in deref_classes:
+        from mongoengine import dereference as module
+        import_classes = deref_classes
+    else:
+        raise ValueError('No import set for: ' % cls_name)
+
+    for cls in import_classes:
+        _class_registry[cls] = getattr(module, cls)
+
+    return _class_registry.get(cls_name)

mongoengine/dereference.py

@@ -31,10 +31,10 @@ class DeReference(object):
             items = [i for i in items]
 
         self.max_depth = max_depth
-
         doc_type = None
+
         if instance and instance._fields:
-            doc_type = instance._fields[name]
+            doc_type = instance._fields.get(name)
             if hasattr(doc_type, 'field'):
                 doc_type = doc_type.field
 
@@ -134,7 +134,7 @@ class DeReference(object):
                         elif doc_type is None:
                             doc = get_document(
                                 ''.join(x.capitalize()
-                                        for x in col.split('_')))._from_son(ref)
+                                    for x in col.split('_')))._from_son(ref)
                         else:
                             doc = doc_type._from_son(ref)
                         object_map[doc.id] = doc
@@ -166,7 +166,7 @@ class DeReference(object):
                 return self.object_map.get(items['_ref'].id, items)
             elif '_types' in items and '_cls' in items:
                 doc = get_document(items['_cls'])._from_son(items)
-                doc._data = self._attach_objects(doc._data, depth, doc, name)
+                doc._data = self._attach_objects(doc._data, depth, doc, None)
                 return doc
 
         if not hasattr(items, 'items'):

mongoengine/django/auth.py

@@ -3,6 +3,8 @@ import datetime
 from mongoengine import *
 
 from django.utils.encoding import smart_str
+from django.contrib.auth.models import _user_get_all_permissions
+from django.contrib.auth.models import _user_has_perm
 from django.contrib.auth.models import AnonymousUser
 from django.utils.translation import ugettext_lazy as _
 
@@ -104,6 +106,25 @@ class User(Document):
         """
         return check_password(raw_password, self.password)
 
+    def get_all_permissions(self, obj=None):
+        return _user_get_all_permissions(self, obj)
+
+    def has_perm(self, perm, obj=None):
+        """
+        Returns True if the user has the specified permission. This method
+        queries all available auth backends, but returns immediately if any
+        backend returns True. Thus, a user who has permission from a single
+        auth backend is assumed to have permission in general. If an object is
+        provided, permissions for this specific object are checked.
+        """
+
+        # Active superusers have all permissions.
+        if self.is_active and self.is_superuser:
+            return True
+
+        # Otherwise we need to check the backends.
+        return _user_has_perm(self, perm, obj)
+
     @classmethod
     def create_user(cls, username, password, email=None):
         """Create (and save) a new user with the given username, password and

mongoengine/document.py

@@ -25,6 +25,14 @@ class EmbeddedDocument(BaseDocument):
     collection.  :class:`~mongoengine.EmbeddedDocument`\ s should be used as
     fields on :class:`~mongoengine.Document`\ s through the
     :class:`~mongoengine.EmbeddedDocumentField` field type.
+
+    A :class:`~mongoengine.EmbeddedDocument` subclass may be itself subclassed,
+    to create a specialised version of the embedded document that will be
+    stored in the same collection. To facilitate this behaviour, `_cls` and
+    `_types` fields are added to documents (hidden though the MongoEngine
+    interface though). To disable this behaviour and remove the dependence on
+    the presence of `_cls` and `_types`, set :attr:`allow_inheritance` to
+    ``False`` in the :attr:`meta` dictionary.
     """
 
     # The __metaclass__ attribute is removed by 2to3 when running with Python3
@@ -353,7 +361,12 @@ class Document(BaseDocument):
         id_field = self._meta['id_field']
         obj = self.__class__.objects(
                 **{id_field: self[id_field]}
-              ).first().select_related(max_depth=max_depth)
+              ).limit(1).select_related(max_depth=max_depth)
+        if obj:
+            obj = obj[0]
+        else:
+            msg = "Reloaded document has been deleted"
+            raise OperationError(msg)
         for field in self._fields:
             setattr(self, field, self._reload(field, obj[field]))
         if self._dynamic:

mongoengine/fields.py

@@ -1,10 +1,12 @@
 import datetime
 import decimal
+import itertools
 import re
 import time
+import urllib2
+import urlparse
 import uuid
 import warnings
-import itertools
 from operator import itemgetter
 
 import gridfs
@@ -101,25 +103,30 @@ class URLField(StringField):
     .. versionadded:: 0.3
     """
 
-    URL_REGEX = re.compile(
-        r'^https?://'
-        r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+[A-Z]{2,6}\.?|'
-        r'localhost|'
-        r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})'
-        r'(?::\d+)?'
-        r'(?:/?|[/?]\S+)$', re.IGNORECASE
-    )
+    _URL_REGEX = re.compile(
+        r'^(?:http|ftp)s?://' # http:// or https://
+        r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}\.?)|' #domain...
+        r'localhost|' #localhost...
+        r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})' # ...or ip
+        r'(?::\d+)?' # optional port
+        r'(?:/?|[/?]\S+)$', re.IGNORECASE)
 
-    def __init__(self, verify_exists=False, **kwargs):
+    def __init__(self, verify_exists=False, url_regex=None, **kwargs):
         self.verify_exists = verify_exists
+        self.url_regex = url_regex or self._URL_REGEX
         super(URLField, self).__init__(**kwargs)
 
     def validate(self, value):
-        if not URLField.URL_REGEX.match(value):
+        if not self.url_regex.match(value):
             self.error('Invalid URL: %s' % value)
+            return
 
         if self.verify_exists:
-            import urllib2
+            warnings.warn(
+                "The URLField verify_exists argument has intractable security "
+                "and performance issues. Accordingly, it has been deprecated.",
+            DeprecationWarning
+            )
             try:
                 request = urllib2.Request(value)
                 urllib2.urlopen(request)
@@ -709,6 +716,10 @@ class ReferenceField(BaseField):
 
         Bar.register_delete_rule(Foo, 'bar', NULLIFY)
 
+    .. note ::
+        `reverse_delete_rules` do not trigger pre / post delete signals to be
+        triggered.
+
     .. versionchanged:: 0.5 added `reverse_delete_rule`
     """
 
@@ -766,7 +777,7 @@ class ReferenceField(BaseField):
     def to_mongo(self, document):
         if isinstance(document, DBRef):
             if not self.dbref:
-                return "%s" % DBRef.id
+                return DBRef.id
             return document
         elif not self.dbref and isinstance(document, basestring):
             return document
@@ -788,7 +799,7 @@ class ReferenceField(BaseField):
             collection = self.document_type._get_collection_name()
             return DBRef(collection, id_)
 
-        return "%s" % id_
+        return id_
 
     def to_python(self, value):
         """Convert a MongoDB-compatible type to a Python type.

mongoengine/queryset.py

@@ -398,6 +398,7 @@ class QuerySet(object):
             or a **-** to determine the index ordering
         """
         index_spec = QuerySet._build_index_spec(self._document, key_or_list)
+        index_spec = index_spec.copy()
         fields = index_spec.pop('fields')
         index_spec['drop_dups'] = drop_dups
         index_spec['background'] = background
@@ -472,7 +473,9 @@ class QuerySet(object):
 
         # Ensure document-defined indexes are created
         if self._document._meta['index_specs']:
-            for spec in self._document._meta['index_specs']:
+            index_spec = self._document._meta['index_specs']
+            for spec in index_spec:
+                spec = spec.copy()
                 fields = spec.pop('fields')
                 types_indexed = types_indexed or includes_types(fields)
                 opts = index_opts.copy()
@@ -498,8 +501,10 @@ class QuerySet(object):
         """
         if isinstance(spec, basestring):
             spec = {'fields': [spec]}
-        if isinstance(spec, (list, tuple)):
-            spec = {'fields': spec}
+        elif isinstance(spec, (list, tuple)):
+            spec = {'fields': list(spec)}
+        elif isinstance(spec, dict):
+            spec = dict(spec)
 
         index_list = []
         direction = None
@@ -510,6 +515,10 @@ class QuerySet(object):
         use_types = allow_inheritance and not spec.get('sparse', False)
 
         for key in spec['fields']:
+            # If inherited spec continue
+            if isinstance(key, (list, tuple)):
+                continue
+
             # Get ASCENDING direction from +, DESCENDING from -, and GEO2D from *
             direction = pymongo.ASCENDING
             if key.startswith("-"):
@@ -1386,6 +1395,8 @@ class QuerySet(object):
         """
         operators = ['set', 'unset', 'inc', 'dec', 'pop', 'push', 'push_all',
                      'pull', 'pull_all', 'add_to_set']
+        match_operators = ['ne', 'gt', 'gte', 'lt', 'lte', 'in', 'nin', 'mod',
+                           'all', 'size', 'exists', 'not']
 
         mongo_update = {}
         for key, value in update.items():
@@ -1409,6 +1420,10 @@ class QuerySet(object):
                 elif op == 'add_to_set':
                     op = op.replace('_to_set', 'ToSet')
 
+            match = None
+            if parts[-1] in match_operators:
+                match = parts.pop()
+
             if _doc_cls:
                 # Switch field names to proper names [set in Field(name='foo')]
                 fields = QuerySet._lookup_field(_doc_cls, parts)
@@ -1442,16 +1457,22 @@ class QuerySet(object):
                     elif field.required or value is not None:
                         value = field.prepare_query_value(op, value)
 
+            if match:
+                match = '$' + match
+                value = {match: value}
+
             key = '.'.join(parts)
 
             if not op:
-                raise InvalidQueryError("Updates must supply an operation eg: set__FIELD=value")
+                raise InvalidQueryError("Updates must supply an operation "
+                                        "eg: set__FIELD=value")
 
             if 'pull' in op and '.' in key:
                 # Dot operators don't work on pull operations
                 # it uses nested dict syntax
                 if op == 'pullAll':
-                    raise InvalidQueryError("pullAll operations only support a single field depth")
+                    raise InvalidQueryError("pullAll operations only support "
+                                            "a single field depth")
 
                 parts.reverse()
                 for key in parts:

python-mongoengine.spec

@@ -5,7 +5,7 @@
 %define srcname mongoengine
 
 Name:           python-%{srcname}
-Version:        0.7.0
+Version:        0.7.5
 Release:        1%{?dist}
 Summary:        A Python Document-Object Mapper for working with MongoDB
 

tests/test_dereference.py

@@ -1,7 +1,7 @@
 from __future__ import with_statement
 import unittest
 
-from bson import DBRef
+from bson import DBRef, ObjectId
 
 from mongoengine import *
 from mongoengine.connection import get_db
@@ -84,6 +84,7 @@ class FieldTest(unittest.TestCase):
 
         group = Group(members=User.objects)
         group.save()
+        group.reload()  # Confirm reload works
 
         with query_counter() as q:
             self.assertEqual(q, 0)
@@ -187,8 +188,8 @@ class FieldTest(unittest.TestCase):
         self.assertEqual(group.members, [user])
 
         raw_data = Group._get_collection().find_one()
-        self.assertTrue(isinstance(raw_data['author'], basestring))
-        self.assertTrue(isinstance(raw_data['members'][0], basestring))
+        self.assertTrue(isinstance(raw_data['author'], ObjectId))
+        self.assertTrue(isinstance(raw_data['members'][0], ObjectId))
 
     def test_recursive_reference(self):
         """Ensure that ReferenceFields can reference their own documents.

tests/test_document.py

@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 from __future__ import with_statement
 import bson
 import os
@@ -85,6 +86,22 @@ class DocumentTest(unittest.TestCase):
         # Ensure Document isn't treated like an actual document
         self.assertFalse(hasattr(Document, '_fields'))
 
+    def test_repr(self):
+        """Ensure that unicode representation works
+        """
+        class Article(Document):
+            title = StringField()
+
+            def __unicode__(self):
+                return self.title
+
+        Article.drop_collection()
+
+        Article(title=u'привет мир').save()
+
+        self.assertEqual('<Article: привет мир>', repr(Article.objects.first()))
+        self.assertEqual('[<Article: привет мир>]', repr(Article.objects.all()))
+
     def test_collection_naming(self):
         """Ensure that a collection with a specified name may be used.
         """
@@ -338,7 +355,6 @@ class DocumentTest(unittest.TestCase):
                 meta = {'allow_inheritance': False}
         self.assertRaises(ValueError, create_employee_class)
 
-
     def test_allow_inheritance_abstract_document(self):
         """Ensure that abstract documents can set inheritance rules and that
         _cls and _types will not be used.
@@ -366,6 +382,31 @@ class DocumentTest(unittest.TestCase):
 
         Animal.drop_collection()
 
+    def test_allow_inheritance_embedded_document(self):
+
+        # Test the same for embedded documents
+        class Comment(EmbeddedDocument):
+            content = StringField()
+            meta = {'allow_inheritance': False}
+
+        def create_special_comment():
+            class SpecialComment(Comment):
+                pass
+
+        self.assertRaises(ValueError, create_special_comment)
+
+        comment = Comment(content='test')
+        self.assertFalse('_cls' in comment.to_mongo())
+        self.assertFalse('_types' in comment.to_mongo())
+
+        class Comment(EmbeddedDocument):
+            content = StringField()
+            meta = {'allow_inheritance': True}
+
+        comment = Comment(content='test')
+        self.assertTrue('_cls' in comment.to_mongo())
+        self.assertTrue('_types' in comment.to_mongo())
+
     def test_document_inheritance(self):
         """Ensure mutliple inheritance of abstract docs works
         """
@@ -396,6 +437,9 @@ class DocumentTest(unittest.TestCase):
                 'indexes': ['name']
             }
 
+        self.assertEqual(Animal._meta['index_specs'],
+                         [{'fields': [('_types', 1), ('name', 1)]}])
+
         Animal.drop_collection()
 
         dog = Animal(name='dog')
@@ -417,6 +461,9 @@ class DocumentTest(unittest.TestCase):
                 'allow_inheritance': False,
                 'indexes': ['name']
             }
+
+        self.assertEqual(Animal._meta['index_specs'],
+                         [{'fields': [('name', 1)]}])
         collection.update({}, {"$unset": {"_types": 1, "_cls": 1}},  multi=True)
 
         # Confirm extra data is removed
@@ -634,6 +681,12 @@ class DocumentTest(unittest.TestCase):
                 'allow_inheritance': True
             }
 
+        self.assertEqual(BlogPost._meta['index_specs'],
+                         [{'fields': [('_types', 1), ('addDate', -1)]},
+                          {'fields': [('tags', 1)]},
+                          {'fields': [('_types', 1), ('category', 1),
+                                      ('addDate', -1)]}])
+
         BlogPost.drop_collection()
 
         info = BlogPost.objects._collection.index_information()
@@ -657,6 +710,13 @@ class DocumentTest(unittest.TestCase):
             title = StringField()
             meta = {'indexes': ['title']}
 
+        self.assertEqual(ExtendedBlogPost._meta['index_specs'],
+                         [{'fields': [('_types', 1), ('addDate', -1)]},
+                          {'fields': [('tags', 1)]},
+                          {'fields': [('_types', 1), ('category', 1),
+                                      ('addDate', -1)]},
+                          {'fields': [('_types', 1), ('title', 1)]}])
+
         BlogPost.drop_collection()
 
         list(ExtendedBlogPost.objects)
@@ -669,6 +729,46 @@ class DocumentTest(unittest.TestCase):
 
         BlogPost.drop_collection()
 
+    def test_inherited_index(self):
+        """Ensure index specs are inhertited correctly"""
+
+        class A(Document):
+            title = StringField()
+            meta = {
+                'indexes': [
+                        {
+                        'fields': ('title',),
+                        },
+                ],
+                'allow_inheritance': True,
+                }
+
+        class B(A):
+            description = StringField()
+
+        self.assertEqual(A._meta['index_specs'], B._meta['index_specs'])
+        self.assertEqual([{'fields': [('_types', 1), ('title', 1)]}],
+                         A._meta['index_specs'])
+
+    def test_build_index_spec_is_not_destructive(self):
+
+        class MyDoc(Document):
+            keywords = StringField()
+
+            meta = {
+                'indexes': ['keywords'],
+                'allow_inheritance': False
+            }
+
+        self.assertEqual(MyDoc._meta['index_specs'],
+                         [{'fields': [('keywords', 1)]}])
+
+        # Force index creation
+        MyDoc.objects._ensure_indexes()
+
+        self.assertEqual(MyDoc._meta['index_specs'],
+                        [{'fields': [('keywords', 1)]}])
+
     def test_db_field_load(self):
         """Ensure we load data correctly
         """
@@ -729,6 +829,9 @@ class DocumentTest(unittest.TestCase):
                 'allow_inheritance': False
             }
 
+        self.assertEqual([{'fields': [('rank.title', 1)]}],
+                        Person._meta['index_specs'])
+
         Person.drop_collection()
 
         # Indexes are lazy so use list() to perform query
@@ -747,6 +850,10 @@ class DocumentTest(unittest.TestCase):
                     '*location.point',
                 ],
             }
+
+        self.assertEqual([{'fields': [('location.point', '2d')]}],
+                        Place._meta['index_specs'])
+
         Place.drop_collection()
 
         info = Place.objects._collection.index_information()
@@ -772,6 +879,10 @@ class DocumentTest(unittest.TestCase):
                 ],
             }
 
+        self.assertEqual([{'fields': [('addDate', -1)], 'unique': True,
+                          'sparse': True, 'types': False}],
+                        BlogPost._meta['index_specs'])
+
         BlogPost.drop_collection()
 
         info = BlogPost.objects._collection.index_information()

tests/test_fields.py

@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 from __future__ import with_statement
 import datetime
 import os
@@ -7,7 +8,7 @@ import tempfile
 
 from decimal import Decimal
 
-from bson import Binary, DBRef
+from bson import Binary, DBRef, ObjectId
 import gridfs
 
 from nose.plugins.skip import SkipTest
@@ -1104,7 +1105,7 @@ class FieldTest(unittest.TestCase):
         p = Person.objects.get(name="Ross")
         self.assertEqual(p.parent, p1)
 
-    def test_str_reference_fields(self):
+    def test_objectid_reference_fields(self):
 
         class Person(Document):
             name = StringField()
@@ -1117,7 +1118,7 @@ class FieldTest(unittest.TestCase):
 
         col = Person._get_collection()
         data = col.find_one({'name': 'Ross'})
-        self.assertEqual(data['parent'], "%s" % p1.pk)
+        self.assertEqual(data['parent'], p1.pk)
 
         p = Person.objects.get(name="Ross")
         self.assertEqual(p.parent, p1)

tests/test_queryset.py

@@ -230,6 +230,30 @@ class QuerySetTest(unittest.TestCase):
 
         Blog.drop_collection()
 
+    def test_chaining(self):
+        class A(Document):
+            pass
+
+        class B(Document):
+            a = ReferenceField(A)
+
+        A.drop_collection()
+        B.drop_collection()
+
+        a1 = A().save()
+        a2 = A().save()
+
+        B(a=a1).save()
+
+        # Works
+        q1 = B.objects.filter(a__in=[a1, a2], a=a1)._query
+
+        # Doesn't work
+        q2 = B.objects.filter(a__in=[a1, a2])
+        q2 = q2.filter(a=a1)._query
+
+        self.assertEqual(q1, q2)
+
     def test_update_write_options(self):
         """Test that passing write_options works"""
 
@@ -414,6 +438,30 @@ class QuerySetTest(unittest.TestCase):
         self.assertEqual(post.comments[0].by, 'joe')
         self.assertEqual(post.comments[0].votes.score, 4)
 
+    def test_updates_can_have_match_operators(self):
+
+        class Post(Document):
+            title = StringField(required=True)
+            tags = ListField(StringField())
+            comments = ListField(EmbeddedDocumentField("Comment"))
+
+        class Comment(EmbeddedDocument):
+            content = StringField()
+            name = StringField(max_length=120)
+            vote = IntField()
+
+        Post.drop_collection()
+
+        comm1 = Comment(content="very funny indeed", name="John S", vote=1)
+        comm2 = Comment(content="kind of funny", name="Mark P", vote=0)
+
+        Post(title='Fun with MongoEngine', tags=['mongodb', 'mongoengine'],
+             comments=[comm1, comm2]).save()
+
+        Post.objects().update_one(pull__comments__vote__lt=1)
+
+        self.assertEqual(1, len(Post.objects.first().comments))
+
     def test_mapfield_update(self):
         """Ensure that the MapField can be updated."""
         class Member(EmbeddedDocument):