Updated for pymongo

* Also made GridFS replace test pass

.gitignore

@@ -1,4 +1,9 @@
 *.pyc
 .*.swp
+*.egg
 docs/.build
 docs/_build
+build/
+dist/
+mongoengine.egg-info/
+env/

AUTHORS

@@ -0,0 +1,5 @@
+Harry Marr <harry@hmarr.com>
+Matt Dennewitz <mattdennewitz@gmail.com>
+Deepak Thukral <iapain@yahoo.com>
+Florian Schlachter <flori@n-schlachter.de>
+Steve Challis <steve@stevechallis.com>

MANIFEST.in

@@ -1,6 +1,6 @@
+include MANIFEST.in
 include README.rst
 include LICENSE
+include AUTHORS
 recursive-include docs *
-prune docs/_build/*
-recursive-include tests *
-recursive-exclude * *.pyc *.swp
+prune docs/_build

README.rst

@@ -15,7 +15,7 @@ a `tutorial <http://hmarr.com/mongoengine/tutorial.html>`_, a `user guide
 Installation
 ============
 If you have `setuptools <http://peak.telecommunity.com/DevCenter/setuptools>`_
-you can use ``easy_install mongoengine``. Otherwise, you can download the
+you can use ``easy_install -U mongoengine``. Otherwise, you can download the
 source from `GitHub <http://github.com/hmarr/mongoengine>`_ and run ``python
 setup.py install``.
 
@@ -82,6 +82,14 @@ Tests
 To run the test suite, ensure you are running a local instance of MongoDB on
 the standard port, and run ``python setup.py test``.
 
+Community
+=========
+- `MongoEngine Users mailing list 
+  <http://groups.google.com/group/mongoengine-users>`_
+- `MongoEngine Developers mailing list 
+  <http://groups.google.com/group/mongoengine-dev>`_
+- `#mongoengine IRC channel <irc://irc.freenode.net/mongoengine>`_
+
 Contributing
 ============
 The source is available on `GitHub <http://github.com/hmarr/mongoengine>`_ - to

docs/apireference.rst

@@ -20,6 +20,9 @@ Documents
 
 .. autoclass:: mongoengine.EmbeddedDocument
    :members:
+   
+.. autoclass:: mongoengine.document.MapReduceDocument
+  :members:
 
 Querying
 ========
@@ -27,6 +30,8 @@ Querying
 .. autoclass:: mongoengine.queryset.QuerySet
    :members:
 
+   .. automethod:: mongoengine.queryset.QuerySet.__call__
+   
 .. autofunction:: mongoengine.queryset.queryset_manager
 
 Fields
@@ -34,18 +39,32 @@ Fields
 
 .. autoclass:: mongoengine.StringField
 
+.. autoclass:: mongoengine.URLField
+
 .. autoclass:: mongoengine.IntField
 
 .. autoclass:: mongoengine.FloatField
 
+.. autoclass:: mongoengine.DecimalField
+
 .. autoclass:: mongoengine.BooleanField
 
 .. autoclass:: mongoengine.DateTimeField
 
 .. autoclass:: mongoengine.EmbeddedDocumentField
 
+.. autoclass:: mongoengine.DictField
+
 .. autoclass:: mongoengine.ListField
 
+.. autoclass:: mongoengine.BinaryField
+
 .. autoclass:: mongoengine.ObjectIdField
 
 .. autoclass:: mongoengine.ReferenceField
+
+.. autoclass:: mongoengine.GenericReferenceField
+
+.. autoclass:: mongoengine.FileField
+
+.. autoclass:: mongoengine.GeoPointField

docs/changelog.rst

@@ -2,10 +2,92 @@
 Changelog
 =========
 
-Changes is v0.1.3
+Changes in v0.4
+===============
+- Added ``GridFSStorage`` Django storage backend
+- Added ``FileField`` for GridFS support
+- New Q-object implementation, which is no longer based on Javascript
+- Added ``SortedListField``
+- Added ``EmailField``
+- Added ``GeoPointField``
+- Added ``exact`` and ``iexact`` match operators to ``QuerySet``
+- Added ``get_document_or_404`` and ``get_list_or_404`` Django shortcuts
+- Added new query operators for Geo queries
+- Added ``not`` query operator
+- Added new update operators: ``pop`` and ``add_to_set``
+- Added ``__raw__`` query parameter
+- Added support for custom querysets
+- Fixed document inheritance primary key issue
+- Added support for querying by array element position
+- Base class can now be defined for ``DictField``
+- Fixed MRO error that occured on document inheritance
+- Added ``QuerySet.distinct``, ``QuerySet.create``, ``QuerySet.snapshot``,
+  ``QuerySet.timeout`` and ``QuerySet.all``
+- Subsequent calls to ``connect()`` now work
+- Introduced ``min_length`` for ``StringField``
+- Fixed multi-process connection issue
+- Other minor fixes
+
+Changes in v0.3
+===============
+- Added MapReduce support
+- Added ``contains``, ``startswith`` and ``endswith`` query operators (and
+  case-insensitive versions that are prefixed with 'i') 
+- Deprecated fields' ``name`` parameter, replaced with ``db_field``
+- Added ``QuerySet.only`` for only retrieving specific fields
+- Added ``QuerySet.in_bulk()`` for bulk querying using ids
+- ``QuerySet``\ s now have a ``rewind()`` method, which is called automatically
+  when the iterator is exhausted, allowing ``QuerySet``\ s to be reused
+- Added ``DictField``
+- Added ``URLField``
+- Added ``DecimalField``
+- Added ``BinaryField``
+- Added ``GenericReferenceField``
+- Added ``get()`` and ``get_or_create()`` methods to ``QuerySet``
+- ``ReferenceField``\ s may now reference the document they are defined on
+  (recursive references) and documents that have not yet been defined
+- ``Document`` objects may now be compared for equality (equal if _ids are
+  equal and documents are of same type)
+- ``QuerySet`` update methods now have an ``upsert`` parameter
+- Added field name substitution for Javascript code (allows the user to use the
+  Python names for fields in JS, which are later substituted for the real field
+  names)
+- ``Q`` objects now support regex querying
+- Fixed bug where referenced documents within lists weren't properly
+  dereferenced
+- ``ReferenceField``\ s may now be queried using their _id
+- Fixed bug where ``EmbeddedDocuments`` couldn't be non-polymorphic
+- ``queryset_manager`` functions now accept two arguments -- the document class
+  as the first and the queryset as the second
+- Fixed bug where ``QuerySet.exec_js`` ignored ``Q`` objects
+- Other minor fixes
+
+Changes in v0.2.2
+=================
+- Fixed bug that prevented indexes from being used on ``ListField``\ s
+- ``Document.filter()`` added as an alias to ``Document.__call__()``
+- ``validate()`` may now be used on ``EmbeddedDocument``\ s
+
+Changes in v0.2.1
+=================
+- Added a MongoEngine backend for Django sessions
+- Added ``force_insert`` to ``Document.save()``
+- Improved querying syntax for ``ListField`` and ``EmbeddedDocumentField``
+- Added support for user-defined primary keys (``_id`` in MongoDB)
+
+Changes in v0.2
+===============
+- Added ``Q`` class for building advanced queries
+- Added ``QuerySet`` methods for atomic updates to documents
+- Fields may now specify ``unique=True`` to enforce uniqueness across a 
+  collection
+- Added option for default document ordering
+- Fixed bug in index definitions
+
+Changes in v0.1.3
 =================
 - Added Django authentication backend
-- Added Document.meta support for indexes, which are ensured just before 
+- Added ``Document.meta`` support for indexes, which are ensured just before 
   querying takes place
 - A few minor bugfixes
 
@@ -15,8 +97,8 @@ Changes in v0.1.2
 - Query values may be processed before before being used in queries
 - Made connections lazy
 - Fixed bug in Document dictionary-style access
-- Added BooleanField
-- Added Document.reload method
+- Added ``BooleanField``
+- Added ``Document.reload()`` method
 
 
 Changes in v0.1.1

docs/conf.py

@@ -22,7 +22,7 @@ sys.path.append(os.path.abspath('..'))
 
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = ['sphinx.ext.autodoc']
+extensions = ['sphinx.ext.autodoc', 'sphinx.ext.todo']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -38,7 +38,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'MongoEngine'
-copyright = u'2009, Harry Marr'
+copyright = u'2009-2010, Harry Marr'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/django.rst

@@ -19,7 +19,7 @@ MongoDB but still use many of the Django authentication infrastucture (such as
 the :func:`login_required` decorator and the :func:`authenticate` function). To
 enable the MongoEngine auth backend, add the following to you **settings.py**
 file::
-    
+
     AUTHENTICATION_BACKENDS = (
         'mongoengine.django.auth.MongoEngineBackend',
     )
@@ -27,3 +27,61 @@ file::
 The :mod:`~mongoengine.django.auth` module also contains a 
 :func:`~mongoengine.django.auth.get_user` helper function, that takes a user's
 :attr:`id` and returns a :class:`~mongoengine.django.auth.User` object.
+
+.. versionadded:: 0.1.3
+
+Sessions
+========
+Django allows the use of different backend stores for its sessions. MongoEngine
+provides a MongoDB-based session backend for Django, which allows you to use
+sessions in you Django application with just MongoDB. To enable the MongoEngine
+session backend, ensure that your settings module has
+``'django.contrib.sessions.middleware.SessionMiddleware'`` in the
+``MIDDLEWARE_CLASSES`` field  and ``'django.contrib.sessions'`` in your
+``INSTALLED_APPS``. From there, all you need to do is add the following line
+into you settings module::
+
+    SESSION_ENGINE = 'mongoengine.django.sessions'
+
+.. versionadded:: 0.2.1
+
+Storage
+=======
+With MongoEngine's support for GridFS via the :class:`~mongoengine.FileField`,
+it is useful to have a Django file storage backend that wraps this. The new
+storage module is called :class:`~mongoengine.django.GridFSStorage`. Using it
+is very similar to using the default FileSystemStorage.::
+
+    fs = mongoengine.django.GridFSStorage()
+
+    filename = fs.save('hello.txt', 'Hello, World!')
+
+All of the `Django Storage API methods
+<http://docs.djangoproject.com/en/dev/ref/files/storage/>`_ have been
+implemented except :func:`path`. If the filename provided already exists, an
+underscore and a number (before # the file extension, if one exists) will be
+appended to the filename until the generated filename doesn't exist. The
+:func:`save` method will return the new filename.::
+
+    >>> fs.exists('hello.txt')
+    True
+    >>> fs.open('hello.txt').read()
+    'Hello, World!'
+    >>> fs.size('hello.txt')
+    13
+    >>> fs.url('hello.txt')
+    'http://your_media_url/hello.txt'
+    >>> fs.open('hello.txt').name
+    'hello.txt'
+    >>> fs.listdir()
+    ([], [u'hello.txt'])
+
+All files will be saved and retrieved in GridFS via the :class::`FileDocument`
+document, allowing easy access to the files without the GridFSStorage
+backend.::
+
+    >>> from mongoengine.django.storage import FileDocument
+    >>> FileDocument.objects()
+    [<FileDocument: FileDocument object>]
+
+.. versionadded:: 0.4

docs/guide/connecting.rst

@@ -0,0 +1,20 @@
+.. _guide-connecting:
+
+=====================
+Connecting to MongoDB
+=====================
+To connect to a running instance of :program:`mongod`, use the
+:func:`~mongoengine.connect` function. The first argument is the name of the
+database to connect to. If the database does not exist, it will be created. If
+the database requires authentication, :attr:`username` and :attr:`password`
+arguments may be provided::
+
+    from mongoengine import connect
+    connect('project1', username='webapp', password='pwd123')
+
+By default, MongoEngine assumes that the :program:`mongod` instance is running
+on **localhost** on port **27017**. If MongoDB is running elsewhere, you may
+provide :attr:`host` and :attr:`port` arguments to
+:func:`~mongoengine.connect`::
+
+    connect('project1', host='192.168.1.35', port=12345)

docs/guide/defining-documents.rst

@@ -0,0 +1,381 @@
+==================
+Defining documents
+==================
+In MongoDB, a **document** is roughly equivalent to a **row** in an RDBMS. When
+working with relational databases, rows are stored in **tables**, which have a
+strict **schema** that the rows follow. MongoDB stores documents in
+**collections** rather than tables - the principle difference is that no schema 
+is enforced at a database level. 
+
+Defining a document's schema
+============================
+MongoEngine allows you to define schemata for documents as this helps to reduce
+coding errors, and allows for utility methods to be defined on fields which may
+be present. 
+
+To define a schema for a document, create a class that inherits from
+:class:`~mongoengine.Document`. Fields are specified by adding **field
+objects** as class attributes to the document class::
+
+    from mongoengine import *
+    import datetime
+    
+    class Page(Document):
+        title = StringField(max_length=200, required=True)
+        date_modified = DateTimeField(default=datetime.datetime.now)
+
+Fields
+======
+By default, fields are not required. To make a field mandatory, set the
+:attr:`required` keyword argument of a field to ``True``. Fields also may have
+validation constraints available (such as :attr:`max_length` in the example
+above). Fields may also take default values, which will be used if a value is
+not provided. Default values may optionally be a callable, which will be called
+to retrieve the value (such as in the above example). The field types available 
+are as follows:
+
+* :class:`~mongoengine.StringField`
+* :class:`~mongoengine.URLField`
+* :class:`~mongoengine.IntField`
+* :class:`~mongoengine.FloatField`
+* :class:`~mongoengine.DecimalField`
+* :class:`~mongoengine.DateTimeField`
+* :class:`~mongoengine.ListField`
+* :class:`~mongoengine.DictField`
+* :class:`~mongoengine.ObjectIdField`
+* :class:`~mongoengine.EmbeddedDocumentField`
+* :class:`~mongoengine.ReferenceField`
+* :class:`~mongoengine.GenericReferenceField`
+* :class:`~mongoengine.BooleanField`
+* :class:`~mongoengine.FileField`
+* :class:`~mongoengine.EmailField`
+* :class:`~mongoengine.SortedListField`
+* :class:`~mongoengine.BinaryField`
+* :class:`~mongoengine.GeoPointField`
+
+Field arguments
+---------------
+Each field type can be customized by keyword arguments.  The following keyword 
+arguments can be set on all fields:
+
+:attr:`db_field` (Default: None)
+    The MongoDB field name.
+
+:attr:`name` (Default: None)
+    The mongoengine field name.
+
+:attr:`required` (Default: False)
+    If set to True and the field is not set on the document instance, a
+    :class:`~mongoengine.base.ValidationError` will be raised when the document is
+    validated.
+
+:attr:`default` (Default: None)
+    A value to use when no value is set for this field.
+
+    The definion of default parameters follow `the general rules on Python
+    <http://docs.python.org/reference/compound_stmts.html#function-definitions>`__,
+    which means that some care should be taken when dealing with default mutable objects 
+    (like in :class:`~mongoengine.ListField` or :class:`~mongoengine.DictField`)::
+
+        class ExampleFirst(Document):
+            # Default an empty list
+            values = ListField(IntField(), default=list)
+
+        class ExampleSecond(Document):
+            # Default a set of values
+            values = ListField(IntField(), default=lambda: [1,2,3])
+
+        class ExampleDangerous(Document):
+            # This can make an .append call to  add values to the default (and all the following objects),
+            # instead to just an object
+            values = ListField(IntField(), default=[1,2,3])
+    
+
+:attr:`unique` (Default: False)
+    When True, no documents in the collection will have the same value for this
+    field.
+
+:attr:`unique_with` (Default: None)
+    A field name (or list of field names) that when taken together with this
+    field, will not have two documents in the collection with the same value.
+
+:attr:`primary_key` (Default: False)
+    When True, use this field as a primary key for the collection.
+
+:attr:`choices` (Default: None)
+    An iterable of choices to which the value of this field should be limited.
+    
+
+List fields
+-----------
+MongoDB allows the storage of lists of items. To add a list of items to a
+:class:`~mongoengine.Document`, use the :class:`~mongoengine.ListField` field
+type. :class:`~mongoengine.ListField` takes another field object as its first
+argument, which specifies which type elements may be stored within the list::
+
+    class Page(Document):
+        tags = ListField(StringField(max_length=50))
+
+Embedded documents
+------------------
+MongoDB has the ability to embed documents within other documents. Schemata may
+be defined for these embedded documents, just as they may be for regular
+documents. To create an embedded document, just define a document as usual, but
+inherit from :class:`~mongoengine.EmbeddedDocument` rather than 
+:class:`~mongoengine.Document`::
+
+    class Comment(EmbeddedDocument):
+        content = StringField()
+
+To embed the document within another document, use the
+:class:`~mongoengine.EmbeddedDocumentField` field type, providing the embedded
+document class as the first argument::
+
+    class Page(Document):
+        comments = ListField(EmbeddedDocumentField(Comment))
+
+    comment1 = Comment('Good work!')
+    comment2 = Comment('Nice article!')
+    page = Page(comments=[comment1, comment2])
+
+Dictionary Fields
+-----------------
+Often, an embedded document may be used instead of a dictionary -- generally
+this is recommended as dictionaries don't support validation or custom field
+types. However, sometimes you will not know the structure of what you want to
+store; in this situation a :class:`~mongoengine.DictField` is appropriate::
+    
+    class SurveyResponse(Document):
+        date = DateTimeField()
+        user = ReferenceField(User)
+        answers = DictField()
+
+    survey_response = SurveyResponse(date=datetime.now(), user=request.user)
+    response_form = ResponseForm(request.POST)
+    survey_response.answers = response_form.cleaned_data()   
+    survey_response.save()
+
+Reference fields
+----------------
+References may be stored to other documents in the database using the
+:class:`~mongoengine.ReferenceField`. Pass in another document class as the
+first argument to the constructor, then simply assign document objects to the
+field::
+    
+    class User(Document):
+        name = StringField()
+
+    class Page(Document):
+        content = StringField()
+        author = ReferenceField(User)
+
+    john = User(name="John Smith")
+    john.save()
+
+    post = Page(content="Test Page")
+    post.author = john
+    post.save()
+
+The :class:`User` object is automatically turned into a reference behind the
+scenes, and dereferenced when the :class:`Page` object is retrieved.
+
+To add a :class:`~mongoengine.ReferenceField` that references the document
+being defined, use the string ``'self'`` in place of the document class as the
+argument to :class:`~mongoengine.ReferenceField`'s constructor. To reference a
+document that has not yet been defined, use the name of the undefined document
+as the constructor's argument::
+
+    class Employee(Document):
+        name = StringField()
+        boss = ReferenceField('self')
+        profile_page = ReferenceField('ProfilePage')
+
+    class ProfilePage(Document):
+        content = StringField()
+
+Generic reference fields
+''''''''''''''''''''''''
+A second kind of reference field also exists,
+:class:`~mongoengine.GenericReferenceField`. This allows you to reference any
+kind of :class:`~mongoengine.Document`, and hence doesn't take a 
+:class:`~mongoengine.Document` subclass as a constructor argument::
+
+    class Link(Document):
+        url = StringField()
+        
+    class Post(Document):
+        title = StringField()
+        
+    class Bookmark(Document):
+        bookmark_object = GenericReferenceField()
+
+    link = Link(url='http://hmarr.com/mongoengine/')
+    link.save()
+
+    post = Post(title='Using MongoEngine')
+    post.save()
+
+    Bookmark(bookmark_object=link).save()
+    Bookmark(bookmark_object=post).save()
+
+.. note::
+   Using :class:`~mongoengine.GenericReferenceField`\ s is slightly less
+   efficient than the standard :class:`~mongoengine.ReferenceField`\ s, so if
+   you will only be referencing one document type, prefer the standard 
+   :class:`~mongoengine.ReferenceField`.
+
+Uniqueness constraints
+----------------------
+MongoEngine allows you to specify that a field should be unique across a
+collection by providing ``unique=True`` to a :class:`~mongoengine.Field`\ 's
+constructor. If you try to save a document that has the same value for a unique
+field as a document that is already in the database, a 
+:class:`~mongoengine.OperationError` will be raised. You may also specify
+multi-field uniqueness constraints by using :attr:`unique_with`, which may be
+either a single field name, or a list or tuple of field names::
+
+    class User(Document):
+        username = StringField(unique=True)
+        first_name = StringField()
+        last_name = StringField(unique_with='first_name')
+
+Skipping Document validation on save
+------------------------------------
+You can also skip the whole document validation process by setting 
+``validate=False`` when caling the :meth:`~mongoengine.document.Document.save` 
+method::
+
+    class Recipient(Document):
+        name = StringField()
+        email = EmailField()
+    
+    recipient = Recipient(name='admin', email='root@localhost')
+    recipient.save()               # will raise a ValidationError while
+    recipient.save(validate=False) # won't
+
+Document collections
+====================
+Document classes that inherit **directly** from :class:`~mongoengine.Document`
+will have their own **collection** in the database. The name of the collection
+is by default the name of the class, coverted to lowercase (so in the example
+above, the collection would be called `page`). If you need to change the name
+of the collection (e.g. to use MongoEngine with an existing database), then
+create a class dictionary attribute called :attr:`meta` on your document, and
+set :attr:`collection` to the name of the collection that you want your
+document class to use::
+
+    class Page(Document):
+        title = StringField(max_length=200, required=True)
+        meta = {'collection': 'cmsPage'}
+
+Capped collections
+------------------
+A :class:`~mongoengine.Document` may use a **Capped Collection** by specifying
+:attr:`max_documents` and :attr:`max_size` in the :attr:`meta` dictionary.
+:attr:`max_documents` is the maximum number of documents that is allowed to be
+stored in the collection, and :attr:`max_size` is the maximum size of the
+collection in bytes. If :attr:`max_size` is not specified and
+:attr:`max_documents` is, :attr:`max_size` defaults to 10000000 bytes (10MB).
+The following example shows a :class:`Log` document that will be limited to 
+1000 entries and 2MB of disk space::
+
+    class Log(Document):
+        ip_address = StringField()
+        meta = {'max_documents': 1000, 'max_size': 2000000}
+
+Indexes
+=======
+You can specify indexes on collections to make querying faster. This is done
+by creating a list of index specifications called :attr:`indexes` in the
+:attr:`~mongoengine.Document.meta` dictionary, where an index specification may
+either be a single field name, or a tuple containing multiple field names. A
+direction may be specified on fields by prefixing the field name with a **+**
+or a **-** sign. Note that direction only matters on multi-field indexes. ::
+
+    class Page(Document):
+        title = StringField()
+        rating = StringField()
+        meta = {
+            'indexes': ['title', ('title', '-rating')]
+        }
+
+.. note::
+   Geospatial indexes will be automatically created for all 
+   :class:`~mongoengine.GeoPointField`\ s
+        
+Ordering
+========
+A default ordering can be specified for your
+:class:`~mongoengine.queryset.QuerySet` using the :attr:`ordering` attribute of
+:attr:`~mongoengine.Document.meta`.  Ordering will be applied when the
+:class:`~mongoengine.queryset.QuerySet` is created, and can be overridden by
+subsequent calls to :meth:`~mongoengine.queryset.QuerySet.order_by`. ::
+
+    from datetime import datetime
+
+    class BlogPost(Document):
+        title = StringField()
+        published_date = DateTimeField()
+
+        meta = {
+            'ordering': ['-published_date']
+        }
+
+    blog_post_1 = BlogPost(title="Blog Post #1")
+    blog_post_1.published_date = datetime(2010, 1, 5, 0, 0 ,0)
+
+    blog_post_2 = BlogPost(title="Blog Post #2") 
+    blog_post_2.published_date = datetime(2010, 1, 6, 0, 0 ,0)
+
+    blog_post_3 = BlogPost(title="Blog Post #3")
+    blog_post_3.published_date = datetime(2010, 1, 7, 0, 0 ,0)
+
+    blog_post_1.save()
+    blog_post_2.save()
+    blog_post_3.save()
+
+    # get the "first" BlogPost using default ordering
+    # from BlogPost.meta.ordering
+    latest_post = BlogPost.objects.first() 
+    assert latest_post.title == "Blog Post #3"
+
+    # override default ordering, order BlogPosts by "published_date"
+    first_post = BlogPost.objects.order_by("+published_date").first()
+    assert first_post.title == "Blog Post #1"
+
+Document inheritance
+====================
+To create a specialised type of a :class:`~mongoengine.Document` you have
+defined, you may subclass it and add any extra fields or methods you may need.
+As this is new class is not a direct subclass of
+:class:`~mongoengine.Document`, it will not be stored in its own collection; it
+will use the same collection as its superclass uses. This allows for more
+convenient and efficient retrieval of related documents::
+
+    # Stored in a collection named 'page'
+    class Page(Document):
+        title = StringField(max_length=200, required=True)
+
+    # Also stored in the collection named 'page'
+    class DatedPage(Page):
+        date = DateTimeField()
+
+Working with existing data
+--------------------------
+To enable correct retrieval of documents involved in this kind of heirarchy,
+two extra attributes are stored on each document in the database: :attr:`_cls`
+and :attr:`_types`. These are hidden from the user through the MongoEngine
+interface, but may not be present if you are trying to use MongoEngine with 
+an existing database. For this reason, you may disable this inheritance
+mechansim, removing the dependency of :attr:`_cls` and :attr:`_types`, enabling
+you to work with existing databases. To disable inheritance on a document
+class, set :attr:`allow_inheritance` to ``False`` in the :attr:`meta`
+dictionary::
+
+    # Will work with data in an existing collection named 'cmsPage'
+    class Page(Document):
+        title = StringField(max_length=200, required=True)
+        meta = {
+            'collection': 'cmsPage',
+            'allow_inheritance': False,
+        }

docs/guide/document-instances.rst

@@ -0,0 +1,72 @@
+===================
+Documents instances
+===================
+To create a new document object, create an instance of the relevant document
+class, providing values for its fields as its constructor keyword arguments.
+You may provide values for any of the fields on the document::
+    
+    >>> page = Page(title="Test Page")
+    >>> page.title
+    'Test Page'
+
+You may also assign values to the document's fields using standard object 
+attribute syntax::
+
+    >>> page.title = "Example Page"
+    >>> page.title
+    'Example Page'
+
+Saving and deleting documents
+=============================
+To save the document to the database, call the
+:meth:`~mongoengine.Document.save` method. If the document does not exist in
+the database, it will be created. If it does already exist, it will be
+updated.
+
+To delete a document, call the :meth:`~mongoengine.Document.delete` method.
+Note that this will only work if the document exists in the database and has a
+valide :attr:`id`.
+
+.. seealso::
+    :ref:`guide-atomic-updates`
+
+Document IDs
+============
+Each document in the database has a unique id. This may be accessed through the
+:attr:`id` attribute on :class:`~mongoengine.Document` objects. Usually, the id
+will be generated automatically by the database server when the object is save,
+meaning that you may only access the :attr:`id` field once a document has been
+saved::
+
+    >>> page = Page(title="Test Page")
+    >>> page.id
+    >>> page.save()
+    >>> page.id
+    ObjectId('123456789abcdef000000000')
+
+Alternatively, you may define one of your own fields to be the document's
+"primary key" by providing ``primary_key=True`` as a keyword argument to a
+field's constructor. Under the hood, MongoEngine will use this field as the
+:attr:`id`; in fact :attr:`id` is actually aliased to your primary key field so
+you may still use :attr:`id` to access the primary key if you want::
+
+    >>> class User(Document):
+    ...     email = StringField(primary_key=True)
+    ...     name = StringField()
+    ...
+    >>> bob = User(email='bob@example.com', name='Bob')
+    >>> bob.save()
+    >>> bob.id == bob.email == 'bob@example.com'
+    True
+
+You can also access the document's "primary key" using the :attr:`pk` field; in
+is an alias to :attr:`id`::
+
+    >>> page = Page(title="Another Test Page")
+    >>> page.save()
+    >>> page.id == page.pk
+
+.. note::
+   If you define your own primary key field, the field implicitly becomes
+   required, so a :class:`ValidationError` will be thrown if you don't provide
+   it.

docs/guide/gridfs.rst

@@ -0,0 +1,83 @@
+======
+GridFS
+======
+
+.. versionadded:: 0.4
+
+Writing
+-------
+
+GridFS support comes in the form of the :class:`~mongoengine.FileField` field
+object. This field acts as a file-like object and provides a couple of
+different ways of inserting and retrieving data. Arbitrary metadata such as
+content type can also be stored alongside the files. In the following example,
+a document is created to store details about animals, including a photo::
+
+    class Animal(Document):
+        genus = StringField()
+        family = StringField()
+        photo = FileField()
+
+    marmot = Animal('Marmota', 'Sciuridae')
+
+    marmot_photo = open('marmot.jpg', 'r')      # Retrieve a photo from disk
+    marmot.photo = marmot_photo                 # Store photo in the document
+    marmot.photo.content_type = 'image/jpeg'    # Store metadata
+
+    marmot.save()
+
+Another way of writing to a :class:`~mongoengine.FileField` is to use the
+:func:`put` method. This allows for metadata to be stored in the same call as
+the file::
+
+    marmot.photo.put(marmot_photo, content_type='image/jpeg')
+
+    marmot.save()
+
+Retrieval
+---------
+
+So using the :class:`~mongoengine.FileField` is just like using any other
+field. The file can also be retrieved just as easily::
+
+    marmot = Animal.objects(genus='Marmota').first()
+    photo = marmot.photo.read()
+    content_type = marmot.photo.content_type
+
+Streaming
+---------
+
+Streaming data into a :class:`~mongoengine.FileField` is achieved in a
+slightly different manner.  First, a new file must be created by calling the
+:func:`new_file` method. Data can then be written using :func:`write`::
+
+    marmot.photo.new_file()
+    marmot.photo.write('some_image_data')
+    marmot.photo.write('some_more_image_data')
+    marmot.photo.close()
+
+    marmot.photo.save()
+
+Deletion
+--------
+
+Deleting stored files is achieved with the :func:`delete` method::
+
+    marmot.photo.delete()
+
+.. note::
+    The FileField in a Document actually only stores the ID of a file in a
+    separate GridFS collection. This means that deleting a document
+    with a defined FileField does not actually delete the file. You must be
+    careful to delete any files in a Document as above before deleting the
+    Document itself.
+
+
+Replacing files
+---------------
+
+Files can be replaced with the :func:`replace` method. This works just like
+the :func:`put` method so even metadata can (and should) be replaced::
+
+    another_marmot = open('another_marmot.png', 'r')
+    marmot.photo.replace(another_marmot, content_type='image/png')

docs/guide/index.rst

@@ -0,0 +1,13 @@
+==========
+User Guide
+==========
+
+.. toctree::
+   :maxdepth: 2
+
+   installing
+   connecting
+   defining-documents
+   document-instances
+   querying
+   gridfs

docs/guide/installing.rst

@@ -0,0 +1,31 @@
+======================
+Installing MongoEngine
+======================
+To use MongoEngine, you will need to download `MongoDB <http://mongodb.org/>`_
+and ensure it is running in an accessible location. You will also need
+`PyMongo <http://api.mongodb.org/python>`_ to use MongoEngine, but if you
+install MongoEngine using setuptools, then the dependencies will be handled for
+you.
+
+MongoEngine is available on PyPI, so to use it you can use 
+:program:`easy_install`:
+    
+.. code-block:: console
+
+    # easy_install mongoengine
+
+Alternatively, if you don't have setuptools installed, `download it from PyPi 
+<http://pypi.python.org/pypi/mongoengine/>`_ and run
+
+.. code-block:: console
+
+    # python setup.py install
+
+To use the bleeding-edge version of MongoEngine, you can get the source from
+`GitHub <http://github.com/hmarr/mongoengine/>`_ and install it as above:
+    
+.. code-block:: console
+
+    # git clone git://github.com/hmarr/mongoengine
+    # cd mongoengine
+    # python setup.py install

docs/guide/querying.rst

@@ -0,0 +1,475 @@
+=====================
+Querying the database
+=====================
+:class:`~mongoengine.Document` classes have an :attr:`objects` attribute, which
+is used for accessing the objects in the database associated with the class.
+The :attr:`objects` attribute is actually a
+:class:`~mongoengine.queryset.QuerySetManager`, which creates and returns a new
+a new :class:`~mongoengine.queryset.QuerySet` object on access. The
+:class:`~mongoengine.queryset.QuerySet` object may may be iterated over to
+fetch documents from the database::
+
+    # Prints out the names of all the users in the database
+    for user in User.objects:
+        print user.name
+
+.. note::
+   Once the iteration finishes (when :class:`StopIteration` is raised),
+   :meth:`~mongoengine.queryset.QuerySet.rewind` will be called so that the
+   :class:`~mongoengine.queryset.QuerySet` may be iterated over again. The
+   results of the first iteration are *not* cached, so the database will be hit
+   each time the :class:`~mongoengine.queryset.QuerySet` is iterated over.
+
+Filtering queries
+=================
+The query may be filtered by calling the
+:class:`~mongoengine.queryset.QuerySet` object with field lookup keyword 
+arguments. The keys in the keyword arguments correspond to fields on the
+:class:`~mongoengine.Document` you are querying::
+
+    # This will return a QuerySet that will only iterate over users whose
+    # 'country' field is set to 'uk'
+    uk_users = User.objects(country='uk')
+
+Fields on embedded documents may also be referred to using field lookup syntax
+by using a double-underscore in place of the dot in object attribute access
+syntax::
+
+    # This will return a QuerySet that will only iterate over pages that have
+    # been written by a user whose 'country' field is set to 'uk'
+    uk_pages = Page.objects(author__country='uk')
+
+Querying lists
+--------------
+On most fields, this syntax will look up documents where the field specified
+matches the given value exactly, but when the field refers to a
+:class:`~mongoengine.ListField`, a single item may be provided, in which case
+lists that contain that item will be matched::
+
+    class Page(Document):
+        tags = ListField(StringField())
+
+    # This will match all pages that have the word 'coding' as an item in the
+    # 'tags' list
+    Page.objects(tags='coding')
+
+Raw queries
+-----------
+It is possible to provide a raw PyMongo query as a query parameter, which will
+be integrated directly into the query. This is done using the ``__raw__``
+keyword argument::
+
+    Page.objects(__raw__={'tags': 'coding'})
+
+.. versionadded:: 0.4
+
+Query operators
+===============
+Operators other than equality may also be used in queries; just attach the
+operator name to a key with a double-underscore::
+
+    # Only find users whose age is 18 or less
+    young_users = Users.objects(age__lte=18)
+
+Available operators are as follows:
+
+* ``ne`` -- not equal to
+* ``lt`` -- less than
+* ``lte`` -- less than or equal to
+* ``gt`` -- greater than
+* ``gte`` -- greater than or equal to
+* ``not`` -- negate a standard check, may be used before other operators (e.g.
+  ``Q(age__not__mod=5)``)
+* ``in`` -- value is in list (a list of values should be provided)
+* ``nin`` -- value is not in list (a list of values should be provided)
+* ``mod`` -- ``value % x == y``, where ``x`` and ``y`` are two provided values
+* ``all`` -- every item in list of values provided is in array
+* ``size`` -- the size of the array is 
+* ``exists`` -- value for field exists
+
+The following operators are available as shortcuts to querying with regular
+expressions:
+
+* ``exact`` -- string field exactly matches value
+* ``iexact`` -- string field exactly matches value (case insensitive)
+* ``contains`` -- string field contains value
+* ``icontains`` -- string field contains value (case insensitive)
+* ``startswith`` -- string field starts with value
+* ``istartswith`` -- string field starts with value (case insensitive)
+* ``endswith`` -- string field ends with value
+* ``iendswith`` -- string field ends with value (case insensitive)
+
+.. versionadded:: 0.3
+
+There are a few special operators for performing geographical queries, that
+may used with :class:`~mongoengine.GeoPointField`\ s:
+
+* ``within_distance`` -- provide a list containing a point and a maximum
+  distance (e.g. [(41.342, -87.653), 5])
+* ``within_box`` -- filter documents to those within a given bounding box (e.g.
+  [(35.0, -125.0), (40.0, -100.0)])
+* ``near`` -- order the documents by how close they are to a given point
+
+.. versionadded:: 0.4
+
+Querying by position
+====================
+It is possible to query by position in a list by using a numerical value as a
+query operator. So if you wanted to find all pages whose first tag was ``db``,
+you could use the following query::
+
+    BlogPost.objects(tags__0='db')
+
+.. versionadded:: 0.4
+
+Limiting and skipping results
+=============================
+Just as with traditional ORMs, you may limit the number of results returned, or
+skip a number or results in you query.
+:meth:`~mongoengine.queryset.QuerySet.limit` and
+:meth:`~mongoengine.queryset.QuerySet.skip` and methods are available on
+:class:`~mongoengine.queryset.QuerySet` objects, but the prefered syntax for
+achieving this is using array-slicing syntax::
+
+    # Only the first 5 people
+    users = User.objects[:5]
+
+    # All except for the first 5 people
+    users = User.objects[5:]
+
+    # 5 users, starting from the 10th user found
+    users = User.objects[10:15]
+
+You may also index the query to retrieve a single result. If an item at that
+index does not exists, an :class:`IndexError` will be raised. A shortcut for
+retrieving the first result and returning :attr:`None` if no result exists is
+provided (:meth:`~mongoengine.queryset.QuerySet.first`)::
+
+    >>> # Make sure there are no users
+    >>> User.drop_collection()
+    >>> User.objects[0]
+    IndexError: list index out of range
+    >>> User.objects.first() == None
+    True
+    >>> User(name='Test User').save()
+    >>> User.objects[0] == User.objects.first()
+    True
+
+Retrieving unique results
+-------------------------
+To retrieve a result that should be unique in the collection, use
+:meth:`~mongoengine.queryset.QuerySet.get`. This will raise
+:class:`~mongoengine.queryset.DoesNotExist` if no document matches the query,
+and :class:`~mongoengine.queryset.MultipleObjectsReturned` if more than one
+document matched the query.
+
+A variation of this method exists, 
+:meth:`~mongoengine.queryset.Queryset.get_or_create`, that will create a new
+document with the query arguments if no documents match the query. An 
+additional keyword argument, :attr:`defaults` may be provided, which will be
+used as default values for the new document, in the case that it should need
+to be created::
+
+    >>> a, created = User.objects.get_or_create(name='User A', defaults={'age': 30})
+    >>> b, created = User.objects.get_or_create(name='User A', defaults={'age': 40})
+    >>> a.name == b.name and a.age == b.age
+    True
+
+Default Document queries
+========================
+By default, the objects :attr:`~mongoengine.Document.objects` attribute on a
+document returns a :class:`~mongoengine.queryset.QuerySet` that doesn't filter
+the collection -- it returns all objects. This may be changed by defining a
+method on a document that modifies a queryset. The method should accept two
+arguments -- :attr:`doc_cls` and :attr:`queryset`. The first argument is the
+:class:`~mongoengine.Document` class that the method is defined on (in this
+sense, the method is more like a :func:`classmethod` than a regular method),
+and the second argument is the initial queryset. The method needs to be
+decorated with :func:`~mongoengine.queryset.queryset_manager` in order for it
+to be recognised. ::
+
+    class BlogPost(Document):
+        title = StringField()
+        date = DateTimeField()
+
+        @queryset_manager
+        def objects(doc_cls, queryset):
+            # This may actually also be done by defining a default ordering for
+            # the document, but this illustrates the use of manager methods
+            return queryset.order_by('-date')
+
+You don't need to call your method :attr:`objects` -- you may define as many
+custom manager methods as you like::
+
+    class BlogPost(Document):
+        title = StringField()
+        published = BooleanField()
+
+        @queryset_manager
+        def live_posts(doc_cls, queryset):
+            return queryset.filter(published=True)
+
+    BlogPost(title='test1', published=False).save()
+    BlogPost(title='test2', published=True).save()
+    assert len(BlogPost.objects) == 2
+    assert len(BlogPost.live_posts) == 1
+
+Custom QuerySets
+================
+Should you want to add custom methods for interacting with or filtering
+documents, extending the :class:`~mongoengine.queryset.QuerySet` class may be
+the way to go. To use a custom :class:`~mongoengine.queryset.QuerySet` class on
+a document, set ``queryset_class`` to the custom class in a
+:class:`~mongoengine.Document`\ s ``meta`` dictionary::
+
+    class AwesomerQuerySet(QuerySet):
+        pass
+
+    class Page(Document):
+        meta = {'queryset_class': AwesomerQuerySet}
+
+.. versionadded:: 0.4
+
+Aggregation
+===========
+MongoDB provides some aggregation methods out of the box, but there are not as
+many as you typically get with an RDBMS. MongoEngine provides a wrapper around
+the built-in methods and provides some of its own, which are implemented as
+Javascript code that is executed on the database server.
+
+Counting results
+----------------
+Just as with limiting and skipping results, there is a method on
+:class:`~mongoengine.queryset.QuerySet` objects -- 
+:meth:`~mongoengine.queryset.QuerySet.count`, but there is also a more Pythonic
+way of achieving this::
+
+    num_users = len(User.objects)
+
+Further aggregation
+-------------------
+You may sum over the values of a specific field on documents using
+:meth:`~mongoengine.queryset.QuerySet.sum`::
+
+    yearly_expense = Employee.objects.sum('salary')
+
+.. note::
+   If the field isn't present on a document, that document will be ignored from
+   the sum.
+
+To get the average (mean) of a field on a collection of documents, use
+:meth:`~mongoengine.queryset.QuerySet.average`::
+
+    mean_age = User.objects.average('age')
+
+As MongoDB provides native lists, MongoEngine provides a helper method to get a
+dictionary of the frequencies of items in lists across an entire collection --
+:meth:`~mongoengine.queryset.QuerySet.item_frequencies`. An example of its use
+would be generating "tag-clouds"::
+
+    class Article(Document):
+        tag = ListField(StringField())
+
+    # After adding some tagged articles...
+    tag_freqs = Article.objects.item_frequencies('tag', normalize=True)
+
+    from operator import itemgetter
+    top_tags = sorted(tag_freqs.items(), key=itemgetter(1), reverse=True)[:10]
+
+Retrieving a subset of fields
+=============================
+Sometimes a subset of fields on a :class:`~mongoengine.Document` is required,
+and for efficiency only these should be retrieved from the database. This issue
+is especially important for MongoDB, as fields may often be extremely large
+(e.g. a :class:`~mongoengine.ListField` of
+:class:`~mongoengine.EmbeddedDocument`\ s, which represent the comments on a
+blog post. To select only a subset of fields, use
+:meth:`~mongoengine.queryset.QuerySet.only`, specifying the fields you want to
+retrieve as its arguments. Note that if fields that are not downloaded are
+accessed, their default value (or :attr:`None` if no default value is provided)
+will be given::
+
+    >>> class Film(Document):
+    ...     title = StringField()
+    ...     year = IntField()
+    ...     rating = IntField(default=3)
+    ...
+    >>> Film(title='The Shawshank Redemption', year=1994, rating=5).save()
+    >>> f = Film.objects.only('title').first()
+    >>> f.title
+    'The Shawshank Redemption'
+    >>> f.year   # None
+    >>> f.rating # default value
+    3
+
+If you later need the missing fields, just call
+:meth:`~mongoengine.Document.reload` on your document.
+
+Advanced queries
+================
+Sometimes calling a :class:`~mongoengine.queryset.QuerySet` object with keyword
+arguments can't fully express the query you want to use -- for example if you
+need to combine a number of constraints using *and* and *or*. This is made 
+possible in MongoEngine through the :class:`~mongoengine.queryset.Q` class.
+A :class:`~mongoengine.queryset.Q` object represents part of a query, and
+can be initialised using the same keyword-argument syntax you use to query
+documents. To build a complex query, you may combine 
+:class:`~mongoengine.queryset.Q` objects using the ``&`` (and) and ``|`` (or)
+operators. To use a :class:`~mongoengine.queryset.Q` object, pass it in as the
+first positional argument to :attr:`Document.objects` when you filter it by
+calling it with keyword arguments::
+
+    # Get published posts
+    Post.objects(Q(published=True) | Q(publish_date__lte=datetime.now()))
+
+    # Get top posts
+    Post.objects((Q(featured=True) & Q(hits__gte=1000)) | Q(hits__gte=5000))
+
+.. warning::
+   Only use these advanced queries if absolutely necessary as they will execute
+   significantly slower than regular queries. This is because they are not
+   natively supported by MongoDB -- they are compiled to Javascript and sent
+   to the server for execution.
+
+Server-side javascript execution
+================================
+Javascript functions may be written and sent to the server for execution. The
+result of this is the return value of the Javascript function. This
+functionality is accessed through the
+:meth:`~mongoengine.queryset.QuerySet.exec_js` method on
+:meth:`~mongoengine.queryset.QuerySet` objects. Pass in a string containing a
+Javascript function as the first argument.
+
+The remaining positional arguments are names of fields that will be passed into
+you Javascript function as its arguments. This allows functions to be written
+that may be executed on any field in a collection (e.g. the
+:meth:`~mongoengine.queryset.QuerySet.sum` method, which accepts the name of
+the field to sum over as its argument). Note that field names passed in in this
+manner are automatically translated to the names used on the database (set
+using the :attr:`name` keyword argument to a field constructor).
+
+Keyword arguments to :meth:`~mongoengine.queryset.QuerySet.exec_js` are
+combined into an object called :attr:`options`, which is available in the
+Javascript function. This may be used for defining specific parameters for your
+function.
+
+Some variables are made available in the scope of the Javascript function:
+
+* ``collection`` -- the name of the collection that corresponds to the
+  :class:`~mongoengine.Document` class that is being used; this should be
+  used to get the :class:`Collection` object from :attr:`db` in Javascript
+  code
+* ``query`` -- the query that has been generated by the
+  :class:`~mongoengine.queryset.QuerySet` object; this may be passed into
+  the :meth:`find` method on a :class:`Collection` object in the Javascript
+  function
+* ``options`` -- an object containing the keyword arguments passed into
+  :meth:`~mongoengine.queryset.QuerySet.exec_js`
+
+The following example demonstrates the intended usage of
+:meth:`~mongoengine.queryset.QuerySet.exec_js` by defining a function that sums
+over a field on a document (this functionality is already available throught
+:meth:`~mongoengine.queryset.QuerySet.sum` but is shown here for sake of
+example)::
+
+    def sum_field(document, field_name, include_negatives=True):
+        code = """
+        function(sumField) {
+            var total = 0.0;
+            db[collection].find(query).forEach(function(doc) {
+                var val = doc[sumField];
+                if (val >= 0.0 || options.includeNegatives) {
+                    total += val;
+                }
+            });
+            return total;
+        }
+        """
+        options = {'includeNegatives': include_negatives}
+        return document.objects.exec_js(code, field_name, **options)
+
+As fields in MongoEngine may use different names in the database (set using the
+:attr:`db_field` keyword argument to a :class:`Field` constructor), a mechanism
+exists for replacing MongoEngine field names with the database field names in
+Javascript code. When accessing a field on a collection object, use
+square-bracket notation, and prefix the MongoEngine field name with a tilde.
+The field name that follows the tilde will be translated to the name used in
+the database. Note that when referring to fields on embedded documents,
+the name of the :class:`~mongoengine.EmbeddedDocumentField`, followed by a dot,
+should be used before the name of the field on the embedded document. The
+following example shows how the substitutions are made::
+
+    class Comment(EmbeddedDocument):
+        content = StringField(db_field='body')
+
+    class BlogPost(Document):
+        title = StringField(db_field='doctitle')
+        comments = ListField(EmbeddedDocumentField(Comment), name='cs')
+
+    # Returns a list of dictionaries. Each dictionary contains a value named
+    # "document", which corresponds to the "title" field on a BlogPost, and
+    # "comment", which corresponds to an individual comment. The substitutions
+    # made are shown in the comments.
+    BlogPost.objects.exec_js("""
+    function() {
+        var comments = [];
+        db[collection].find(query).forEach(function(doc) {
+            // doc[~comments] -> doc["cs"]
+            var docComments = doc[~comments];
+
+            for (var i = 0; i < docComments.length; i++) {
+                // doc[~comments][i] -> doc["cs"][i]
+                var comment = doc[~comments][i];
+
+                comments.push({
+                    // doc[~title] -> doc["doctitle"]
+                    'document': doc[~title],
+
+                    // comment[~comments.content] -> comment["body"]
+                    'comment': comment[~comments.content]
+                });
+            }
+        });
+        return comments;
+    }
+    """)
+
+.. _guide-atomic-updates:
+
+Atomic updates
+==============
+Documents may be updated atomically by using the
+:meth:`~mongoengine.queryset.QuerySet.update_one` and
+:meth:`~mongoengine.queryset.QuerySet.update` methods on a 
+:meth:`~mongoengine.queryset.QuerySet`. There are several different "modifiers"
+that you may use with these methods:
+
+* ``set`` -- set a particular value
+* ``unset`` -- delete a particular value (since MongoDB v1.3+)
+* ``inc`` -- increment a value by a given amount
+* ``dec`` -- decrement a value by a given amount
+* ``pop`` -- remove the last item from a list
+* ``push`` -- append a value to a list
+* ``push_all`` -- append several values to a list
+* ``pop`` -- remove the first or last element of a list
+* ``pull`` -- remove a value from a list
+* ``pull_all`` -- remove several values from a list
+* ``add_to_set`` -- add value to a list only if its not in the list already
+
+The syntax for atomic updates is similar to the querying syntax, but the 
+modifier comes before the field, not after it::
+
+    >>> post = BlogPost(title='Test', page_views=0, tags=['database'])
+    >>> post.save()
+    >>> BlogPost.objects(id=post.id).update_one(inc__page_views=1)
+    >>> post.reload()  # the document has been changed, so we need to reload it
+    >>> post.page_views
+    1
+    >>> BlogPost.objects(id=post.id).update_one(set__title='Example Post')
+    >>> post.reload()
+    >>> post.title
+    'Example Post'
+    >>> BlogPost.objects(id=post.id).update_one(push__tags='nosql')
+    >>> post.reload()
+    >>> post.tags
+    ['database', 'nosql']

docs/index.rst

@@ -1,20 +1,28 @@
+==============================
 MongoEngine User Documentation
-=======================================
+==============================
 
 MongoEngine is an Object-Document Mapper, written in Python for working with 
 MongoDB. To install it, simply run
 
 .. code-block:: console
 
-    # easy_install mongoengine
+    # pip install -U mongoengine
 
 The source is available on `GitHub <http://github.com/hmarr/mongoengine>`_.
 
+To get help with using MongoEngine, use the `MongoEngine Users mailing list
+<http://groups.google.com/group/mongoengine-users>`_ or come chat on the
+`#mongoengine IRC channel <irc://irc.freenode.net/mongoengine>`_.
+
+If you are interested in contributing, join the developers' `mailing list 
+<http://groups.google.com/group/mongoengine-dev>`_.
+
 .. toctree::
    :maxdepth: 2
 
    tutorial
-   userguide
+   guide/index
    apireference
    django
    changelog

docs/userguide.rst

@@ -1,517 +0,0 @@
-==========
-User Guide
-==========
-
-Installing
-==========
-MongoEngine is available on PyPI, so to use it you can use 
-:program:`easy_install`
-    
-.. code-block:: console
-
-    # easy_install mongoengine
-
-Alternatively, if you don't have setuptools installed, `download it from PyPi 
-<http://pypi.python.org/pypi/mongoengine/>`_ and run
-
-.. code-block:: console
-
-    # python setup.py install
-
-.. _guide-connecting:
-
-Connecting to MongoDB
-=====================
-To connect to a running instance of :program:`mongod`, use the
-:func:`~mongoengine.connect` function. The first argument is the name of the
-database to connect to. If the database does not exist, it will be created. If
-the database requires authentication, :attr:`username` and :attr:`password`
-arguments may be provided::
-
-    from mongoengine import connect
-    connect('project1', username='webapp', password='pwd123')
-
-By default, MongoEngine assumes that the :program:`mongod` instance is running
-on **localhost** on port **27017**. If MongoDB is running elsewhere, you may
-provide :attr:`host` and :attr:`port` arguments to
-:func:`~mongoengine.connect`::
-
-    connect('project1', host='192.168.1.35', port=12345)
-
-Defining documents
-==================
-In MongoDB, a **document** is roughly equivalent to a **row** in an RDBMS. When
-working with relational databases, rows are stored in **tables**, which have a
-strict **schema** that the rows follow. MongoDB stores documents in
-**collections** rather than tables - the principle difference is that no schema 
-is enforced at a database level. 
-
-Defining a document's schema
-----------------------------
-MongoEngine allows you to define schemata for documents as this helps to reduce
-coding errors, and allows for utility methods to be defined on fields which may
-be present. 
-
-To define a schema for a document, create a class that inherits from
-:class:`~mongoengine.Document`. Fields are specified by adding **field
-objects** as class attributes to the document class::
-
-    from mongoengine import *
-    import datetime
-    
-    class Page(Document):
-        title = StringField(max_length=200, required=True)
-        date_modified = DateTimeField(default=datetime.now)
-
-Fields
-------
-By default, fields are not required. To make a field mandatory, set the
-:attr:`required` keyword argument of a field to ``True``. Fields also may have
-validation constraints available (such as :attr:`max_length` in the example
-above). Fields may also take default values, which will be used if a value is
-not provided. Default values may optionally be a callable, which will be called
-to retrieve the value (such as in the above example). The field types available 
-are as follows:
-
-* :class:`~mongoengine.StringField`
-* :class:`~mongoengine.IntField`
-* :class:`~mongoengine.FloatField`
-* :class:`~mongoengine.DateTimeField`
-* :class:`~mongoengine.ListField`
-* :class:`~mongoengine.ObjectIdField`
-* :class:`~mongoengine.EmbeddedDocumentField`
-* :class:`~mongoengine.ReferenceField`
-
-List fields
-^^^^^^^^^^^
-MongoDB allows the storage of lists of items. To add a list of items to a
-:class:`~mongoengine.Document`, use the :class:`~mongoengine.ListField` field
-type. :class:`~mongoengine.ListField` takes another field object as its first
-argument, which specifies which type elements may be stored within the list::
-
-    class Page(Document):
-        tags = ListField(StringField(max_length=50))
-
-Embedded documents
-^^^^^^^^^^^^^^^^^^
-MongoDB has the ability to embed documents within other documents. Schemata may
-be defined for these embedded documents, just as they may be for regular
-documents. To create an embedded document, just define a document as usual, but
-inherit from :class:`~mongoengine.EmbeddedDocument` rather than 
-:class:`~mongoengine.Document`::
-
-    class Comment(EmbeddedDocument):
-        content = StringField()
-
-To embed the document within another document, use the
-:class:`~mongoengine.EmbeddedDocumentField` field type, providing the embedded
-document class as the first argument::
-
-    class Page(Document):
-        comments = ListField(EmbeddedDocumentField(Comment))
-
-    comment1 = Comment('Good work!')
-    comment2 = Comment('Nice article!')
-    page = Page(comments=[comment1, comment2])
-
-Reference fields
-^^^^^^^^^^^^^^^^
-References may be stored to other documents in the database using the
-:class:`~mongoengine.ReferenceField`. Pass in another document class as the
-first argument to the constructor, then simply assign document objects to the
-field::
-    
-    class User(Document):
-        name = StringField()
-
-    class Page(Document):
-        content = StringField()
-        author = ReferenceField(User)
-
-    john = User(name="John Smith")
-    john.save()
-
-    post = Page(content="Test Page")
-    post.author = john
-    post.save()
-
-The :class:`User` object is automatically turned into a reference behind the
-scenes, and dereferenced when the :class:`Page` object is retrieved.
-
-Uniqueness constraints
-^^^^^^^^^^^^^^^^^^^^^^
-MongoEngine allows you to specify that a field should be unique across a
-collection by providing ``unique=True`` to a :class:`~mongoengine.Field`\ 's
-constructor. If you try to save a document that has the same value for a unique
-field as a document that is already in the database, a 
-:class:`~mongoengine.OperationError` will be raised. You may also specify
-multi-field uniqueness constraints by using :attr:`unique_with`, which may be
-either a single field name, or a list or tuple of field names::
-
-    class User(Document):
-        username = StringField(unique=True)
-        first_name = StringField()
-        last_name = StringField(unique_with='last_name')
-
-Document collections
---------------------
-Document classes that inherit **directly** from :class:`~mongoengine.Document`
-will have their own **collection** in the database. The name of the collection
-is by default the name of the class, coverted to lowercase (so in the example
-above, the collection would be called `page`). If you need to change the name
-of the collection (e.g. to use MongoEngine with an existing database), then
-create a class dictionary attribute called :attr:`meta` on your document, and
-set :attr:`collection` to the name of the collection that you want your
-document class to use::
-
-    class Page(Document):
-        title = StringField(max_length=200, required=True)
-        meta = {'collection': 'cmsPage'}
-
-Capped collections
-^^^^^^^^^^^^^^^^^^
-A :class:`~mongoengine.Document` may use a **Capped Collection** by specifying
-:attr:`max_documents` and :attr:`max_size` in the :attr:`meta` dictionary.
-:attr:`max_documents` is the maximum number of documents that is allowed to be
-stored in the collection, and :attr:`max_size` is the maximum size of the
-collection in bytes. If :attr:`max_size` is not specified and
-:attr:`max_documents` is, :attr:`max_size` defaults to 10000000 bytes (10MB).
-The following example shows a :class:`Log` document that will be limited to 
-1000 entries and 2MB of disk space::
-
-    class Log(Document):
-        ip_address = StringField()
-        meta = {'max_documents': 1000, 'max_size': 2000000}
-
-Indexes
--------
-You can specify indexes on collections to make querying faster. This is done
-by creating a list of index specifications called :attr:`indexes` in the
-:attr:`~mongoengine.Document.meta` dictionary, where an index specification may
-either be a single field name, or a tuple containing multiple field names. A
-direction may be specified on fields by prefixing the field name with a **+**
-or a **-** sign. Note that direction only matters on multi-field indexes. ::
-
-    class Page(Document):
-        title = StringField()
-        rating = StringField()
-        meta = {
-            'indexes': ['title', ('title', '-rating')]
-        }
-        
-Ordering
---------
-A default ordering can be specified for your
-:class:`~mongoengine.queryset.QuerySet` using the :attr:`ordering` attribute of
-:attr:`~mongoengine.Document.meta`.  Ordering will be applied when the
-:class:`~mongoengine.queryset.QuerySet` is created, and can be overridden by
-subsequent calls to :meth:`~mongoengine.queryset.QuerySet.order_by`. ::
-
-    from datetime import datetime
-
-    class BlogPost(Document):
-        title = StringField()
-        published_date = DateTimeField()
-
-        meta = {
-            'ordering': ['-published_date']
-        }
-
-    blog_post_1 = BlogPost(title="Blog Post #1")
-    blog_post_1.published_date = datetime(2010, 1, 5, 0, 0 ,0))
-
-    blog_post_2 = BlogPost(title="Blog Post #2") 
-    blog_post_2.published_date = datetime(2010, 1, 6, 0, 0 ,0))
-
-    blog_post_3 = BlogPost(title="Blog Post #3")
-    blog_post_3.published_date = datetime(2010, 1, 7, 0, 0 ,0))
-
-    blog_post_1.save()
-    blog_post_2.save()
-    blog_post_3.save()
-
-    # get the "first" BlogPost using default ordering
-    # from BlogPost.meta.ordering
-    latest_post = BlogPost.objects.first() 
-    self.assertEqual(latest_post.title, "Blog Post #3")
-
-    # override default ordering, order BlogPosts by "published_date"
-    first_post = BlogPost.objects.order_by("+published_date").first()
-    self.assertEqual(first_post.title, "Blog Post #1")
-
-Document inheritance
---------------------
-To create a specialised type of a :class:`~mongoengine.Document` you have
-defined, you may subclass it and add any extra fields or methods you may need.
-As this is new class is not a direct subclass of
-:class:`~mongoengine.Document`, it will not be stored in its own collection; it
-will use the same collection as its superclass uses. This allows for more
-convenient and efficient retrieval of related documents::
-
-    # Stored in a collection named 'page'
-    class Page(Document):
-        title = StringField(max_length=200, required=True)
-
-    # Also stored in the collection named 'page'
-    class DatedPage(Page):
-        date = DateTimeField()
-
-Working with existing data
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-To enable correct retrieval of documents involved in this kind of heirarchy,
-two extra attributes are stored on each document in the database: :attr:`_cls`
-and :attr:`_types`. These are hidden from the user through the MongoEngine
-interface, but may not be present if you are trying to use MongoEngine with 
-an existing database. For this reason, you may disable this inheritance
-mechansim, removing the dependency of :attr:`_cls` and :attr:`_types`, enabling
-you to work with existing databases. To disable inheritance on a document
-class, set :attr:`allow_inheritance` to ``False`` in the :attr:`meta`
-dictionary::
-
-    # Will work with data in an existing collection named 'cmsPage'
-    class Page(Document):
-        title = StringField(max_length=200, required=True)
-        meta = {
-            'collection': 'cmsPage',
-            'allow_inheritance': False,
-        }
-
-Documents instances
-===================
-To create a new document object, create an instance of the relevant document
-class, providing values for its fields as its constructor keyword arguments.
-You may provide values for any of the fields on the document::
-    
-    >>> page = Page(title="Test Page")
-    >>> page.title
-    'Test Page'
-
-You may also assign values to the document's fields using standard object 
-attribute syntax::
-
-    >>> page.title = "Example Page"
-    >>> page.title
-    'Example Page'
-
-Saving and deleting documents
------------------------------
-To save the document to the database, call the
-:meth:`~mongoengine.Document.save` method. If the document does not exist in
-the database, it will be created. If it does already exist, it will be
-updated.
-
-To delete a document, call the :meth:`~mongoengine.Document.delete` method.
-Note that this will only work if the document exists in the database and has a
-valide :attr:`id`.
-
-Document IDs
-------------
-Each document in the database has a unique id. This may be accessed through the
-:attr:`id` attribute on :class:`~mongoengine.Document` objects. Usually, the id
-will be generated automatically by the database server when the object is save,
-meaning that you may only access the :attr:`id` field once a document has been
-saved::
-
-    >>> page = Page(title="Test Page")
-    >>> page.id
-    >>> page.save()
-    >>> page.id
-    ObjectId('123456789abcdef000000000')
-
-Alternatively, you may explicitly set the :attr:`id` before you save the
-document, but the id must be a valid PyMongo :class:`ObjectId`.
-
-Querying the database
-=====================
-:class:`~mongoengine.Document` classes have an :attr:`objects` attribute, which
-is used for accessing the objects in the database associated with the class.
-The :attr:`objects` attribute is actually a
-:class:`~mongoengine.queryset.QuerySetManager`, which creates and returns a new
-a new :class:`~mongoengine.queryset.QuerySet` object on access. The
-:class:`~mongoengine.queryset.QuerySet` object may may be iterated over to
-fetch documents from the database::
-
-    # Prints out the names of all the users in the database
-    for user in User.objects:
-        print user.name
-
-Filtering queries
------------------
-The query may be filtered by calling the
-:class:`~mongoengine.queryset.QuerySet` object with field lookup keyword 
-arguments. The keys in the keyword arguments correspond to fields on the
-:class:`~mongoengine.Document` you are querying::
-
-    # This will return a QuerySet that will only iterate over users whose
-    # 'country' field is set to 'uk'
-    uk_users = User.objects(country='uk')
-
-Fields on embedded documents may also be referred to using field lookup syntax
-by using a double-underscore in place of the dot in object attribute access
-syntax::
-    
-    # This will return a QuerySet that will only iterate over pages that have
-    # been written by a user whose 'country' field is set to 'uk'
-    uk_pages = Page.objects(author__country='uk')
-
-Querying lists
-^^^^^^^^^^^^^^
-On most fields, this syntax will look up documents where the field specified
-matches the given value exactly, but when the field refers to a
-:class:`~mongoengine.ListField`, a single item may be provided, in which case
-lists that contain that item will be matched::
-
-    class Page(Document):
-        tags = ListField(StringField())
-
-    # This will match all pages that have the word 'coding' as an item in the
-    # 'tags' list
-    Page.objects(tags='coding')
-
-Query operators
----------------
-Operators other than equality may also be used in queries; just attach the
-operator name to a key with a double-underscore::
-    
-    # Only find users whose age is 18 or less
-    young_users = Users.objects(age__lte=18)
-
-Available operators are as follows:
-
-* ``neq`` -- not equal to
-* ``lt`` -- less than
-* ``lte`` -- less than or equal to
-* ``gt`` -- greater than
-* ``gte`` -- greater than or equal to
-* ``in`` -- value is in list (a list of values should be provided)
-* ``nin`` -- value is not in list (a list of values should be provided)
-* ``mod`` -- ``value % x == y``, where ``x`` and ``y`` are two provided values
-* ``all`` -- every item in array is in list of values provided
-* ``size`` -- the size of the array is 
-* ``exists`` -- value for field exists
-
-Limiting and skipping results
------------------------------
-Just as with traditional ORMs, you may limit the number of results returned, or
-skip a number or results in you query.
-:meth:`~mongoengine.queryset.QuerySet.limit` and
-:meth:`~mongoengine.queryset.QuerySet.skip` and methods are available on
-:class:`~mongoengine.queryset.QuerySet` objects, but the prefered syntax for
-achieving this is using array-slicing syntax::
-
-    # Only the first 5 people
-    users = User.objects[:5]
-
-    # All except for the first 5 people
-    users = User.objects[5:]
-
-    # 5 users, starting from the 10th user found
-    users = User.objects[10:15]
-
-Aggregation
------------
-MongoDB provides some aggregation methods out of the box, but there are not as
-many as you typically get with an RDBMS. MongoEngine provides a wrapper around
-the built-in methods and provides some of its own, which are implemented as
-Javascript code that is executed on the database server.
-
-Counting results
-^^^^^^^^^^^^^^^^
-Just as with limiting and skipping results, there is a method on
-:class:`~mongoengine.queryset.QuerySet` objects -- 
-:meth:`~mongoengine.queryset.QuerySet.count`, but there is also a more Pythonic
-way of achieving this::
-
-    num_users = len(User.objects)
-
-Further aggregation
-^^^^^^^^^^^^^^^^^^^
-You may sum over the values of a specific field on documents using
-:meth:`~mongoengine.queryset.QuerySet.sum`::
-
-    yearly_expense = Employee.objects.sum('salary')
-
-.. note::
-   If the field isn't present on a document, that document will be ignored from
-   the sum.
-
-To get the average (mean) of a field on a collection of documents, use
-:meth:`~mongoengine.queryset.QuerySet.average`::
-
-    mean_age = User.objects.average('age')
-
-As MongoDB provides native lists, MongoEngine provides a helper method to get a
-dictionary of the frequencies of items in lists across an entire collection --
-:meth:`~mongoengine.queryset.QuerySet.item_frequencies`. An example of its use
-would be generating "tag-clouds"::
-
-    class Article(Document):
-        tag = ListField(StringField())
-
-    # After adding some tagged articles...
-    tag_freqs = Article.objects.item_frequencies('tag', normalize=True)
-
-    from operator import itemgetter
-    top_tags = sorted(tag_freqs.items(), key=itemgetter(1), reverse=True)[:10]
-
-Advanced queries
-----------------
-Sometimes calling a :class:`~mongoengine.queryset.QuerySet` object with keyword
-arguments can't fully express the query you want to use -- for example if you
-need to combine a number of constraints using *and* and *or*. This is made 
-possible in MongoEngine through the :class:`~mongoengine.queryset.Q` class.
-A :class:`~mongoengine.queryset.Q` object represents part of a query, and
-can be initialised using the same keyword-argument syntax you use to query
-documents. To build a complex query, you may combine 
-:class:`~mongoengine.queryset.Q` objects using the ``&`` (and) and ``|`` (or)
-operators. To use :class:`~mongoengine.queryset.Q` objects, pass them in
-as positional arguments to :attr:`Document.objects` when you filter it by
-calling it with keyword arguments::
-
-    # Get published posts
-    Post.objects(Q(published=True) | Q(publish_date__lte=datetime.now()))
-
-    # Get top posts
-    Post.objects((Q(featured=True) & Q(hits__gte=1000)) | Q(hits__gte=5000))
-
-.. warning::
-   Only use these advanced queries if absolutely necessary as they will execute
-   significantly slower than regular queries. This is because they are not
-   natively supported by MongoDB -- they are compiled to Javascript and sent
-   to the server for execution.
-
-Atomic updates
---------------
-Documents may be updated atomically by using the
-:meth:`~mongoengine.queryset.QuerySet.update_one` and
-:meth:`~mongoengine.queryset.QuerySet.update` methods on a 
-:meth:`~mongoengine.queryset.QuerySet`. There are several different "modifiers"
-that you may use with these methods:
-
-* ``set`` -- set a particular value
-* ``unset`` -- delete a particular value (since MongoDB v1.3+)
-* ``inc`` -- increment a value by a given amount
-* ``dec`` -- decrement a value by a given amount
-* ``push`` -- append a value to a list
-* ``push_all`` -- append several values to a list
-* ``pull`` -- remove a value from a list
-* ``pull_all`` -- remove several values from a list
-
-The syntax for atomic updates is similar to the querying syntax, but the 
-modifier comes before the field, not after it::
-    
-    >>> post = BlogPost(title='Test', page_views=0, tags=['database'])
-    >>> post.save()
-    >>> BlogPost.objects(id=post.id).update_one(inc__page_views=1)
-    >>> post.reload()  # the document has been changed, so we need to reload it
-    >>> post.page_views
-    1
-    >>> BlogPost.objects(id=post.id).update_one(set__title='Example Post')
-    >>> post.reload()
-    >>> post.title
-    'Example Post'
-    >>> BlogPost.objects(id=post.id).update_one(push__tags='nosql')
-    >>> post.reload()
-    >>> post.tags
-    ['database', 'nosql']
-

mongoengine/__init__.py

@@ -12,7 +12,7 @@ __all__ = (document.__all__ + fields.__all__ + connection.__all__ +
 
 __author__ = 'Harry Marr'
 
-VERSION = (0, 2, 0)
+VERSION = (0, 4, 1)
 
 def get_version():
     version = '%s.%s' % (VERSION[0], VERSION[1])

mongoengine/base.py

@@ -1,8 +1,17 @@
 from queryset import QuerySet, QuerySetManager
+from queryset import DoesNotExist, MultipleObjectsReturned
 
+import sys
+import bson
 import pymongo
 
 
+_document_registry = {}
+
+def get_document(name):
+    return _document_registry[name]
+
+
 class ValidationError(Exception):
     pass
 
@@ -11,17 +20,30 @@ class BaseField(object):
     """A base class for fields in a MongoDB document. Instances of this class
     may be added to subclasses of `Document` to define a document's schema.
     """
-    
-    def __init__(self, name=None, required=False, default=None, unique=False,
-                 unique_with=None):
-        self.name = name
-        self.required = required
+
+    # Fields may have _types inserted into indexes by default
+    _index_with_types = True
+    _geo_index = False
+
+    def __init__(self, db_field=None, name=None, required=False, default=None,
+                 unique=False, unique_with=None, primary_key=False,
+                 validation=None, choices=None):
+        self.db_field = (db_field or name) if not primary_key else '_id'
+        if name:
+            import warnings
+            msg = "Fields' 'name' attribute deprecated in favour of 'db_field'"
+            warnings.warn(msg, DeprecationWarning)
+        self.name = None
+        self.required = required or primary_key
         self.default = default
         self.unique = bool(unique or unique_with)
         self.unique_with = unique_with
+        self.primary_key = primary_key
+        self.validation = validation
+        self.choices = choices
 
     def __get__(self, instance, owner):
-        """Descriptor for retrieving a value from a field in a document. Do 
+        """Descriptor for retrieving a value from a field in a document. Do
         any necessary conversion between Python and MongoDB types.
         """
         if instance is None:
@@ -52,7 +74,7 @@ class BaseField(object):
         """
         return self.to_python(value)
 
-    def prepare_query_value(self, value):
+    def prepare_query_value(self, op, value):
         """Prepare a value that is being used in a query for PyMongo.
         """
         return value
@@ -62,25 +84,47 @@ class BaseField(object):
         """
         pass
 
+    def _validate(self, value):
+        # check choices
+        if self.choices is not None:
+            if value not in self.choices:
+                raise ValidationError("Value must be one of %s."
+                    % unicode(self.choices))
+
+        # check validation argument
+        if self.validation is not None:
+            if callable(self.validation):
+                if not self.validation(value):
+                    raise ValidationError('Value does not match custom' \
+                                          'validation method.')
+            else:
+                raise ValueError('validation argument must be a callable.')
+
+        self.validate(value)
 
 class ObjectIdField(BaseField):
     """An field wrapper around MongoDB's ObjectIds.
     """
-    
+
     def to_python(self, value):
-        return str(value)
+        return value
+        # return unicode(value)
 
     def to_mongo(self, value):
-        if not isinstance(value, pymongo.objectid.ObjectId):
-            return pymongo.objectid.ObjectId(value)
+        if not isinstance(value, bson.objectid.ObjectId):
+            try:
+                return bson.objectid.ObjectId(unicode(value))
+            except Exception, e:
+                #e.message attribute has been deprecated since Python 2.6
+                raise ValidationError(unicode(e))
         return value
 
-    def prepare_query_value(self, value):
+    def prepare_query_value(self, op, value):
         return self.to_mongo(value)
 
     def validate(self, value):
         try:
-            pymongo.objectid.ObjectId(str(value))
+            bson.objectid.ObjectId(unicode(value))
         except:
             raise ValidationError('Invalid Object ID')
 
@@ -98,6 +142,7 @@ class DocumentMetaclass(type):
         doc_fields = {}
         class_name = [name]
         superclasses = {}
+        simple_class = True
         for base in bases:
             # Include all fields present in superclasses
             if hasattr(base, '_fields'):
@@ -106,19 +151,66 @@ class DocumentMetaclass(type):
                 # Get superclasses from superclass
                 superclasses[base._class_name] = base
                 superclasses.update(base._superclasses)
+
+            if hasattr(base, '_meta'):
+                # Ensure that the Document class may be subclassed -
+                # inheritance may be disabled to remove dependency on
+                # additional fields _cls and _types
+                if base._meta.get('allow_inheritance', True) == False:
+                    raise ValueError('Document %s may not be subclassed' %
+                                     base.__name__)
+                else:
+                    simple_class = False
+
+        meta = attrs.get('_meta', attrs.get('meta', {}))
+
+        if 'allow_inheritance' not in meta:
+            meta['allow_inheritance'] = True
+
+        # Only simple classes - direct subclasses of Document - may set
+        # allow_inheritance to False
+        if not simple_class and not meta['allow_inheritance']:
+            raise ValueError('Only direct subclasses of Document may set '
+                             '"allow_inheritance" to False')
+        attrs['_meta'] = meta
+
         attrs['_class_name'] = '.'.join(reversed(class_name))
         attrs['_superclasses'] = superclasses
 
         # Add the document's fields to the _fields attribute
         for attr_name, attr_value in attrs.items():
             if hasattr(attr_value, "__class__") and \
-                issubclass(attr_value.__class__, BaseField):
-                if not attr_value.name:
-                    attr_value.name = attr_name
+               issubclass(attr_value.__class__, BaseField):
+                attr_value.name = attr_name
+                if not attr_value.db_field:
+                    attr_value.db_field = attr_name
                 doc_fields[attr_name] = attr_value
         attrs['_fields'] = doc_fields
 
-        return super_new(cls, name, bases, attrs)
+        new_class = super_new(cls, name, bases, attrs)
+        for field in new_class._fields.values():
+            field.owner_document = new_class
+
+        module = attrs.get('__module__')
+
+        base_excs = tuple(base.DoesNotExist for base in bases
+                          if hasattr(base, 'DoesNotExist')) or (DoesNotExist,)
+        exc = subclass_exception('DoesNotExist', base_excs, module)
+        new_class.add_to_class('DoesNotExist', exc)
+
+        base_excs = tuple(base.MultipleObjectsReturned for base in bases
+                          if hasattr(base, 'MultipleObjectsReturned'))
+        base_excs = base_excs or (MultipleObjectsReturned,)
+        exc = subclass_exception('MultipleObjectsReturned', base_excs, module)
+        new_class.add_to_class('MultipleObjectsReturned', exc)
+
+        global _document_registry
+        _document_registry[name] = new_class
+
+        return new_class
+
+    def add_to_class(self, name, value):
+        setattr(self, name, value)
 
 
 class TopLevelDocumentMetaclass(DocumentMetaclass):
@@ -128,63 +220,69 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
 
     def __new__(cls, name, bases, attrs):
         super_new = super(TopLevelDocumentMetaclass, cls).__new__
-        # Classes defined in this package are abstract and should not have 
+        # Classes defined in this package are abstract and should not have
         # their own metadata with DB collection, etc.
-        # __metaclass__ is only set on the class with the __metaclass__ 
+        # __metaclass__ is only set on the class with the __metaclass__
         # attribute (i.e. it is not set on subclasses). This differentiates
         # 'real' documents from the 'Document' class
         if attrs.get('__metaclass__') == TopLevelDocumentMetaclass:
             return super_new(cls, name, bases, attrs)
 
         collection = name.lower()
-        
-        simple_class = True
+
+        id_field = None
+        base_indexes = []
+        base_meta = {}
 
         # Subclassed documents inherit collection from superclass
         for base in bases:
             if hasattr(base, '_meta') and 'collection' in base._meta:
-                # Ensure that the Document class may be subclassed - 
-                # inheritance may be disabled to remove dependency on 
-                # additional fields _cls and _types
-                if base._meta.get('allow_inheritance', True) == False:
-                    raise ValueError('Document %s may not be subclassed' %
-                                     base.__name__)
-                else:
-                    simple_class = False
                 collection = base._meta['collection']
 
+                # Propagate index options.
+                for key in ('index_background', 'index_drop_dups', 'index_opts'):
+                   if key in base._meta:
+                      base_meta[key] = base._meta[key]
+
+                id_field = id_field or base._meta.get('id_field')
+                base_indexes += base._meta.get('indexes', [])
+
         meta = {
             'collection': collection,
-            'allow_inheritance': True,
             'max_documents': None,
             'max_size': None,
             'ordering': [], # default ordering applied at runtime
-            'indexes': [] # indexes to be ensured at runtime
+            'indexes': [], # indexes to be ensured at runtime
+            'id_field': id_field,
+            'index_background': False,
+            'index_drop_dups': False,
+            'index_opts': {},
+            'queryset_class': QuerySet,
         }
+        meta.update(base_meta)
 
         # Apply document-defined meta options
         meta.update(attrs.get('meta', {}))
-        
-        # Only simple classes - direct subclasses of Document - may set
-        # allow_inheritance to False
-        if not simple_class and not meta['allow_inheritance']:
-            raise ValueError('Only direct subclasses of Document may set '
-                             '"allow_inheritance" to False')
         attrs['_meta'] = meta
 
-        attrs['id'] = ObjectIdField(name='_id')
-
         # Set up collection manager, needs the class to have fields so use
         # DocumentMetaclass before instantiating CollectionManager object
         new_class = super_new(cls, name, bases, attrs)
-        new_class.objects = QuerySetManager()
 
-        # Generate a list of indexes needed by uniqueness constraints
+        # Provide a default queryset unless one has been manually provided
+        if not hasattr(new_class, 'objects'):
+            new_class.objects = QuerySetManager()
+
+        user_indexes = [QuerySet._build_index_spec(new_class, spec)
+                        for spec in meta['indexes']] + base_indexes
+        new_class._meta['indexes'] = user_indexes
+
         unique_indexes = []
         for field_name, field in new_class._fields.items():
+            # Generate a list of indexes needed by uniqueness constraints
             if field.unique:
                 field.required = True
-                unique_fields = [field_name]
+                unique_fields = [field.db_field]
 
                 # Add any unique_with fields to the back of the index spec
                 if field.unique_with:
@@ -197,17 +295,34 @@ class TopLevelDocumentMetaclass(DocumentMetaclass):
                         parts = other_name.split('.')
                         # Lookup real name
                         parts = QuerySet._lookup_field(new_class, parts)
-                        name_parts = [part.name for part in parts]
+                        name_parts = [part.db_field for part in parts]
                         unique_with.append('.'.join(name_parts))
                         # Unique field should be required
                         parts[-1].required = True
                     unique_fields += unique_with
 
                 # Add the new index to the list
-                index = [(field, pymongo.ASCENDING) for field in unique_fields]
+                index = [(f, pymongo.ASCENDING) for f in unique_fields]
                 unique_indexes.append(index)
+
+            # Check for custom primary key
+            if field.primary_key:
+                current_pk = new_class._meta['id_field']
+                if current_pk and current_pk != field_name:
+                    raise ValueError('Cannot override primary key field')
+
+                if not current_pk:
+                    new_class._meta['id_field'] = field_name
+                    # Make 'Document.id' an alias to the real primary key field
+                    new_class.id = field
+
         new_class._meta['unique_indexes'] = unique_indexes
 
+        if not new_class._meta['id_field']:
+            new_class._meta['id_field'] = 'id'
+            new_class._fields['id'] = ObjectIdField(db_field='_id')
+            new_class.id = new_class._fields['id']
+
         return new_class
 
 
@@ -215,14 +330,36 @@ class BaseDocument(object):
 
     def __init__(self, **values):
         self._data = {}
+        # Assign default values to instance
+        for attr_name in self._fields.keys():
+            # Use default value if present
+            value = getattr(self, attr_name, None)
+            setattr(self, attr_name, value)
         # Assign initial values to instance
-        for attr_name, attr_value in self._fields.items():
-            if attr_name in values:
+        for attr_name in values.keys():
+            try:
                 setattr(self, attr_name, values.pop(attr_name))
-            else:
-                # Use default value if present
-                value = getattr(self, attr_name, None)
-                setattr(self, attr_name, value)
+            except AttributeError:
+                pass
+
+    def validate(self):
+        """Ensure that all fields' values are valid and that required fields
+        are present.
+        """
+        # Get a list of tuples of field names and their current values
+        fields = [(field, getattr(self, name))
+                  for name, field in self._fields.items()]
+
+        # Ensure that each field is matched to a valid value
+        for field, value in fields:
+            if value is not None:
+                try:
+                    field._validate(value)
+                except (ValueError, AttributeError, AssertionError), e:
+                    raise ValidationError('Invalid value for field of type "%s": %s'
+                                          % (field.__class__.__name__, value))
+            elif field.required:
+                raise ValidationError('Field "%s" is required' % field.name)
 
     @classmethod
     def _get_subclasses(cls):
@@ -239,6 +376,16 @@ class BaseDocument(object):
             all_subclasses.update(subclass._get_subclasses())
         return all_subclasses
 
+    @apply
+    def pk():
+        """Primary key alias
+        """
+        def fget(self):
+            return getattr(self, self._meta['id_field'])
+        def fset(self, value):
+            return setattr(self, self._meta['id_field'], value)
+        return property(fget, fset)
+
     def __iter__(self):
         return iter(self._fields)
 
@@ -289,17 +436,19 @@ class BaseDocument(object):
         for field_name, field in self._fields.items():
             value = getattr(self, field_name, None)
             if value is not None:
-                data[field.name] = field.to_mongo(value)
+                data[field.db_field] = field.to_mongo(value)
         # Only add _cls and _types if allow_inheritance is not False
         if not (hasattr(self, '_meta') and
                 self._meta.get('allow_inheritance', True) == False):
             data['_cls'] = self._class_name
             data['_types'] = self._superclasses.keys() + [self._class_name]
+        if data.has_key('_id') and not data['_id']:
+            del data['_id']
         return data
-    
+
     @classmethod
     def _from_son(cls, son):
-        """Create an instance of a Document (subclass) from a PyMongo SOM.
+        """Create an instance of a Document (subclass) from a PyMongo SON.
         """
         # get the class name from the document, falling back to the given
         # class if unavailable
@@ -322,8 +471,28 @@ class BaseDocument(object):
                 return None
             cls = subclasses[class_name]
 
-        for field_name, field in cls._fields.items():
-            if field.name in data:
-                data[field_name] = field.to_python(data[field.name])
+        present_fields = data.keys()
 
-        return cls(**data)
+        for field_name, field in cls._fields.items():
+            if field.db_field in data:
+                value = data[field.db_field]
+                data[field_name] = (value if value is None
+                                    else field.to_python(value))
+
+        obj = cls(**data)
+        obj._present_fields = present_fields
+        return obj
+
+    def __eq__(self, other):
+        if isinstance(other, self.__class__) and hasattr(other, 'id'):
+            if self.id == other.id:
+                return True
+        return False
+
+if sys.version_info < (2, 5):
+    # Prior to Python 2.5, Exception was an old-style class
+    def subclass_exception(name, parents, unused):
+        return types.ClassType(name, parents, {})
+else:
+    def subclass_exception(name, parents, module):
+        return type(name, parents, {'__module__': module})

mongoengine/connection.py

@@ -1,62 +1,71 @@
 from pymongo import Connection
-
+import multiprocessing
 
 __all__ = ['ConnectionError', 'connect']
 
 
-_connection_settings = {
+_connection_defaults = {
     'host': 'localhost',
     'port': 27017,
-    'pool_size': 1,
 }
-_connection = None
+_connection = {}
+_connection_settings = _connection_defaults.copy()
 
 _db_name = None
 _db_username = None
 _db_password = None
-_db = None
+_db = {}
 
 
 class ConnectionError(Exception):
     pass
 
 
-def _get_connection():
+def _get_connection(reconnect=False):
     global _connection
+    identity = get_identity()
     # Connect to the database if not already connected
-    if _connection is None:
+    if _connection.get(identity) is None or reconnect:
         try:
-            _connection = Connection(**_connection_settings)
+            _connection[identity] = Connection(**_connection_settings)
         except:
             raise ConnectionError('Cannot connect to the database')
-    return _connection
+    return _connection[identity]
 
-def _get_db():
+def _get_db(reconnect=False):
     global _db, _connection
+    identity = get_identity()
     # Connect if not already connected
-    if _connection is None:
-        _connection = _get_connection()
+    if _connection.get(identity) is None or reconnect:
+        _connection[identity] = _get_connection(reconnect=reconnect)
 
-    if _db is None:
+    if _db.get(identity) is None or reconnect:
         # _db_name will be None if the user hasn't called connect()
         if _db_name is None:
             raise ConnectionError('Not connected to the database')
 
         # Get DB from current connection and authenticate if necessary
-        _db = _connection[_db_name]
+        _db[identity] = _connection[identity][_db_name]
         if _db_username and _db_password:
-            _db.authenticate(_db_username, _db_password)
+            _db[identity].authenticate(_db_username, _db_password)
 
-    return _db
+    return _db[identity]
 
+def get_identity():
+    identity = multiprocessing.current_process()._identity
+    identity = 0 if not identity else identity[0]
+    return identity
+    
 def connect(db, username=None, password=None, **kwargs):
     """Connect to the database specified by the 'db' argument. Connection 
     settings may be provided here as well if the database is not running on
     the default port on localhost. If authentication is needed, provide
     username and password arguments as well.
     """
-    global _connection_settings, _db_name, _db_username, _db_password
-    _connection_settings.update(kwargs)
+    global _connection_settings, _db_name, _db_username, _db_password, _db
+    _connection_settings = dict(_connection_defaults, **kwargs)
     _db_name = db
     _db_username = username
     _db_password = password
+    return _get_db(reconnect=True)
+

mongoengine/django/auth.py

@@ -30,6 +30,10 @@ class User(Document):
     is_active = BooleanField(default=True)
     is_superuser = BooleanField(default=False)
     last_login = DateTimeField(default=datetime.datetime.now)
+    date_joined = DateTimeField(default=datetime.datetime.now)
+
+    def __unicode__(self):
+        return self.username
 
     def get_full_name(self):
         """Returns the users first and last names, separated by a space.
@@ -53,6 +57,8 @@ class User(Document):
         salt = get_hexdigest(algo, str(random()), str(random()))[:5]
         hash = get_hexdigest(algo, salt, raw_password)
         self.password = '%s$%s$%s' % (algo, salt, hash)
+        self.save()
+        return self
 
     def check_password(self, raw_password):
         """Checks the user's password against a provided password - always use
@@ -68,11 +74,26 @@ class User(Document):
         """Create (and save) a new user with the given username, password and
         email address.
         """
-        user = User(username=username, email=email)
+        now = datetime.datetime.now()
+
+        # Normalize the address by lowercasing the domain part of the email
+        # address.
+        if email is not None:
+            try:
+                email_name, domain_part = email.strip().split('@', 1)
+            except ValueError:
+                pass
+            else:
+                email = '@'.join([email_name, domain_part.lower()])
+
+        user = User(username=username, email=email, date_joined=now)
         user.set_password(password)
         user.save()
         return user
 
+    def get_and_delete_messages(self):
+        return []
+
 
 class MongoEngineBackend(object):
     """Authenticate using MongoEngine and mongoengine.django.auth.User.

mongoengine/django/sessions.py

@@ -0,0 +1,63 @@
+from django.contrib.sessions.backends.base import SessionBase, CreateError
+from django.core.exceptions import SuspiciousOperation
+from django.utils.encoding import force_unicode
+
+from mongoengine.document import Document
+from mongoengine import fields
+from mongoengine.queryset import OperationError
+
+from datetime import datetime
+
+
+class MongoSession(Document):
+    session_key = fields.StringField(primary_key=True, max_length=40)
+    session_data = fields.StringField()
+    expire_date = fields.DateTimeField()
+    
+    meta = {'collection': 'django_session', 'allow_inheritance': False}
+
+
+class SessionStore(SessionBase):
+    """A MongoEngine-based session store for Django.
+    """
+
+    def load(self):
+        try:
+            s = MongoSession.objects(session_key=self.session_key,
+                                     expire_date__gt=datetime.now())[0]
+            return self.decode(force_unicode(s.session_data))
+        except (IndexError, SuspiciousOperation):
+            self.create()
+            return {}
+
+    def exists(self, session_key):
+        return bool(MongoSession.objects(session_key=session_key).first())
+
+    def create(self):
+        while True:
+            self.session_key = self._get_new_session_key()
+            try:
+                self.save(must_create=True)
+            except CreateError:
+                continue
+            self.modified = True
+            self._session_cache = {}
+            return
+
+    def save(self, must_create=False):
+        s = MongoSession(session_key=self.session_key)
+        s.session_data = self.encode(self._get_session(no_load=must_create))
+        s.expire_date = self.get_expiry_date()
+        try:
+            s.save(force_insert=must_create, safe=True)
+        except OperationError:
+            if must_create:
+                raise CreateError
+            raise
+
+    def delete(self, session_key=None):
+        if session_key is None:
+            if self.session_key is None:
+                return
+            session_key = self.session_key
+        MongoSession.objects(session_key=session_key).delete()

mongoengine/django/shortcuts.py

@@ -0,0 +1,45 @@
+from django.http import Http404
+from mongoengine.queryset import QuerySet
+from mongoengine.base import BaseDocument
+
+def _get_queryset(cls):
+    """Inspired by django.shortcuts.*"""
+    if isinstance(cls, QuerySet):
+        return cls
+    else:
+        return cls.objects
+
+def get_document_or_404(cls, *args, **kwargs):
+    """
+    Uses get() to return an document, or raises a Http404 exception if the document
+    does not exist.
+
+    cls may be a Document or QuerySet object. All other passed
+    arguments and keyword arguments are used in the get() query.
+
+    Note: Like with get(), an MultipleObjectsReturned will be raised if more than one
+    object is found.
+
+    Inspired by django.shortcuts.*
+    """
+    queryset = _get_queryset(cls)
+    try:
+        return queryset.get(*args, **kwargs)
+    except queryset._document.DoesNotExist:
+        raise Http404('No %s matches the given query.' % queryset._document._class_name)
+
+def get_list_or_404(cls, *args, **kwargs):
+    """
+    Uses filter() to return a list of documents, or raise a Http404 exception if
+    the list is empty.
+
+    cls may be a Document or QuerySet object. All other passed
+    arguments and keyword arguments are used in the filter() query.
+
+    Inspired by django.shortcuts.*
+    """
+    queryset = _get_queryset(cls)
+    obj_list = list(queryset.filter(*args, **kwargs))
+    if not obj_list:
+        raise Http404('No %s matches the given query.' % queryset._document._class_name)
+    return obj_list

mongoengine/django/storage.py

@@ -0,0 +1,112 @@
+import os
+import itertools
+import urlparse
+
+from mongoengine import *
+from django.conf import settings
+from django.core.files.storage import Storage
+from django.core.exceptions import ImproperlyConfigured
+
+
+class FileDocument(Document):
+    """A document used to store a single file in GridFS.
+    """
+    file = FileField()
+
+
+class GridFSStorage(Storage):
+    """A custom storage backend to store files in GridFS
+    """
+
+    def __init__(self, base_url=None):
+
+        if base_url is None:
+            base_url = settings.MEDIA_URL
+        self.base_url = base_url
+        self.document = FileDocument
+        self.field = 'file'
+
+    def delete(self, name):
+        """Deletes the specified file from the storage system.
+        """
+        if self.exists(name):
+            doc = self.document.objects.first()
+            field = getattr(doc, self.field)
+            self._get_doc_with_name(name).delete()  # Delete the FileField
+            field.delete()                          # Delete the FileDocument
+
+    def exists(self, name):
+        """Returns True if a file referened by the given name already exists in the
+        storage system, or False if the name is available for a new file.
+        """
+        doc = self._get_doc_with_name(name)
+        if doc:
+            field = getattr(doc, self.field)
+            return bool(field.name)
+        else:
+            return False
+
+    def listdir(self, path=None):
+        """Lists the contents of the specified path, returning a 2-tuple of lists;
+        the first item being directories, the second item being files.
+        """
+        def name(doc):
+            return getattr(doc, self.field).name
+        docs = self.document.objects
+        return [], [name(d) for d in docs if name(d)]
+
+    def size(self, name):
+        """Returns the total size, in bytes, of the file specified by name.
+        """
+        doc = self._get_doc_with_name(name)
+        if doc:
+            return getattr(doc, self.field).length
+        else:
+            raise ValueError("No such file or directory: '%s'" % name)
+
+    def url(self, name):
+        """Returns an absolute URL where the file's contents can be accessed
+        directly by a web browser.
+        """
+        if self.base_url is None:
+            raise ValueError("This file is not accessible via a URL.")
+        return urlparse.urljoin(self.base_url, name).replace('\\', '/')
+
+    def _get_doc_with_name(self, name):
+        """Find the documents in the store with the given name
+        """
+        docs = self.document.objects
+        doc = [d for d in docs if getattr(d, self.field).name == name]
+        if doc:
+            return doc[0]
+        else:
+            return None
+
+    def _open(self, name, mode='rb'):
+        doc = self._get_doc_with_name(name)
+        if doc:
+            return getattr(doc, self.field)
+        else:
+            raise ValueError("No file found with the name '%s'." % name)
+
+    def get_available_name(self, name):
+        """Returns a filename that's free on the target storage system, and
+        available for new content to be written to.
+        """
+        file_root, file_ext = os.path.splitext(name)
+        # If the filename already exists, add an underscore and a number (before
+        # the file extension, if one exists) to the filename until the generated
+        # filename doesn't exist.
+        count = itertools.count(1)
+        while self.exists(name):
+            # file_ext includes the dot.
+            name = os.path.join("%s_%s%s" % (file_root, count.next(), file_ext))
+
+        return name
+
+    def _save(self, name, content):
+        doc = self.document()
+        getattr(doc, self.field).put(content, filename=name)
+        doc.save()
+
+        return name

mongoengine/django/tests.py

@@ -0,0 +1,21 @@
+#coding: utf-8
+from django.test import TestCase
+from django.conf import settings
+
+from mongoengine import connect
+
+class MongoTestCase(TestCase):
+    """
+    TestCase class that clear the collection between the tests
+    """
+    db_name = 'test_%s' % settings.MONGO_DATABASE_NAME
+    def __init__(self, methodName='runtest'):
+        self.db = connect(self.db_name)
+        super(MongoTestCase, self).__init__(methodName)
+
+    def _post_teardown(self):
+        super(MongoTestCase, self)._post_teardown()
+        for collection in self.db.collection_names():
+            if collection == 'system.indexes':
+                continue
+            self.db.drop_collection(collection)

mongoengine/document.py

@@ -15,7 +15,7 @@ class EmbeddedDocument(BaseDocument):
     fields on :class:`~mongoengine.Document`\ s through the
     :class:`~mongoengine.EmbeddedDocumentField` field type.
     """
-    
+
     __metaclass__ = DocumentMetaclass
 
 
@@ -40,69 +40,76 @@ class Document(BaseDocument):
     presence of `_cls` and `_types`, set :attr:`allow_inheritance` to
     ``False`` in the :attr:`meta` dictionary.
 
-    A :class:`~mongoengine.Document` may use a **Capped Collection** by 
+    A :class:`~mongoengine.Document` may use a **Capped Collection** by
     specifying :attr:`max_documents` and :attr:`max_size` in the :attr:`meta`
     dictionary. :attr:`max_documents` is the maximum number of documents that
-    is allowed to be stored in the collection, and :attr:`max_size` is the 
-    maximum size of the collection in bytes. If :attr:`max_size` is not 
-    specified and :attr:`max_documents` is, :attr:`max_size` defaults to 
+    is allowed to be stored in the collection, and :attr:`max_size` is the
+    maximum size of the collection in bytes. If :attr:`max_size` is not
+    specified and :attr:`max_documents` is, :attr:`max_size` defaults to
     10000000 bytes (10MB).
 
     Indexes may be created by specifying :attr:`indexes` in the :attr:`meta`
-    dictionary. The value should be a list of field names or tuples of field 
+    dictionary. The value should be a list of field names or tuples of field
     names. Index direction may be specified by prefixing the field names with
     a **+** or **-** sign.
     """
 
     __metaclass__ = TopLevelDocumentMetaclass
 
-    def save(self, safe=True):
+    def save(self, safe=True, force_insert=False, validate=True):
         """Save the :class:`~mongoengine.Document` to the database. If the
         document already exists, it will be updated, otherwise it will be
         created.
+
+        If ``safe=True`` and the operation is unsuccessful, an
+        :class:`~mongoengine.OperationError` will be raised.
+
+        :param safe: check if the operation succeeded before returning
+        :param force_insert: only try to create a new document, don't allow
+            updates of existing documents
+        :param validate: validates the document; set to ``False`` for skiping
         """
-        self.validate()
+        if validate:
+            self.validate()
         doc = self.to_mongo()
         try:
-            object_id = self.__class__.objects._collection.save(doc, safe=safe)
+            collection = self.__class__.objects._collection
+            if force_insert:
+                object_id = collection.insert(doc, safe=safe)
+            else:
+                object_id = collection.save(doc, safe=safe)
         except pymongo.errors.OperationFailure, err:
-            raise OperationError('Tried to save duplicate unique keys (%s)'
-                                  % str(err))
-        self.id = self._fields['id'].to_python(object_id)
+            message = 'Could not save document (%s)'
+            if u'duplicate key' in unicode(err):
+                message = u'Tried to save duplicate unique keys (%s)'
+            raise OperationError(message % unicode(err))
+        id_field = self._meta['id_field']
+        self[id_field] = self._fields[id_field].to_python(object_id)
 
-    def delete(self):
+    def delete(self, safe=False):
         """Delete the :class:`~mongoengine.Document` from the database. This
         will only take effect if the document has been previously saved.
+
+        :param safe: check if the operation succeeded before returning
         """
-        object_id = self._fields['id'].to_mongo(self.id)
-        self.__class__.objects(id=object_id).delete()
+        id_field = self._meta['id_field']
+        object_id = self._fields[id_field].to_mongo(self[id_field])
+        try:
+            self.__class__.objects(**{id_field: object_id}).delete(safe=safe)
+        except pymongo.errors.OperationFailure, err:
+            message = u'Could not delete document (%s)' % err.message
+            raise OperationError(message)
 
     def reload(self):
         """Reloads all attributes from the database.
+
+        .. versionadded:: 0.1.2
         """
-        obj = self.__class__.objects(id=self.id).first()
+        id_field = self._meta['id_field']
+        obj = self.__class__.objects(**{id_field: self[id_field]}).first()
         for field in self._fields:
             setattr(self, field, obj[field])
 
-    def validate(self):
-        """Ensure that all fields' values are valid and that required fields
-        are present.
-        """
-        # Get a list of tuples of field names and their current values
-        fields = [(field, getattr(self, name)) 
-                  for name, field in self._fields.items()]
-
-        # Ensure that each field is matched to a valid value
-        for field, value in fields:
-            if value is not None:
-                try:
-                    field.validate(value)
-                except (ValueError, AttributeError, AssertionError), e:
-                    raise ValidationError('Invalid value for field of type "' +
-                                          field.__class__.__name__ + '"')
-            elif field.required:
-                raise ValidationError('Field "%s" is required' % field.name)
-
     @classmethod
     def drop_collection(cls):
         """Drops the entire collection associated with this
@@ -110,3 +117,43 @@ class Document(BaseDocument):
         """
         db = _get_db()
         db.drop_collection(cls._meta['collection'])
+
+
+class MapReduceDocument(object):
+    """A document returned from a map/reduce query.
+
+    :param collection: An instance of :class:`~pymongo.Collection`
+    :param key: Document/result key, often an instance of
+                :class:`~bson.objectid.ObjectId`. If supplied as
+                an ``ObjectId`` found in the given ``collection``,
+                the object can be accessed via the ``object`` property.
+    :param value: The result(s) for this key.
+
+    .. versionadded:: 0.3
+    """
+
+    def __init__(self, document, collection, key, value):
+        self._document = document
+        self._collection = collection
+        self.key = key
+        self.value = value
+
+    @property
+    def object(self):
+        """Lazy-load the object referenced by ``self.key``. ``self.key``
+        should be the ``primary_key``.
+        """
+        id_field = self._document()._meta['id_field']
+        id_field_type = type(id_field)
+
+        if not isinstance(self.key, id_field_type):
+            try:
+                self.key = id_field_type(self.key)
+            except:
+                raise Exception("Could not cast key as %s" % \
+                                id_field_type.__name__)
+
+        if not hasattr(self, "_key_object"):
+            self._key_object = self._document.objects.with_id(self.key)
+            return self._key_object
+        return self._key_object

mongoengine/fields.py

@@ -1,26 +1,39 @@
-from base import BaseField, ObjectIdField, ValidationError
+from base import BaseField, ObjectIdField, ValidationError, get_document
 from document import Document, EmbeddedDocument
 from connection import _get_db
- 
+from operator import itemgetter
+
 import re
 import pymongo
+import bson.dbref
+import bson.son
+import bson.binary
 import datetime
+import decimal
+import gridfs
+import warnings
+import types
 
 
 __all__ = ['StringField', 'IntField', 'FloatField', 'BooleanField',
-           'DateTimeField', 'EmbeddedDocumentField', 'ListField', 
-           'ObjectIdField', 'ReferenceField', 'ValidationError']
+           'DateTimeField', 'EmbeddedDocumentField', 'ListField', 'DictField',
+           'ObjectIdField', 'ReferenceField', 'ValidationError',
+           'DecimalField', 'URLField', 'GenericReferenceField', 'FileField',
+           'BinaryField', 'SortedListField', 'EmailField', 'GeoPointField']
+
+RECURSIVE_REFERENCE_CONSTANT = 'self'
 
 
 class StringField(BaseField):
     """A unicode string field.
     """
-    
-    def __init__(self, regex=None, max_length=None, **kwargs):
+
+    def __init__(self, regex=None, max_length=None, min_length=None, **kwargs):
         self.regex = re.compile(regex) if regex else None
         self.max_length = max_length
+        self.min_length = min_length
         super(StringField, self).__init__(**kwargs)
-    
+
     def to_python(self, value):
         return unicode(value)
 
@@ -30,6 +43,9 @@ class StringField(BaseField):
         if self.max_length is not None and len(value) > self.max_length:
             raise ValidationError('String value is too long')
 
+        if self.min_length is not None and len(value) < self.min_length:
+            raise ValidationError('String value is too short')
+
         if self.regex is not None and self.regex.match(value) is None:
             message = 'String value did not match validation regex'
             raise ValidationError(message)
@@ -37,6 +53,79 @@ class StringField(BaseField):
     def lookup_member(self, member_name):
         return None
 
+    def prepare_query_value(self, op, value):
+        if not isinstance(op, basestring):
+            return value
+
+        if op.lstrip('i') in ('startswith', 'endswith', 'contains', 'exact'):
+            flags = 0
+            if op.startswith('i'):
+                flags = re.IGNORECASE
+                op = op.lstrip('i')
+
+            regex = r'%s'
+            if op == 'startswith':
+                regex = r'^%s'
+            elif op == 'endswith':
+                regex = r'%s$'
+            elif op == 'exact':
+                regex = r'^%s$'
+
+            # escape unsafe characters which could lead to a re.error
+            value = re.escape(value)
+            value = re.compile(regex % value, flags)
+        return value
+
+
+class URLField(StringField):
+    """A field that validates input as an URL.
+
+    .. versionadded:: 0.3
+    """
+
+    URL_REGEX = re.compile(
+        r'^https?://'
+        r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+[A-Z]{2,6}\.?|'
+        r'localhost|'
+        r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})'
+        r'(?::\d+)?'
+        r'(?:/?|[/?]\S+)$', re.IGNORECASE
+    )
+
+    def __init__(self, verify_exists=False, **kwargs):
+        self.verify_exists = verify_exists
+        super(URLField, self).__init__(**kwargs)
+
+    def validate(self, value):
+        if not URLField.URL_REGEX.match(value):
+            raise ValidationError('Invalid URL: %s' % value)
+
+        if self.verify_exists:
+            import urllib2
+            try:
+                request = urllib2.Request(value)
+                response = urllib2.urlopen(request)
+            except Exception, e:
+                message = 'This URL appears to be a broken link: %s' % e
+                raise ValidationError(message)
+
+
+class EmailField(StringField):
+    """A field that validates input as an E-Mail-Address.
+
+    .. versionadded:: 0.4
+    """
+
+    EMAIL_REGEX = re.compile(
+        r"(^[-!#$%&'*+/=?^_`{}|~0-9A-Z]+(\.[-!#$%&'*+/=?^_`{}|~0-9A-Z]+)*"  # dot-atom
+        r'|^"([\001-\010\013\014\016-\037!#-\[\]-\177]|\\[\001-011\013\014\016-\177])*"' # quoted-string
+        r')@(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+[A-Z]{2,6}\.?$', re.IGNORECASE # domain
+    )
+
+    def validate(self, value):
+        if not EmailField.EMAIL_REGEX.match(value):
+            raise ValidationError('Invalid Mail-address: %s' % value)
+
 
 class IntField(BaseField):
     """An integer field.
@@ -45,12 +134,15 @@ class IntField(BaseField):
     def __init__(self, min_value=None, max_value=None, **kwargs):
         self.min_value, self.max_value = min_value, max_value
         super(IntField, self).__init__(**kwargs)
-    
+
     def to_python(self, value):
         return int(value)
 
     def validate(self, value):
-        assert isinstance(value, (int, long))
+        try:
+            value = int(value)
+        except:
+            raise ValidationError('%s could not be converted to int' % value)
 
         if self.min_value is not None and value < self.min_value:
             raise ValidationError('Integer value is too small')
@@ -66,11 +158,13 @@ class FloatField(BaseField):
     def __init__(self, min_value=None, max_value=None, **kwargs):
         self.min_value, self.max_value = min_value, max_value
         super(FloatField, self).__init__(**kwargs)
-    
+
     def to_python(self, value):
         return float(value)
 
     def validate(self, value):
+        if isinstance(value, int):
+            value = float(value)
         assert isinstance(value, float)
 
         if self.min_value is not None and value < self.min_value:
@@ -79,11 +173,46 @@ class FloatField(BaseField):
         if self.max_value is not None and value > self.max_value:
             raise ValidationError('Float value is too large')
 
+class DecimalField(BaseField):
+    """A fixed-point decimal number field.
+
+    .. versionadded:: 0.3
+    """
+
+    def __init__(self, min_value=None, max_value=None, **kwargs):
+        self.min_value, self.max_value = min_value, max_value
+        super(DecimalField, self).__init__(**kwargs)
+
+    def to_python(self, value):
+        if not isinstance(value, basestring):
+            value = unicode(value)
+        return decimal.Decimal(value)
+
+    def to_mongo(self, value):
+        return unicode(value)
+
+    def validate(self, value):
+        if not isinstance(value, decimal.Decimal):
+            if not isinstance(value, basestring):
+                value = str(value)
+            try:
+                value = decimal.Decimal(value)
+            except Exception, exc:
+                raise ValidationError('Could not convert to decimal: %s' % exc)
+
+        if self.min_value is not None and value < self.min_value:
+            raise ValidationError('Decimal value is too small')
+
+        if self.max_value is not None and value > self.max_value:
+            raise ValidationError('Decimal value is too large')
+
 
 class BooleanField(BaseField):
     """A boolean field type.
+
+    .. versionadded:: 0.1.2
     """
-    
+
     def to_python(self, value):
         return bool(value)
 
@@ -100,36 +229,50 @@ class DateTimeField(BaseField):
 
 
 class EmbeddedDocumentField(BaseField):
-    """An embedded document field. Only valid values are subclasses of 
-    :class:`~mongoengine.EmbeddedDocument`. 
+    """An embedded document field. Only valid values are subclasses of
+    :class:`~mongoengine.EmbeddedDocument`.
     """
 
-    def __init__(self, document, **kwargs):
-        if not issubclass(document, EmbeddedDocument):
-            raise ValidationError('Invalid embedded document class provided '
-                                  'to an EmbeddedDocumentField')
-        self.document = document
+    def __init__(self, document_type, **kwargs):
+        if not isinstance(document_type, basestring):
+            if not issubclass(document_type, EmbeddedDocument):
+                raise ValidationError('Invalid embedded document class '
+                                      'provided to an EmbeddedDocumentField')
+        self.document_type_obj = document_type
         super(EmbeddedDocumentField, self).__init__(**kwargs)
-    
+
+    @property
+    def document_type(self):
+        if isinstance(self.document_type_obj, basestring):
+            if self.document_type_obj == RECURSIVE_REFERENCE_CONSTANT:
+                self.document_type_obj = self.owner_document
+            else:
+                self.document_type_obj = get_document(self.document_type_obj)
+        return self.document_type_obj
+
     def to_python(self, value):
-        if not isinstance(value, self.document):
-            return self.document._from_son(value)
+        if not isinstance(value, self.document_type):
+            return self.document_type._from_son(value)
         return value
 
     def to_mongo(self, value):
-        return self.document.to_mongo(value)
+        return self.document_type.to_mongo(value)
 
     def validate(self, value):
-        """Make sure that the document instance is an instance of the 
+        """Make sure that the document instance is an instance of the
         EmbeddedDocument subclass provided when the document was defined.
         """
         # Using isinstance also works for subclasses of self.document
-        if not isinstance(value, self.document):
+        if not isinstance(value, self.document_type):
             raise ValidationError('Invalid embedded document instance '
                                   'provided to an EmbeddedDocumentField')
+        self.document_type.validate(value)
 
     def lookup_member(self, member_name):
-        return self.document._fields.get(member_name)
+        return self.document_type._fields.get(member_name)
+
+    def prepare_query_value(self, op, value):
+        return self.to_mongo(value)
 
 
 class ListField(BaseField):
@@ -137,13 +280,53 @@ class ListField(BaseField):
     of the field to be used as a list in the database.
     """
 
+    # ListFields cannot be indexed with _types - MongoDB doesn't support this
+    _index_with_types = False
+
     def __init__(self, field, **kwargs):
         if not isinstance(field, BaseField):
             raise ValidationError('Argument to ListField constructor must be '
                                   'a valid field')
         self.field = field
+        kwargs.setdefault('default', lambda: [])
         super(ListField, self).__init__(**kwargs)
 
+    def __get__(self, instance, owner):
+        """Descriptor to automatically dereference references.
+        """
+        if instance is None:
+            # Document class being used rather than a document object
+            return self
+
+        if isinstance(self.field, ReferenceField):
+            referenced_type = self.field.document_type
+            # Get value from document instance if available
+            value_list = instance._data.get(self.name)
+            if value_list:
+                deref_list = []
+                for value in value_list:
+                    # Dereference DBRefs
+                    if isinstance(value, (bson.dbref.DBRef)):
+                        value = _get_db().dereference(value)
+                        deref_list.append(referenced_type._from_son(value))
+                    else:
+                        deref_list.append(value)
+                instance._data[self.name] = deref_list
+
+        if isinstance(self.field, GenericReferenceField):
+            value_list = instance._data.get(self.name)
+            if value_list:
+                deref_list = []
+                for value in value_list:
+                    # Dereference DBRefs
+                    if isinstance(value, (dict, bson.son.SON)):
+                        deref_list.append(self.field.dereference(value))
+                    else:
+                        deref_list.append(value)
+                instance._data[self.name] = deref_list
+
+        return super(ListField, self).__get__(instance, owner)
+
     def to_python(self, value):
         return [self.field.to_python(item) for item in value]
 
@@ -159,13 +342,75 @@ class ListField(BaseField):
 
         try:
             [self.field.validate(item) for item in value]
-        except:
-            raise ValidationError('All items in a list field must be of the '
-                                  'specified type')
+        except Exception, err:
+            raise ValidationError('Invalid ListField item (%s)' % str(item))
+
+    def prepare_query_value(self, op, value):
+        if op in ('set', 'unset'):
+            return [self.field.prepare_query_value(op, v) for v in value]
+        return self.field.prepare_query_value(op, value)
 
     def lookup_member(self, member_name):
         return self.field.lookup_member(member_name)
 
+    def _set_owner_document(self, owner_document):
+        self.field.owner_document = owner_document
+        self._owner_document = owner_document
+
+    def _get_owner_document(self, owner_document):
+        self._owner_document = owner_document
+
+    owner_document = property(_get_owner_document, _set_owner_document)
+
+
+class SortedListField(ListField):
+    """A ListField that sorts the contents of its list before writing to
+    the database in order to ensure that a sorted list is always
+    retrieved.
+
+    .. versionadded:: 0.4
+    """
+
+    _ordering = None
+
+    def __init__(self, field, **kwargs):
+        if 'ordering' in kwargs.keys():
+            self._ordering = kwargs.pop('ordering')
+        super(SortedListField, self).__init__(field, **kwargs)
+
+    def to_mongo(self, value):
+        if self._ordering is not None:
+            return sorted([self.field.to_mongo(item) for item in value],
+                          key=itemgetter(self._ordering))
+        return sorted([self.field.to_mongo(item) for item in value])
+
+
+class DictField(BaseField):
+    """A dictionary field that wraps a standard Python dictionary. This is
+    similar to an embedded document, but the structure is not defined.
+
+    .. versionadded:: 0.3
+    """
+
+    def __init__(self, basecls=None, *args, **kwargs):
+        self.basecls = basecls or BaseField
+        assert issubclass(self.basecls, BaseField)
+        kwargs.setdefault('default', lambda: {})
+        super(DictField, self).__init__(*args, **kwargs)
+
+    def validate(self, value):
+        """Make sure that a list of valid fields is being used.
+        """
+        if not isinstance(value, dict):
+            raise ValidationError('Only dictionaries may be used in a '
+                                  'DictField')
+
+        if any(('.' in k or '$' in k) for k in value):
+            raise ValidationError('Invalid dictionary key name - keys may not '
+                                  'contain "." or "$" characters')
+
+    def lookup_member(self, member_name):
+        return self.basecls(db_field=member_name)
 
 class ReferenceField(BaseField):
     """A reference to a document that will be automatically dereferenced on
@@ -173,13 +418,22 @@ class ReferenceField(BaseField):
     """
 
     def __init__(self, document_type, **kwargs):
-        if not issubclass(document_type, Document):
-            raise ValidationError('Argument to ReferenceField constructor '
-                                  'must be a top level document class')
-        self.document_type = document_type
-        self.document_obj = None
+        if not isinstance(document_type, basestring):
+            if not issubclass(document_type, (Document, basestring)):
+                raise ValidationError('Argument to ReferenceField constructor '
+                                      'must be a document class or a string')
+        self.document_type_obj = document_type
         super(ReferenceField, self).__init__(**kwargs)
 
+    @property
+    def document_type(self):
+        if isinstance(self.document_type_obj, basestring):
+            if self.document_type_obj == RECURSIVE_REFERENCE_CONSTANT:
+                self.document_type_obj = self.owner_document
+            else:
+                self.document_type_obj = get_document(self.document_type_obj)
+        return self.document_type_obj
+
     def __get__(self, instance, owner):
         """Descriptor to allow lazy dereferencing.
         """
@@ -190,36 +444,262 @@ class ReferenceField(BaseField):
         # Get value from document instance if available
         value = instance._data.get(self.name)
         # Dereference DBRefs
-        if isinstance(value, (pymongo.dbref.DBRef)):
+        if isinstance(value, (bson.dbref.DBRef)):
             value = _get_db().dereference(value)
             if value is not None:
                 instance._data[self.name] = self.document_type._from_son(value)
-        
+
         return super(ReferenceField, self).__get__(instance, owner)
 
     def to_mongo(self, document):
-        if isinstance(document, (str, unicode, pymongo.objectid.ObjectId)):
-            # document may already be an object id
-            id_ = document
-        else:
+        id_field_name = self.document_type._meta['id_field']
+        id_field = self.document_type._fields[id_field_name]
+
+        if isinstance(document, Document):
             # We need the id from the saved object to create the DBRef
             id_ = document.id
             if id_ is None:
                 raise ValidationError('You can only reference documents once '
                                       'they have been saved to the database')
+        else:
+            id_ = document
 
-        # id may be a string rather than an ObjectID object
-        if not isinstance(id_, pymongo.objectid.ObjectId):
-            id_ = pymongo.objectid.ObjectId(id_)
-
+        id_ = id_field.to_mongo(id_)
         collection = self.document_type._meta['collection']
-        return pymongo.dbref.DBRef(collection, id_)
-    
-    def prepare_query_value(self, value):
+        return bson.dbref.DBRef(collection, id_)
+
+    def prepare_query_value(self, op, value):
         return self.to_mongo(value)
 
     def validate(self, value):
-        assert isinstance(value, (self.document_type, pymongo.dbref.DBRef))
+        assert isinstance(value, (self.document_type, bson.dbref.DBRef))
 
     def lookup_member(self, member_name):
         return self.document_type._fields.get(member_name)
+
+
+class GenericReferenceField(BaseField):
+    """A reference to *any* :class:`~mongoengine.document.Document` subclass
+    that will be automatically dereferenced on access (lazily).
+
+    .. versionadded:: 0.3
+    """
+
+    def __get__(self, instance, owner):
+        if instance is None:
+            return self
+
+        value = instance._data.get(self.name)
+        if isinstance(value, (dict, bson.son.SON)):
+            instance._data[self.name] = self.dereference(value)
+
+        return super(GenericReferenceField, self).__get__(instance, owner)
+
+    def dereference(self, value):
+        doc_cls = get_document(value['_cls'])
+        reference = value['_ref']
+        doc = _get_db().dereference(reference)
+        if doc is not None:
+            doc = doc_cls._from_son(doc)
+        return doc
+
+    def to_mongo(self, document):
+        id_field_name = document.__class__._meta['id_field']
+        id_field = document.__class__._fields[id_field_name]
+
+        if isinstance(document, Document):
+            # We need the id from the saved object to create the DBRef
+            id_ = document.id
+            if id_ is None:
+                raise ValidationError('You can only reference documents once '
+                                      'they have been saved to the database')
+        else:
+            id_ = document
+
+        id_ = id_field.to_mongo(id_)
+        collection = document._meta['collection']
+        ref = bson.dbref.DBRef(collection, id_)
+        return {'_cls': document.__class__.__name__, '_ref': ref}
+
+    def prepare_query_value(self, op, value):
+        return self.to_mongo(value)
+
+
+class BinaryField(BaseField):
+    """A binary data field.
+    """
+
+    def __init__(self, max_bytes=None, **kwargs):
+        self.max_bytes = max_bytes
+        super(BinaryField, self).__init__(**kwargs)
+
+    def to_mongo(self, value):
+        return bson.binary.Binary(value)
+
+    def to_python(self, value):
+        # Returns str not unicode as this is binary data
+        return str(value)
+
+    def validate(self, value):
+        assert isinstance(value, str)
+
+        if self.max_bytes is not None and len(value) > self.max_bytes:
+            raise ValidationError('Binary value is too long')
+
+
+class GridFSError(Exception):
+    pass
+
+
+class GridFSProxy(object):
+    """Proxy object to handle writing and reading of files to and from GridFS
+
+    .. versionadded:: 0.4
+    """
+
+    def __init__(self, grid_id=None):
+        self.fs = gridfs.GridFS(_get_db())  # Filesystem instance
+        self.newfile = None                 # Used for partial writes
+        self.grid_id = grid_id              # Store GridFS id for file
+
+    def __getattr__(self, name):
+        obj = self.get()
+        if name in dir(obj):
+            return getattr(obj, name)
+        raise AttributeError
+
+    def __get__(self, instance, value):
+        return self
+
+    def get(self, id=None):
+        if id:
+            self.grid_id = id
+        try:
+            return self.fs.get(id or self.grid_id)
+        except:
+            # File has been deleted
+            return None
+
+    def new_file(self, **kwargs):
+        self.newfile = self.fs.new_file(**kwargs)
+        self.grid_id = self.newfile._id
+
+    def put(self, file, **kwargs):
+        if self.grid_id:
+            raise GridFSError('This document already has a file. Either delete '
+                              'it or call replace to overwrite it')
+        self.grid_id = self.fs.put(file, **kwargs)
+
+    def write(self, string):
+        if self.grid_id:
+            if not self.newfile:
+                raise GridFSError('This document already has a file. Either '
+                                  'delete it or call replace to overwrite it')
+        else:
+            self.new_file()
+        self.newfile.write(string)
+
+    def writelines(self, lines):
+        if not self.newfile:
+            self.new_file()
+            self.grid_id = self.newfile._id
+        self.newfile.writelines(lines)
+
+    def read(self):
+        try:
+            return self.get().read()
+        except:
+            return None
+
+    def delete(self):
+        # Delete file from GridFS, FileField still remains
+        self.fs.delete(self.grid_id)
+        self.grid_id = None
+
+    def replace(self, file, **kwargs):
+        self.delete()
+        self.put(file, **kwargs)
+
+    def close(self):
+        if self.newfile:
+            self.newfile.close()
+        else:
+            msg = "The close() method is only necessary after calling write()"
+            warnings.warn(msg)
+
+
+class FileField(BaseField):
+    """A GridFS storage field.
+
+    .. versionadded:: 0.4
+    """
+
+    def __init__(self, **kwargs):
+        super(FileField, self).__init__(**kwargs)
+
+    def __get__(self, instance, owner):
+        if instance is None:
+            return self
+
+        # Check if a file already exists for this model
+        grid_file = instance._data.get(self.name)
+        self.grid_file = grid_file
+        if self.grid_file:
+            return self.grid_file
+        return GridFSProxy()
+
+    def __set__(self, instance, value):
+        if isinstance(value, file) or isinstance(value, str):
+            # using "FileField() = file/string" notation
+            grid_file = instance._data.get(self.name)
+            # If a file already exists, delete it
+            if grid_file:
+                try:
+                    grid_file.delete()
+                except:
+                    pass
+                # Create a new file with the new data
+                grid_file.put(value)
+            else:
+                # Create a new proxy object as we don't already have one
+                instance._data[self.name] = GridFSProxy()
+                instance._data[self.name].put(value)
+        else:
+            instance._data[self.name] = value
+
+    def to_mongo(self, value):
+        # Store the GridFS file id in MongoDB
+        if isinstance(value, GridFSProxy) and value.grid_id is not None:
+            return value.grid_id
+        return None
+
+    def to_python(self, value):
+        if value is not None:
+            return GridFSProxy(value)
+
+    def validate(self, value):
+        if value.grid_id is not None:
+            assert isinstance(value, GridFSProxy)
+            assert isinstance(value.grid_id, bson.objectid.ObjectId)
+
+
+class GeoPointField(BaseField):
+    """A list storing a latitude and longitude.
+
+    .. versionadded:: 0.4
+    """
+
+    _geo_index = True
+
+    def validate(self, value):
+        """Make sure that a geo-value is of type (x, y)
+        """
+        if not isinstance(value, (list, tuple)):
+            raise ValidationError('GeoPointField can only accept tuples or '
+                                  'lists of (x, y)')
+
+        if not len(value) == 2:
+            raise ValidationError('Value must be a two-dimensional point.')
+        if (not isinstance(value[0], (float, int)) and
+            not isinstance(value[1], (float, int))):
+            raise ValidationError('Both values in point must be float or int.')

tests/document.py

@@ -1,5 +1,6 @@
 import unittest
-import datetime
+from datetime import datetime
+import bson
 import pymongo
 
 from mongoengine import *
@@ -7,7 +8,7 @@ from mongoengine.connection import _get_db
 
 
 class DocumentTest(unittest.TestCase):
-    
+
     def setUp(self):
         connect(db='mongoenginetest')
         self.db = _get_db()
@@ -38,7 +39,7 @@ class DocumentTest(unittest.TestCase):
             name = name_field
             age = age_field
             non_field = True
-        
+
         self.assertEqual(Person._fields['name'], name_field)
         self.assertEqual(Person._fields['age'], age_field)
         self.assertFalse('non_field' in Person._fields)
@@ -60,7 +61,7 @@ class DocumentTest(unittest.TestCase):
 
         mammal_superclasses = {'Animal': Animal}
         self.assertEqual(Mammal._superclasses, mammal_superclasses)
-        
+
         dog_superclasses = {
             'Animal': Animal,
             'Animal.Mammal': Mammal,
@@ -68,7 +69,7 @@ class DocumentTest(unittest.TestCase):
         self.assertEqual(Dog._superclasses, dog_superclasses)
 
     def test_get_subclasses(self):
-        """Ensure that the correct list of subclasses is retrieved by the 
+        """Ensure that the correct list of subclasses is retrieved by the
         _get_subclasses method.
         """
         class Animal(Document): pass
@@ -78,15 +79,15 @@ class DocumentTest(unittest.TestCase):
         class Dog(Mammal): pass
 
         mammal_subclasses = {
-            'Animal.Mammal.Dog': Dog, 
+            'Animal.Mammal.Dog': Dog,
             'Animal.Mammal.Human': Human
         }
         self.assertEqual(Mammal._get_subclasses(), mammal_subclasses)
-        
+
         animal_subclasses = {
             'Animal.Fish': Fish,
             'Animal.Mammal': Mammal,
-            'Animal.Mammal.Dog': Dog, 
+            'Animal.Mammal.Dog': Dog,
             'Animal.Mammal.Human': Human
         }
         self.assertEqual(Animal._get_subclasses(), animal_subclasses)
@@ -124,9 +125,14 @@ class DocumentTest(unittest.TestCase):
 
         self.assertTrue('name' in Employee._fields)
         self.assertTrue('salary' in Employee._fields)
-        self.assertEqual(Employee._meta['collection'], 
+        self.assertEqual(Employee._meta['collection'],
                          self.Person._meta['collection'])
 
+        # Ensure that MRO error is not raised
+        class A(Document): pass
+        class B(A): pass
+        class C(B): pass
+
     def test_allow_inheritance(self):
         """Ensure that inheritance may be disabled on simple classes and that
         _cls and _types will not be used.
@@ -141,7 +147,7 @@ class DocumentTest(unittest.TestCase):
             class Dog(Animal):
                 pass
         self.assertRaises(ValueError, create_dog_class)
-        
+
         # Check that _cls etc aren't present on simple documents
         dog = Animal(name='dog')
         dog.save()
@@ -157,6 +163,20 @@ class DocumentTest(unittest.TestCase):
                 meta = {'allow_inheritance': False}
         self.assertRaises(ValueError, create_employee_class)
 
+        # Test the same for embedded documents
+        class Comment(EmbeddedDocument):
+            content = StringField()
+            meta = {'allow_inheritance': False}
+
+        def create_special_comment():
+            class SpecialComment(Comment):
+                pass
+        self.assertRaises(ValueError, create_special_comment)
+
+        comment = Comment(content='test')
+        self.assertFalse('_cls' in comment.to_mongo())
+        self.assertFalse('_types' in comment.to_mongo())
+
     def test_collection_name(self):
         """Ensure that a collection with a specified name may be used.
         """
@@ -167,7 +187,7 @@ class DocumentTest(unittest.TestCase):
         class Person(Document):
             name = StringField()
             meta = {'collection': collection}
-        
+
         user = Person(name="Test User")
         user.save()
         self.assertTrue(collection in self.db.collection_names())
@@ -181,11 +201,42 @@ class DocumentTest(unittest.TestCase):
         Person.drop_collection()
         self.assertFalse(collection in self.db.collection_names())
 
+    def test_inherited_collections(self):
+        """Ensure that subclassed documents don't override parents' collections.
+        """
+        class Drink(Document):
+            name = StringField()
+
+        class AlcoholicDrink(Drink):
+            meta = {'collection': 'booze'}
+
+        class Drinker(Document):
+            drink = GenericReferenceField()
+
+        Drink.drop_collection()
+        AlcoholicDrink.drop_collection()
+        Drinker.drop_collection()
+
+        red_bull = Drink(name='Red Bull')
+        red_bull.save()
+
+        programmer = Drinker(drink=red_bull)
+        programmer.save()
+
+        beer = AlcoholicDrink(name='Beer')
+        beer.save()
+
+        real_person = Drinker(drink=beer)
+        real_person.save()
+
+        self.assertEqual(Drinker.objects[0].drink.name, red_bull.name)
+        self.assertEqual(Drinker.objects[1].drink.name, beer.name)
+
     def test_capped_collection(self):
         """Ensure that capped collections work properly.
         """
         class Log(Document):
-            date = DateTimeField(default=datetime.datetime.now)
+            date = DateTimeField(default=datetime.now)
             meta = {
                 'max_documents': 10,
                 'max_size': 90000,
@@ -211,7 +262,7 @@ class DocumentTest(unittest.TestCase):
         # Check that the document cannot be redefined with different options
         def recreate_log_document():
             class Log(Document):
-                date = DateTimeField(default=datetime.datetime.now)
+                date = DateTimeField(default=datetime.now)
                 meta = {
                     'max_documents': 11,
                 }
@@ -225,11 +276,13 @@ class DocumentTest(unittest.TestCase):
         """Ensure that indexes are used when meta[indexes] is specified.
         """
         class BlogPost(Document):
-            date = DateTimeField(name='addDate', default=datetime.datetime.now)
+            date = DateTimeField(db_field='addDate', default=datetime.now)
             category = StringField()
+            tags = ListField(StringField())
             meta = {
                 'indexes': [
-                    '-date', 
+                    '-date',
+                    'tags',
                     ('category', '-date')
                 ],
             }
@@ -237,14 +290,32 @@ class DocumentTest(unittest.TestCase):
         BlogPost.drop_collection()
 
         info = BlogPost.objects._collection.index_information()
-        self.assertEqual(len(info), 4) # _id, types, '-date', ('cat', 'date')
+        # _id, types, '-date', 'tags', ('cat', 'date')
+        self.assertEqual(len(info), 5)
 
         # Indexes are lazy so use list() to perform query
         list(BlogPost.objects)
         info = BlogPost.objects._collection.index_information()
-        self.assertTrue([('_types', 1), ('category', 1), ('addDate', -1)] 
-                        in info.values())
-        self.assertTrue([('_types', 1), ('addDate', -1)] in info.values())
+        info = [value['key'] for key, value in info.iteritems()]
+        self.assertTrue([('_types', 1), ('category', 1), ('addDate', -1)]
+                        in info)
+        self.assertTrue([('_types', 1), ('addDate', -1)] in info)
+        # tags is a list field so it shouldn't have _types in the index
+        self.assertTrue([('tags', 1)] in info)
+
+        class ExtendedBlogPost(BlogPost):
+            title = StringField()
+            meta = {'indexes': ['title']}
+
+        BlogPost.drop_collection()
+
+        list(ExtendedBlogPost.objects)
+        info = ExtendedBlogPost.objects._collection.index_information()
+        info = [value['key'] for key, value in info.iteritems()]
+        self.assertTrue([('_types', 1), ('category', 1), ('addDate', -1)]
+                        in info)
+        self.assertTrue([('_types', 1), ('addDate', -1)] in info)
+        self.assertTrue([('_types', 1), ('title', 1)] in info)
 
         BlogPost.drop_collection()
 
@@ -265,7 +336,7 @@ class DocumentTest(unittest.TestCase):
         self.assertRaises(OperationError, post2.save)
 
         class Date(EmbeddedDocument):
-            year = IntField(name='yr')
+            year = IntField(db_field='yr')
 
         class BlogPost(Document):
             title = StringField()
@@ -287,6 +358,56 @@ class DocumentTest(unittest.TestCase):
 
         BlogPost.drop_collection()
 
+    def test_custom_id_field(self):
+        """Ensure that documents may be created with custom primary keys.
+        """
+        class User(Document):
+            username = StringField(primary_key=True)
+            name = StringField()
+
+        User.drop_collection()
+
+        self.assertEqual(User._fields['username'].db_field, '_id')
+        self.assertEqual(User._meta['id_field'], 'username')
+
+        def create_invalid_user():
+            User(name='test').save() # no primary key field
+        self.assertRaises(ValidationError, create_invalid_user)
+
+        def define_invalid_user():
+            class EmailUser(User):
+                email = StringField(primary_key=True)
+        self.assertRaises(ValueError, define_invalid_user)
+
+        class EmailUser(User):
+            email = StringField()
+
+        user = User(username='test', name='test user')
+        user.save()
+
+        user_obj = User.objects.first()
+        self.assertEqual(user_obj.id, 'test')
+        self.assertEqual(user_obj.pk, 'test')
+
+        user_son = User.objects._collection.find_one()
+        self.assertEqual(user_son['_id'], 'test')
+        self.assertTrue('username' not in user_son['_id'])
+
+        User.drop_collection()
+
+        user = User(pk='mongo', name='mongo user')
+        user.save()
+
+        user_obj = User.objects.first()
+        self.assertEqual(user_obj.id, 'mongo')
+        self.assertEqual(user_obj.pk, 'mongo')
+
+        user_son = User.objects._collection.find_one()
+        self.assertEqual(user_son['_id'], 'mongo')
+        self.assertTrue('username' not in user_son['_id'])
+
+        User.drop_collection()
+
     def test_creation(self):
         """Ensure that document may be created using keyword arguments.
         """
@@ -337,10 +458,29 @@ class DocumentTest(unittest.TestCase):
         """
         class Comment(EmbeddedDocument):
             content = StringField()
-        
+
         self.assertTrue('content' in Comment._fields)
         self.assertFalse('id' in Comment._fields)
-        self.assertFalse(hasattr(Comment, '_meta'))
+        self.assertFalse('collection' in Comment._meta)
+
+    def test_embedded_document_validation(self):
+        """Ensure that embedded documents may be validated.
+        """
+        class Comment(EmbeddedDocument):
+            date = DateTimeField()
+            content = StringField(required=True)
+
+        comment = Comment()
+        self.assertRaises(ValidationError, comment.validate)
+
+        comment.content = 'test'
+        comment.validate()
+
+        comment.date = 4
+        self.assertRaises(ValidationError, comment.validate)
+
+        comment.date = datetime.now()
+        comment.validate()
 
     def test_save(self):
         """Ensure that a document may be saved in the database.
@@ -353,7 +493,17 @@ class DocumentTest(unittest.TestCase):
         person_obj = collection.find_one({'name': 'Test User'})
         self.assertEqual(person_obj['name'], 'Test User')
         self.assertEqual(person_obj['age'], 30)
-        self.assertEqual(str(person_obj['_id']), person.id)
+        self.assertEqual(person_obj['_id'], person.id)
+        # Test skipping validation on save
+        class Recipient(Document):
+            email = EmailField(required=True)
+
+        recipient = Recipient(email='root@localhost')
+        self.assertRaises(ValidationError, recipient.save)
+        try:
+            recipient.save(validate=False)
+        except ValidationError:
+            fail()
 
     def test_delete(self):
         """Ensure that document may be deleted using the delete method.
@@ -368,7 +518,7 @@ class DocumentTest(unittest.TestCase):
         """Ensure that a document may be saved with a custom _id.
         """
         # Create person object and save it to the database
-        person = self.Person(name='Test User', age=30, 
+        person = self.Person(name='Test User', age=30,
                              id='497ce96f395f2f052a494fd4')
         person.save()
         # Ensure that the object is in the database with the correct _id
@@ -376,6 +526,18 @@ class DocumentTest(unittest.TestCase):
         person_obj = collection.find_one({'name': 'Test User'})
         self.assertEqual(str(person_obj['_id']), '497ce96f395f2f052a494fd4')
 
+    def test_save_custom_pk(self):
+        """Ensure that a document may be saved with a custom _id using pk alias.
+        """
+        # Create person object and save it to the database
+        person = self.Person(name='Test User', age=30,
+                             pk='497ce96f395f2f052a494fd4')
+        person.save()
+        # Ensure that the object is in the database with the correct _id
+        collection = self.db[self.Person._meta['collection']]
+        person_obj = collection.find_one({'name': 'Test User'})
+        self.assertEqual(str(person_obj['_id']), '497ce96f395f2f052a494fd4')
+
     def test_save_list(self):
         """Ensure that a list field may be properly saved.
         """
@@ -404,7 +566,7 @@ class DocumentTest(unittest.TestCase):
         BlogPost.drop_collection()
 
     def test_save_embedded_document(self):
-        """Ensure that a document with an embedded document field may be 
+        """Ensure that a document with an embedded document field may be
         saved in the database.
         """
         class EmployeeDetails(EmbeddedDocument):
@@ -430,7 +592,7 @@ class DocumentTest(unittest.TestCase):
     def test_save_reference(self):
         """Ensure that a document reference field may be saved in the database.
         """
-        
+
         class BlogPost(Document):
             meta = {'collection': 'blogpost_1'}
             content = StringField()
@@ -449,8 +611,8 @@ class DocumentTest(unittest.TestCase):
         post_obj = BlogPost.objects.first()
 
         # Test laziness
-        self.assertTrue(isinstance(post_obj._data['author'], 
-                                   pymongo.dbref.DBRef))
+        self.assertTrue(isinstance(post_obj._data['author'],
+                                   bson.dbref.DBRef))
         self.assertTrue(isinstance(post_obj.author, self.Person))
         self.assertEqual(post_obj.author.name, 'Test User')
 

tests/fields.py

@@ -1,5 +1,9 @@
 import unittest
 import datetime
+from decimal import Decimal
+
+import pymongo
+import gridfs
 
 from mongoengine import *
 from mongoengine.connection import _get_db
@@ -79,6 +83,19 @@ class FieldTest(unittest.TestCase):
         person.name = 'Shorter name'
         person.validate()
 
+    def test_url_validation(self):
+        """Ensure that URLFields validate urls properly.
+        """
+        class Link(Document):
+            url = URLField()
+
+        link = Link()
+        link.url = 'google'
+        self.assertRaises(ValidationError, link.validate)
+
+        link.url = 'http://www.google.com:8080'
+        link.validate()
+        
     def test_int_validation(self):
         """Ensure that invalid values cannot be assigned to int fields.
         """
@@ -106,12 +123,38 @@ class FieldTest(unittest.TestCase):
         person.height = 1.89
         person.validate()
 
-        person.height = 2
+        person.height = '2.0'
         self.assertRaises(ValidationError, person.validate)
         person.height = 0.01
         self.assertRaises(ValidationError, person.validate)
         person.height = 4.0
         self.assertRaises(ValidationError, person.validate)
+        
+    def test_decimal_validation(self):
+        """Ensure that invalid values cannot be assigned to decimal fields.
+        """
+        class Person(Document):
+            height = DecimalField(min_value=Decimal('0.1'), 
+                                  max_value=Decimal('3.5'))
+
+        Person.drop_collection()
+
+        person = Person()
+        person.height = Decimal('1.89')
+        person.save()
+        person.reload()
+        self.assertEqual(person.height, Decimal('1.89'))
+
+        person.height = '2.0'
+        person.save()
+        person.height = 0.01
+        self.assertRaises(ValidationError, person.validate)
+        person.height = Decimal('0.01')
+        self.assertRaises(ValidationError, person.validate)
+        person.height = Decimal('4.0')
+        self.assertRaises(ValidationError, person.validate)
+
+        Person.drop_collection()
 
     def test_boolean_validation(self):
         """Ensure that invalid values cannot be assigned to boolean fields.
@@ -146,6 +189,9 @@ class FieldTest(unittest.TestCase):
     def test_list_validation(self):
         """Ensure that a list field only accepts lists with valid elements.
         """
+        class User(Document):
+            pass
+
         class Comment(EmbeddedDocument):
             content = StringField()
 
@@ -153,6 +199,7 @@ class FieldTest(unittest.TestCase):
             content = StringField()
             comments = ListField(EmbeddedDocumentField(Comment))
             tags = ListField(StringField())
+            authors = ListField(ReferenceField(User))
 
         post = BlogPost(content='Went for a walk today...')
         post.validate()
@@ -167,15 +214,75 @@ class FieldTest(unittest.TestCase):
         post.tags = ('fun', 'leisure')
         post.validate()
 
-        comments = [Comment(content='Good for you'), Comment(content='Yay.')]
-        post.comments = comments
-        post.validate()
-
         post.comments = ['a']
         self.assertRaises(ValidationError, post.validate)
         post.comments = 'yay'
         self.assertRaises(ValidationError, post.validate)
 
+        comments = [Comment(content='Good for you'), Comment(content='Yay.')]
+        post.comments = comments
+        post.validate()
+
+        post.authors = [Comment()]
+        self.assertRaises(ValidationError, post.validate)
+
+        post.authors = [User()]
+        post.validate()
+
+    def test_sorted_list_sorting(self):
+        """Ensure that a sorted list field properly sorts values.
+        """
+        class Comment(EmbeddedDocument):
+            order = IntField()
+            content = StringField()
+
+        class BlogPost(Document):
+            content = StringField()
+            comments = SortedListField(EmbeddedDocumentField(Comment),
+                                       ordering='order')
+            tags = SortedListField(StringField())
+
+        post = BlogPost(content='Went for a walk today...')
+        post.save()
+
+        post.tags = ['leisure', 'fun']
+        post.save()
+        post.reload()
+        self.assertEqual(post.tags, ['fun', 'leisure'])
+        
+        comment1 = Comment(content='Good for you', order=1)
+        comment2 = Comment(content='Yay.', order=0)
+        comments = [comment1, comment2]
+        post.comments = comments
+        post.save()
+        post.reload()
+        self.assertEqual(post.comments[0].content, comment2.content)
+        self.assertEqual(post.comments[1].content, comment1.content)
+
+        BlogPost.drop_collection()
+
+    def test_dict_validation(self):
+        """Ensure that dict types work as expected.
+        """
+        class BlogPost(Document):
+            info = DictField()
+
+        post = BlogPost()
+        post.info = 'my post'
+        self.assertRaises(ValidationError, post.validate)
+
+        post.info = ['test', 'test']
+        self.assertRaises(ValidationError, post.validate)
+
+        post.info = {'$title': 'test'}
+        self.assertRaises(ValidationError, post.validate)
+
+        post.info = {'the.title': 'test'}
+        self.assertRaises(ValidationError, post.validate)
+
+        post.info = {'title': 'test'}
+        post.validate()
+
     def test_embedded_document_validation(self):
         """Ensure that invalid embedded documents cannot be assigned to
         embedded document fields.
@@ -184,7 +291,7 @@ class FieldTest(unittest.TestCase):
             content = StringField()
 
         class PersonPreferences(EmbeddedDocument):
-            food = StringField()
+            food = StringField(required=True)
             number = IntField()
 
         class Person(Document):
@@ -195,9 +302,14 @@ class FieldTest(unittest.TestCase):
         person.preferences = 'My Preferences'
         self.assertRaises(ValidationError, person.validate)
 
+        # Check that only the right embedded doc works
         person.preferences = Comment(content='Nice blog post...')
         self.assertRaises(ValidationError, person.validate)
 
+        # Check that the embedded doc is valid
+        person.preferences = PersonPreferences()
+        self.assertRaises(ValidationError, person.validate)
+
         person.preferences = PersonPreferences(food='Cheese', number=47)
         self.assertEqual(person.preferences.food, 'Cheese')
         person.validate()
@@ -258,7 +370,419 @@ class FieldTest(unittest.TestCase):
 
         User.drop_collection()
         BlogPost.drop_collection()
+    
+    def test_list_item_dereference(self):
+        """Ensure that DBRef items in ListFields are dereferenced.
+        """
+        class User(Document):
+            name = StringField()
 
+        class Group(Document):
+            members = ListField(ReferenceField(User))
+
+        User.drop_collection()
+        Group.drop_collection()
+
+        user1 = User(name='user1')
+        user1.save()
+        user2 = User(name='user2')
+        user2.save()
+
+        group = Group(members=[user1, user2])
+        group.save()
+
+        group_obj = Group.objects.first()
+
+        self.assertEqual(group_obj.members[0].name, user1.name)
+        self.assertEqual(group_obj.members[1].name, user2.name)
+
+        User.drop_collection()
+        Group.drop_collection()
+
+    def test_recursive_reference(self):
+        """Ensure that ReferenceFields can reference their own documents.
+        """
+        class Employee(Document):
+            name = StringField()
+            boss = ReferenceField('self')
+            friends = ListField(ReferenceField('self'))
+
+        bill = Employee(name='Bill Lumbergh')
+        bill.save()
+
+        michael = Employee(name='Michael Bolton')
+        michael.save()
+
+        samir = Employee(name='Samir Nagheenanajar')
+        samir.save()
+
+        friends = [michael, samir]
+        peter = Employee(name='Peter Gibbons', boss=bill, friends=friends)
+        peter.save()
+
+        peter = Employee.objects.with_id(peter.id)
+        self.assertEqual(peter.boss, bill)
+        self.assertEqual(peter.friends, friends)
+
+    def test_recursive_embedding(self):
+        """Ensure that EmbeddedDocumentFields can contain their own documents.
+        """
+        class Tree(Document):
+            name = StringField()
+            children = ListField(EmbeddedDocumentField('TreeNode'))
+
+        class TreeNode(EmbeddedDocument):
+            name = StringField()
+            children = ListField(EmbeddedDocumentField('self'))
+        
+        tree = Tree(name="Tree")
+
+        first_child = TreeNode(name="Child 1")
+        tree.children.append(first_child)
+
+        second_child = TreeNode(name="Child 2")
+        first_child.children.append(second_child)
+        
+        third_child = TreeNode(name="Child 3")
+        first_child.children.append(third_child)
+
+        tree.save()
+
+        tree_obj = Tree.objects.first()
+        self.assertEqual(len(tree.children), 1)
+        self.assertEqual(tree.children[0].name, first_child.name)
+        self.assertEqual(tree.children[0].children[0].name, second_child.name)
+        self.assertEqual(tree.children[0].children[1].name, third_child.name)
+
+    def test_undefined_reference(self):
+        """Ensure that ReferenceFields may reference undefined Documents.
+        """
+        class Product(Document):
+            name = StringField()
+            company = ReferenceField('Company')
+
+        class Company(Document):
+            name = StringField()
+
+        ten_gen = Company(name='10gen')
+        ten_gen.save()
+        mongodb = Product(name='MongoDB', company=ten_gen)
+        mongodb.save()
+
+        obj = Product.objects(company=ten_gen).first()
+        self.assertEqual(obj, mongodb)
+        self.assertEqual(obj.company, ten_gen)
+
+    def test_reference_query_conversion(self):
+        """Ensure that ReferenceFields can be queried using objects and values
+        of the type of the primary key of the referenced object.
+        """
+        class Member(Document):
+            user_num = IntField(primary_key=True)
+
+        class BlogPost(Document):
+            title = StringField()
+            author = ReferenceField(Member)
+
+        Member.drop_collection()
+        BlogPost.drop_collection()
+
+        m1 = Member(user_num=1)
+        m1.save()
+        m2 = Member(user_num=2)
+        m2.save()
+
+        post1 = BlogPost(title='post 1', author=m1)
+        post1.save()
+
+        post2 = BlogPost(title='post 2', author=m2)
+        post2.save()
+
+        post = BlogPost.objects(author=m1).first()
+        self.assertEqual(post.id, post1.id)
+
+        post = BlogPost.objects(author=m2).first()
+        self.assertEqual(post.id, post2.id)
+
+        Member.drop_collection()
+        BlogPost.drop_collection()
+        
+    def test_generic_reference(self):
+        """Ensure that a GenericReferenceField properly dereferences items.
+        """
+        class Link(Document):
+            title = StringField()
+            meta = {'allow_inheritance': False}
+            
+        class Post(Document):
+            title = StringField()
+            
+        class Bookmark(Document):
+            bookmark_object = GenericReferenceField()
+            
+        Link.drop_collection()
+        Post.drop_collection()
+        Bookmark.drop_collection()
+    
+        link_1 = Link(title="Pitchfork")
+        link_1.save()
+    
+        post_1 = Post(title="Behind the Scenes of the Pavement Reunion")
+        post_1.save()
+        
+        bm = Bookmark(bookmark_object=post_1)
+        bm.save()
+        
+        bm = Bookmark.objects(bookmark_object=post_1).first()
+        
+        self.assertEqual(bm.bookmark_object, post_1)
+        self.assertTrue(isinstance(bm.bookmark_object, Post))
+        
+        bm.bookmark_object = link_1
+        bm.save()
+        
+        bm = Bookmark.objects(bookmark_object=link_1).first()
+        
+        self.assertEqual(bm.bookmark_object, link_1)
+        self.assertTrue(isinstance(bm.bookmark_object, Link))
+    
+        Link.drop_collection()
+        Post.drop_collection()
+        Bookmark.drop_collection()
+
+    def test_generic_reference_list(self):
+        """Ensure that a ListField properly dereferences generic references.
+        """
+        class Link(Document):
+            title = StringField()
+    
+        class Post(Document):
+            title = StringField()
+    
+        class User(Document):
+            bookmarks = ListField(GenericReferenceField())
+    
+        Link.drop_collection()
+        Post.drop_collection()
+        User.drop_collection()
+    
+        link_1 = Link(title="Pitchfork")
+        link_1.save()
+    
+        post_1 = Post(title="Behind the Scenes of the Pavement Reunion")
+        post_1.save()
+    
+        user = User(bookmarks=[post_1, link_1])
+        user.save()
+        
+        user = User.objects(bookmarks__all=[post_1, link_1]).first()
+        
+        self.assertEqual(user.bookmarks[0], post_1)
+        self.assertEqual(user.bookmarks[1], link_1)
+        
+        Link.drop_collection()
+        Post.drop_collection()
+        User.drop_collection()
+
+    def test_binary_fields(self):
+        """Ensure that binary fields can be stored and retrieved.
+        """
+        class Attachment(Document):
+            content_type = StringField()
+            blob = BinaryField()
+
+        BLOB = '\xe6\x00\xc4\xff\x07'
+        MIME_TYPE = 'application/octet-stream'
+
+        Attachment.drop_collection()
+
+        attachment = Attachment(content_type=MIME_TYPE, blob=BLOB)
+        attachment.save()
+
+        attachment_1 = Attachment.objects().first()
+        self.assertEqual(MIME_TYPE, attachment_1.content_type)
+        self.assertEqual(BLOB, attachment_1.blob)
+
+        Attachment.drop_collection()
+
+    def test_binary_validation(self):
+        """Ensure that invalid values cannot be assigned to binary fields.
+        """
+        class Attachment(Document):
+            blob = BinaryField()
+
+        class AttachmentRequired(Document):
+            blob = BinaryField(required=True)
+
+        class AttachmentSizeLimit(Document):
+            blob = BinaryField(max_bytes=4)
+
+        Attachment.drop_collection()
+        AttachmentRequired.drop_collection()
+        AttachmentSizeLimit.drop_collection()
+
+        attachment = Attachment()
+        attachment.validate()
+        attachment.blob = 2
+        self.assertRaises(ValidationError, attachment.validate)
+
+        attachment_required = AttachmentRequired()
+        self.assertRaises(ValidationError, attachment_required.validate)
+        attachment_required.blob = '\xe6\x00\xc4\xff\x07'
+        attachment_required.validate()
+
+        attachment_size_limit = AttachmentSizeLimit(blob='\xe6\x00\xc4\xff\x07')
+        self.assertRaises(ValidationError, attachment_size_limit.validate)
+        attachment_size_limit.blob = '\xe6\x00\xc4\xff'
+        attachment_size_limit.validate()
+
+        Attachment.drop_collection()
+        AttachmentRequired.drop_collection()
+        AttachmentSizeLimit.drop_collection()
+
+    def test_choices_validation(self):
+        """Ensure that value is in a container of allowed values.
+        """
+        class Shirt(Document):
+            size = StringField(max_length=3, choices=('S','M','L','XL','XXL'))
+
+        Shirt.drop_collection()
+
+        shirt = Shirt()
+        shirt.validate()
+
+        shirt.size = "S"
+        shirt.validate()
+
+        shirt.size = "XS"
+        self.assertRaises(ValidationError, shirt.validate)
+
+        Shirt.drop_collection()
+
+    def test_file_fields(self):
+        """Ensure that file fields can be written to and their data retrieved
+        """
+        class PutFile(Document):
+            file = FileField()
+
+        class StreamFile(Document):
+            file = FileField()
+
+        class SetFile(Document):
+            file = FileField()
+
+        text = 'Hello, World!'
+        more_text = 'Foo Bar'
+        content_type = 'text/plain'
+
+        PutFile.drop_collection()
+        StreamFile.drop_collection()
+        SetFile.drop_collection()
+
+        putfile = PutFile()
+        putfile.file.put(text, content_type=content_type)
+        putfile.save()
+        putfile.validate()
+        result = PutFile.objects.first()
+        self.assertTrue(putfile == result)
+        self.assertEquals(result.file.read(), text)
+        self.assertEquals(result.file.content_type, content_type)
+        result.file.delete() # Remove file from GridFS
+
+        streamfile = StreamFile()
+        streamfile.file.new_file(content_type=content_type)
+        streamfile.file.write(text)
+        streamfile.file.write(more_text)
+        streamfile.file.close()
+        streamfile.save()
+        streamfile.validate()
+        result = StreamFile.objects.first()
+        self.assertTrue(streamfile == result)
+        self.assertEquals(result.file.read(), text + more_text)
+        self.assertEquals(result.file.content_type, content_type)
+        result.file.delete()
+
+        # Ensure deleted file returns None
+        self.assertTrue(result.file.read() == None)
+
+        setfile = SetFile()
+        setfile.file = text
+        setfile.save()
+        setfile.validate()
+        result = SetFile.objects.first()
+        self.assertTrue(setfile == result)
+        self.assertEquals(result.file.read(), text)
+
+        # Try replacing file with new one
+        result.file.replace(more_text)
+        result.save()
+        result.validate()
+        result = SetFile.objects.first()
+        self.assertTrue(setfile == result)
+        self.assertEquals(result.file.read(), more_text)
+        result.file.delete() 
+
+        PutFile.drop_collection()
+        StreamFile.drop_collection()
+        SetFile.drop_collection()
+
+        # Make sure FileField is optional and not required
+        class DemoFile(Document):
+            file = FileField()
+        d = DemoFile.objects.create()
+
+    def test_file_uniqueness(self):
+        """Ensure that each instance of a FileField is unique
+        """
+        class TestFile(Document):
+            name = StringField()
+            file = FileField()
+
+        # First instance
+        testfile = TestFile()
+        testfile.name = "Hello, World!"
+        testfile.file.put('Hello, World!')
+        testfile.save()
+
+        # Second instance
+        testfiledupe = TestFile()
+        data = testfiledupe.file.read() # Should be None
+
+        self.assertTrue(testfile.name != testfiledupe.name)
+        self.assertTrue(testfile.file.read() != data)
+
+        TestFile.drop_collection()
+
+    def test_geo_indexes(self):
+        """Ensure that indexes are created automatically for GeoPointFields.
+        """
+        class Event(Document):
+            title = StringField()
+            location = GeoPointField()
+
+        Event.drop_collection()
+        event = Event(title="Coltrane Motion @ Double Door",
+                      location=[41.909889, -87.677137])
+        event.save()
+
+        info = Event.objects._collection.index_information()
+        self.assertTrue(u'location_2d' in info)
+        self.assertTrue(info[u'location_2d']['key'] == [(u'location', u'2d')])
+
+        Event.drop_collection()
+
+    def test_ensure_unique_default_instances(self):
+        """Ensure that every field has it's own unique default instance."""
+        class D(Document):
+            data = DictField()
+            data2 = DictField(default=lambda: {})
+
+        d1 = D()
+        d1.data['foo'] = 'bar'
+        d1.data2['foo'] = 'bar'
+        d2 = D()
+        self.assertEqual(d2.data, {})
+        self.assertEqual(d2.data2, {})
 
 if __name__ == '__main__':
     unittest.main()