Updates to documentation in prep for 0.5

docs/guide/defining-documents.rst

@@ -36,22 +36,25 @@ are as follows:
 
 * :class:`~mongoengine.StringField`
 * :class:`~mongoengine.URLField`
+* :class:`~mongoengine.EmailField`
 * :class:`~mongoengine.IntField`
 * :class:`~mongoengine.FloatField`
 * :class:`~mongoengine.DecimalField`
 * :class:`~mongoengine.DateTimeField`
+* :class:`~mongoengine.ComplexDateTimeField`
 * :class:`~mongoengine.ListField`
+* :class:`~mongoengine.SortedListField`
 * :class:`~mongoengine.DictField`
+* :class:`~mongoengine.MapField`
 * :class:`~mongoengine.ObjectIdField`
-* :class:`~mongoengine.EmbeddedDocumentField`
 * :class:`~mongoengine.ReferenceField`
 * :class:`~mongoengine.GenericReferenceField`
+* :class:`~mongoengine.EmbeddedDocumentField`
 * :class:`~mongoengine.BooleanField`
 * :class:`~mongoengine.FileField`
-* :class:`~mongoengine.EmailField`
-* :class:`~mongoengine.SortedListField`
 * :class:`~mongoengine.BinaryField`
 * :class:`~mongoengine.GeoPointField`
+* :class:`~mongoengine.SequenceField`
 
 Field arguments
 ---------------
@@ -105,6 +108,12 @@ arguments can be set on all fields:
 :attr:`choices` (Default: None)
     An iterable of choices to which the value of this field should be limited.
 
+:attr:`help_text` (Default: None)
+    Optional help text to output with the field - used by form libraries
+
+:attr:`verbose` (Default: None)
+    Optional human-readable name for the field - used by form libraries
+
 
 List fields
 -----------
@@ -155,6 +164,9 @@ store; in this situation a :class:`~mongoengine.DictField` is appropriate::
     survey_response.answers = response_form.cleaned_data()
     survey_response.save()
 
+Dictionaries can store complex data, other dictionaries, lists, references to
+other objects, so are the most flexible field type available.
+
 Reference fields
 ----------------
 References may be stored to other documents in the database using the
@@ -272,6 +284,7 @@ kind of :class:`~mongoengine.Document`, and hence doesn't take a
     Bookmark(bookmark_object=post).save()
 
 .. note::
+
    Using :class:`~mongoengine.GenericReferenceField`\ s is slightly less
    efficient than the standard :class:`~mongoengine.ReferenceField`\ s, so if
    you will only be referencing one document type, prefer the standard
@@ -369,6 +382,7 @@ If a dictionary is passed then the following options are available:
     Whether the index should be sparse.
 
 .. note::
+
    Geospatial indexes will be automatically created for all
    :class:`~mongoengine.GeoPointField`\ s
 

docs/guide/document-instances.rst

@@ -29,6 +29,7 @@ already exist, then any changes will be updated atomically.  For example::
     >>> page.save()  # Performs an atomic set on the title field.
 
 .. note::
+
     Changes to documents are tracked and on the whole perform `set` operations.
 
     * ``list_field.pop(0)`` - *sets* the resulting list
@@ -78,6 +79,7 @@ is an alias to :attr:`id`::
     >>> page.id == page.pk
 
 .. note::
+
    If you define your own primary key field, the field implicitly becomes
    required, so a :class:`ValidationError` will be thrown if you don't provide
    it.

docs/guide/gridfs.rst

@@ -66,6 +66,7 @@ Deleting stored files is achieved with the :func:`delete` method::
     marmot.photo.delete()
 
 .. note::
+
     The FileField in a Document actually only stores the ID of a file in a
     separate GridFS collection. This means that deleting a document
     with a defined FileField does not actually delete the file. You must be

docs/guide/installing.rst

@@ -1,31 +1,31 @@
 ======================
 Installing MongoEngine
 ======================
+
 To use MongoEngine, you will need to download `MongoDB <http://mongodb.org/>`_
 and ensure it is running in an accessible location. You will also need
 `PyMongo <http://api.mongodb.org/python>`_ to use MongoEngine, but if you
 install MongoEngine using setuptools, then the dependencies will be handled for
 you.
 
-MongoEngine is available on PyPI, so to use it you can use 
+MongoEngine is available on PyPI, so to use it you can use :program:`pip`:
-:program:`easy_install`:
 
 .. code-block:: console
 
-    # easy_install mongoengine
+    $ pip install mongoengine
 
 Alternatively, if you don't have setuptools installed, `download it from PyPi
 <http://pypi.python.org/pypi/mongoengine/>`_ and run
 
 .. code-block:: console
 
-    # python setup.py install
+    $ python setup.py install
 
 To use the bleeding-edge version of MongoEngine, you can get the source from
 `GitHub <http://github.com/hmarr/mongoengine/>`_ and install it as above:
 
 .. code-block:: console
 
-    # git clone git://github.com/hmarr/mongoengine
+    $ git clone git://github.com/hmarr/mongoengine
-    # cd mongoengine
+    $ cd mongoengine
-    # python setup.py install
+    $ python setup.py install

docs/guide/querying.rst

@@ -14,6 +14,7 @@ fetch documents from the database::
         print user.name
 
 .. note::
+
    Once the iteration finishes (when :class:`StopIteration` is raised),
    :meth:`~mongoengine.queryset.QuerySet.rewind` will be called so that the
    :class:`~mongoengine.queryset.QuerySet` may be iterated over again. The
@@ -39,29 +40,6 @@ syntax::
     # been written by a user whose 'country' field is set to 'uk'
     uk_pages = Page.objects(author__country='uk')
 
-Querying lists
---------------
-On most fields, this syntax will look up documents where the field specified
-matches the given value exactly, but when the field refers to a
-:class:`~mongoengine.ListField`, a single item may be provided, in which case
-lists that contain that item will be matched::
-
-    class Page(Document):
-        tags = ListField(StringField())
-
-    # This will match all pages that have the word 'coding' as an item in the
-    # 'tags' list
-    Page.objects(tags='coding')
-
-Raw queries
------------
-It is possible to provide a raw PyMongo query as a query parameter, which will
-be integrated directly into the query. This is done using the ``__raw__``
-keyword argument::
-
-    Page.objects(__raw__={'tags': 'coding'})
-
-.. versionadded:: 0.4
 
 Query operators
 ===============
@@ -99,26 +77,67 @@ expressions:
 * ``endswith`` -- string field ends with value
 * ``iendswith`` -- string field ends with value (case insensitive)
 
-.. versionadded:: 0.3
-
 There are a few special operators for performing geographical queries, that
 may used with :class:`~mongoengine.GeoPointField`\ s:
 
 * ``within_distance`` -- provide a list containing a point and a maximum
   distance (e.g. [(41.342, -87.653), 5])
+* ``within_spherical_distance`` -- Same as above but using the spherical geo model
+  (e.g. [(41.342, -87.653), 5/earth_radius])
+* ``near`` -- order the documents by how close they are to a given point
+* ``near_sphere`` -- Same as above but using the spherical geo model
 * ``within_box`` -- filter documents to those within a given bounding box (e.g.
   [(35.0, -125.0), (40.0, -100.0)])
-* ``near`` -- order the documents by how close they are to a given point
+* ``within_polygon`` -- filter documents to those within a given polygon (e.g.
+  [(41.91,-87.69), (41.92,-87.68), (41.91,-87.65), (41.89,-87.65)]).
+  .. note:: Requires Mongo Server 2.0
 
-.. versionadded:: 0.4
 
-Querying by position
+Querying lists
-====================
+--------------
+On most fields, this syntax will look up documents where the field specified
+matches the given value exactly, but when the field refers to a
+:class:`~mongoengine.ListField`, a single item may be provided, in which case
+lists that contain that item will be matched::
+
+    class Page(Document):
+        tags = ListField(StringField())
+
+    # This will match all pages that have the word 'coding' as an item in the
+    # 'tags' list
+    Page.objects(tags='coding')
+
 It is possible to query by position in a list by using a numerical value as a
 query operator. So if you wanted to find all pages whose first tag was ``db``,
 you could use the following query::
 
-    BlogPost.objects(tags__0='db')
+    Page.objects(tags__0='db')
+
+If you only want to fetch part of a list eg: you want to paginate a list, then
+the `slice` operator is required::
+
+    # comments - skip 5, limit 10
+    Page.objects.fields(slice__comments=[5, 10])
+
+For updating documents, if you don't know the position in a list, you can use
+the $ positional operator ::
+
+    Post.objects(comments__by="joe").update(**{'inc__comments__$__votes': 1})
+
+However, this doesn't map well to the syntax so you can alos use a capital S instead ::
+
+    Post.objects(comments__by="joe").update(inc__comments__S__votes=1)
+
+    .. note:: Due to Mongo currently the $ operator only applies to the first matched item in the query.
+
+
+Raw queries
+-----------
+It is possible to provide a raw PyMongo query as a query parameter, which will
+be integrated directly into the query. This is done using the ``__raw__``
+keyword argument::
+
+    Page.objects(__raw__={'tags': 'coding'})
 
 .. versionadded:: 0.4
 
@@ -270,6 +289,7 @@ You may sum over the values of a specific field on documents using
     yearly_expense = Employee.objects.sum('salary')
 
 .. note::
+
    If the field isn't present on a document, that document will be ignored from
    the sum.
 
@@ -318,6 +338,11 @@ will be given::
     >>> f.rating # default value
     3
 
+.. note::
+
+    The :meth:`~mongoengine.queryset.QuerySet.exclude` is the opposite of
+    :meth:`~mongoengine.queryset.QuerySet.only` if you want to exclude a field.
+
 If you later need the missing fields, just call
 :meth:`~mongoengine.Document.reload` on your document.
 
@@ -341,6 +366,67 @@ calling it with keyword arguments::
     # Get top posts
     Post.objects((Q(featured=True) & Q(hits__gte=1000)) | Q(hits__gte=5000))
 
+.. _guide-atomic-updates:
+
+Atomic updates
+==============
+Documents may be updated atomically by using the
+:meth:`~mongoengine.queryset.QuerySet.update_one` and
+:meth:`~mongoengine.queryset.QuerySet.update` methods on a
+:meth:`~mongoengine.queryset.QuerySet`. There are several different "modifiers"
+that you may use with these methods:
+
+* ``set`` -- set a particular value
+* ``unset`` -- delete a particular value (since MongoDB v1.3+)
+* ``inc`` -- increment a value by a given amount
+* ``dec`` -- decrement a value by a given amount
+* ``pop`` -- remove the last item from a list
+* ``push`` -- append a value to a list
+* ``push_all`` -- append several values to a list
+* ``pop`` -- remove the first or last element of a list
+* ``pull`` -- remove a value from a list
+* ``pull_all`` -- remove several values from a list
+* ``add_to_set`` -- add value to a list only if its not in the list already
+
+The syntax for atomic updates is similar to the querying syntax, but the
+modifier comes before the field, not after it::
+
+    >>> post = BlogPost(title='Test', page_views=0, tags=['database'])
+    >>> post.save()
+    >>> BlogPost.objects(id=post.id).update_one(inc__page_views=1)
+    >>> post.reload()  # the document has been changed, so we need to reload it
+    >>> post.page_views
+    1
+    >>> BlogPost.objects(id=post.id).update_one(set__title='Example Post')
+    >>> post.reload()
+    >>> post.title
+    'Example Post'
+    >>> BlogPost.objects(id=post.id).update_one(push__tags='nosql')
+    >>> post.reload()
+    >>> post.tags
+    ['database', 'nosql']
+
+.. note ::
+
+    In version 0.5 the :meth:`~mongoengine.Document.save` runs atomic updates
+    on changed documents by tracking changes to that document.
+
+The positional operator allows you to update list items without knowing the
+index position, therefore making the update a single atomic operation.  As we
+cannot use the `$` syntax in keyword arguments it has been mapped to `S`::
+
+    >>> post = BlogPost(title='Test', page_views=0, tags=['database', 'mongo'])
+    >>> post.save()
+    >>> BlogPost.objects(id=post.id, tags='mongo').update(set__tags__S='mongodb')
+    >>> post.reload()
+    >>> post.tags
+    ['database', 'mongodb']
+
+.. note ::
+    Currently only top level lists are handled, future versions of mongodb /
+    pymongo plan to support nested positional operators.  See `The $ positional
+    operator <http://www.mongodb.org/display/DOCS/Updating#Updating-The%24positionaloperator>`_.
+
 Server-side javascript execution
 ================================
 Javascript functions may be written and sent to the server for execution. The
@@ -443,59 +529,3 @@ following example shows how the substitutions are made::
         return comments;
     }
     """)
-
-.. _guide-atomic-updates:
-
-Atomic updates
-==============
-Documents may be updated atomically by using the
-:meth:`~mongoengine.queryset.QuerySet.update_one` and
-:meth:`~mongoengine.queryset.QuerySet.update` methods on a
-:meth:`~mongoengine.queryset.QuerySet`. There are several different "modifiers"
-that you may use with these methods:
-
-* ``set`` -- set a particular value
-* ``unset`` -- delete a particular value (since MongoDB v1.3+)
-* ``inc`` -- increment a value by a given amount
-* ``dec`` -- decrement a value by a given amount
-* ``pop`` -- remove the last item from a list
-* ``push`` -- append a value to a list
-* ``push_all`` -- append several values to a list
-* ``pop`` -- remove the first or last element of a list
-* ``pull`` -- remove a value from a list
-* ``pull_all`` -- remove several values from a list
-* ``add_to_set`` -- add value to a list only if its not in the list already
-
-The syntax for atomic updates is similar to the querying syntax, but the
-modifier comes before the field, not after it::
-
-    >>> post = BlogPost(title='Test', page_views=0, tags=['database'])
-    >>> post.save()
-    >>> BlogPost.objects(id=post.id).update_one(inc__page_views=1)
-    >>> post.reload()  # the document has been changed, so we need to reload it
-    >>> post.page_views
-    1
-    >>> BlogPost.objects(id=post.id).update_one(set__title='Example Post')
-    >>> post.reload()
-    >>> post.title
-    'Example Post'
-    >>> BlogPost.objects(id=post.id).update_one(push__tags='nosql')
-    >>> post.reload()
-    >>> post.tags
-    ['database', 'nosql']
-
-The positional operator allows you to update list items without knowing the
-index position, therefore making the update a single atomic operation.  As we
-cannot use the `$` syntax in keyword arguments it has been mapped to `S`::
-
-    >>> post = BlogPost(title='Test', page_views=0, tags=['database', 'mongo'])
-    >>> post.save()
-    >>> BlogPost.objects(id=post.id, tags='mongo').update(set__tags__S='mongodb')
-    >>> post.reload()
-    >>> post.tags
-    ['database', 'mongodb']
-
-.. note ::
-    Currently only top level lists are handled, future versions of mongodb /
-    pymongo plan to support nested positional operators.  See `The $ positional
-    operator <http://www.mongodb.org/display/DOCS/Updating#Updating-The%24positionaloperator>`_.

docs/index.rst

@@ -2,35 +2,62 @@
 MongoEngine User Documentation
 ==============================
 
-MongoEngine is an Object-Document Mapper, written in Python for working with
+**MongoEngine** is an Object-Document Mapper, written in Python for working with
 MongoDB. To install it, simply run
 
 .. code-block:: console
 
     # pip install -U mongoengine
 
-The source is available on `GitHub <http://github.com/hmarr/mongoengine>`_.
+:doc:`tutorial`
+  Start here for a quick overview.
+
+:doc:`guide/index`
+  The Full guide to MongoEngine
+
+:doc:`apireference`
+  The complete API documentation.
+
+:doc:`django`
+  Using MongoEngine and Django
+
+Community
+---------
 
 To get help with using MongoEngine, use the `MongoEngine Users mailing list
 <http://groups.google.com/group/mongoengine-users>`_ or come chat on the
 `#mongoengine IRC channel <irc://irc.freenode.net/mongoengine>`_.
 
-If you are interested in contributing, join the developers' `mailing list
+Contributing
+------------
+
+The source is available on `GitHub <http://github.com/hmarr/mongoengine>`_ and
+contributions are always encouraged. Contributions can be as simple as
+minor tweaks to this documentation. To contribute, fork the project on
+`GitHub <http://github.com/hmarr/mongoengine>`_ and send a
+pull request.
+
+Also, you can join the developers' `mailing list
 <http://groups.google.com/group/mongoengine-dev>`_.
 
+Changes
+-------
+See the :doc:`changelog` for a full list of changes to MongoEngine.
+
 .. toctree::
-   :maxdepth: 2
+   :hidden:
 
    tutorial
    guide/index
    apireference
    django
    changelog
-   upgrading
+   upgrade
 
 Indices and tables
-==================
+------------------
 
 * :ref:`genindex`
+* :ref:`modindex`
 * :ref:`search`
 

docs/upgrade.rst

@@ -5,24 +5,33 @@ Upgrading
 0.4 to 0.5
 ===========
 
-There have been the following backwards incompatibilities from 0.4 to 0.5:
+There have been the following backwards incompatibilities from 0.4 to 0.5.  The
+main areas of changed are: choices in fields, map_reduce and collection names.
 
-# Choice options:
+Choice options:
+--------------
 
 Are now expected to be an iterable of tuples, with  the first element in each
 tuple being the actual value to be stored. The second element is the
 human-readable name for the option.
 
-# PyMongo / MongoDB
 
-map reduce now requires pymongo 1.11+ More methods now use map_reduce as db.eval
+PyMongo / MongoDB
-is not supported for sharding - the following have been changed:
+-----------------
 
-    * sum
+map reduce now requires pymongo 1.11+- The pymongo merge_output and reduce_output
-    * average
+parameters, have been depreciated.
-    * item_frequencies
 
-#. Default collection naming.
+More methods now use map_reduce as db.eval is not supported for sharding as such
+the following have been changed:
+
+    * :meth:`~mongoengine.queryset.QuerySet.sum`
+    * :meth:`~mongoengine.queryset.QuerySet.average`
+    * :meth:`~mongoengine.queryset.QuerySet.item_frequencies`
+
+
+Default collection naming
+-------------------------
 
 Previously it was just lowercase, its now much more pythonic and readable as its
 lowercase and underscores, previously ::

mongoengine/queryset.py

@@ -744,7 +744,7 @@ class QuerySet(object):
 
         :param write_options: optional extra keyword arguments used if we
             have to create a new document.
-            Passes any write_options onto :meth:`~mongoengine.document.Document.save`
+            Passes any write_options onto :meth:`~mongoengine.Document.save`
 
         .. versionadded:: 0.3
         """
@@ -901,9 +901,11 @@ class QuerySet(object):
         Returns an iterator yielding
         :class:`~mongoengine.document.MapReduceDocument`.
 
-        .. note:: Map/Reduce changed in server version **>= 1.7.4**. The PyMongo
+        .. note::
-           :meth:`~pymongo.collection.Collection.map_reduce` helper requires
+
-           PyMongo version **>= 1.11**.
+            Map/Reduce changed in server version **>= 1.7.4**. The PyMongo
+            :meth:`~pymongo.collection.Collection.map_reduce` helper requires
+            PyMongo version **>= 1.11**.
 
         .. versionchanged:: 0.5
            - removed ``keep_temp`` keyword argument, which was only relevant
@@ -1070,8 +1072,7 @@ class QuerySet(object):
         and `.exclude()` to manipulate which fields to retrieve.  Fields also
         allows for a greater level of control for example:
 
-        Retrieving a Subrange of Array Elements
+        Retrieving a Subrange of Array Elements:
-        ---------------------------------------
 
         You can use the $slice operator to retrieve a subrange of elements in
         an array ::
@@ -1500,6 +1501,7 @@ class QuerySet(object):
         This is useful for generating tag clouds, or searching documents.
 
         .. note::
+
             Can only do direct simple mappings and cannot map across
             :class:`~mongoengine.ReferenceField` or
             :class:`~mongoengine.GenericReferenceField` for more complex